What makes a good-quality research study?

Choosing the right research studies to explore can make a huge difference to CPD – but what should teachers be looking out for?

Judging the quality of a study is not an exact science, but there are certain hallmarks that we can look for to figure out how much faith we should place in the findings of a piece of research.

There are two aspects that it’s important to think about here: the design of the study, and whether this makes the research capable of answering the question that you are interested in; and any problems with the study that might represent “threats” to the validity of the findings.

When trying to judge a study, you are ultimately trying to assess whether it has been well designed, well conducted and well reported. It should be based on a clear research question, and it should use a sound research methodology, with limitations fully acknowledged. The data should be collected and analysed carefully, and all the analyses that were conducted should be reported transparently.

EEF view: what does the research say?

If you want to know about the effectiveness of an approach in the classroom, when thinking about the design of a study it’s important to consider what comparison the research is making.

Pupils tend to make progress through time, and it is important that the research is able to compare whether progress for pupils that received an intervention is greater or less than pupils that are continuing with usual teaching practice. In this series, we’ve explored randomised controlled trials and quasi-experimental designs, both of which use comparative groups to make an estimate of the impact of an approach.

Once you’re happy with the design of the study, there are a number of factors that might cause us to lose confidence, even in the most rigorous designs:

Sample size
If we’re just comparing one school receiving an intervention with another that doesn’t, then there’s a risk that we are just capturing the differences between the specific schools, rather than the intervention. But if we compare 100 schools receiving an intervention with 100 that do not, we learn much more about the effects of the intervention itself.
Balance
We also need to consider how similar the comparison groups are. Even if we compare 100 schools receiving the intervention with 100 that do not, if the 100 intervention schools all have much lower prior attainment than the comparison group, it will decrease our confidence in the result.
Attrition
This refers to the number of pupils or schools that drop out of a trial. If no one is left in the trial at the end we won’t find out much about the intervention (apart from the fact that it’s not very popular). Similarly, if all of the pupils with low prior attainment drop out of the trial, then the balance that we had between the intervention and control groups disappears.

To capture these aspects, the EEF awards padlocks to the evaluation reports we publish. Findings lose padlocks for design (if there is no comparison group then studies cannot receive any padlocks); the statistical power (which comes from sample size); the imbalance between groups; and the number of pupils that drop out of the study.

There are other potential threats to validity that it’s important to consider; for example, around the importance of independence.

Study quality is one consideration when thinking about research. Alongside these questions on methodology and study quality it is important to consider relevance. Even the most rigorous study might not be helpful if the findings are not transferable to your context.

How does it work in the classroom?

Speaking to Tes, Professor Becky Francis, chief executive of the Education Endowment Foundation, said that in order for research evidence to have a “positive and empowering effect”, it is vital that all of those involved in education - from teachers and school leaders to policymakers - take the time to really “question, critique and discriminate” between the many claims of evidence-informed approaches that they encounter, rather than simply accepting them at face value.

As responsible professionals, teachers must “exercise professional judgement when assessing the rigour with which research evidence has been produced”, she said, and take time to consider the relevance of a research study to the particular context. Thinking about the feasibility of implementing evidenced approaches in your environment is one way of doing this, she offered, and so it may also be helpful to ask some simple but significant questions:

Will the approach need to be adapted to fit my local context?
How much organisational capacity might it require to embed the practice? Can we afford to make this commitment?
Are teachers and others likely to want to adopt the practice?

Further reading:

Recent

What makes a good-quality research study?

Read more