One of the biggest challenges in education research is determining how far findings are applicable to new contexts. 

For example, you might read about an interesting study done in higher education. Will the findings also hold in a primary school? Or you may hear about research findings from the United States. How far would the same apply in an English school?

There are so many different variables that come together in educational settings - it can be quite challenging to know “what works”.

It is therefore tempting to try to “control” for as many variables as possible in education research.

Studies that do this can be valuable. For example, a large-scale randomised controlled trial (RCT) might tell us something about the effects and scalability of a programme implemented across dozens of schools. 

However, RCTs often suffer from distortions, such as attrition, spillover between classes or inconsistencies in how the programme was implemented. 

Smaller, qualitative studies, meanwhile, might fit better within schools’ daily practices. Yet there will often be so many variables that it will be hard to generalise.

Handling education research with care

Julia Moeller and colleagues recently wrote an interesting pre-print about this. The article describes the tension between our desire for generalisability and the need to acknowledge the variability, or “heterogeneity’, that exists in research.

One source of this heterogeneity, the authors note, is person-specific: individuals in a sample might differ from another sample in ways that are relevant for the study; for example, with regard to psychological and demographic characteristics. 

The extent to which sample-specific findings can be generalised will therefore depend on how representative a sample is. There are many variables that differ between individuals - personality, age and culture, to name a few - and it will be hard to control for them all.

But there is another source of heterogeneity that might be even more challenging to control: time-specificity. As the authors note, “Over time, phenomena can emerge, disappear or fluctuate.” This poses a challenge when we look at findings over time, even at different points in the same study. 

Context is also notoriously difficult to control. I do quite a lot of research in an international comparative context. This poses immense challenges around how we even begin to compare one country with another. Education systems differ and, within countries, systems and policies are constantly changing. 

We can’t, therefore, just assume that findings will replicate - and when they do, we can’t assume they are more trustworthy or can be generalised to further settings as a result.

So, where does this leave us? The key message is that we always need to be modest and cautious when talking about research. There is uncertainty in any conclusion; this ought to be communicated. And when considering replicability, we also need to consider generalisability.

Researcher David C Berliner has called education research “the hardest science of them all”. We would do well to keep those words in mind when reading and applying research.

Christian Bokhove is associate professor in mathematics education at the University of Southampton and a specialist in research methodologies