Part of being a researcher is accepting that your mountain is very often someone else’s molehill. So the willingness of schools to contribute to research in education is striking. To date, more than half of schools across England have taken part in randomised controlled trials (RCTs) to evaluate the impact of educational interventions on pupil outcomes.

However, in spite of the unwavering support from the teaching profession, misconceptions and confusions surrounding these trials still exist. Something we used to hear a lot of in the early trials we ran was disappointment from schools assigned to the “control” condition - the group that does not receive the intervention being assessed by the research.

It was quite common for these teachers to override the random nature of the trial and deliver the intervention to those pupils whom they felt might benefit most.

Our response was always: “No one knows whether the intervention works. If you deliver it to this pupil, you disrupt the trial and make it less likely that we can detect a genuine effect.”

It’s a “greater good” argument, which relies on an understanding of the bigger picture. So, perhaps it would be helpful to examine the bigger picture.

RCTs, particularly in the field of medicine, are seen as the gold standard of evaluation research. It may come as a surprise, however, to learn that the origins of the RCT span at least as far back in education as in medical research. In fact, this year marks the 100th anniversary of the first education RCT.

The original 1919 paper, written by Robert Alexander Cummins of Iowa, states: “The factor of the teacher was equalized by random selection of the classes, which made up the two groups.”

It concluded that reducing the length of lessons for pupils in US grades 3 to 8 led to improvements in pupil performance because it resulted in “more intensive application … in their effort to get as much done as possible”. (It should be noted, however, that the correct analysis techniques for Cummins’ data had not yet been invented, so we should take his conclusion with a large pinch of salt.)

The most powerful stronghold for education trials remains their birthplace in the US. Indeed, a recent systematic review (Connolly et al, 2018) found that more than half of all English language education RCTs were conducted in North America, with a little less than a third undertaken in Europe.

However, contributions from the US have not always been a given. For a period of 20 years, from the 1980s to the early 2000s, experimental research was almost abandoned altogether there. This move away from the “experimenting society” of the 1960s and 1970s was driven largely by the absence of treatment effects from randomised experiments of most of the interventions evaluated. Put simply: they stopped testing new approaches to education because very few of these approaches worked.

Yet rigorous evidence of nothing - of no effect of educational interventions or, indeed, of harm - is still important scientific knowledge. Policymakers and practitioners can cease to promote the use of these interventions. Funders can make better use of public money. Schools can stop wasting resources on something that has no impact, and researchers can move on to evaluate other promising interventions that may prove to be of benefit.

An interviewee in our evaluation of the Helping Handwriting Shine programme said that their reason for participating in the trial stemmed from frustration with the money wasted by schools on interventions that were ineffective, and from a desire to make an active contribution to tackling this issue. By their own admission, they had to resist the temptation to deliver the intervention to “control” pupils in the school who, in their view, may have benefited from it, in order to protect the integrity of the trial.

A key lesson we can learn from history is therefore to embrace negative findings as enthusiastically as we tend to embrace positive ones.

Increase in use of RCTs

The past decade has seen an increased use of RCTs in the UK to evaluate what works in education. One of the first large-scale pragmatic trials of a curriculum intervention in England looked at Every Child Counts, the last UK Labour government’s flagship numeracy policy for pupils in primary school (Torgerson et al, 2013). It was funded by the Department for Education, possibly partly as a result of pressure from a small number of MPs to use RCTs to evaluate government education policy. It was funded, designed and conducted just before the Education Endowment Foundation (EEF) was established in 2011.

This opened the floodgates. Within a short period of time, the EEF was funding and overseeing more than 100 RCTs.

No one is quite sure what forces aligned to create this unprecedented change in England’s approach to education research. Possibly it was prompted by leading academics. However, it would not have happened without the political will to allow evaluators sufficient power to use the most appropriate design for important questions of policy and practice.

The EEF has not only concerned itself with establishing whether interventions in English schools actually help children but has also made great progress in funding research that aims to avert science’s replication crisis. This is the problem of research studies proving difficult - or, indeed, impossible - to reproduce, thus undermining their integrity.

Teachers often ask what they can do to help find out whether all those interventions that claim to make a vast difference to children’s learning actually do what they claim. But they are asking the wrong question. We should, instead, ask what teachers are already doing.

When the EEF started funding education RCTs on a large scale in 2011, many researchers were worried that schools would not sign up to these trials. Such concerns proved unfounded since, to date, more than half of English schools have participated in a randomised control trial.

But what does it take for a school to contribute successfully to these trials?

Some may be surprised to learn that fidelity to the intervention being tested is not top of the list. As evaluators, we are interested in every kind of implementation, from delivery exactly as the developer intended through to adaptations and outright rejection of the programme. The most important aspect is that teachers follow the assessment protocol at the end (and also at the beginning) of the evaluation, so that the impact can be measured. If schools drop out before the assessment task at the end of an evaluation, or don’t complete the assessment task for whatever reason, it makes it harder to draw conclusions about impact. This is true both of the control group and the intervention group.

Difficult to prioritise tests

Outcomes are often measured by a quick attainment test, supported by external administrators. But, when a school is under myriad other pressures, it can be difficult for it to prioritise these tests. Teachers sometimes associate an intervention that they have grown to dislike with the evaluation itself, resulting in withdrawal from the study. This can be catastrophic since, by losing a school, the whole process of randomisation is corrupted, and we can no longer be confident in the comparisons made between intervention and control groups.

By allowing administrators into schools to test children at the end of an experiment, whatever their experience of the intervention, teachers can still contribute to the evidence. When resources are constrained, it becomes even more important that we show what has not worked, so that we can avoid wasting future time and money on it.

Despite the huge support given by teachers, experimentation in schools is under threat. There is a risk, for example, that because of the explosion in the funding of RCTs, evaluators may increasingly find it difficult to recruit schools to take part in trials owing to competing pressures on teachers’ time.

Cutting out research activity is an easy way for teachers to reduce workload as other pressures mount. However, researchers are powerless without the support of teachers.

Meanwhile, as was shown in the US in the 1980s and 1990s, it is easy for politicians and civil servants to assume that no improvement as a result of an intervention implies a problematic research design. They can then use this as an excuse to stop funding trials.

A concerted effort by the funding community, researchers and teachers alike should help to mitigate such risk in future, and preserve the education RCT for as long as people feel the need to use interventions to help children learn.

Ben Styles is head of the Education Trials Unit at the National Foundation for Educational Research. Carole Torgerson is professor of education at Durham University. They will be speaking at the One Hundred Years of Education Trials: No Significant Difference? event, which takes place at the Royal Statistical Society, London, on 23 September. For further details, go to bit.ly/RCTs100years

This article originally appeared in the 23 August 2019 issue under the headline “Trials, tribulations and centenary celebrations”