Faced with uncertainty around how to solve a problem, many will choose to fall back on what we consider our best bet.
This is certainly true of the push to make teaching more evidence informed. Much of the focus has been on the best bets of education research: practices that seem to work in most contexts.
One rather broad umbrella term that falls into this category is “retrieval practice”.
A recent systematic review of the literature tried to find out if it was as good a bet as it seemed (Agarwal, Nunes and Blunt, 2021). The authors looked at 50 classroom-based experiments on retrieval practice.
What they found was that retrieval practice almost universally seems to work. Effect sizes were medium to large and almost all studies had positive values as effect sizes.
But does that mean that retrieval practice works in every context? After all, a recent Education Endowment Foundation report into cognitive science principles in the classroom found that retrieval practice would benefit from further research into its “impact in real classroom conditions”.
In Agarwal et al’s review, the researchers first looked at different features of the studies. For example, 40 per cent were conducted in a secondary school, 40 per cent in an undergraduate setting and 20 per cent in medical schools. None was conducted in primary schools. Few studies were conducted in non-science subjects, with the majority of studies being for science and psychology.
Comparison conditions also varied. Some studies compared retrieval practice to teacher-led lessons without retrieval practice, while others compared it to asking students to re-read materials alone. Most of the experiments (64 per cent) were conducted with a sample size of fewer than 100 students. In addition, 94 per cent of the studies were conducted in the US and Western Europe, the so-called “Weird” countries (western, educated, industrialised, rich, democratic).
The most common formats for retrieval practice were multiple-choice questions and short-answer questions, and the researchers found that the approach worked best if the test questions that students went on to complete had a similar format.
Finally, the researchers highlighted one particular challenge with retrieval practice: a so-called “practice effect”. Because students are simply learning the test questions and answers, it is not always straightforward to see if the advantages of retrieval really transfer to other knowledge domains, even those quite close to what had been practised.
Agarwal et al conclude that the diversity of the settings makes it hard to compare contexts. In addition, some contexts are clearly under-studied and need further research, they point out.
However, they also add that, despite their expectations that their review would provide some practical guidance on optimal conditions for retrieval practice, it turned out that nearly all conditions in classrooms yielded a benefit. That means that teachers can feel confident in implementing retrieval practice without worrying too much about the precise format or timing of retrieval interventions.
How can we square this with the findings of the EEF review? It turns out that the EEF only looked at studies that compared retrieval practice with restudy, while Agarwal and colleagues included more comparison conditions. This further underlines how important it is to know the inclusion and exclusion criteria in meta-reviews, even for practices that we might consider to be our best bets.
Christian Bokhove is associate professor in maths education at the University of Southampton and a specialist in research methodologies
This article originally appeared in the 30 July 2021 issue under the headline “Take a punt on retrieval practice - it’s a safe bet”
