Transparency takes bravery. In the short term, it might have been easier if Ofsted chief inspector Amanda Spielman and her research team, led by Professor Daniel Muijs, had chosen to hide the working behind the new Ofsted framework, which was published in draft form two weeks ago. Instead, when the consultation was launched, they also published a fully referenced document, setting out the rationale and sources supporting their proposals.

Ironically, by sharing the evidence base on which the framework is built, Ofsted has made it easier for others to challenge its work. But this openness is hugely valuable, and worthy of serious praise. Evidence can democratise education, and it is undoubtedly a step forward for the inspectorate to set out the studies that will inform its new approach.

Also, the positive impact of an expectation that school and system leaders base their decisions on evidence - and that they set out clearly what this evidence is - cannot be underestimated.

But evidence is as valuable when it is used to start a conversation as when it ends one. Engaging in debate about the quality and application of research is a key part of being truly evidence-informed. Indeed, peer review is the foundation of good research, and plays a crucial role in helping us to reach the strongest and most robust conclusions.

So, in this spirit, how does Ofsted’s evidence review fare?

What could be rated ‘good’?

Research on: Memory

Research on memory plays a prominent role in the review, rightly in my view. Supporting students to remember key concepts and ideas is a fundamental part of teaching.

The broad evidence base on memory that is covered, which includes studies from the lab and the classroom, provides insights that are relevant to all educators, and that are justifiably generating wide interest among teachers and researchers alike.

When new models of learning are introduced, it is helpful to draw on different types of research, from developmental studies in universities to trials in schools, and from a broad range of disciplines, including psychology, sociology and neuroscience.

But it is also reasonable to ask about the relevance of particular findings to English schools, particularly in areas that are new or where the evidence base is evolving quickly. We should ask detailed questions about what extra information is needed to move from lab studies to specific decisions about lesson or curriculum design, even when those studies appear to show great promise.

Overall, the review handles this challenge of context well. It is upfront about the need for more evaluations that take findings from cognitive science and apply them in the classroom, for example by citing ongoing research on spaced practice funded by the Education Endowment Foundation (EEF).

Approaches such as spaced learning and retrieval practice hold promise, and ideas from neuroscience and cognitive psychology, such as the distinction between long- and short-term memory, can help teachers understand how children learn.

But understanding exactly how to apply these ideas every day in the classroom is not a trivial endeavour. We know this from our own experiences at the EEF, where we funded £6 million worth of research on projects informed by neuroscience. All the studies funded had evidence of promise from lab studies, but when they were taken to the classroom several failed to convert this promise to an impact on learning.

For me, we are ready for the next step in conversations about memory. The first step is to make more fine-grained distinctions between those practices based on insights from cognitive science that have been transferred successfully to the classroom, across a range of subjects, and those that are at an earlier stage of development.

It is not wrong to talk, as the review does, about the promise of interleaving, which involves deliberately mixing up the order in which content is taught. But subject leaders would be right to ask for more classroom studies before using this finding to justify a decision to transform their schemes of work.

For example, in their helpful recent book, Understanding How We Learn, Yana Weinstein, Megan Sumeracki and Oliver Caviglioli argue that there are a number of outstanding questions about whether subjects, topics or question types should be interleaved, highlighting a study that interleaved science and language instruction but did not enhance learning.

Ofsted’s review undoubtedly begins to provide this detail, for example by flagging up the fact that the evidence related to interleaving is currently strongest within mathematics, and, in my view, this is the type of nuance we need to move forward.

Research On: Differentiation

Ofsted has historically struggled with differentiation.

The belief that the perfect Ofsted lesson should contain three “must, should, could” learning objectives, complete with attendant resources and activities, did damage both to teachers’ professionalism and their wellbeing. It created a type of managerialism that squeezed out thoughtful practice and, at its worst, capped expectations of children who needed to be challenged and supported most.

But this evidence review makes a significant step forward. After acknowledging the fact that children are almost certain to make progress at different rates, it draws a helpful distinction between differentiating by outcomes and differentiated support.

While providing more focused support to children who are not making progress is recommended, perhaps by providing targeted catch-up, creating individualised learning plans or a multitude of differentiated resources is not.

This distinction, which is supported by evidence on individualised instruction in the EEF’s Teaching and Learning Toolkit, and by a recent meta-analysis combining 78 studies conducted in primary schools, is helpful both pedagogically and from the perspective of workload. Indeed, by focusing on “adapting teaching”, the review appears to be moving away from the term “differentiation” altogether. This is ambitious, but I think worth a shot.

In the same section of the review, the authors take the opportunity to state clearly that there is no evidence supporting the idea that pupils have distinct and identifiable learning styles, a finding supported by several high-quality studies. Given the stubbornness of the myth, this point is very helpful.

Research on: Group work

There is a risk that schools end up with broad curricula, but narrow pedagogy. In reality, in both areas, breadth matters.

That’s why I was pleased to see group work appear in Ofsted’s evidence review. Some might make a lazy presumption that teaching a well-structured, sequenced curriculum demands uniform lessons. But, in fact, a wealth of studies exploring the impact of collaborative and peer learning suggest that this couldn’t be further from the truth, and it is encouraging to see this noted in the review.

Also heartening is the review’s clear statement that in order for group work to be effective, the design and application of activities is key. There is great value in this focus on implementation, which is repeated in other sections and reflected in the framework itself. As Dylan Wiliam memorably said, “Everything works somewhere, and nothing works everywhere,” which is why the detail is so essential.

In the case of group work, the review highlights the importance of clearly structuring tasks and ensuring that ground rules are in place to enable students to work together effectively, both factors that are highlighted in studies in the EEF Toolkit. Effective collaborative learning is much more than just sitting pupils together and hoping for the best, and it is reasonable to underline the reality that getting it right requires effort and planning.

What could be rated ‘requires improvement’?

Research on: Reading

Nowhere is breadth more important than in the teaching of reading. However, on this dimension, I am concerned that the review presents mixed messages that could lead schools towards an overly narrow approach.

The review rightly devotes considerable space to research on reading, and helpfully sets out five essential components of reading instruction: phonemic awareness, phonics, fluency, vocabulary and comprehension.

However, the text itself lacks balance, and could be misinterpreted by schools.

The section on reading places heavy emphasis on phonics. In many ways, this is justified. The evidence for phonics is extremely strong. Indeed, it is the only section of the EEF’s Toolkit where the evidence is so strong that it’s been awarded the maximum evidence strength rating of five padlocks. While it would be useful to conduct more research on phonics in the early years, new studies in primary and secondary schools underline the positive impact phonics can undeniably have.

However, the review gives substantially less space to other aspects of reading. In particular, I would have liked to see more coverage of reading comprehension, including research related to the classroom strategies that appear to be most effective.

While the review does highlight the potential of “direct instruction of comprehension strategies”, elsewhere, readers appear to be warned away from “developing generic reading comprehension strategies rather than the subject knowledge required for understanding”.

The danger here is in creating a false dichotomy. Undoubtedly, and as established by a large number of high-quality studies, subject knowledge supports understanding. But an equally substantial evidence base - eight meta-analyses in the Toolkit - supports instruction in reading comprehension strategies. Some have argued that these strategies can be taught quickly and then ignored, but, in my view, the evidence on this is not strong.

The key point, though, is that it is not a competition. Both background knowledge and strategy instruction can improve comprehension, and, more broadly, each of the elements introduced above can support children’s reading development as part of a balanced approach.

When considering language more broadly, the evidence in favour of activities about communication and talk is also compelling. Again, the pitfall to avoid is adopting a belief that choices about literacy instruction are mutually exclusive.

Research on: Metacognition

In educational research, errors of omission are much more common, and harder to stop, than errors of commission. When reviewing a large body of literature - and Ofsted could hardly have chosen a broader one - ensuring that all relevant spheres of study are included is very tricky.

However, an area I would have liked to have seen more coverage of is metacognition.

There is strong evidence that developing students’ knowledge of metacognitive strategies and ability to self-regulate - such as how to plan, monitor and evaluate learning - improves outcomes. By modelling how experts break down and think about complex tasks like writing an essay or drawing a portrait, teachers can develop students’ self-belief and understanding of how to succeed. Scaffolded tasks, such as worked examples and guided practice, allow pupils to develop their metacognitive and cognitive skills without being overloaded.

The research base related to these approaches has grown over many years, and has been widely popularised by US academics such as Barak Rosenshine and John Dunlovsky. But this evidence has also been extended by recent work in England, including trials in primary and secondary schools, and guidance on metacognition and self-regulation published in 2018 by the EEF. In fact, and for full disclosure, the review underpinning this guidance was co-authored by Muijs himself.

Including metacognition within the review would also have provided an opportunity to challenge an unhelpful binary between knowledge and skills. A widely held misconception is that metacognition is a general skill that can be separated from subject knowledge. In fact, metacognitive strategies are most effective when they are introduced in relation to specific tasks and content. When they are introduced effectively, they can help students to master this knowledge more quickly than might otherwise have been the case.

Methodology

My final point sounds nerdy, but it’s worth making anyway. There is a risk, when conducting research, and in particular when completing a review that draws on a wide range of different studies, that you search for evidence to support a view you already hold. Less deliberate, but possibly more common, is the danger of being drawn towards a conclusion by an unconscious bias or because of our natural tendency to seek patterns, even in the messiest data.

One way to try to reduce these risks is to make the process of conducting research as structured and systematic as possible. In the case of a programme evaluation, it might mean preregistering a trial, which requires the researcher to state as clearly as possible how the results will be analysed once the data is in. In the case of a review, it might mean specifying the terms that will be used to search for evidence and setting clear criteria for which studies will be included or set aside.

This approach, taken by organisations like Nice in healthcare and the EEF in the construction of our Toolkit, reduces the risk of missing important studies or of getting a distorted picture of an area; for example, because the most prominent studies are the most positive.

There are, of course, reasons why taking such steps might be impractical, particularly for a project with as broad a scope as Ofsted’s new inspection framework. It should also be reiterated that the publication of any review at all is a big step forward. But now that we have seen that the bar can be raised, we should strive to keep pushing it higher.

Final thoughts

Transparency, as I said, takes bravery, but it can build trust.

By setting out the evidence behind its new proposals, Ofsted has created the conditions for a professional debate about teaching and learning that would not otherwise have been possible. In doing so, it has also responded to an increasing appetite for evidence from teachers and school leaders.

Showing your working does not mean that disagreement will end. But it makes it much more likely that challenges will be constructive, and, in turn, that they will generate light instead of heat. Now, more than ever, this should be celebrated.

Sir Kevan Collins is chief executive of the Education Endowment Foundation

This article first appeared in the Tes magazine issue of 1 February 2019 under the headline “Marking Ofsted’s homework”