Evaluating complex human activities is hard, especially if it happens in the context of high-stakes accountability.
Many important decisions - driving tests, judicial sentencing, penalties in football matches - rely on human judgement. This inevitably leads to some inconsistency, with different individuals interpreting the evidence in different ways.
In education, school inspections come with high-stakes consequences. And, as school inspections also involve human judgement, it stands to reason that their outcomes depend, at least in part, on the characteristics of the inspector or inspectors assigned.
This is what I, along with John Jerrim and Sam Sims, am studying in our Nuffield Foundation-funded project Inspecting the Inspectorate. We recently published our first working paper in the project, which looked at the Ofsted grades awarded by 1,376 different inspectors across 35,751 school inspections conducted between 2012 and 2019. Interestingly, we think it’s the first time this was looked at by an independent team of researchers.
We used publicly available data, including some from the excellent Watchsted.
Insights into Ofsted school inspections
So, what did we find? We found that, even when comparing primary schools that are similar on all characteristics, female inspectors were more likely to award a low Ofsted grade, while there were no clear differences in the grades awarded by male and female inspectors inspecting secondary schools.
As well as gender differences, we also found that freelance Ofsted inspectors were more likely to award a higher Ofsted grade than inspectors employed on a permanent contract.
Of course, Ofsted inspectors on a permanent contract are also more likely to be assigned to inspect lower-performing schools, which can explain some - although seemingly not all - of this difference.
It’s also worth noting that we found no differences in the inspection grades awarded by inspectors with more or less experience, working inside or outside of their home region, or when inspecting schools within their primary/secondary school specialism.
More by Christian Bokhove:
One important thing to keep in mind is that judgements are often determined by a combination of factors. For example, a school might be in a challenging area but still have students from many different backgrounds on roll. To help us understand this so-called “joint effect”, we looked at how different factors interact.
For instance, we found that a female HMI working with one other inspector has a 49 per cent chance of awarding a primary school an “inadequate” or “requires improvement” grade. This is notably higher than for a male Ofsted inspector working alone, where the chance of an “inadequate” or “requires improvement” grade is just 32 per cent. The difference in outcomes between these two hypothetical inspectors is most notable at the “inadequate” grade.
There were a lot of reactions to our work last week. Some were rooted in feelings about Ofsted itself; others were more about the methodology. And there absolutely are some limitations to our work.
For example, a new inspection framework was introduced by Ofsted in September 2019, which put less emphasis on performance in national examinations and more on the quality of the curriculum. It is unclear what consequences this might have had; in time we will try to look at this, when enough inspections have taken place.
We also conducted our analyses only with publicly available data. It’s fair to say that several variables we used could be questioned. For example, we determined the gender of inspectors based on their names, and only looked at the gender of the lead inspector.
This is why one of the most important recommendations from our current work is that Ofsted makes more data and information about inspections available and that more internal and external scrutiny and analysis takes place on such a high-stakes topic. We were struck by the fact that there actually wasn’t all that much work on it.
Christian Bokhove is associate professor in mathematics education at the University of Southampton and a specialist in research methodologies
