A research expert has questioned how Ofsted can reliably inspect teaching, after a new study showed that observers assessing the same lesson can produce different findings.
Professor John Jerrim, professor of education and social statistics at UCL Institute of Education, has shared findings from his upcoming paper exclusively with Tes. It assesses how teacher self-efficacy is linked to student-, teacher- and expert-rater views of lesson quality.
The report comes after Ofsted told sector leaders last month that teaching could be assessed as a separate category under draft plans for its new inspection framework.
How would Ofsted assess teaching quality?
Professor Jerrim found that when two observers are making judgements about the same lessons, there are only “moderate levels of agreement”.
For example, when judging classroom management within the same lesson, the correlation across raters stands at just 0.34. For this report, a correlation of 1 indicates a perfect correlation and 0 equals no correlation. Values below 0.3 are generally considered to be weak, and values between 0.3 and 0.6 moderate.
The report says that it is clear that “there are still discrepancies in the views of expert observers judging exactly the same lesson”.
Judging teaching quality, even for experts looking at the same lesson and with very clear criteria, is “really hard to do”, Professor Jerrim told Tes, adding that there is a lot of “inconsistency and disagreement”.
“Given the limited time Ofsted has to inspect, and often small inspection teams, I don’t think they could judge teaching from lesson observations with a high enough degree of consistency or reliability,” he said.
Professor Jerrim said the paper, which will be published in the American Educational Research Journal this month, raises the question of how a potential “teaching” Ofsted judgement should be formed: “How exactly do inspectorates judge quality of teaching, if not basing it on observations of lessons? And how reliable or consistent are these alternatives?” he asked.
Lee Owston, Ofsted’s national director for education, said last month that the school watchdog will not inspect individual staff members under its new report-card model.
Expert observer judgements ‘threatened by bias’
Professor Jerrim’s report is based on a study carried out in eight different countries, including England.
Expert ratings are considered the most objective measure of teaching quality, according to the report, but it warns that this rating method is “not without its disadvantages”.
These types of external observer judgements are based on a limited number of classroom observations, which can have an impact on the accuracy of the ratings, the report advises.
The presence of observers may “alter teachers’ behaviour in the class” and their ratings are also “threatened by bias associated with the characteristics of observers, teachers and classroom settings”.
To judge consistency between different expert observers, Professor Jerrim calculated the average score across segments that each rater gave to each lesson; this then found the correlation between the two observers that were rating the same lesson.
Different expert observers also tend to make different judgements about aspects of teaching quality when basing them on different lessons given by the same teacher, the report finds.
Ofsted declined to comment.
For the latest education news and analysis delivered every weekday morning, sign up for the Tes Daily newsletter
