Let’s give credit when credit’s due. In its report Do two inspectors inspecting the same school make consistent decisions? Ofsted has tried to meet head-on one of the major criticisms levelled at the inspection process - the inconsistency of inspection judgements.

Although the report’s “questionable”  title is a tad misleading, the inquiry on which it is based represents a refreshingly more open attitude than the “no problems, no warts” assumption of previous Ofsted inspection regimes.

The report has a number of commendable features.

It is suitably detailed and provides a clear account of how the inquiry was conducted, including the inevitable compromises between the ideal and the practical that had to be made in the real-life, non-laboratory circumstances of “live” inspections.

Also, it does not claim too much; its exploratory status and findings are made clear both in the report itself and the chief inspector’s comments on it.

A particularly noteworthy feature is its tentative tone: for example: “The small number of inspections and the specific context of the sample mean the results of the study should not be generalised more broadly.” Or again, “There is reasonable security that the 24 completed inspections were carried out independently.”

Gone are the cast-iron certainties that dominate the text of school inspection reports. Inspectors could benefit from being more cautious about their findings, as this report is.

The authors acknowledge the subjectivity inevitable in the inspection process. They also valuably demonstrate the inquiry’s concerns to protect the participating schools from undue pressure, though pressure there inevitably was.

Also of note was the use (albeit limited) of outside observers to check on how far the participating inspectors worked independently when arriving at their judgements,

Unsurprisingly, (but perhaps a little prematurely) the report makes much of the main finding that in just over 90 per cent of cases there was agreement between inspectors on the overall  judgement they had to make - ie, whether the school remained good or whether more evidence needed to be collected.

‘Almost too good to be true’

But remember this was only a binary choice. That finding of 90 per cent-plus is good, almost too good to be true, but then two 10-year olds faced with the same decision would probably come out with at least around 50 per cent agreement.

Appropriately the chief inspector recognises that the inquiry is but the start of a more wide-ranging, evidence-informed process. With its focus on short inspections of primary schools of above-average size, the study is a relatively easy place to start, though that “easiness” is complex enough, as the report illustrates.

The inquiry’s approach needs to be replicated - initially, in more than 24 primary schools. It’s probably unlikely that that replication will result in another 90 per cent-plus figure, but it may and it would be good news if it approximated that percentage.

Ofsted needs to tackle the reliability of inspector judgements in full inspections of both primary, special and secondary schools where more than binary judgments are involved.

Such wider replication would be logistically very complex and demanding of Ofsted’s resources, but there will be pressure to undertake these studies now that the reliability genie is out of the inspection bottle.

Rightly or wrongly, there is more concern in schools over the reliability of Ofsted inspectors’ judgements, rather than those of HMI but such Ofsted inspectors’ judgements were not used in this initial inquiry. They need to feature prominently in any future reliability studies.

The report attributes the high-level of inter-inspector agreement in part to Ofsted’s own quality assurance procedures and to its training programmes, but provides no evidence to support that assertion. Here, the report might be claiming too much.

So, to quote the report’s title, do two inspectors inspecting the same school make consistent decisions?  The tentative answer is positive but only in respect of short inspections of above-average-size primary schools, each undertaken by two HMIs who have undergone unspecified training and who are faced with just a binary choice - the school remains good or more evidence is needed.

That’s certainly an endorsement but inevitably a very limited  one.

And then there is the even more vexed and problematic issue of the validity of inspection processes - an area which the chief inspector would like to explore.

She would do well to heed the words of Geoffrey Vickers in his book, The Art of Judgment: value judgments “cannot be proved correct or incorrect; they can only be approved as right or condemned as wrong by the exercise of another value judgment”.

Inspection is value-saturated; its value judgments cannot be proved, only justified.

Colin Richards is a former primary school teacher, university professor and HMI

Want to keep up with the latest education news and opinion? Follow TES on Twitter and like TES on Facebook