Ofsted has come under fire for claiming that its team carries out “as much peer review as we can do”, despite the fact that it has chosen not to focus on publishing its work in academic journals.

Donna Neil, Ofsted’s head of research, said in a series of posts on social media that the “sheer number of people” reviewing the watchdog’s work compensated for the fact that it is not subject to scrutiny in peer-reviewed journals.

But Professor Dorothy Bishop, from the University of Oxford, told Tes that the decision was “a bit of a cop-out”.

Podcast: Dorothy Bishop on how the teaching and assessment of young children run contrary to developmental psychology research

Research: Teachers taught to rate grades over knowledge

Inspections: School finances could face even more Ofsted scrutiny

Dr Neill wrote on Twitter: “Our research is conducted in small teams of highly qualified/experienced researchers. Within the team we peer review.

“We have advisory panels that include a wide range of stakeholders such as: academics, professionals in the field of study, third sector orgs, parents and children. These panels critique and provide feedback on our scope, methods and key findings.

“We also receive a lot of internal peer review of our work from policy colleagues, our data and insight department, deputy directors and directors. And every report is personally reviewed and signed off by Amanda (who will read references and challenge us on their methods too!).

“The sheer number of people reviewing our work, the range of people from all sorts of backgrounds and perspectives, both academic and otherwise, and the detail with which reviewers scrutinise our final publications is just about as much peer review as we can do.”

But Professor Bishop, an expert in developmental neuropsychology, took issue with the fact that Ofsted relies on its own team, as well as specially selected “advisory panels”, to scrutinise its findings.

“I would say that you really do need to have somebody who’s not connected with the research to be evaluating it,” she said. “That would normally count as peer review.

“There’s two reasons why it’s a shame if they’re not publishing in journals. One is they’re not getting the more severe scrutiny from peer review, which I think is usually tougher.

“And the other is it’s probably limiting the readership, as well, of the work. Because it’s in a journal, it’s more likely to be read by a wider range of people.

“I think they leave themselves open to the charge that they may be not getting as thorough, or perhaps as rigorous, a scrutiny as they might benefit from.”

Dr Neill made the comments after Daniel Mujis, Ofsted’s deputy director of research and evaluation, wrote in a blog for the watchdog: “Although we may in future publish some of our work in academic journals, this is not our primary aim.”

Professor Stephen Gorard, from Durham University’s School of Education, told Tes: “Perhaps the really key thing for the traditional peer review is, of course, that the author - or the author’s organisation, or the author’s funder - doesn’t get to pick who the reviewers are.

“It’s all very well to have user groups, and all that kind of stuff, but they are picking these other friends of Ofsted. It doesn’t come to someone like me.

“If there’s any pressure on the people...to show that Ofsted has been successful, or Ofsted’s judgement is fair or whatever, then the peer review goes out of the wall. It’s not peer review anymore, it’s something else.

“If there’s a danger that there’s unconscious, or even deliberate, bias, because of conflict and vested interest, then it’s not proper peer review. But it doesn’t mean that other peer review processes are immune to that, either.”

Dr Neill also said that journals should consider “improving” their peer review system to include those who their research is about, or will affect - but Mr Gorard said this could distort results.

He said: “What should be happening, is that judging it solely on the quality of the research. And you should care not one bit what the actual answer is, but only are the implications drawn correct. 

“As soon as you involve users, with the best will in the world, that disappears entirely. So someone suggesting that suggests to me that they don’t understand that distinction.”

Ofsted was approached for comment.