An influential finding from the world’s best-known education study showing a link between greater school freedom and better results is wrong, according to new research.
The Programme for International Student Assessment (Pisa) study has concluded that when principals are given more independence over spending, their schools achieve better academic results, as long as those results are published. “Autonomy matters when combined with accountability,” the study first advised in 2010. The recommendation has influenced education ministers and was repeated in the latest Pisa report in December.
But an academic from an international exam board has carried out a more sophisticated analysis, concluding that Pisa got it wrong. Tom Benton, principal research officer for Cambridge Assessment, found no statistically significant association between the amount of freedom given to schools over how to spend their budgets and their academic results. “With schools in developed nations there doesn’t seem to be a relationship,” he told TES. “I find the (Pisa) evidence quite unconvincing.”
The news will be a blow to politicians including Michael Gove, England’s education secretary, who has used the finding to help justify his controversial academies expansion policy and his free schools programme, both of which are designed to give school leaders more freedom.
The finding follows significant challenges last year to the credibility of Pisa, which is helping to shape education policy in a growing number of countries. Academics claimed that the country rankings produced by Pisa were “useless” and “meaningless” because the statistical model used to calculate them was either unsuitable or “utterly wrong”. A further study cast doubt on the quality of Pisa data collected through school questionnaires.
The importance of freedom for schools has been a particularly influential Pisa finding. In January 2012, Mr Gove said Pisa had “clearly” shown that “the greater the amount of autonomy at school level, with headteachers and principals free to determine how pupils are taught and how budgets are spent, the greater the potential there has been for all-round improvement”.
However, Dr Benton notes in his report that when Pisa 2009 (published in 2010) looked at individual school autonomy, it did not find any significant link between curriculum freedom and students’ results.
Pisa did find that “within countries where schools are held to account for their results through posting achievement data publicly, schools that enjoy greater autonomy in resource allocation tend to do better than those with less autonomy”.
But Dr Benton looked at the same data that Pisa based this claim on and found no such link. He examined the figures for state and independent schools separately and concluded that the relationship did not exist in either sector.
The researcher believes he has reached a different conclusion because his analysis is “more fine-grained”. When considering other factors that might explain individual school performance, Dr Benton took into account the fact that the difference between achievement in private and state schools varied “significantly” between countries - a fact that the Pisa calculation glossed over.
He also included the significant variation of other background factors, such as gender and whether the school in question was in the state or private sector - another point Pisa had not accounted for. Each of those factors alone would account for his differing conclusion, Dr Benton told TES.
Tim Oates, group director of assessment research and development at Cambridge Assessment, who advised Mr Gove on the curriculum, said that governments should follow the example of Germany, which conducted detailed analysis into Pisa findings. “There is not much point in people bashing (each other) over the head over the topline Pisa findings... it doesn’t really work,” he said.
Andreas Schleicher, deputy education director of the Organisation for Economic Cooperation and Development, which runs Pisa, stood by its analysis. “It makes no sense to separate out public and private schools from an analysis of the impact of autonomy and accountability on pupil performance... since that is one of the main expressions of school autonomy in many countries,” he said. “Even then the nature of public and private schools varies considerably from one country to another, and retaining that diversity is important in analysis such as this.”
A spokesperson for England’s Department for Education did not comment on the new research, but restated the department’s belief in the importance of school autonomy.
