Some low-achieving students have been “systematically excluded” from the Programme for International Student Assessment (Pisa) data for England, a leading academic has warned, undermining the global rankings, which are heavily relied upon as a basis for education policy decisions.

This means the 2018 data could have “potential biases”, leading to some scores becoming “inflated”, according to John Jerrim, a professor at University College London, who wrote England’s official country report for Pisa 2015.

For example. Professor Jerrim’s analysis of the 2018 maths score for England shows it could be around 11 points too high, as low-achieving students are underrepresented in the sample.

Exclusive: ‘Investigate serious flaws in England’s Pisa data’

Explainer: What is the Pisa test and what does it measure?

Research: Big questions on the quality of England’s ‘odd’ Pisa data

Background: Gove accused of building on shaky Pisa foundations

Pisa 2018: Will Pisa results be Gove’s report card?

Professor Jerrim has called for an independent review into the data with an open letter to the Office for Statistics Regulation.

Run by the Organisation for Economic Cooperation and Development, Pisa tests reading, maths and science skills in 15-year-olds every three years in a number of education systems around the world.

Concerns over the reliability of Pisa data

Its outcome and rankings have been very influential in the education policy making in England. Michael Gove, when he was education secretary, used England’s performance in the rankings as a justification for sweeping school reforms.

But in a paper published today online and in the forthcoming edition of Review of Education, Professor Jerrim, who has written extensively about Pisa, found that the data in the four nations in the UK has “serious flaws”.

He said: “Pisa is meant to be a representative study of 15-year-olds across the UK. But there are serious flaws with some children being excluded from the study, schools being unwilling to participate, and some pupils not showing up for the test.

“In England and Wales, there is clear evidence that some lower-achieving pupils have been systematically excluded. While what has happened in Scotland is, frankly, a bit of a mess.”

According to the paper, average Pisa scores in England and Wales would likely be around 10 to 15 points lower if a more representative sample of 15 year olds had taken the test.

For Wales, this could mean its performance would fall significantly below the international average, taking Wales on a par with countries such as Malta and Belarus

In England one of the problems was the number of schools, and the number of students within participating schools, that chose to not take the test in 2018 - a concern Professor Jerrim had raised in 2019 after the publication of the Pisa report.

An investigation into the potential bias that this situation could have created in the data was conducted but not published, and Professor Jerrim obtained information via a freedom of information request.

He found that schools with lower levels of historic GCSE performance were more likely to refuse to participate in Pisa 2018.

He also compared the GCSE grades of the students participating in Pisa with the national average.

What he found was that lower-achieving students were underrepresented in the sample: 21.3 per cent of the Pisa 2018 sample for England failed to achieve a grade 4 in their GCSEs, compared with 28.5 per cent of all students nationwide in 2019. 

And he found that the Pisa 2018 cohort had notably more students at grade 5 and above than the nationwide figures.

Across the whole of the UK, Professor Jerrim found that around 40 per cent of students are not included in the Pisa data, one of the highest percentages in participating countries. 

But a spokesperson at the OECD said that this is not the case. They said: “According to the Pisa Technical Report, the United Kingdom overstated the size of the target population by about 6 per cent (corresponding to the proportion of students which were later found to be ineligible, in participating schools), and removed about 5 per cent of the target population through deliberate exclusions at school or student level.

“The remaining, and largest portion of the difference between the initial target population and the Pisa sample (before non-response adjustments) is simply the compound effect of school non-participation and student non-response.

“This reflects the behaviour of schools and students, and certainly not the intention of countries or their data-collection agencies.”

The OECD added that Professor Jerrim’s estimate that scores in England and Wales have been inflated by the unrepresentative sample is “based on a number of assumptions and cannot be confirmed by the OECD.”

They said: “At the same time, the OECD cannot rule out that exclusions and non-response at student level resulted in a small upward bias, in particular based on the new evidence provided by Jerrim, which corresponds to a form of non-response bias analysis using auxiliary data. 

“Such data (GCSE results for Pisa participants and non-participants) were not available at the time of data adjudication and are not available for most other countries.

“England and Wales are, in this respect, in the privileged position of having national examination results available for most of the Pisa cohort.

“Few other countries have national examinations around the age of 16. The OECD will ensure that this evidence is considered in future discussions about the appropriate sampling standards and supports efforts to assess the quality of Pisa even beyond the publication of the datasets.”

Commenting on the paper, a spokesperson for the National Foundation for Education Research, which compiled the England Pisa national report, said: “NFER recruited the sample using the rigorous procedures defined by the OECD and followed by all participating countries.

“The sample of schools and pupils which participated in Pisa 2018 met these standards. This paper does, however, raise some interesting points, which we would be happy to explore further.”

A Department for Education spokesperson said: ”​The Pisa 2018 data for England met the standards of the OECD and was fully accepted as valid and suitable for its international reports. Our data is subject to rigorous scrutiny processes and the OECD was satisfied that the school data was representative of the sample schools chosen.”