Lower-attaining pupils have done better in English and maths compared with pupils “on the borderline” since the Progress 8 performance measure was introduced, research has shown.

The research, published by the University of Bristol alongside FFT Education Datalab, interrogated whether Progress 8 had “encouraged schools to work more equitably” by focusing less on borderline pupils.

Progress 8 became one of the Department for Education’s main performance measures for schools from 2015-16.

• Background: How does Progress 8 work?
• Criticism: ‘Progress 8 is biased and fundamentally flawed’
• Covid: Can Progress 8 survive coronavirus?

Before this, schools had been measured on the proportion of pupils who achieved at least five good (A* to C) GCSE grades including English and maths.

This performance measure was replaced amid concerns that it had encouraged schools to focus disproportionately on pupils who were on the C/D borderline.

As part of the research, FFT analysed how attainment changed for “borderline pupils” relative to other pupils after the introduction of Progress 8.

To do this, FFT split pupils into three groups.

In a new blog post, Dave Thomson, chief statistician at FFT, and Professor Simon Burgess of the University of Bristol said this was done by creating a proxy group based on attainment at key stage 2.

They added: “We define the borderline group as pupils with a 40-60 per cent probability of achieving grade C or above (later grade 4 or above) in English and mathematics in each year.

“We then define two other groups: those with a higher than 60 per cent probability and those with a lower than 40 per cent probability.”

FFT analysed data between 2012 and 2017, looking at the change in standardised scores in English and maths for pupils in the above-borderline and below-borderline groups relative to the borderline group before and after the introduction of Progress 8. 

The research analysed data from 2014 as researchers thought they “might see schools begin to anticipate its introduction”.

Results show that there was a gain of 0.01 standard deviations (SD) in GCSE English and maths for the above-borderline group and 0.07SD for the below-borderline group, meaning those pupils in the latter group made more relative progress.

The authors highlighted that “the effect for the below-borderline group increased in each of the first three years since Progress 8 was introduced”.

The graph above can be viewed on FFT’s blog.

The authors added that “0.07SD means very little in the real world”, explaining that “in 2018, it translates to 0.25 points in English and maths scores (equivalent to an eighth of a grade in both subjects)”.

Mr Thomson and Professor Burgess said that the effects are “small but not trivial”.

They also said that while Progress 8 is “clearly not perfect”, it is “much more equitable to pupils than threshold measures of raw attainment”.

Outcomes were measured using standardised scores in GCSE English Language and maths. 

Mr Thomson and Professor Burgess said they were “cautious in presenting these results”.

“In the paper, we note the issue of trends subsequent to announcement but before implementation. We therefore judge the results to be supportive of the hypothesis but not clinching.”

The provisional Progress 8 measures for secondary schools will be published by the DfE this Thursday, for the first time since the onset of the Covid pandemic.

But FFT said it was “questionable” how much the “measures of school performance” would be able to be analysed this year “given the differential effects of the pandemic on attendance”.