I am frequently staggered by the questionable quality of education policy and its analysis in this country. The latest example came, sadly, in a report put out by an influential thinktank, despite its reliance on impressive statistical work led by rightly respected figures in the field. 

Yet the report, English education: world class?, by the Education Policy Institute, setting out how UK GCSE grades need to improve for our pupils to match the world’s “best-performing” countries, comes across as headline-chasing nonsense on stilts. It’s vulnerable to serious criticism from a range of perspectives. 

If I understand its methodology correctly - though the detail set out in the document seems to leave some questions - the paper saw the researchers looking at the results of individual pupils in England who took the last round of the OECD’s Programme for International Student Assessment (Pisa) tests in reading, maths and science in autumn 2015.

England’s national pupil database was then used to look up what grades those same pupils went on to achieve in GCSEs taken at the end of that academic year.

Some seemingly impressive and detailed statistical techniques were then used to work out, from this, what pupils internationally - including the other UK nations - would have achieved if they had sat GCSEs here, given each country’s Pisa scores. 

Pisa problems

All this may be fair enough, in terms of an interesting statistical exercise. The problem comes in the interpretation.

For the paper went on to suggest, oddly, how “GCSE scores” would need to improve in each of the home nations - though, of course, Scotland does not have GCSEs - in order to match those achieved in national GCSE grades in this study by the five “top-performing” countries in Pisa. 

Here are the problems:

1. The first is the most glaring. The paper offers no attempt to consider whether what it says are “ambitious yet realistic benchmarks” in terms of improved GCSE performance from pupils in England are achievable, given the way the GCSE grading system, at least as it now operates in England, actually works.

This is because this 43-page document offers no discussion of how GCSE grades are allocated. More specifically, it does not discuss comparable outcomes, the system that, in recent years, has effectively capped GCSE grades to combat the previous perceived problem of grade inflation.

This means that exam boards, as overseen by the regulator Ofqual, have to hand out grades in broadly the same proportions year-on-year unless they have good evidence that ability levels or underlying levels of pupil understanding - raising questions as to how this is to be measured - have changed. In reality, this puts a serious drag on possible grade improvements.

Again, it is surprising this was not even discussed in the report.

2. The report assumes that if UK GCSE results were to improve, gains in Pisa scores would, too. But GCSE results in England did indeed rise, dramatically, in the 2000s. Yet England’s Pisa scores, as the report states, remained “stagnant” over the period. So the idea that the one improvement - even if it were statistically possible - would lead to the other must, again, be subject at least to serious questioning. Again, this non-linkage of the two sets of data in the past is not discussed.

3. Seeking to make comparisons between the performance of different nations of the UK based on their pupils’ Pisa scores, the report says Northern Ireland pupils need to improve slightly more than those in England in order to match “top-performing” nations. 

But in the real world of actual GCSE exams, Northern Ireland has long had better results than England: for example, published exam board figures from last year showed that, in English, 78 per cent of pupils in Northern Ireland achieved A*-C, and 22 per cent A*-A, compared to 59.5 per cent and 13.5 per cent in England.

In fact, the consistently higher GCSE results of Northern Ireland pupils compared to those in England, set against the fact that the Pisa scores for the two countries are broadly similar, must surely raise more questions about the ability to link the two datasets.

4. A final major problem is the report’s willingness to view the Pisa results without sufficient scepticism, given the range of criticisms that have been thrown at them. These include sampling questions, the non-measurement of creative subjects and the surely huge contribution to national scores that a range of background factors including high levels of after-school tuition in some Far Eastern countries may be making. The report seems to imply improved Pisa performance should be an end in itself. Politicians may see this as fine; outsiders should surely at least subject such aims to rigorous critical analysis.

5. Overall, the report could be read as simply restating the fact that the UK nations perform worse on Pisa than others around the globe in a slightly different way. Given the problems with using GCSE results in this way, one has to wonder why they bothered. They might just have reprinted different nations’ 2015 Pisa scores.

The Education Policy Institute has made a useful contribution to policy analysis over the year or so of its existence. I, therefore, offer these criticisms reluctantly. It may be that this was an interesting statistical challenge whose results were simply too interesting to resist putting out on GCSE results week.

But these findings will be used, again, as a way of beating English schools over the head. So the report’s authors should expect them to be subject to questioning. For sure, our education system is far from perfect. But if criticisms are to be made, they need to be on a far more solid basis than this. 

Warwick Mansell is a freelance education journalist and author of Education by Numbers. You can read his back catalogue here. He tweets @warwickmansell

Want to keep up with the latest education news and opinion? Follow Tes on Twitter and like Tes on Facebook.