This week has seen the publication of the results of the 2018 Programme for International Student Assessment (Pisa) survey, including country rankings in reading, maths and science. In Scotland, these results have been much anticipated, following the “Pisa shock” of 2015.
The 2018 results have shown a modest improvement in the reading score - from 493 in 2015 to 504, effectively bringing the country’s performance back to the levels in 2006, 2009 and 2012 (but below earlier scores in 2000 and 2003). Scores in maths and science have remained stagnant since 2005, with marginal declines (from 491 to 489 in maths and 497 to 490 in science).
Inevitably, these results have been used to score political points in the immediate run-up to the general election.
Education secretary John Swinney tweeted this:
The reaction to the results has been more negative in the media. The Sun, for example, stated that “despite the up-tick in reading, performance in maths and science has continued to fall”.
So who is right? What do the Pisa results tell us about Scotland? Is there really evidence of a decline in standards, and can this be attributed to Curriculum for Excellence (CfE) and the SNP, as many are claiming?
Pisa results 2018: What happened in Scotland?
Pisa results: No sign of attainment gap cash ‘making an impact’
Pisa: Are the Scottish and English education systems less different than we thought?
It is important to note that the Pisa study is based on a sample, so the measures it produces have sampling errors and cannot be automatically generalised towards all 15-year-old students in Scotland. Therefore, prior to describing a change in a Pisa score in 2018 as an “increase” or a “decrease”, it should be first checked whether the change is statistically significant.
Pisa 2018: How accurate are the results?
The alleged “decrease” in maths and sciences attainment in Scotland compared with 2015 is not statistically significant. The difference in numbers falls within the margin of error in this sort of survey, and the best that can be said is that there is no change between 2015 and 2018 in these subjects.
Moreover, in international comparative terms, these performances fall pretty much in the middle ground in the league table, suggesting an average performance by Scotland. Reading scores have increased from 2015 to 2018, but the difference is small, and also may be a one-off fluctuation. To quote the recent UCL/IoE blog on the UK Pisa results:
“But hold your horses before getting too excited. One good set of results is NOT a trend! And a swing of this size in Pisa can simply be a result of changes in methodology.”
Thus, the media hype about Scotland’s decline in maths and science is not especially warranted, as the evidence is actually pretty underwhelming. Conversely, claims about boosts in reading are slightly more credible - although we note that evidence that this is directly linked to CfE or other interventions such as the Attainment Challenge is limited at best. While the scores have largely remained stable in recent years, there are some interesting nuances in the data.
First, the claim that that only five countries (Canada, Estonia, Finland, Ireland and South Korea) have better scores than Scotland in reading is technically true, albeit with methodological caveats. And yet Scotland is in a company of another 12 countries that have their reading scores in the same confidence interval (where the true population parameter lies).
Perhaps more remarkable is that only three countries have reading scores for boys higher than Scotland (10 countries have reading scores for boys within the same confidence interval). However, Scotland is doing less well when reading proficiency levels are considered; quite a few countries with average reading scores lower than in Scotland have a higher proportion of students achieving reading proficiency level 5 or 6 (including England, Slovenia, United States, Australia, Norway, Poland and Israel).
There are some interesting trends over time that merit further comment. There was, for example, a sharp drop in the reading score from 2000 to 2006 and then stable results between 2006 and 2012, a drop in 2015, before the score bounced back to the level of 2012. If we accept the 2015 result as a fluctuation (which may be a methodological issue), we can safely say that since 2006 there has been little change.
Similarly in maths, we can again see a sharp drop between 2003 and 2006 (well before the introduction of CfE) and similar scores (differences are not statistically significant) since 2006. The big decline in scores took place before the introduction of Curriculum for Excellence (and indeed before the period of SNP government), and therefore it is inaccurate to suggest (as many media outlets seem to be doing) that the decline is due to the current curriculum reforms.
Science is the only area where the drop in attainment might be attributed to CfE: here we see stable results between 2006 and 2012, then a sharp drop between 2012 and 2015, and then a slight (but not statistically significant) decline in 2018. It is not clear why. We can, of course, speculate: changes to the specification of content, increased formulaic teaching to the test, lack of accessibility to triple science for the senior phase, and a decline in practical work are possible suspects. There is, however, a need for more research.
Another very interesting finding relates to immigrant children. Their educational attainment is often considered as an indicator of the success of immigrant integration. Here, the news for Scotland is very positive. In reading, second-generation immigrant students performed higher than or similar to all OECD (Organisation for Economic Cooperation and Development) countries, with only Singapore of the non-OECD countries having a higher performance. Performance among first-generation immigrant students in Scotland was also higher than or similar to all OECD countries.
In maths, second-generation immigrant students in Scotland performed higher than or similar to all OECD countries, with only Singapore and Macao (China) of the non-OECD countries having a higher performance . Performance among first-generation immigrant students in Scotland was also higher than or similar to all OECD countries in maths.
And in science, second-generation immigrant students in Scotland performed higher than or similar to all OECD countries, with only Singapore and Macao (China) of the non-OECD countries having a higher performance. Performance among first-generation immigrant students was also higher than or similar to all OECD countries in science.
It is not unusual for immigrant children to perform better than a country’s majority population children in Stem subjects. Yet, the fact that they are able to perform so well in Scotland might offer some insights into why native Scottish children are not doing equally well. One of the reasons could be a lack of interest and motivation - indicating an important area for policy development.
Professor Mark Priestley and Dr Marina Shapira are lecturers and researchers at the University of Stirling. This is a version of a post that originally appeared on Prof Priestley’s blog
