Every few years, we are treated to a barrage of articles detailing how countries are performing in international studies such as Pisa (Programme for International Student Assessment), Pirls (Progress in International Reading Literacy Study) and Timss (Trends in International Mathematics and Science Study).
The reaction in political terms depends on how the country is doing: did it perform relatively well? If so, then ascribe this to policy changes. Did the country do relatively poorly or was there a decline? Blame the previous government and use it as a reason for implementing your favoured educational change.
Such political games have caused some people to be very dismissive about the results of international studies. But I think that such results can be useful - when handled correctly.
We have to accept that, given the complex challenge of comparing countries at an international level, analysis is difficult. It might seem easy to study this data, but this is not the case. Noting within-country changes, taking socioeconomic factors into account as well, can be insightful regarding policy changes, but there is a delay, and correlation is not causation.
The best way, in my opinion, to compare countries is to look at a broad range of sources: take all the information points you can to put the Pisa, Timms or other study into a larger landscape of data. Otherwise, we risk distorting the picture.
One area where this is apparent is school climate. Teachers, students and principals will all differ in their views on bullying, behaviour, classroom climate and so on. We need to be more aware of this when we take headline figures as being generalisable. Quantitative and qualitative sources paint a picture and can tell us interesting things. But these should always be contextualised.
This is not to criticise the studies. It is very hard to do international studies well. The large studies that the Organisation for Economic Co-operation and Development and International Association for the Evaluation of Educational Achievement provide are certainly top quality when it comes to methodology. However, interpretation of their findings is a different matter altogether. Results from studies such as these should be the starting point of a discussion, not the end point.
Christian Bokhove is a lecturer in mathematics education at the University of Southampton
