“We find it very difficult to actually motivate students to take the Pisa exam very seriously. And given the fact that so much is dependent on the Pisa results in terms of policy and the future of education it really makes us worry about that.”

Those off-the-cuff remarks from a head about her anxiety over pupil performance in the Programme for International Student Assessment offer a potent, if unwitting, reminder about the malign power of testing and accountability to warp teachers’ priorities.

On the surface her comments, at the Global Education and Skills Seminar in Dubai on last weekend might seem perfectly reasonable. Pisa is a test after all and this head, also from Dubai, just wants her pupils to do well in it.

Quick read: ’No evidence more testing drives pupil anxiety,’ says Pisa boss

Academic advice: ‘Ignore Pisa entirely,’ argues top academic

Long read: Does Pisa really tell us anything useful about schools?

And she’s right - Pisa has become increasingly important in many countries, and influential on policy - a lot can depend on it. But it shouldn’t do in the way this head suggests. Firstly the study is not designed to rate the performance of individual schools, it is a sampling exercise designed to show how a country’s entire schools system works.

No need to worry

So if you’re a head - national pride apart - there should be no reason at all to worry about Pisa performance. In fact you may well actually benefit from a poor Pisa showing by your country as it could prompt your government to invest more in education.

But that wasn’t how this head saw it. The fact that her pupils weren’t motivated by Pisa - because they “don’t see their results” - was a real concern for her whole teaching staff, she said. 

The head’s words demonstrate the frightening tendency for high stakes tests to exert far more influence over the classroom than is appropriate. In England’s primaries pupils can be put through hours of preparation for SATs that will make no difference at all to their futures. And now Pisa - a high level international study designed to give insights to policymakers in government - is apparently leading to teacher anxiety at classroom level.

It is a ridiculous situation. But to be fair to Andreas Schleicher, the OECD official who designed and runs Pisa, it is not something he has encouraged - well not directly anyway.      

Pisa doesn’t want cramming

“I have a lot of sympathy with this issue,” he responded after the anxious head explained her difficulty “for us at ground level to motivate students to take it seriously” due to them not receiving their Pisa results.

“But what you refer to as a problem is actually a deliberate design feature of Pisa,” Mr Schleicher said. “I could have solved that very easily. I could have given every student their results, I could have produced training material, I could have reproduced the same kind of problem that many education systems have - that students cram just what they need to know for the test and then do well.

“The test of truth in life is not you know what you learn in school but are you willing and motivated to continue in a low stakes situation to learn? That is really what we wanted to test.

“We intentionally did not create the link between immediate rewards and a test but we wanted to see to what extent were students were willing and able to engage with real life problems - that is the test of truth for all of us every day in life.”

In other words the OECD went out of its way to design an assessment that would not end up simply measuring cramming or teaching to the test. It would instead examine how well equipped students were to apply the knowledge they had acquired in schools so that policymakers could respond accordingly.

‘Useless’ rankings

So Pisa is blameless for this classroom anxiety then? Not entirely. The underlying reason for the head’s worry was highlighted earlier at the same meeting when Portugal’s secretary of state for education, Joao Costa complained about the headline country rankings produced by Pisa. He argued that by acting as a rating agency for the schools systems - “the Standard & Poor’s of education” - the study was obstructing its own usefulness.  

“The data provided by Pisa is very, very rich, it is extremely rich,” he said. “But the risk is that we only look at Pisa’s ranking of countries and this is the most useless part of Pisa.”

The simplistic country rankings were diverting attention away from the much more subtle insights that Pisa might offer. They were getting in the way.

And you can see his point. The study was designed to be diagnostic, not a contest that countries compete in to win. You don’t get any prizes for being top of Pisa, maybe a few edu tourists but nothing else. This is supposed to be measure of what makes a good education system - but not an end in itself.

The confidence intervals that qualify the country rankings published by Pisa show they are often not that reliable anyway - as the OECD has admitted “large variation in single (country) ranking positions is likely” depending on the sampling. But inevitably it is that simple metric that has come to dominate the agenda for journalists and ministers. And now, frighteningly, it seems the Pisa ranking hysteria is feeding down to individual schools.

What’s the point?

If anyone, in any part of a schools system, starts going out of their way for the sole purpose of trying to get a better Pisa score then they really ought to ask themselves why. If you get your country to punch above its true standard by trying to put extra pressure on teachers and pupils to do better in the tests then what have you achieved?  A slightly higher Pisa score at best (although Mr Schliecher has said the tests are designed so that prepping doesn’t work).

But what’s the point in that? You wouldn’t have actually got yourself a better schools system by inflating your Pisa score, just put pupils and teachers under a bit more pressure.

Of course when the latest Pisa results are published later this year ministers around the globe will note their country’s ranking position and claim credit for it, make excuses for it, or promise to act on it. That is the reality. These simplistic rankings have become politically impossible to ignore. But if ministers really want their school systems to thrive then they need to ensure their teachers do not succumb to the same Pisa rankings pressure.