International comparisons of education systems are not useful because they are plagued by missing data and the private tuition of students, and because policies cannot be simply transplanted to other countries, a world-respected education academic has said.

Speaking last week at the Best in Class summit organised by the Sutton Trust in New York, Dylan Wiliam, emeritus professor of the UCL Institute of Education, poured criticism on international league tables like the Organisation for Economic Co-operation and Development’s Programme for International Student Assessment (Pisa).

Appearing on a panel that included the boss of Pisa, Andreas Schleicher, Professor Wiliam said: “Like everybody else in this room, I am really interested in how we raise levels of achievement in general and to close the achievement gap.

“Probably unlike most people in this room, I am not particularly convinced about the usefulness of international comparisons for directing this effort in a particular country for a variety of reasons.”

He then went on to list what he considered to be a number of major flaws with international comparisons, including:

• Missing data – “We know that a number of students were not sampled in Shanghai for example, so we can’t be sure that those are representative of all the students living in Shanghai,” he said.
• Private tuition giving a misleading impression – “Singapore is clearly very successful,” Professor Wiliam said. But he added: “I happen to think it’s mostly private tuition. We know that in [South] Korea for example if you take out the effects of tuition, the performance of kids in Korea could well be below that of the United States.”

Professor Wiliam also pointed out that some systems celebrated for their high performance actually appear to be stagnating. “Ontario’s results have basically been flat for 20 years,” he said. “The reason that’s quite important is because if what they were doing in Ontario was good educationally, then as children have been exposed to one, two, three, four, five years of this system, the results should be going up, and they’re not. And we see results in Finland actually going down."

He said that, even if problems with Pisa could be overcome, he was still “sceptical” about the value of international comparisons because education policies cannot easily be transplanted to other countries.

“Even if we know why those countries are high performing, could we implement that in the UK or England? The answer is probably not.

“I’m very sceptical about even if we can figure out good research from this, whether these would provide us with solutions.”

After the summit, Professor Wiliam told Tes: "The problem is that people claim to know why particular countries get particular results, and there are far too many variables, and far too few data points, for this to be possible."