“Pisa does not presume to tell countries what they should do. Pisa’s strength lies in telling countries what everybody else is doing.”

This statement in Andreas Schleicher’s latest book, World Class: how to build a 21st-century school system, exemplifies the self-contradicting nature of the Pisa enterprise.

For nearly two decades, this triennial assessment program has been telling education systems what they should do, despite its claim of the opposite. In this book, Schleicher, the chief orchestrator of Pisa, brings together what Pisa has been telling, without telling, what education systems should do in order to become successful and secure better Pisa outcomes. Many of the recommendations, however, are confusing and meaningless at best and destructive at worst because they are drawn from self-contradicting evidence.

The cornerstone of the Pisa enterprise is its claim to be able to identify successful education systems and extract lessons from these systems for others to emulate. But the evidence it uses to judge success is inconsistent. Until 2015, Pisa has identified successful education systems based on student performance on its assessment. In 2015, the enterprise began to examine student wellbeing and found that “students in low-achieving countries tend to report higher levels of life satisfaction than students in high-achieving countries.” In other words, education systems that are successful in producing good test scores are not necessarily successful in cultivating the wellbeing of young people. If wellbeing is an educational outcome that is worth pursuing, Pisa's previous successful systems are therefore not actually successful. If so, what these systems are doing or have been doing should be avoided instead of being promoted. In other words, when the outcome measure changes so too does the rank ordering of countries in Pisa and the parameters of success. 

Recruiting top graduates into teaching has been a widely promoted policy recommendation drawn from Pisa data because, as Schleicher reiterates, “the quality of a school system will never exceed the quality of its teachers.” He also dismisses “only top graduates should become teachers” as a myth, but at the same time encourages governments to recruit from highest performing segments of the population because that’s what industries and organisations do. This is part of the human capital argument that underpins and justifies both Pisa and the pursuit of 21st-century skills, Then Schleicher brings evidence to show that, actually, students' Pisa performance can exceed their teachers: “in some countries … teachers’ proficiency in numeracy is average, but their students are top performers in the Pisa mathematics test. In addition, in most high-performing countries, students score above what would be expected based solely on the average knowledge and skills of the teachers in those countries.”

In his book, Schleicher likes to discount the importance of culture as an influential factor in educational performance and achievement. Schleicher treats the belief that “some countries do better in education because of their culture” as a myth that needs to debunked. But at the same time, he points out that the cultural differences in belief in effort versus innate ability between the Eastern Asian countries (eg, Singapore and China) and Western countries (eg, France and the US) are powerful factors affecting students Pisa performance.  Schleicher also uses the rapid improvement in some countries (eg, Qatar, Macao and Portugal) to show that education can improve irrespective of culture and without cultural change. “These countries and economies did not change their culture, or the composition of their populations.” But at the same time, he believes these results show that culture “can also be created – through thoughtful policy and practice.”

Pisa has led to many other inconsistent and self-contradicting findings and recommendations with regard to virtually all aspects of education, such as the value of high-stakes exams, national standards and curriculum, investment, and class sizes. The inconsistencies arise from the lack of consideration of an inescapable, but generally neglected, fact about education: the side effects. In his latest book, What Works May Hurt: side effects in education, Yong Zhao argues that education policies and practices, like medical products, can have adverse side effects while simultaneously achieving beneficial results for a number of reasons.

First, a policy or practice that is effective in accomplishing one goal can impede the realisation of other goals. A strategy that improves short-term mastery of materials can cause damages to creativity and transfer, for instance. In the case of Pisa, even if the factors that make an education system successful in achieving high test scores could be identified with precision, it is perfectly possible that they might hurt student wellbeing. East Asian education systems, for example, have been found to be very effective in producing excellent performances in tests AND hurting students’ social-emotional, psychological, and physical well-being, resulting in less life satisfaction, less positive attitude, and lower levels of confidence. 

Second, policies and practices that benefit some students can hurt others. This is evidenced by the inconsistent finding of the quality of teachers in Pisa. Top graduates do not necessarily benefit all students. It has been found that teachers with high academic performances in secondary schools have been found to benefit high performing students but hurt low performing students.

Third, policies and practices that work well in one context may cause harm in others. The unintended consequences of education reform are well documented and evidenced. School choice, for example, may benefit children whose parents are able to take advantage of the choice but hurt those parents are not in the position to participate. In the Pisa case, this explains why some, actually only a very small number, countries have seen rapid improvements after adopting policies endorsed by Pisa, while many others have not seen improvement. The countries performing less well, after adopting Pisa strategies, tend not to be in the international limelight and the side effects, or negative effects, of policy interference, are less well known. This means that cultural contexts do indeed play a major role in the effectiveness and effects of educational policies and practices.

Pisa has been a major provider of educational cures but it has never discussed the side effects of its prescriptions. It is unlikely that it will voluntarily study and disclose the potential harms of its recommended strategies. Thus it is up to policymakers, education professionals, parents, and students to watch for negative side effects because they exist. What works can hurt. 

Yong Zhao is the global chair at the University of Bath, Professor Alma Harris is a professor of educational leadership and policy and Michelle Jones is a lecturer at the University of Bath