Ever since Curriculum for Excellence (CfE) was conceived, it has contained, as some of its central aims, attributes that it would clearly be desirable to develop in young people but that are tough to measure.

In Scotland, we might want to mould young people so that when they emerge from the education system, they embody the four capacities: successful learners, confident individuals, responsible citizens and effective contributors. But beyond successful learners - which, it is generally assumed, the exam system measures - how do we know whether or not we are succeeding? And how do we know if the Scottish curriculum - which was overhauled in order to nurture young people equipped to adapt to a fast-changing world - is actually working?

Well, the Organisation for Economic Cooperation and Development (OECD) has come up with an assessment of global competence that it designed to answer a critical question: are we equipping young people to thrive in an interconnected world?

In other words, at a time when countries around the world are grappling with a global pandemic, how well are schools preparing their pupils to be able to collaborate with people from different disciplines, cultures and value systems, at the same time as being autonomous thinkers?

The good news for Scotland is that, while the OECD’s traditional three-yearly Pisa (Programme for International Student Assessment) tests of 15-year-olds’ grasp of reading, science and maths have left us with little to celebrate, this assessment, carried out at the same time as Pisa 2018, gives us a few things that we can boast about. And the huge appetite for its results was evident when we published a story about them online in October: our first tweet about the story was one of the most liked and retweeted ever from @TesScotland, with thousands of people engaging with the story and many saying the findings of the OECD report were a vindication of CfE.

However, there are also some good reasons why Scotland should not get carried away with these findings. We’ll get to those, but let’s start with what went well.

One key aspect of the assessment was a cognitive test that focused on three areas: the ability to evaluate information, formulate arguments and explain issues and situations; the ability to identify and analyse multiple perspectives; and the ability to evaluate actions and consequences.

Scottish students’ mean score on the global competence cognitive test was 534. Only two countries scored more highly: Canada (554) and Singapore (576). (The region of Hong Kong also scored more highly, achieving a mean of 542.)

In the assessment, students were faced with a number of different scenarios in which they had to consider the impact of stereotyping, look at situations from different perspectives, evaluate actions and consequences and weigh up sources to sort fact from opinion.

In one question, for example, they were told of the experience of the Nigerian writer Chimamanda Ngozi Adichie, who went to study at an American university. Adichie’s American roommate was surprised that she could speak English and knew how to work the stove, and was disappointed to discover that she was listening to American pop as opposed to “tribal music”. Students had to explain why the American was surprised and why stereotypes can mislead.

Another scenario involved the Refugee Olympic Team, which competed in the Olympic Games for the first time in 2016. Students were told that a fictional athlete on the team, Felix, had won a medal but his decision to accept it for the Refugee Olympic Team had sparked a debate about whether the medal should actually have been added to the haul of his host country or his home country. The students had to pick out the strongest arguments to support each country’s perspective, as well as consider the perspective of the athlete himself.

According to the OECD report: “At the highest level of proficiency in global competence, students are able to analyse and understand multiple perspectives. They are able to examine and evaluate large amounts of information without much support provided in the unit’s scenario. Students can effectively explain situations that require complex thinking and extrapolation and can build models of the situation described in the stimulus.”

On average, across all countries, 4 per cent of students attained the highest level of proficiency, Level 5, in global competence. The largest proportions of students who scored at this level were found in Singapore (22 per cent), Canada (15 per cent) and Scotland (12 per cent). Meanwhile, 10.5 per cent of students taking the assessment in Scotland were assessed as below Level 1, the lowest level of proficiency, against an average of 26.5 per cent across all countries.

There was no significant difference in the performance of girls and boys in Scotland, with the report stating: “Girls outperformed boys in all countries and economies except Scotland (United Kingdom), where the difference was not statistically significant.”

On average, across the 27 participating countries and economies - which did not include any other part of the UK - the least-disadvantaged students outperformed their disadvantaged peers by 75 points. However, in Scotland, the difference between the two groups was above average, at 89 points.

The other aspect of the assessment was students’ self-reported knowledge, attitudes, skills and dispositions. Scottish students demonstrated some of the most positive attitudes towards immigrants and showed a healthy respect for people from other cultures.

Some 91.3 per cent of Scottish students agreed that immigrant children should have the same opportunities for education as other children in the country, against an OECD average of 85.1 per cent; and 87.5 per cent agreed that immigrants should have all the same rights as everyone else in the country (against an OECD average of 80.2 per cent).

Meanwhile, 87.2 per cent of Scottish 15-year-olds answered “very much” or “mostly” in response to the statement: “I respect people from other cultures as equal human beings.” And 86.3 per cent said they treated all people with respect regardless of their cultural background.

But there are some caveats.

While the questionnaire involved 65 countries - not a million miles away from the 79 that took part in the standard 2018 Pisa process - the number of countries that participated in the cognitive test in which Scotland made the top three was significantly lower. Only 27 countries and economies took part and just 11 were from the OECD, so a lot of the countries that do well in the traditional Pisa assessments of maths, reading and science were not represented.

The result could be painted as being top of the class on the day when most of the high flyers were off ill. And while Scotland was in the top three, Singapore and Canada did significantly better - and these are the two countries that Tarek Mostafa, who leads on the global competence report for the OECD, says provide “relevant benchmarks to compare Scotland to”.

While Scotland’s 15-year-olds had a positive attitude to immigrants, they had a below-average interest in learning about other cultures, particularly concerning the religions of the world - only 29.3 per cent of Scottish students said the statement “I want to learn more about the religions of the world” described them, against an OECD average of 40.1 per cent.

Rowena Arshad, chair in multicultural and antiracist education at the University of Edinburgh and co-director of the Centre for Education for Racial Equality in Scotland is generally sceptical about the value of Pisa, believing it to be a “narrow” measure of educational achievement. She has suggested in the past that while Scotland might not be a top Pisa performer, “we may well be top of the league table for young people who care about each other and who have community”.

Some of the results in the new Pisa study appear to bear this out, but she explains why it is hard to draw conclusions.

“You could argue that some essential life skills needed for the future, such as a respect for difference and human rights, have been instilled,” says Arshad. A valid interpretation of the results, she says, is that they “bode well for us producing skilled workers…able to live and work in diverse workplaces and societies”.

Arshad adds: “However, this set of results also tell us that this same set of Scottish pupils appear to show below-average interest in learning about other cultures. Does this mean that while human rights - for example, for immigrants - are accepted and that there is awareness that it is right to show respect for others, in practice, a parochial mentality continues? We just do not know.”

Another disappointing result for Scotland was on “cognitive adaptability”. It could be argued that one of the key aims of CfE was to create young people capable of adapting to change. When the idea behind our now-not-so-new curriculum was being explained, the need to do things differently was often justified in terms of the world of work that young people were going to enter at the end of their schooling. The oft-repeated line was that the jobs many young people in school would be doing did not exist yet, so the goal of the system had to be to create rounded human beings capable of taking on a range of different challenges. CfE was also built on the assumption that, in the future, young people would have to be lifelong learners and that there was no longer any such thing as a job for life.

Well, Pisa 2018 asked students about their ability to adapt to new situations - and, overall, on this measure Scottish students were found to be below average.

A total of 56.9 per cent of Scottish students answered “very much” or “mostly” in response to the statement “I can deal with unusual situations” against an OECD average of 58.9 per cent. And 69.2 per cent agreed that “I can change my behaviour to meet the needs of new situations” when the OECD average was 67.1 per cent. Some 45.4 per cent of Scottish students said, “I can adapt easily to a new culture”, compared with the OECD average of 49.3 per cent.

The report says: “In 28 out of the 65 countries/economies that took the questionnaire, boys reported greater cognitive adaptability than girls. The largest gaps in favour of boys were observed in Costa Rica, France, Greece, Iceland, Korea and Scotland (United Kingdom).”

Overall, global competence was measured by nine different indices. Scotland was above the OECD average for three: attitudes towards immigrants, respect for other cultures and student awareness of global issues. Scotland was below average for five indices: global mindedness, cognitive adaptability, perspective taking, student interest in other cultures and self-efficacy. Scotland was similar to the OECD average for awareness of intercultural communication.

So, why do the results on the global cognitive test differ from the results of the questionnaires? The University of Edinburgh’s Lindsay Paterson, a professor of education policy, says: “I suspect that the reason for the discrepancy in Scotland is that the test deals with hypothetical case studies, which don’t depend strongly on knowledge. They are tests of competence, not, as it were, of capacity.”

The explanation from the OECD’s Mostafa is simply that the two different assessments measure different things, “therefore it is difficult to compare the two”.

He says: “Some countries do well on the cognitive aspects but less so on the attitudinal aspects or the opposite. In general, the two are positively correlated - or, in other words, students who have positive intercultural attitudes tend to perform well on the test.”

Paterson is also concerned that where Scottish students do well, “the picture is not so good on the role which schools might have played in this”. He points out that Scottish students report that they are less likely to learn about the interconnectedness of countries’ economies (Scotland 39.6 per cent; OECD average 54.9 per cent); about solving conflict with classmates (Scotland 52.4 per cent; OECD average 64.4 per cent); and about different cultures (Scotland 69.7 per cent; OECD average 75.6 per cent). They are also less likely to report being asked by teachers for their own opinion (Scotland 40.9 per cent; OECD average 45.7 per cent).

Meanwhile, teachers in Scotland - 1,445 of whom took part in a separate survey - are also generally much less likely to report covering key topics such as climate change and global warming or the causes of poverty in their lessons.

Paterson adds: “Overall, on what schools do, Scotland records some of the greatest levels of socioeconomic inequality in access to learning activities on global issues.”

He concludes: “Scottish schools are less important as sources for pupils’ knowledge and understanding of global issues than are schools in other countries [and] Scottish schools are particularly poor at providing knowledge about global issues to pupils in socially deprived circumstances.”

Yong Zhao, a foundation distinguished professor in the School of Education at the University of Kansas - who has argued that countries should ignore Pisa because it fails to measure what matters - takes issue with the idea that global competence “can be or should be tested as a global concept”. Zhao thinks that schools should help their pupils to understand that “we are all in this together”, but argues that this concept “cannot be easily measured, especially across different cultures”.

“So, let’s take water as an example,” he says. “If people are located in the Middle East, saving water is very important as a way to save the environment, but if you are in Bangladesh, how to navigate and avoid drowning is more relevant. They are all dealing with the same global issue of water - but may have different reactions.”

Many people dislike Pisa because of both the huge influence that it wields and the focus on reading, maths and science. Arshad describes Pisa as having “dominated government meetings, academic conferences, journals and the media for the past two decades”, but says the narrow focus on 15-year-old students could “provide a distorted view of education and, indeed, what the ‘excellence’ and ‘deficit’ factors are for each country”.

But there are few signs that the global competence survey is going to have anything like the same influence. There is, however, support for what seems like a genuine attempt to measure what matters, as opposed to what is easy to assess - an approach that experts like the University of Glasgow’s Louise Hayward have been extolling the virtue of for years, not least because what gets tested often skews what gets covered in the classroom.

Reading matters, and so does science and maths. But in Scotland, according to CfE, knowledge about key subject areas is not all our schools are setting out to impart.

The charity AFS Global aims to develop “active global citizens determined to make the world a better place”. It supported the launch of the global competence survey results because it believes global competence “will shape our future as much as reading, maths and science does”.

“Ultimately, we must thrive together or we must fail as one,” said Daniel Obst, the president and CEO of AFS Intercultural Programs at the launch in October.

As countries around the world continue to grapple with the coronavirus pandemic, it is a statement that is hard to argue with, and while this assessment is - as the OECD itself admits - a “still experimental approach”, at least it is an attempt to ascertain whether we are equipping future generations for all the challenges we know they will face. It also gives Scottish schools some hints about where they need to focus their attention if they are to be true to the aims of CfE.

One result, however, is almost certainly out of date. When Scottish 15-year-olds sat the test back in 2018, around 40 per cent said they were unaware of global health issues such as pandemics.

Unfortunately, schools can safely assume that Scottish teenagers are now up to speed on that front.

Emma Seith is a reporter at Tes Scotland. She tweets @Emma_Seith

This article originally appeared in the 27 November 2020 issue under the headline “A vote of competence”