The debate about whether the data that the Scottish government uses to track attainment - particularly in primary schools - is reliable has been reignited, thanks to comments made by the new director of education in Glasgow, Douglas Hutchison.
In a recent interview with The Herald, Mr Hutchison - who has been in post since mid-January - made the case for reintroducing the Scottish Survey of Literacy and Numeracy (SSLN), which tested a sample of Scottish pupils, focusing on literacy and numeracy in alternate years, until it ran for the last time in 2017.
Then last week - just a few minutes after Mr Hutchison had rounded off a wide-ranging interview with this publication by making similar comments - the issue was raised in the Scottish Parliament.
Mr Hutchison - whose interview with Tes Scotland will be published in full next week - said that if he were education secretary, he would not be able to sleep at night knowing there was a lack of “reliable [attainment] data until you get to S4, 5 and 6”.
As an education director, he said he did not get anything useful from the teacher judgement data published annually by the Scottish government “at systems level”, and that, if he could, he would “reintroduce the SSLN and ditch the SNSA” (Scottish National Standardised Assessments).
Questions over student attainment data in Scotland
The Scottish National Standardised Assessments replaced the SSLN and are used to test pupils’ literacy and numeracy skills in P1, P4, P7 and S3 - but the data is not published publicly. Instead, the tests are supposed to inform teacher judgements about whether pupils are hitting the expected level for their age and stage. This data is what the government uses to track attainment, and is known as the Achievement of Curriculum for Excellence (CfE) Levels data.
But Mr Hutchison told Tes Scotland that he was concerned about the reliability, validity and the consistency of this data.
“Something like the SSLN was better at giving you that sense of the whole - how the whole system is improving or declining, in a probably slightly more objective way,” he said.
In the Scottish Parliament, referring to the SSLN, Conservative MSP Sandesh Gulhane asked this of education secretary Shirley-Anne Somerville: “Why is it so difficult for the cabinet secretary to reinstate that well-regarded survey?”
In her response, Ms Somerville cited the 2015 Organisation for Economic Cooperation and Development (OECD) review of Curriculum for Excellence, which said that the “light sampling” of the SSLN was not providing “sufficient evidence for stakeholders to use in their own evaluative activities or for national agencies to identify with confidence the areas of strength”.
The 2015 report added: “Nor has it allowed identification of those aspects or localities where intervention might be needed.”
But it seems that the new system for tracking attainment in Scotland is not what the OECD had in mind either.
In its latest review of CfE, published last year, the OECD said this of the Achievement of Curriculum for Excellence (CfE) Levels data: “While this data is interesting, reporting it on a national scale and tracking small changes in percentages as evidence of improvement or otherwise may not be giving the system the robust data needed to monitor student achievement.”
The report flagged up “the long-established National Assessment of Mathematics and Reading Skills (NAMER), running in Ireland since 1972” as an example of good practice. It recommended that Scotland “redevelop a sample-based evaluation system to collect robust and reliable data necessary to support curriculum reviews and decision making”.
The Scottish government said it is fulfilling this recommendation by exploring options “for a sample survey approach to assessing progress across the four CfE capacities” - and Ms Somerville repeated this in the Scottish Parliament last week.
What is unclear is whether the sample-survey approach that the government is exploring will address the concerns about the reliability of our literacy and numeracy attainment data.
However, another assertion from Ms Somerville - that the government remains “committed to teacher professional judgment as the means of assessing progress in the broad general education phase” - does not give much hope.
But the key word in the OECD’s recommendation is “redevelop” - it says Scotland should “redevelop a sample-based evaluation system”. What else could this refer to, if not the SSLN?
And the OECD and Glasgow’s new education director - who is also the president of national education directors’ association ADES - are not the only ones concerned about the robustness of the data that the government uses to track pupil performance.
Professor Becky Francis, chief executive of the Education Endowment Foundation - the English charity dedicated to breaking the link between family income and educational achievement - gave evidence to the Scottish Parliament’s Education, Children and Young People Committee last week as part of its ongoing Scottish Attainment Challenge inquiry.
Professor Francis said she was surprised that, when it came to assessing learning loss during the pandemic, there had been no assessment in Scotland “that goes beyond teachers and is run by an external organisation”.
Surely, these more recent comments from academics, education directors and the OECD itself trump those made by the OECD all the way back in 2015?
Dr Gulhane accused the government of resisting the reintroduction of the SSLN because it wanted to “hide its failings”. But actually, what the OECD seems to be saying in its 2021 review is that the data Scotland collects just now could be making it hard for us to track improvement.
After it questioned the robustness of the Scottish attainment data in its report, the OECD added: “Some observed that the practice of reporting on levels might be giving rise to an impression of a rather static system or one that is, at best, inert and, at worst, not improving.”
So there is a fundamental problem here: strip away the political rhetoric on all sides and the current dearth of attainment data makes it impossible to tell for sure whether things are going well or not.
Emma Seith is a senior reporter at Tes Scotland. She tweets @Emma_Seith
