On Scottish Qualifications Authority results day yesterday, the SQA also published a methodology report on how it awarded students their grades.
Statistician Professor Guy Nason, of Imperial College London’s department of mathematics, has analysed the SQA document in a paper that includes the following key observations:
1. The SQA process of awarding grades should be like a driving test - but it’s not
“In a national exam, a student might have their grade changed, but this would only be loosely influenced by other students as the whole national cohort is used to form grade boundaries,” says Professor Nason. “However, in the SQA’s new standardisation process, a student might have their grade changed as a result of what students and teachers in their local centre had been doing. By contrast, if you take a driving test in the UK, it is a national test set to national standards. Your result should not depend on what has been happening in your local town.”
SQA exams results day 2020: Poorest far more likely to have Higher pass downgraded
See how SQA results day panned out: Live blog
‘Exam results’ in the Covid era: ‘Sense of injustice on whole other level’
2. Even for a leading statistician, it is difficult to understand some of what the SQA means
On one passage of the SQA document, Professor Nason asks: “Is this statement saying, ‘We tried different fudge factors to get the outcome we want’? It is essential that the SQA publishes full algorithmic details about what is going on here, preferably with anonymised sample data for us to even begin to understand what is being attempted, and then we can come to some assessment of whether it is suitable and fair.”
3. It was an advantage to take a course in a school or college with no history of offering that course in the past
“If your teachers are typical, then there will be a tendency to over-predict and, since you are in a new centre, your grades will not be modified [as they are elsewhere],” says Professor Nason. He adds that “the differing treatments of new, recent and established centres again means that we have to question whether these are ‘national exams’ with all students being treated in the same way”.
4. SQA needs to publish more details about its methodology
Professor Nason says: “In due course, we would expect to see the full publication of algorithms and sample data sets so that the community can come to a mature understanding of what has happened and, hopefully, feed positive alterations into future models. Reading about the SQA’s process is enlightening, but the wordy explanation is sometimes confusing and ambiguous.”
5. The priorities of the SQA’s system of standardising results are seriously questionable
“The problem at the heart of the statistical standardisation is that it can be simultaneously unfair to individuals, but also maintain the integrity of the system,” says Professor Nason. “However, if system integrity damages the life chances of individuals, then it is not much of a system.”
