When describing the results of research, “significance” does not mean that the result is noteworthy or important. Instead, significance (or statistical significance) has a technical meaning that is used to describe the statistical uncertainty of a result.
Statistical uncertainty is influenced by a number of factors - but the most prominent is the size of a sample relative to the population overall.
This is similar to when conducting a political poll. If you ask 100 people, you may have a bigger margin for error than if you were to ask 100,000 people.
A result can only be described as statistically significant when it surpasses a particular statistical threshold and thereby rejects the idea that there is no difference in impact between the two approaches you’re comparing.
Where can I see this in action?
All studies have their limitations, and the size of a sample relative to the overall population is one that many researchers have to contend with.
A study published in September 2022, for example, looked at how confirmation bias presents in teenagers in comparison with adults. Lead researcher Gabriele Chierchia used a dataset already in existence, which had just 77 participants aged between 11 and 32 years. This, he identified, is a clear limitation to his findings.
In meta-analysis or systematic reviews, statistical significance can also be affected by the number of studies that can be included. When talking to Tes about her meta-analysis on the use of “guided play” in the classroom, Elizabeth Byrne, a research associate with the Centre for Research on Play in Education, Development and Learning (Pedal), said that there was a “huge variation in the way that guided play was implemented”.
“Lots of different outcome measures were also used and this meant that not all of the studies we looked at in the review could be included in the statistical analysis,” she explained.
The Education Endowment Foundation (EEF) is an independent charity dedicated to breaking the link between family income and educational achievement.
To achieve this, it summarises the best available evidence for teachers; its Teaching and Learning Toolkit, for example, is used by 70 per cent of secondary schools.
The charity also generates new evidence of “what works” to improve teaching and learning, by funding independent evaluations of high-potential projects, and supports teachers and senior leaders to use the evidence to achieve the maximum possible benefit for young people.
