In recent years there has been a lot of interest in making science more open.
We have seen an increasing number of “open access” journal articles; this means that they are available for everyone to read for free, rather than being behind a paywall.
Many articles are now also made available as so-called “pre-prints” - these are essentially preview copies of a study, released either before or after peer review.
On the face of it, these are positive steps that make education research more accessible.
However, I would recommend being critical of the studies we read, as both types of articles have their challenges.
The challenges of education research
For example, I often see criticisms of the quality of the review process. “How can something like this ever get through peer review?” I see people complain on Twitter. In most cases, however, peer review improves the final published article. On the whole, it’s probably the best we can do from a myriad of worse options.
More from Christian Bokhove:
Another common criticism I see levelled at research is around publication bias (a phenomenon where studies that report statistically significant results are more likely to be published in peer-reviewed journals than those reporting nonsignificant results).
I must admit, this is something I have often raged against myself, especially if a study’s outcomes are unsurprising. When I see a null result for a randomised controlled trial that cost a million pounds to run, only to confirm something I already suspected would not work, I naturally wonder whether we could have used that money for better things. At the same time, only publishing the exciting results is also not good, as we then only look at those studies.
A third challenge that I encounter is the issue of tracing back claims made about a particular piece of research. It is custom to use references to support claims, but following those references up is not always straightforward.
Sometimes, I find myself wondering, “Hmm, is this really what that study said?”, only to disappear down a rabbit hole of references that cite even more references, of sometimes dubious origins.
How do we know which studies to trust?
So, what does all of this mean? Does it suggest that education research can’t be trusted?
No, but it does mean that we need to be cautious consumers of research. And while it might not always be easy to recognise which studies we can trust and which we can’t, there are clues that can help us to determine reliability.
For example, the issue of tracing back claims might be partially mitigated by increasingly open data analysis and reporting practices. More and more articles now come with attached data, plus analysis scripts that allow researchers and other interested parties to look at the data for themselves and better understand how the researchers came to their conclusions.
Another thing to note regarding the reliability of journals and articles is the citation count. This is partially expressed in the impact factor (a measure of the frequency that the average article in a publication has been cited in a given year). Citation rates are not foolproof, though, with even poor studies getting a lot of airplay. Nevertheless, popular journals also get a lot of attention, with vibrant online communities putting research results under scrutiny.
With pre-prints now available, and the existence of websites like PubPeer, which allows people to comment on published research, we now have more opportunities than ever to share constructive feedback and engage in sensible debate.
Unfortunately, this is not how it always plays out, with some “debates” descending into all-out wars.
And yet claims that research is poor are often unwarranted. There will always be challenges surrounding studies, but we can do something about those challenges.
Above all, we have to remember that being a critical consumer (and creator) of research means finding ways to deal with the uncertainty of social science.
Christian Bokhove is associate professor in mathematics education at the University of Southampton and a specialist in research methodologies
Get the biggest education stories of the week delivered straight to your inbox every Friday with the Tes magazine newsletter. If you’re a subscriber, visit your preference centre and check the ‘magazine newsletter’ option
