As editor of a journal on mathematics education research, I know it can be hard to evaluate journal articles.
I would say this is especially true in education. As it’s such an interdisciplinary field, it can be really hard to separate the wheat from the chaff.
To complicate matters, recent technological developments have made it easier for anyone to create professional-looking websites or open-access articles - including those with nefarious intentions.
How to tell if education research is trustworthy
One way that journals try to safeguard quality is by using peer review. This is by no means a guarantee that poor research will never make it through, but it at least provides some kind of filter.
Peer review refers to when several reviewers - “peers” of the author, who also specialise in the field - have a close look at the article and provide feedback on it.
The reviews can vary from constructive evaluations to rather underwhelming ones that do little to improve an article’s quality. This is one of the downsides to peer review: there is an arbitrary element to it, which can sometimes result in good work being held back while poor work makes it through.
As a process, peer review has good intentions, but there are still things that can go wrong.
For example, most journals conduct “double-blind” peer review, whereby authors don’t know who reviewed them and the reviewers don’t know who authored a paper.
In practice, though, there are often many clues as to who has written an article, especially since there may be only a few specialists in the world on the topic being explored.
Read more, from Tes Explains:
Another challenge is to get the balance right between being critical and being constructive.
There is no doubt that social science is improved by the input of “critical friends”, but peer review is not the only option.
For example, Daniel Lakens has written about what is called the “red team” approach, which integrates criticism into every step of the research process.
In this approach, the “red team” plays devil’s advocate, trying to find holes and errors in ongoing work. The idea is to challenge dominant assumptions, improving the quality of a project in the process.
Another recent development has been the process of “open review”, in which journals include the reviews of published articles. I often find it very interesting to read how authors have responded to reviewers’ comments, as this provides insight into the process of writing the article.
There are also now “post-review” opportunities to take a critical look at research. The website PubPeer, for instance, allows people to post anonymous comments about published papers and has been essential in uncovering recent problems with the falsification of images in articles, and the use of “paper mills”.
According to the Committee on Publication Ethics, these mills are “profit oriented, unofficial and potentially illegal organisations that produce and sell fraudulent manuscripts that seem to resemble genuine research”.
This just goes to show how hard it is to recognise fraudulent research, even for the experts.
Telltale signs
So, what could teachers look at when trying to determine whether a piece of research is trustworthy or not?
One thing to pay attention to is the turnaround time of the peer review. Although this can vary greatly within and between journals, the expected turnaround time is an important variable. For example, short turnaround times might indicate less or non-existent peer review.
Another thing to consider might be whether there is a professional organisation and/or publisher behind a journal. Note that, in most cases, there will be an editorial board that is independent of such organisations.
If a journal article does not seem to fit within the remit of a journal, or the journal seems to publish almost everything, this can also be a telltale sign that the article is not to be trusted.
Ultimately, it might not always be easy to separate the wheat from the chaff - but we must if we want to keep science rigorous and trustworthy. And teachers can absolutely be a part of that process.
Christian Bokhove is professor in mathematics education at the University of Southampton and a specialist in research methodologies
