The BBC recently reported on the ”reproducibility crisis” facing science. A team at the University of Virginia tried to reproduce the experiments behind five landmark cancer studies, but ended up only being able to confirm the findings of two. This should not have come as a surprise.

Last May a poll by Nature revealed that more than 70 per cent of researchers who responded had tried and failed to reproduce another scientist’s experiments.

This is a problem because, as Professor Malcolm Macleod was quoted as saying, “The issue of replication goes to the heart of the scientific process”. Findings based on experimental science simply cannot be relied upon unless they can be reproduced by others.

This is a tricky problem for the field of education, increasingly hitched as it is to the imperatives of evidence-based practice.

Teachers are under pressure to review their methods in the light of scientifically-conducted research.

The proliferation of toolkits and the emphasis on effect sizes reflects several decades of impatience with education bandwagons, and articulates a desire to embed teaching in a more reliable scientific approach to establishing what actually works in the classroom.

However, studies which claim to be based on scientific method are no help if the method involved is faulty. 

The wider replication crisis is not wholly or even in large part a consequence of fraudulent practice. It more often comes down to questionable research practices (QRPs in the jargon) - including selective reporting and manipulation of outliers in the data.

The “file drawer effect” constitutes a big problem - the tendency to publish positive results but not to publish negative or inconclusive ones.

The problem is that we don’t know about studies that were consigned to the filing cabinet. Yielding to the pressure to publish, even trained researchers might be tempted to jump to false positive conclusions.

Confirmation bias is another problem - even scientists may be predisposed to find what they are looking for.

In the methodological hierarchy of research, pride of place goes to randomised controlled trials (RCTs). These are difficult to achieve in education, and more often findings are based on ‘quasi-experiments’, or even correlation studies.

The latter only allow associations to be inferred between variables, not cause-effect relationships. Meta-syntheses rarely scrutinise the methodology of the studies being combined, and tend to include correlation studies, undermining the appropriateness of the word “effect” in “effect sizes”.

In medicine, even the strongest evidence from clinical studies is treated as “supplementary information”, adding to but not replacing the expertise of practitioners.

Research-based findings should be filtered through teacher-tested practice.

In a very thoughtful recent book, Professor Inez de Florio argues that reducing meta-analyses of research to lists of techniques, and combining these in “recipes” which rank strategies according to effect size, risks undermining the personality of teachers and students.

To the mountain of research, vastly variable in quality and utility, teachers bring that most precious commodity - judgement. For hard-pressed teachers (and with apologies to William Blake), we need songs of evidence and experience.

Dr Kevin Stannard is the director of innovation and learning at the Girls’ Day School Trust. He tweets as @KevinStannard1

For more columns by Kevin, visit his back catalogue

Want to keep up with the latest education news and opinion? Follow TES on Twitter and like TES on Facebook