In the 1200s, the word “wicked” meant awful, coming from “wicca” or “witch”. Then F Scott Fitzgerald used it ironically to mean something wonderful, and the slang stuck. 

But what does it mean when you say the kind of learning that takes place in a classroom is wicked?

It has nothing to do with either of the previous definitions, but instead describes the replicability of research results in the learning environment.

‘Kind’ vs ‘wicked’ learning environments

In his keynote speech at the World Education Summit earlier this year, Dylan Wiliam, emeritus professor of educational assessment at UCL, spoke about “curriculum, pedagogy and assessment, in that order”.

As part of his talk, he explored the limitations of drawing conclusions from educational research.

Wiliam explained how some learning environments can be labelled as “kind”, as it is possible for a person to reliably learn from their experiences there. 

An example of this is tennis: the lessons learned from playing on the court are likely to be replicated the next time you play. Gravity and physics aren’t likely to change, so how you play one shot on one day and get a certain result is likely to happen the next day, too.

In a “wicked” learning environment, however, that replicability cannot be relied upon.

What makes a classroom ‘wicked’?

We spoke to Wiliam to get the detail about what exactly makes a classroom wicked, rather than kind.

1. Lots of ‘noise’

This doesn’t refer to actual noise (although that can be an issue, too) but rather a number of factors that can influence the effectiveness of the teaching - and, in turn, the learning - that takes place.

A teacher can teach the same lesson twice, but in one of those lessons the students are distracted by a fire alarm, or a bee, or the teacher is simply having an off day.  

“What we learn may not be what is true,” Wiliam says. “If we look at a teacher where learning seems to be happening, it could be the teacher who is making the difference, but it might be the case that you’ve got a group of motivated students and the teacher has done nothing to affect them.”

So what does this mean? Ultimately, that the teacher cannot always rely on their own recollections or data on what has worked or what hasn’t.

“The difficulty is [that] we look at the teacher, and we see the results, and we assume it is the teacher causing those results. There is a Latin phrase, ’Post hoc ergo propter hoc’ - ‘After the event, therefore because of the event’.”

Wiliam warns that if you assume that things that happen afterwards are caused by the things that happen beforehand, then you might be looking at a coincidence rather than cause and effect. 

2. Performance vs learning

If you set your students off on a task, get no hands up and they’re all able to complete it, that means they’ve learned it, right? Well, no. 

Performance and learning are two different things, Wiliam says, and we shouldn’t conflate them.

“Performance is how well students do in the learning task, and learning is what is retained some time later,” he says.

“One of the most pervasive assumptions is that if students successfully complete the learning activity without help, they will learn it. So teachers take performance in the learning task as a proxy for learning.”

But performance is often inversely related to learning, Wiliam continues.

“The more students struggle in the learning task, the more they remember, and the less they struggle, the less they learn,” he explains.

But it doesn’t end there. “If they struggle too much, then there is not enough cognitive capacity leftover to create those long-term memories. We need a Goldilocks-type sweet spot: we need students to struggle, but not too much.”

3. Short term vs long term

When we think about performance in the classroom, we can’t just take results from end-of-term or even end-of-year assessments as an indicator that the teaching has been effective. Wiliam says a study by Carell and West, involving Air Force Academy students, indicates that teaching that gets good results after a year won’t necessarily yield good results the following year.

“Teachers who were more effective in the short term were less effective in the long term, because they were more focused on short-term goals rather than laying secure foundations for future learning,” he says.

Therefore, unless you look at the impact of a teaching approach in the long term, you may be drawing incorrect conclusions about its usefulness.

4. We sometimes draw the wrong conclusions

Even in-depth research can lead experts to draw an incorrect conclusion when a significant variable is overlooked, Wiliam says, offering a story from his own experience.

“I spent four years tracking students who were being put into sets, and we found that the earlier setting was introduced, the bigger the spread of achievement, and students in the bottom sets made less progress,” he says.

“At the time, I didn’t understand how big the effect of teacher quality was. I was kind of assuming that teachers were pretty similar. Now I understand that the most effective teachers are far more effective than the least effective teachers.”

As a result, he now doesn’t feel as confident about the conclusions he drew at the time. 

“I don’t know how to interpret those results any more. It could have been the setting which was the cause, or it could be the differential allocation of teachers to groups.”

This can happen to anyone undertaking research, he says.

“Any research finding can potentially be overtaken when you find out that things you didn’t think were important turn out to be important.”

5. Maintaining randomisation can be hard

Randomised controlled trials are often seen as the gold standard in research. But is this always possible to maintain when research takes place in the classroom?

Speaking about a research project designed to look at the role of class sizes, Wiliam recalls how the randomisation was sabotaged by a “drift”.

“Maintaining randomisation was impossible,” he says. “Middle-class parents found out about the experiment, and if they found out their kid was in the larger classes, they campaigned to have their child moved to the small class.

“Although the kids were randomly allocated at the beginning, there was a drift, and it could have been the smaller classes that made the difference, or it could be there was a drift of high-achieving students to the smaller classes.”

This was originally published on 21 April 2021.