Papers were scattered across the classroom and practical equipment lay abused and abandoned across workspaces.

As I cleared up the mess, I conceded that this Year 10 class had got to me: across the group, different students had found ways to disrupt the lesson - sometimes blatantly, often subtly - and I seemed unable to find a strategy to improve it. A run of difficult lessons with them had left my confidence shaken.

Feeling harried, isolated and quietly ashamed, I questioned whether I’d made a huge mistake by becoming a teacher.

I knew that my first year in teaching would be hard, that challenges would sometimes seem overwhelming. But what kept me awake at night wasn’t planning or assessment or my subject knowledge - I worried about classroom management.

I was not alone and, 15 years on, it seems as though not much has changed. Behaviour is still commonly cited as a reason why some new teachers start to lose that glow of enthusiasm and think about quitting.

For example, in a recent ATL teaching union survey of trainees and newly qualified teachers, a quarter of respondents cited challenging pupil behaviour as a reason why they were considering leaving the profession.

And it is a problem that does not always disappear with experience. When I was a trainee, some of the veteran teachers seemed to have an almost magical presence, being able to turn classes from a rowdy mob to hardworking angels by simply walking into a room. But it’s not entirely magic; some of it is probably what psychologists call the exposure effect. While people may say familiarity breeds contempt, the evidence suggests the reverse - people tend to prefer or feel more positively towards things as they become more familiar with them.

But those veteran “magicians” are few and far between: many of those who have been teaching for some time still experience problems. The 2014 Ofsted report Below the Radar found that “many teachers have come to accept some low-level disruption as a part of everyday life in the classroom”.

Aside from the obvious stresses these issues create, the problem is that such disruption acts as a limiting factor on how much learning can take place. If you’re struggling to secure basic cooperation, then inevitably much of your time and effort will not be focused on learning. The working atmosphere in a classroom has a strong impact on pupil outcomes. The research might be summarised like this: learning is much more difficult, if not impossible, in a disorderly environment.

Of course, classroom culture is not the only thing we need to get right for learning to take place - but it is necessary. So what can we do?

When it comes to evidence-based approaches that can help teachers and school leaders tackle low-level disruption, pickings are slim. There’s a great deal we don’t know. But we can explore the research and psychology underpinning many behaviour policies in schools, and more specifically look at the use of rewards and sanctions that almost all schools policies are based upon.

In doing so, we can begin to unpick what should work, and - perhaps more importantly - what most likely will not.

An unknown known

How widespread is the issue of low-level disruption in schools? We’ve got only surveys and inspection reports to guide us.

In 2012, researcher Terry Haydn questioned Ofsted’s claim that 99.7 per cent of schools in England were dealing with behaviour that was “satisfactory” or better. He found this was not reflected in the experiences of teachers; even very experienced teachers reported that they struggled with behaviour at times.

Survey data also suggests a less positive picture. The Below the Radar report highlighted a Pisa (Programme for International Student Assessment) survey finding that pupils were twice as likely as headteachers to report disruption that hindered their learning in maths, and a YouGov survey of parents and teachers suggested that about an hour per day is lost from lessons owing to disruption - roughly equivalent to half a term each year.

Perhaps as a consequence of not having a reliable measure of the problem’s extent, there’s also little evidence-based research into what schools can do about it beyond the reliable, but slow, process of relationship-building.

The Education Endowment Foundation summarises the impact of assorted behaviour interventions, aimed at improving attainment by reducing challenging behaviour as moderately effective.

However, reading behind the headline reveals that effective interventions are predominantly focused on relatively small groups of pupils - for example, those diagnosed with particular special educational needs related to regulating emotions.

The outcomes are also not straightforward - for example, the guidance suggests that anger-management interventions appear, on average, to have some positive impact on behaviour, but a negative impact on pupil attainment.

The evidence base for whole-school strategies is even weaker - with very little on improving low-level classroom disruption.

Admittedly, it’s a difficult area for research, but this leaves schools having to muddle through and implement policies that may, or may not, be effective. Perhaps this is why debate about behaviour in schools so often descends into a “war of anecdotes”.

Without good evidence on what makes a school behaviour policy effective, many schools experiment with systems that appear broadly based on behaviourist principles.

“Behaviourist” is sometimes (I think unfairly) used as a pejorative in education literature, but schools using some sort of system for rewarding or sanctioning behaviour are implicitly drawing on a behaviourist approach - and that is most, if not all, schools.

Despite this, the principles, potential pitfalls and limitations of behaviourism are not necessarily widely appreciated.

So a more explicit understanding of this area of psychology might inform how we manage low-level disruption in our classes.

The basics of behaviourism

Core ideas within the behaviourist approach include reinforcement and punishment.

Very simply, psychologists such as Edward Thorndike and BF Skinner identified that, when an animal receives reinforcement after performing a behaviour, they are more likely to repeat that behaviour in future. Conversely, receiving a punishment after performing a behaviour leads the animal to be less likely to repeat that behaviour.

Skinner described reinforcements and punishments as being “positive” or “negative”. Sometimes these terms are used incorrectly, so here are some examples of what they might look like in schools.

• Positive reinforcement is where you give something pleasant as a reward after a behaviour. It might be something material such as a sweet, a token in the form of a merit, or social approval such as praise.
• Negative reinforcement is where you take away something unpleasant to reward a behaviour - for example, allowing a pupil to skip to the front of the dinner queue.

Confusion commonly arises between negative reinforcement and punishment. (And punishment here shouldn’t be confused with corporal punishment.)

• Positive punishment is where an aversive stimulus follows a behaviour, for example telling a pupil off or using a mild teacher look to express disapproval.
• Negative punishment is where a pleasing stimulus is taken away after a behaviour, such as confiscating a mobile phone or holding a pupil back for a detention.

You’ll see from the examples that most, if not all, teachers routinely use reinforcement and punishment in their classroom practice, whether they recognise it or not.

However, perhaps because much of this application is based on tacit knowledge, I think some potential misconceptions have grown around these techniques. And it is here that we can begin to unpick how we might better approach low-level disruption.

The role of sanctions

Behaviourism has an unfairly harsh reputation, so many teachers are surprised to discover that Skinner was very much against the use of punishment in schools.

He believed that a major disadvantage of punishment was that, even where it is consistently applied, it merely temporarily suppresses an undesirable behaviour. However, the effectiveness of sanctions is likely more complex than Skinner believed.

There’s some evidence to suggest that our attitudes to sanctions depend on how much we trust the people around us. For example, a recent meta-analysis examined whether punishments were effective at promoting cooperation in high or low-trust societies.

Where societies are high-trust (ie, people generally hold the view that the motives of others around them are benevolent) members of that society adhere to norms that encourage cooperation and the punishment of those who defy those cooperative social norms. Where there is a lack of trust and social norms are less strongly shared, punishment appears much less effective.

This implies that in schools (which we’d all hope exemplify high-trust, benevolent environments) the use of appropriate sanctions to support cooperative norms can be a potentially effective strategy.

Another reason why sanctions may be more effective than Skinner thought is owing to a phenomenon called “loss aversion”. The work of Amos Tversky and Daniel Kahneman suggests that there is a reliable asymmetry in our response to positive reinforcement and negative punishment - in that when people weigh up similar gains and losses, they tend to prefer avoiding losses to making gains. Sanctions such as confiscation of a mobile phone or losing some of your breaktime may, in some instances, be more effective than giving merits or rewards.

Psychologists often refer to two types of motivation: intrinsic, which is driven by an internal sense of reward, for example, reading for the sheer pleasure of it; and extrinsic, driven by some sort of externally originating reward, such as reading a book so you can get a merit for finishing it. One concern is that the use of extrinsic motivators such as praise or merits may dampen intrinsic motivation.

There are certainly good reasons to use praise and rewards thoughtfully. For example, when teachers overpraise, they may inadvertently lower expectations; using sympathetic praise in an attempt to protect the feelings of a pupil who is struggling, may inadvertently communicate that they didn’t expect them to achieve. In contrast, a teacher who appears hard to please and perhaps more rarely offers praise, may be communicating to the pupil that they hold high expectations of what they can achieve.

Indeed, on the whole, using praise and rewards to manipulate behaviour is something we should probably avoid. Young people are often sharply aware when an adult is trying to shape their behaviour with controlling praise or contingent rewards, and this can sometimes backfire.

Yet rewards, like sanctions, are complicated. Obviously, we’d love our pupils to find everything they do in school intrinsically motivating - but while that’s true for some things we learn in school, many of the more complex and abstract ideas underpinning the cultural advances of the past few centuries are challenging and (at least initially) we may not be terribly intrinsically motivated to struggle with them. Motivation to chat with friends will always be stronger than the drive to do homework.

If a pupil is highly intrinsically motivated, then diverting that attention to extrinsic rewards is unlikely to be helpful, and will, in some cases, be counterproductive.

On the other hand, it often requires a significant investment of time and effort to develop an intrinsic motivation for something that is challenging (I immediately think of practising scales when learning to play a musical instrument). In these cases, teachers often intuitively recognise that extrinsic motivators are a useful “plan B” to encourage pupils who might initially lack the intrinsic motivation that would encourage them to persist at things they find difficult.

Eventually, as the pupil experiences some success, despite those challenges, they will hopefully develop more intrinsic motivation, which in turn helps them to invest the effort and time to progress even further.

Daniel Willingham, professor of psychology at the University of Virginia, helpfully summarises three key points we should consider if we do use praise.

“Praise should be sincere, meaning that the child has done something praiseworthy,” he says. “The content of the praise should express congratulations (rather than express a wish of something else the child should do). The target of the praise should be not an attribute of the child, but rather an attribute of the child’s behaviour.”

So if, when they are used well, the rewards and sanctions commonly used in schools can be effective, why do we continue to see surveys suggesting that behaviour issues are so widespread?

I suspect there are two major reasons. The first is that, as demonstrated, rewards and sanctions are extremely difficult - if not impossible - to apply consistently. Human beings are fundamentally inconsistent in our judgements. The second is that, in any system of judgement, we are prone to making false-positive (for example when we think a pupil was talking when they weren’t) and false-negative errors (if we give the benefit of doubt as to whether a pupil has their mobile under the table).

Unfortunately, when you try to minimise one of these errors, you inevitably end up increasing the rate of the other type - and both errors have consequences.

If your system makes too many false-positive errors, then sanctions will start to appear somewhat arbitrary and this may undermine a culture of trust in the school. On the other hand, with too many false-negative errors, low-level disruption will probably increase as pupils push the boundaries.

Social call

Being accurate in these judgements probably isn’t too difficult when only one or two pupils are disrupting a lesson, but it rapidly overwhelms a teacher’s ability to keep track when a critical mass of pupils decide not to cooperate. One indicator of this problem might be teachers having a whole section of the board dedicated to keeping track of which pupils have been escalated on the school “consequence system”.

So, if your system solely relies on individual teachers consistently applying a complicated school behaviour policy to provide individual feedback to pupils about behaviour expectations, then it is probably at odds with the realities of human performance.

However, I think this second issue is even more problematic: exclusive reliance on individual-level feedback ignores the social-behavioural processes that underpin low-level disruptive behaviour. Human beings are social animals. We don’t just learn social behaviour from personal feedback in the form of rewards and sanctions, we imitate one another, forming judgements about what is acceptable conduct based on what we see others doing.

Social norms are the (often unwritten) rules about how we behave in social context - for example, the rules of queuing or saying “please” and “thank you”. Like all cultural institutions, schools possess social norms regarding the behaviour of students. In schools, we often articulate these as the ethos and values that we want pupils to embrace. However, we don’t always explicitly articulate the behaviours one would see if those values were successfully adopted.

It’s almost certainly not enough to simply say what we value, such as respect, responsibility or courtesy; we need to be explicit about what these values look like in practice, and demonstrate these values through the expected behaviour and routines we set for a school. For example, we could think about what it would look like if we embodied these values during task transitions, lesson beginnings or moving around the school.

I think this might be a promising way to help promote the classroom culture we want. If we can translate (often abstract) values into concrete behaviours that we want to see in the classroom on a daily basis, and then practise them until they become standard routines that all pupils come to know and expect, this may help to promote the cooperative social norms that may have a stronger influence on classroom climate.

This may be why many schools are adopting universal routines, reinforced by a behaviour policy, to support school-wide social norms for basic classroom conduct (Doug Lemov’s Teach Like a Champion being perhaps the best-known example).

Admittedly, though, there’s relatively little research into how such routines can be implemented effectively, or how long it takes to turn things around in a school where the social norms don’t always support learning. Or, indeed, how we balance the systems that support whole-school behaviour with those to identify and support students who might benefit from additional interventions.

While so much is unknown about what makes school behaviour policies effective, I’ll stick my neck out. If schools don’t purposefully encourage these value-derived norms, then pupils will go ahead and evolve their own. As I found with my own difficult Year 10 class, the norms that emerge may not necessarily support the climate of learning that’s best for the pupils and their teachers.

Nick Rose is a researcher for Teach First. He tweets @nick_j_rose