In the past, we have treated schools as talent refineries. Their job was to identify talent, and let it rise to the top. The demand for skill and talent was sufficiently modest that it did not matter that potentially able individuals were ignored. The demand for talent and skill is now so great, however, that schools have to be talent incubators, even factories. It is not enough to identify talent in our schools any more; we have to create it.

It is important to note here that schools make much less difference than is commonly supposed. In the average school, 15 out of a class of 30 will achieve five good grades at GCSE (including English and mathematics). If those same students went to a so-called “good” school, then 17 out of 30 would reach the same standard; in a so-called “bad” school, then only 13 out of 30 would do so.

If it does not matter very much which school you go to, it matters very much which classrooms in that school you are in. And it’s not class size that makes the difference, nor is it the presence or absence of setting by ability - these have only marginal effects. The only thing that really matters is the quality of the teacher.

In the classrooms of the best teachers, students learn at twice the rate they do in the classrooms of average teachers: they learn in six months what students taught by the average teachers take a year to learn.

In some countries, such as Finland, Japan, and Singapore, teaching is such a high-status occupation that recruitment into the profession is highly selective. For countries in this position, the quality of teacher preparation and of continuing professional development is almost irrelevant. If you can persuade the smartest people in the country to want to be teachers, you will have a great education system.

For countries not in this position, efforts to raise the status of the profession are essential, but changes in the entrants to a profession take a very long time to work through - typically of the order of 30 years - and are of limited impact.

For example, suppose we could immediately raise the threshold for entry into teaching so that, from now on, only those who are better than the lowest-performing one-third of entrants were able to become teachers. Suppose further that, despite this raising of the threshold for entry into the profession, we were still able to recruit as many teachers as we needed. The effect of this - over 30 years - would be to increase teacher quality by just 20 per cent of the current gap between teacher quality in Finland and teacher quality in the UK. In terms of exam results, this would result in just one extra student per class passing an exam every three years.

So, as well as improving the quality of entrants to the teaching profession, we have to make the teachers we have better - what I call the “love the one you’re with” strategy. But professional development has been ineffective, because it has been based on a faulty analysis of the problem.

The standard model of teacher professional development is based on the idea that teachers lack important knowledge. For the past 20 years, most professional development has therefore been designed to address those deficits. The result has been teachers who are more knowledgeable, but no more effective in practice. Changes in what teachers know or believe will not benefit students unless teachers also change what they do in classrooms. We have been focusing on getting teachers to think their way into a new way of acting, whereas it would be far more effective to get teachers to act their way into a new way of thinking.

Unfortunately, changing teacher practice is difficult, because it involves changing long-established habits. On the other hand, we are beginning to learn what kinds of structures need to be in place to help teachers change habits. Teachers need to be able to exercise choice, to find ideas that suit their personal style, and they also need the flexibility to take other people’s ideas and adapt them to work in their own classrooms. Because teachers use a number of well-established routines to manage their classrooms, changing these can make their teaching, at least in the short- term, less fluent, so they need to take small steps as they develop their practice.

Teachers need, however, to be accountable for developing their practice: the evidence is that, left to their own devices, teachers improve their practice slowly, if at all. And, because changes in practice are so difficult, they also need to be given support for change.

The problem is that our schools are inundated with initiatives, and too many try to embrace them all. When everything is a priority, nothing is, and schools have to be selective about where they invest their efforts. When there is clear evidence about what works, to do otherwise is frankly self-indulgent.

Effective leadership is rarely about stopping people doing unproductive things. In most public service organisations, people are, in the main, genuinely interested in doing good. The problem is that when resources are limited (as they always are), then whether something is good is irrelevant. What matters is whether there is something better that could be done with the same resources. That is why leadership is so hard. It requires preventing people from doing good things to give them time to do even better things.

For leaders, the requirement is to create a culture for the continuous improvement of practice, and to keep the focus on a small number of things that are likely to improve outcomes for students.

We need people who are drawn to the profession not because it is easy, but because it is hard - a job that is so difficult that one’s daily experience is of failure, but one where, each day, to quote Samuel Beckett, one can “fail better”.

Dylan Wiliam is deputy director of the Institute of Education at the University of London.

This is an abridged version of a talk given earlier this year.