The boom is over, house prices are turning and the credit crunch is on. From now on, we are going to need more knowledge, creativity and innovation to stay ahead - not a blowout of spend, spend, spend.

Socially, things are almost as bad. City centres are night-time nightmares, and of 21 countries surveyed by Unicef, the United Nations children’s fund, the UK ranked last in child wellbeing. In society and education, it’s time to put the booze and bling behind us and search for something more innovative, inclusive and inspirational instead.

But to know where we are going, we need to understand where we’ve been and what others are doing. From my research with colleagues on more than three decades of educational change, here is a sense of what we have been through and what alternatives already exist.

Loss of autonomy

From the 1960s until the late 1970s, teachers worked in an era of optimism and innovation. Many remain nostalgic for this golden age, though not all in the same way.

Teachers in innovative environments are nostalgic for the freedom and flexibility to develop the curriculum to fit diverse learning needs as part of a mission to change the world. Today’s obsession with targets, testing and inspection, they feel, has stolen this mission from them. At the other end of the spectrum, teachers in traditional schools also resent their lost autonomy and the lack of freedom to teach traditional subjects as they choose, irrespective of how students respond.

The age of optimism had its high points, but there were huge variations in quality that depended on the lottery of leadership and innovation among individual heads.

Much of the 1980s saw efforts to create greater consistency and coherence. The Keith Joseph years provoked passionate debates about secondary school reform. Her Majesty’s Inspectorate produced seven, then eight areas of experience to define balance, breadth and coherence in the curriculum. Vocational education was alive with initiatives. Records of achievement blazed the trail for today’s Assessment for Learning. The Technical and Vocational Education Initiative and city technology colleges created the modular options and tutoring schemes that were the forerunners of personalised learning and personal, social and health education.

In this turbulent time, the best local authorities seized the day to adapt reforms so they benefited all students. But weaker local authorities turned modules into curricular chaos and records of achievement into time-wasting bureaucracy. This time, the fault of inconsistency lay with local authorities.

Impatience with this inconsistency, coupled with political and public nostalgia for tradition, competition and certainty, propelled us into the current age of standardisation and market competition.

Across two systems of government control, with lesser, then greater degrees of resources and support, this period witnessed prescriptive and sometimes punitive reforms in the shape of the national curriculum, Sats, an intrusive Ofsted and strictly scripted national literacy and numeracy strategies.

Some feel this age developed a sense of urgency, secured public support for educational investment and moved the profession in a common and accountable direction. Others claim the goals could have been achieved by less negative means. Whatever the case, there is growing agreement that the strategy of standardisation is exhausted.

Results hit a plateau, primary school parents complained about children’s loss of pleasure in their learning, cheating scandals became widespread, and crises in teacher retention and leadership recruitment pointed to a deficit of motivation. There had to be a better way.

School networks

Promising ways forward already exist. There is a push for more personalised learning, leadership has become a national priority and networks as well as federations are emerging everywhere.

For example, among the 300 or so secondary schools in the Raising AchievementTransforming Learning (Ratl) project of the Specialist Schools and Academies Trust, two-thirds improved at double the rate of the national average through schools working together in networks of lateral pressure and support to implement short, medium and long-term strategies for success.

This strategy of the strong helping the weak in professional cultures of committed and transparent improvement certainly beats standardisation as a way to develop a creative, inclusive and high-performing knowledge society. But the policy priorities of standardisation still linger - in short-term targets, Sats scores, and Ofsted inspections that rely too heavily on numerical achievement data to assess the progress of schools. These have pushed many heads involved to focus more on meeting the short-term pressure to deliver results than on long-term efforts to transform teaching and learning.

The Finnish model

If we want to consider other options, let us consider the best. The world’s number one performer at age 15 in literacy, maths and science is Finland, according to the influential Programme for International Student Assessment tables produced by the Organisation for Economic Co-operation (OECD) and Development. Finland also ranks top in economic competitiveness and corporate transparency.

So how does this nation achieve such remarkable results, including some of the narrowest achievement gaps in the world? A few months ago, I took a team to Finland for the OECD to report on the relationship between leadership and school improvement. This is what we found. After high unemployment because of the loss of its Russian market in the early 1990s, Finland decentralised its education system to help it become a more creative knowledge economy.

The country’s prime minister, university vice-chancellors and corporate chief executive officers meet regularly to steer it towards its high-tech future. But most of the coherence comes from a common vision that connects Finland’s innovative future to its creative, craft-like past. There are more composers per capita in Finland than any other developed country, and all young people engage in creative and performing arts to the end of secondary school.

This shared vision accords high status to teachers as builders of their country’s creative and inclusive future. Though paid only at the OECD average, teaching in Finland is highly competitive, with only a one in 10 chance of acceptance. Retention is very high because conditions are good and teachers feel trusted. Finns control quality at the most important point - the point of entry.

Within the broad steerings of the state, highly qualified teachers create the curriculum together in each municipality. They work in a professional culture of trust, co-operation and responsibility on behalf of all children, not just those in their own classes.

Finally, school heads share resources and support each other through a sense of common responsibility for all the young people in their town or city. They have time to do this, they say, because unlike Anglo-Saxon countries, they are not always having to respond to imposed initiatives.

The Finns know where they have been and feel responsible for where they are going. Highly qualified teachers are attracted by the vision, and publicly valued because they help to achieve it. This is done through collective professional responsibility, not administrative accountability. Performance is the best in the world because Finns emphasise learning, not testing. Indeed, they have no standardised achievement tests, except for confidential sampling and monitoring purposes.

The future

England has no inspiring social or educational vision. Unlike Scotland, Northern Ireland or Wales, which have clearer identities and are less driven by targets and testing, it has no strong sense of who or what it is. Instead, it regresses to arithmetical achievement gaps or vacuous claims to world-class standards as evasions of any vision.

The Prime Minister has spoken passionately about returning to values and vision. There is no better way for this ascetic Scot to do this than by transforming the overly standardised, excessively regulated and unnecessarily competitive education system of the English.

It is time to develop a more inspirational vision by reclaiming some of the freedom and flexibility of the 1970s, capitalising on the innovative curriculum and assessment developments of the 1980s and 1990s, and learning from the present-day successes of our European neighbours. Only then can we grant real responsibility to the profession for realising it. The future of England will not be standardised. Why should its schools be?

Andy Hargreaves is the Brennan Chair in Education at Boston College in the United States, and visiting professor at the London Institute of Education and Manchester University. This article arises from research conducted with Ivor Goodson, Dean Fink, Dennis Shirley and the OECD.

1964-1976: optimism and innovation

Teachers worked with a great deal of freedom and flexibility, but teaching quality varied hugely between schools

1977-1987: the search for consistency

The 1980s saw a wave of curriculum and assessment reforms, but weak leadership at local level often led to chaos and paper-chasing bureaucracy

1988-2008: the era of centralisation

Both Conservative and Labour governments have sought to raise standards and make teachers more accountable through the national curriculum, Ofsted and tests. But there is a growing consensus that a standardised approach no longer works.

Beyond 2010: shared professionalism

Successful systems such as that in Finland (left) are based on high-quality teaching, shared vision and a sense of collective responsibility. This can only be achieved in England by a major shift away from excessive regulation and the culture of competition.