WHEN Pandas first arrived in English schools just over a year ago, they were broadly welcomed as a useful new tool for managers striving to improve results.

The performance and assessment reports (Panda stands for Performance and Assessment) offer heads and governors benchmarking data about how their school’s pupils are achieving in comparison with those from similar economic and social backgrounds in other schools.

The second issue of Pandas arrived last month. But there is mounting concern among heads that they may be doing many schools a serious disservice.

Arthur Williams, head of Immanuel and St Andrew Church of England primary school in Streatham, south London, complains that his most recent report - which is compiled by the Office for Standards in Education using data from the Government and the Qualifications and Curriculum Authority - is based on “crude and unreliable” statistics.

OFSTED compares pupil results after taking into account social and economic factors. It uses the number of pupils on free school meals as its proxy indicator of deprivation: in Immanuel and St Andrew’s case 19.8 per cent.

Working this way puts the school in the second most prosperous grouping, which means its pupils are expected to reach higher levels. But 50 of the children counted on the roll are nursery pupils - part-timers who do not stay for dinners - so their eligibility for free meals is ignored, distorting the overall picture (see box). “We are being compared with a school of 8.5 per cent,” says Mr Williams, “and, based on that, quite sweeping judgments are being made,” about achievement levels.

He says that OFSTED made 83 categorical judgments about his school on the basis of its statistics, including the suggestion - misleading, in his view - that it is below average in science.

In contrast, says Mr Williams, his local education authority - Lambeth - conducts an exercise similar to a Panda but does take account of a weighted formula that recognises pupils having free meals as a percentage of a school’s full-timers.

Lambeth also takes note of two other factors: the number of children whose first language is not English and of the high mobility of pupils in inner-city schools.

Its school profile is presented as information rather than a judgment. Heads and governors are asked to draw their own conclusions and act accordingly. There is authority guidance on how to do this and governors are trained in how to use the school profile.

“If the Panda report were going to be used by inspectors to make judgments about my school,” Mr Williams says, “I would have to say that it does not give a fair and accurate position of the strengths and challenges facing us as a school community.”

Mr Williams is not alone in his criticisms of the OFSTED assessments. Marianne Barton, head of Abbey high school in Redditch, Worcestershire, says that when her school was inspected last November, OFSTED agreed that free school meals were not a reliable indicator. In Worcestershire, hot meals are not provided in the first or middle schools so that pupils are accustomed to going home for lunch.

When the comparatively low take-up of free school meals in many Worcestershire schools is taken together with a relatively high take-up in nearby Birmingham, this distorts the comparative achievement tables.

At the Leys high school in Redditch, Lesley McGuigan, the head, points out that 71 per cent of parents are receiving state benefit but only 18 per cent of pupils have free school meals. Pandas, she adds, might be cuddly on the outside but they are in reality pretty vicious animals.

Ray Westwood, the county’s principal inspector responsible for school effectiveness and improvement, says that its statistics seemed to indicate that the number of children applying for free school meals in the county was roughly half of those eligible.

Worcestershire, he adds, is developing a much more precise pupil-by-pupil comparison based on home postcodes, census data and information from the social services.

Mark Hewlett is principal of Rawlins community college, in Quorn, Leicestershire, a 14 to 18-year-old mixed upper school of 1,200 pupils.

He says his authority has indicated that the take-up of free school meals in the county’s upper schools was 3 per cent below that in the high schools (for 11 to 14-year-olds). This is attributed to older children being more sensitive about stigma associated with claiming such meals.

In his own school, he calculates the take-up of free meals to be 4 to 5 per cent too low. This results in his school being placed in the wrong comparative category in the achievement table published in the OFSTED report which goes out to parents.

“The report gave an entirely erroneous impression of our establishment,” he said. “It indicated well below average achievement when in fact our results are average or arguably above average.”

John Dunford, general secretary of the Secondary Heads Association, says that Pandas purport to be objective statistical assessments. But he says that the use of free meals as a proxy indicator of social and economic disadvantage quite frequently runs into problems.

He believes a better indicator would involve looking at performance against prior achievement. SHA is looking at the question of how you judge a school’s performance and how you calculate added value. Many secondary schools, he adds, use tools such as the cognitive ability tests from the National Foundation for Educational Research to reflect on pupil’s progress.

He points out that it is not easy to find a measure that has national credibility and consistency for the wide variety of secondary schools in England where pupils can start at either 11,12 or 14.

“You can either have prior attainment or a socio-economic indicator, and people say ‘Well, free school meals are a pretty good proxy’. But pretty good is not fair to lots of individual schools and, if it is your school that is going to get a grade D instead of a grade B, then you are in trouble.”

Jeff Holman of the National Association of Head Teachers says that problems arise when misleading or inaccurate information becomes public, for example through the summary school inspection reports.

When this inaccurate data ends up in the summary reports of inspector’s visits parents are misled. The association is unhappy that such information is published in comparative table form.

A spokeswoman for OFSTED said that the Department for Education and Employment is collecting additional information from authorities to allow OFSTED to remove part-time pupils from the benchmarking tables.

However, OFSTED does not consider that narrowing the bands of pupils on free school meals would help, she says.

English as an additional language was once included in the benchmarking exercise but was later excluded by the QCA because that authority felt that it added little extra explanation. Free school meals provide the only nationally collected indicator of poverty available to OFSTED, she said “Pandas have been welcomed by many heads and LEAs because they are found to be useful,” she says.

“But we are not saying that they are 100 per cent perfect. Work is always going on to refine them to help raise pupil attainment.”

THE FREE-LUNCH FACTOR

Pandas place schools in comparative bands of similar schools based on the numbers of pupils taking free meals.

For primary schools, the categories are 0 to 8 per cent; 8 to 20 per cent; 20 to 35, 35 to 50 per cent; and more than 50 per cent.

Immanuel and St Andrew school, Streatham, an inner-city primary, has 19.8 per cent on free meals, putting it in the second richest band. However, the headteacher, Arthur Williams, says this figure seriously distorts the true picture. Once you exclude from the roll the 50 nursery pupils who do not have lunch, the percentage of free-meal pupils rises to 25 per cent, indicating significantly greater social and economic need.

categories as they stand, argues Mr Williams, are just too broad.

Next year, if he can hunt down one extra pupil for free school meals or encourage nursery children to sample lunch, he could move the school up into the next band, which would result in a totally different interpretation of his school’s results and achievements.

Mr Williams is also concerned that Pandas do not take into account the number of pupils in a school for whom English is not a first language.

His 260 pupils cover 14 ethnic groups representing four major world religions. Some 30.2 per cent of his pupils speak English as an additional language.