Philip Nye from FFT Education Datalab explores how some schools are gaming the system through ‘off-rolling’.
In the National Pupil Database (NPD), England has one of the richest education datasets in the world that researchers are able to request access to.
The NPD brings together records from a termly pupil census that all state schools participate in, together with results from statutory tests and assessments that pupils have sat, as well as data on pupil absence, school exclusions and more.
The data can be used to answer any number of research questions – on school accountability measures, admissions and changes in assessment among other areas. Conditions for those wishing to use the data are becoming more challenging, however, threatening the benefit to society that researchers can bring by undertaking novel research using administrative data.
Who’s Left
In June we published a piece of work, Who’s Left 2018, exploring pupil movements on and off the rolls of state secondary schools in England, in response to persistent claims that the loss of pupils is used by some to boost their results in the government’s league tables (performance tables, to give them their formal name).
The work expanded and brought up to date a piece of work we published in 2017, using NPD data to build the best picture of pupil moves yet available, using an original methodology.
In this year’s work, we used pupil-level NPD data, to identify the cohort of 550,000-580,000 pupils who reached the end of their secondary education in each of the last four years, and who had spent at least some time at a mainstream state secondary school.
Data from termly school census records allowed us to identify a group of 8,700 pupils in the most recent cohort for which data is available who had moved into alternative provision – an alternative to mainstream education for pupils who have been excluded, are pregnant, have mental health problems or are otherwise unable to access mainstream education – some way higher than certain official figures imply.
More concerningly, we identified a group of around 22,300 pupils who had left the rolls of state establishments entirely. Taking into account pupils who had moved to the independent sector, and those who we think had left the population legitimately through emigration, for example, we estimated that there were 6,200-7,700 pupils who remained in the England but who had either not taken any exams at age 16, or who had not counted in the government’s school league tables.
We raised concerns that among this group will be pupils who have been off-rolled – encouraged off the roll of a mainstream in an informal exclusion, for the benefit of the school’s exam results.
We don’t think it’s possible to put a precise figure on the amount of off-rolling that has gone on, as we don’t think it’s possible to say whether something is an example of off-rolling from data alone. We do, though, think that England’s Department for Education needs to take a stronger interest in the issue.
Reweighting
As well as analysing the pupil moves picture in England, we carried out an exercise to see what the league table impact of pupils joining or leaving school rolls was.
Again building on an original methodology we developed in last year’s work, we reweighted league tables results according to the amount of time which pupils had spent on-roll at a given school, including any who had spent time there and subsequently left – unlike the government’s published league tables, which more or less only count pupils who remain on-roll shortly before finishing secondary education.
In many cases, the impact of this reweighting was considerable – in some cases improving schools’ results, and in other cases causing them to decrease. Certain groups of schools would be more affected than others – sponsored academies, would overall see a decline in their results – while London’s results would also be affected more than other regions’. We think that there are a number of factors that explain these results, including greater availability of alternatives to mainstream education in certain areas, and different approaches to inclusion and behaviour.
Impact
The work that we have published in 2017 and 2018 has been well-received, opening up a discussion about groups of pupils who are not often talked about and whether, in only counting pupils who remain on-roll shortly before finishing secondary education, school league tables are susceptible to gaming.
Since the piece of work we published in 2017, off-rolling has gained greater prominence as an issue nationally, with wide media coverage. The head of England’s schools inspectorate, Ofsted, has cited our work, expressing her own concerns about this form of league table gaming. Ofsted has also recently published its own work looking at off-rolling, and has announced that it has identified a group of 300 secondary schools which seemed to have lost a disproportionate number of pupils.
It is only as a result of the availability of administrative data that we have been able to shine a light on this issue.
Read FFT Education Datalab’s Who’s Left 2018 research here.
About the author
Philip Nye is FFT Education Datalab’s external affairs manager, and a researcher on inspection, the academy system and other subjects. He leads on Datalab’s public profile, including the team’s use of visualisation. His research interests include multi-academy trusts, free schools, school finance, and Ofsted.