DfE ‘gap’ data doesn’t make sense – is it ‘numerical sleight of hand’?

Janet Downs's picture

The gap between disadvantaged pupils and others continues to narrow,’ says the Department for Education in its commentary on the 2018 Key Stage 2 assessment tests.

The graph on page 1 shows a narrowing of the disadvantage gap from 3.34 in 2011 to 2.90 in 2018 – a difference of 0.44.  It might be thought this was rather a slender reduction.

But you’d be wrong.  According to DfE number crunchers, the gap has narrowed by 13.2% since 2011.

Gap reduction is ‘wily statistics’

I haven’t a clue how the DfE turns 0.44% into 13.2%, but Chris Rolph, writing on the Nottingham Institute of Education blog, attempted to explain similar claims made about large year’s gap reduction figures.     This ‘numerical sleight of hand’ comprised ‘wily statistics – and a hefty dose of rounding up.’

0.09 doesn’t equal three

The DfE also said the gap had decreased by 3% in the last year.  The graph on the first page shows a gap of 2.99 in 2017 and 2.90 today.  I calculate that as 0.09.  Does 0.09=3?   I think the answer is No.

More pupils reach expected standard – but the gap since 2016 remains the same

The proportion of pupils eligible for free school meals (FSM), known as the ‘disadvantaged’, who reached the expected standard in reading, writing and maths tests in 2018 is 46% - that’s 11 percentage points higher than in 2016.  The proportion of all other pupils reaching the expected standard in the same period has also risen by 11 percentage points: from 57% to 68%.

Both proportions have risen.  But the gap remains stableIt was 22 percentage points in 2016.  And it’s 22 percentage points in 2018.  Or 21 percentage points in both years if unrounded figures are used.   

Gap at higher standard has widened since 2016

Worse, the gap between FSM pupils and their advantaged peers reaching the higher standard in reading, writing and maths SATs has widened since 2016.  It was five percentage points difference in 2016 and is now seven percentage points.   

Focussing on gap diverts teacher attention

Narrowing the disadvantage gap is constantly cited by the government as a goal which is being achieved because of recent reforms.  But it appears the numbers don’t support that conclusion.

In any case, there are valid reasons why focussing on the ‘gap’, especially when measured by test results, diverts attention from ‘constructing classrooms that give the greatest chance of success to those most likely to fall behind.’    And those most likely to fall behind are not just those identified by the crude measure of FSM eligibility.

No prizes for guessing which minister will be first to trumpet the 13.2% reduction in gap

Nevertheless, we can expect Tory politicians from the prime minister downwards to constantly repeat that the ‘gap’ has narrowed by 13.2% since 2011.  No prizes for guessing which minister is likely to spout this data first.

UPDATE 16.57  Surprise, surprise!  Schools Standards Minister Nick Gibb has  used the 13.2% figure (although he changed it to 13% and said it was from 2010).   

UPDATE 17 December 2018.  The comments in the previous update about Gibb making 'crass' comments has been removed as I have dealt with this in more detail here.  

Share on Twitter Share on Facebook

Be notified by email of each new post.


John Mountford's picture
Thu, 13/12/2018 - 19:26

Janet, focusing on the gap diverts everyone's attention, not just teachers, who should know better anyway. As to the minister, he wouldn't recognise a valid statistic fron a runner bean, so no surprise there. However, why the hell is someone of this pedigree a minister. It's beyond my understanding.

On to more serious matters. The gap is a falacy. All this attention on narrowing it is b.s. Claims that these are the expected standards are meaningless. Expected for whom and by whom are the questions we need to settle first. For any educator who reads this comment, be advised this preoccupation with attaimnent gaps is a diversion from truth and not only because it diverts attention from the kind of classroom practices that really make a difference to ALL pipils.

There are two ways every child might magically manage to reach the required standard. First, if the measure is sufficiently reduced it might make it possible for children at every attainment level to achieve it. Second, if we could put the brake on those children who are brighter (more intelligent), causing their attainment to stagnate (if this is even possible) thus allowing the others to gain parity (which they are not going to do anyway), then, 'never-say-die'. Unless one of these things happens, nature simply takes its course, and the inevitable spread of cognitive ability persists. The people who expect the gap to be narrowed live in denial of this universal fact. Just like every other human trait, intelligence varies from individual to individual. As Roger Titcombe has repeatedly pointed out on this platform and elsewhere, in any random polulation the natural distribution will reflect the Bell Curve. It is a fact of life some cannot accept when applied to human intelligence and until we move beyond that sticking point we will forever be presented with this kind of meaningless drivel.

Our task is to recognise this and reorganise education to reflect this reality. This is where pedagocical consideration come in and snake-oil remedies get consigned to the history books - classified as a failed social experiment that cost countless individuals access to an education that could have made a REAL difference to their lives. 

Janet Downs's picture
Fri, 14/12/2018 - 08:40

John - it may be no news to us that Gibb and the DfE regularly mislead  but it's important to keep challenging the data.  But the alleged gap reduction of 13.2% will be repeated constantly.  It will be churned by the media without investigation.  Even UKSA said the gap reduction claim was backed up by DfE figures.  But those DfE figures are based on an 'index' which shows a slight fall but which is manipulated by calculating a percentage of a percentage to show the fall is much larger than it is.  

Closing the gap may be a fallacy but it's used as a prominent indicator that post-2010 are working.  Exposing it as a numerical sleight of hand is, therefore, essential.



Add new comment

Already a member? Click here to log in before you comment. Or register with us.