UK Statistics Authority issued a statement today saying that it was “disappointed” in the way the £350m figure (of the cost of the UK’s EU membership) is being used and that it “undermines trust in official statistics”. I am sure that that much is true but it is also true that had UKSA presented the numbers “better” in the first place, the Leave campaign might never have used it in the way it has. UKSA could also have moved more quickly to clear up the ambiguity. Instead it left this statement to the 11th hour and risks making itself look political. A lesson should be learnt by all who feel that such numbers are important to democracy, accountability and good policy making. The reputation of numbers has taken a hit in this debate and every effort should be made to make sure they are presented properly in the future.
The claim/fact/lie that it costs the UK £350 million a week to be a member of the EU and the spats about it between Remain and Leave camps will live in the memory long after the referendum has gone. I hope that the destructive ferocity of the debate about the number – and it will now presumably get even worse – will long be in the minds of UKSA too and will lead to changes of practice beyond writing a few letters and a press release.
The background to the issue is explained in this FullFact blog. An informative summary (but of course viewing it from the Remain side) comes from InFacts. And the BBC wrote about the politics of the £350million. Like it or not we are stuck with it.
So how did we get in this mess?
The number comes from the so-called Pink Book. It’s the flag ship ONS publication that annually sets out the UK’s balance of payments figures, in other words, the overseas flows of money. It used to be pink and used to be a book.
The image below shows table 9.9, from the latest (2014) edition published last October. Total debits (payments to the EU, row 50, final column for 2014) were £19.1bn. Divide that by 52 and you get to something around £350m a week. So the 350 itself is “right” but what does “total debits” mean. And when is it better to use the balance figure (roughly half the size, in row 50) or some other netted off figure (such as row 50 less row 17, as UKSA suggests in its statement)? How is a user of the table, prior to this debate, meant to know which is the “best” figure?
Here are the footnotes to the table, which do not explain this aspect of the figures:
I do not defend the continued use of the number by the Leave camp but a neutral in the cold light of day could feel some sympathy. A researcher who was happy to stumble into table 9.9 when it was published last year would see the gross figure and assume that it was the amount paid. Unless you know something that is quite technical, and involves looking in a place on another website to which you are not directed, why would you strip off some credits to reduce the amount paid over? I still don’t really understand why some credits and debits are or are not included in the best estimate of the government’s net (or gross?) payment. The data are probably very accurate but help is needed to understand the figures. That help was not offered by UKSA (or ONS, its executive arm) last October. Ticking off the Leave campaign for its use of the number would have much more weight and credibility if they had ignored clearly presented caveats.
What could ONS have done?
I would like to think that had ONS/UKSA known then what they know now, they’d have done things differently. Though there is no hint of this in their announcement today.
The minimum starting point would have been to add footnotes to table 9.9. In his statement today, Sir Andrew says “it is vital that official statistics are used accurately, with important caveats and limitations explained”. UKSA should have supplied those important caveats and limitations alongside the data! This could have been a link to Treasury guidance, other sources or just a line or two about the various versions of “net”, all of which are accurate but meaning different things, that could be derived from the table.
Or the rows could have been presented differently. Row 17 could have been slotted in under row 41 as another adjustment. Or if there is some need to present this table exactly as it is presented, produce a “Table 9.9 v2” on the tab/page after showing a version of the table that better reflects the reality, and direct users there, with explanation?
Accounts are presented all the time by many organisations. Sometimes there are constraints in how they can be presented but mostly the aim ought to be to help the reader draw the right conclusion. I went to the UK Statistics Authority annual accounts to see how they present expenditure. Quite well is the short answer. The image below is from page 75 of the 2014/15 accounts and shows how some receipts have been netted off the much larger expenditure. The same could have been done with the UK contributions to the EU budget.
One could say that total UKSA expenditure was £197m (£117m plus £81m in rounded terms). That doesn’t look wrong to me. But UKSA clearly felt that presenting what it calls net operating cost was more meaningful, so it added in the £27m of income, despite it being a section headed “Programme expenditure”. Further down there is a “Total comprehensive expenditure”incorporating other losses and gains (on capital and intangibles).
That said, having missed the opportunity to present the data better last October and realising that a problem was brewing what could UKSA have done? One option was to reissue table 9.9 with footnotes and some explanatory text. It would have been an easy way to sort the problem. But they didn’t.
UKSA has tried to clear up the confusion, once it had set in, for example in this letter in April. It concluded as follows:
Alas, such interventions, given the other options had been passed up, always looked a bit too little too late. Apart from some press comment, the impact of such letters is questionable and they do not reach anyone else coming to the data who is not up to speed with the UKSA “disappointment”.
For the record, ONS has also published (25 May, on a beta site) one of its new “Perspective” features on The UK contribution to the EU budget. It included the chart below which helpfully sets out the numbers. It also had the lines (hidden away in a slideshow): “In 2014 the UK’s official gross payments to the EU amounted to £19.1 billion. However, this amount of money was never actually transferred to the EU …” Why was that information not put in the Pink Book?
So far as the £350m and the referendum goes, the die is cast. Today’s announcement seems unlikely to do anything to change the situation. Both sides have misused numbers in this referendum debate but the Leave campaign will probably feel a bit miffed about this UKSA statement and its timing so long after the data’s release in October. The issue needs to be returned to in the summer, in or out. The reputation of all statistics (and UKSA for that matter) is under threat.
- UKSA is reviewing the code of practice under which statistics are produced and disseminated. We shall hope that the statisticians have learnt that they must be super vigilant about presentation and how it affects the use of data. The new code can highlight that. The ONS must get its house in order.
- UKSA produced a report in 2013, “Official statistics in the context of the referendum on Scottish independence” ahead of the 2014 referendum. Perhaps it should have produced a similar document for this referendum too. It would have flushed out issues such as this before this late stage of the campaign.
Footnote – The UKSA statement of 27 May:
Sir Andrew Dilnot, Chair of the UK Statistics Authority has said today:
- Given the high level of public interest in the European Union referendum debate, it is vital that official statistics are used accurately, with important caveats and limitations explained.
- The UK Statistics Authority is disappointed to note that there continue to be suggestions that the UK contributes £350 million to the EU each week, and that this full amount could be spent elsewhere (see Annex A).
- As we have made clear, the UK’s contribution to the EU is paid after the application of the rebate. We have also pointed out that there are payments received by the UK public and private sectors that are relevant here. The continued use of a gross figure in contexts that imply it is a net figure is misleading and undermines trust in official statistics.