Open data, the ONS and government statistics

If you are hoping for an open data revolution in government that could deliver new knowledge, statistics and understanding of our society and economy, one place that you might hope to see it appear is in the Office for National Statistics. The progress to date has been very disappointing but future prospects are more rosy.

The ONS, and its governing board UKSA, sits at the heart of the government’s statistical system, the GSS, that links all the statistics departments across Whitehall. UKSA and the ONS has little control over the departments but can set an example. As the powerhouse of the machine, you’d expect it to do so.

It would be supported in such actions by the “rules”. The Code of Practice for Official Statistics says (Protocol 3): “Administrative sources should be fully exploited for statistical purposes, subject to adherence to appropriate safeguards.” Practice 3 within that Protocol adds: “Maximise opportunities for the use of administrative data, cross-analysis of sources and for the exchange and re-use of data, to avoid duplicating requests for information.”

Practice 5 of the Protocol is even more explicit about what is required. “Prepare, in consultation with the National Statistician, a Statement of Administrative Sources which identifies the following. a. The administrative systems currently used in the production of official statistics. ……. c. Information on other administrative sources that are not currently used in the production of official statistics but have potential to be so used.”

But, as we shall see, UKSA has not apparently challenged the ONS and others on this point. The National Statistician has even offered guidance on the “Use of Administrative or Management Information”, but it was last updated in January 2011. The statements of administrative sources make interesting reading. Those used by ONS, and links to other departmental sources, were listed in a response to my (undated) FoI question from February this year.

So, what new statistics are there as a result of open data?

My dream was that the many new, expanding and increasingly digitised (known and unknown) data bases in government would yield lots of new statistical series with a fuller, often more accurate picture of society emerging from it. Well, there’s not been many published, in fact, perhaps none in the ONS at least. One sometimes hears ONS staff say that new data has improved the quality of existing series but I haven’t seen that written up or properly justified so who knows.

One area where administrative data (as the civil servants often call the public sector’s potentially open data) could really help improve the nation’s view of itself is the Census. And, admin data used in the Census would allow at least partial updates far more frequent than every decade! Sadly, the consultation on the 2021 Census published last week (on 4 June) was disappointing. It contained just four pages in a 100+ page document on the potential new sources (head for Annex B on page 51).

Most revealingly, it approaches admin data from the wrong end – it looks to see which data published in the last Census could be sourced from admin data when it ought to look at what statistics could be drawn from admin data. That data might then replicate or be different from – or even add to the depth of – the Census. The ONS along with, it has to be said, many of the Census users are apparently blind to the fact that there could be many new descriptive statistics lurking in the admin data.

The ONS, it seems, sees admin data as little more than a way to verify the total population figure from the Census (and possibly help with enumeration). And that phrasing is important – the Census will be used to judge the quality of the admin data, not the other way round! The admin data will certainly not be seen as something to deliver new and quite possibly more accurate data on population characteristics.

It’s a shame that the plans for the 2021 census are unambitious as the public mood to fill in forms is waning and the (understandable) lack of desire of government to spend £1 billion plus on it pointed to a new way. The obvious way was to use some part of the mountain of administrative data. But little has changed which is a shame as the data would have become more accessible and potentially “open” since the idea was first mooted in the mid 1990s in advance of the 2001 census. It looks as if once again the ONS is pushing admin data into the long grass.

What has the UKSA and ONS said about open data?

The GSS does have a web page setting out its policy, good practice and guidance on open data. It includes a thin, three page document. There is also a longer version in an attachment sent to PASC in September 2013. These two documents were the ones referred to by ONS in response to a freedom of information request about the Public Data Principles (which also noted several other ONS-related open data type lists). There was another FoI request from last year which referred to two other UKSA documents from 2011. That FoI request unhelpfully refused to say when open data had been discussed at UKSA board meetings – saying that any discussion would have been noted in the minutes – so, short of spending some hours wading through the minutes, were are left unsure if it has been a hot topic at the board. We have no evidence to hand that the UKSA board has pushed the issue.

Under “good practice” on the ONS web page there is only one ONS case study, on data behind the CPI. It’s a case study I know well as it writes up a two-year battle I had with the ONS to release this data. The document is undated – a particularly irritating UKSA/ONS habit that limits the users’ understanding of what is being read – but was probably produced in 2012. I had requested all the individual price quotes that go into the calculation of the RPI/CPI monthly figures. In the end, the battle was only a partial success as only regional data for some of the categories of goods were published, in six-monthly chunks, following the fear of the disclosure team that the shops where prices were obtained would become identifiable. It’s an odd reading as shop prices are not private in the way that information about, say, personal income or health is. I can walk into the local supermarket and read prices off shelves or from my receipts!

The UKSA/ONS documents make much of the data that the organisation already publishes. It does indeed publish a lot of statistics – that’s its job! But not much of that has to do with open data, and not much has changed in years.

Beyond that, two thoughts. First, the ONS web site relaunch in 2011 was a disaster. This has been widely written up. UKSA claims that it is much better now. There are signs that improvements are now being made but as a response to a disaster the last four years have, in the eyes of many users, been wasted. Second, the arrival of the new baby,, has been an anticlimax. There’s hardly any new data on it, from ONS or elsewhere, and if new data were to be there you wouldn’t know where to look for it.

Lets end on a more optimistic note. What is notable about all the official documents mentioned above is that none is recent! (The Census consultation is the exception but that unambitious path was set a few years ago.) Normally the lack of anything new would be bad news, the sign of not much happening. But perhaps not in this case. In the last year we have had a new National Statistician appointed and several other senior staff who are well-known to take open data and innovation more seriously. I conclude that while very little has happened we should expect, hope for and encourage some significant moves in the period ahead.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s