UK statistics – will it be an open data failure?

The open data map I produced on 10 May showed that the coalition government had been busy with public sector open data. It set up many initiatives to open up data and it felt great at times – there was certainly fantastic ambition. The trouble is that there was no real plan and a shortage of skills, which led to a lack of delivery. Unforgivably, there was very little openness about what was going on and it now looks as if not much has been delivered. We have a new government and it’s a good time to think about what is needed if the supposed benefits of public sector open data are to be seen. 

As Guiseppe Sollazzo, an ex-colleague on ODUG, has written: The British government has “… gone from four to zero advisory panels on open data issues in the space of two years.” He was referring to the Data Strategy Board (wound up in 2013), APPSI and ODUG (wound up earlier this year) and the Public Sector Transparency Board (thought to have been appointed until April 2015).

Are these bodies to be resurrected, replaced or done without?

The buzz about open data in the UK has been ignited by inspiring individuals. The government has talked the talk (and written some cheques) and thousands of disparate developers in start-ups, bedrooms and the public sector, have done things that would have been unimaginable a decade ago. The UK tops a prestigious table for the performance with regard to open government data.

Despite the long list of initiatives, I still don’t have the data I’d like to have. I’d like information, statistics or data that:

  • sheds light onto the state and changing nature of the UK economy and society. This is an area pretty much unaffected by the recent open data activities – we have no new stats of note and little additional insight. Access to existing data has not got much better, indeed has got worse in some cases.
  • describes my local area – about schools, street cleaning, car ownership or parking tickets. This is partly to monitor local services and engage local communities but also to make life easier. These just don’t seem to have become available. If there are apps that would improve my life, I haven’t found them.

Open data in this context is not about personally disclosive data and it only relates to information held by the public sector – those data whose collection has been paid for by the people (tax payers) and are, in aggregate, about them and their activities.

To try to understand what is going on I produced an “Open data map” to show the main players. I have then looked on their web sites to see what they have to say.

I conclude the following themes:

  • Lots of talk and little action. Enormous amount of talk yet there’s been a mixed reaction from the public sector. A relatively small proportion of the information that could have been released has been released.
  • No strategy? Hard to pin down what government open data strategy is. It seems to be fragmented and lacking coherence. Reports are produced and not followed through in any robust way. Various groups have been launched and then faded away.
  • The departments leading the transparency agenda are not transparent. The same applies to the various supporting transparency groups: they are lacking transparency in their structures (who works there, how much does it cost, how do you contact them and what do they do?) and often publish no basic admin info such as minutes of meetings. It is sometimes left to independent individuals or volunteer groups to publish minutes etc and to use FoI to extract information. Some groups are well hidden with only contact details being postal addresses!
  • “Champions” of open data are under-resourced given the huge inertia not to open up data and the many vested interests resisting any change.
  • Among CG departments, some are far more willing to engage than others. I have been told that my characterisation is unfair but I see the Cabinet Office as pro open data (but often not very effective) and BIS largely against (and rather more effective). With some exceptions, many other departments – and notably the ONS and UKSA – seem indifferent. The Prime Minister is apparently keen and has given good speeches but how much action has resulted? Rare are the cases where departments or their leaders have driven change – it seems to be individuals in departments that have produced new products. But the bottom up approach is not going to be effective on the really big issues.
  • Budget or public spending and revenue data lacks structure and context. Opening up of government costs/expenditure has been notable but is unstructured and underused.
  • Barely any core data. There has been a failure to progress big issues of core reference data – maps, weather, and corporate and land registers – from Public Data Group assets. The National Information Infrastructure has stalled under Cabinet Office leadership.
  • The website has been a disaster. It is very hard to find information.
  • The Government has done little to promote the results of open data. More generally on access, there is no place for the public to find outputs or apps. And if data are poor or inaccurate there is no common or reliable way for users’ comments to be fed back.
  • Apps have been for amusement not action. Some apps have made life easier or more entertaining at the margin, but they are mostly of fringe appeal and there are very few new statistics, little gain for policy analysts and public.
  • Government expertise is lacking. Skill levels are too low so defensive attitudes lead to inaction. Within government, the cross cutting, inter-departmental professions (statisticians, economist, social scientists etc) are showing little interest
  • Few in government use open data. OD practices are a long way from business as normal. Freeing up of data has been heavily under-resourced.
  • Much interesting public sector data is now not under direct ministerial control. Arms length bodies are in denial. Some ex-utilities have been privatised. There is a major issue to be addressed – there’s little interest to open under their own steam and it’s hard for the centre to apply pressure yet much of what is important rests there.

There are some areas of considerable achievement in the public sector’s open data world but the direction of travel is often painfully slow – and has probably slowed since 2012. The new government needs a coherent strategy to deliver real change or the open data movement will stall in the face of modest achievements and campaigners’ exhaustion.


4 thoughts on “UK statistics – will it be an open data failure?

  1. I notionally sit on the local public data panel – I think it’s at least a year since we met – the will to have external advisers working on local government data issues seems half hearted. That doesn’t mean their isn’t will elsewhere – cabinet office still, but….


    1. Well, there is no mention of privacy or anonymisation as that was not really what the blogs were about. If there was the willingness and ability to release lots of data, ensuring the right processes are in place to deliver anonymisation would become important. If there was a more coherent plan for open data perhaps anonymisation would find its rightful place in the debate?


      1. The standard rationale for not going 100% open is a privacy one (even though this may be a conveniant excuse sometimes rather than a real reason) so you do need to deal with privacy up front if you are going to have optimal open data.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s