Which parts of the public sector are battling to open up data – or resisting it – in the UK? There are so many moving parts it’s hard to say, really. So, I have produced this “map” to set out the main players and their links to each other. (Blog updated since first publication.)
This first diagram sets the scene. The many silos in the public sector have loads of information that is by and large not made public. There are lots of people – mainly outside the public sector but also in it! – who’d happily look at it and use it for one purpose or another. Open data is about enabling that sharing.
The “map” below is more complex. It shows the main players in the UK public sector – and the links between them. Solid lines imply a closer or more controlling relationship than a dotted line.
The main heavy outer line represents the public sector. The dotted line inside that is central government. The departments of state with a major role in open data are identified as are the main agencies and the principle initiatives set up by them.
Of course, virtually all the departments and agents of government release some information. What they publish or make available varies greatly between them. The triangles show the main conduits of information as part of the open data movement. Some data that used to be public sector data has been privatised and, as things stand, are now outside the public sector.
*** Please send me comments or suggested corrections – email@example.com ***
Here is a list of acronyms:
PS – Declaration of interest: I have been working with PASC on their statistics reports during the last Parliament and have been a member of ODUG since last September.
The rest of this blog is a series of links to the main web pages of the above organisations and other pages that explain the scene.
What is open data?
OKF – and the open data handbook – http://opendatahandbook.org/en/what-is-open-data/
Open data is data that can be freely used, reused and redistributed by anyone – subject only, at most, to the requirement to attribute and share alike.
The full Open Definition gives precise details as to what this means. To summarize the most important:
- Availability and Access:the data must be available as a whole and at no more than a reasonable reproduction cost, preferably by downloading over the internet. The data must also be available in a convenient and modifiable form.
- Reuse and Redistribution:the data must be provided under terms that permit reuse and redistribution including the intermixing with other datasets.
- Universal Participation:everyone must be able to use, reuse and redistribute – there should be no discrimination against fields of endeavour or against persons or groups. For example, ‘non-commercial’ restrictions that would prevent ‘commercial’ use, or restrictions of use for certain purposes (e.g. only in education), are not allowed.
And the open data handbook – http://opendatahandbook.org/
Surveys of open data country performance
Open Data Barometer: – http://webfoundation.org/2015/01/open-data-barometer-second-edition/
http://opendatabarometer.org/ January 2015, 2nd edition
…….. and the Centre for Data Innovation March 2015 report: – http://www2.datainnovation.org/2015-open-data-g8.pdf
IBP Open Budget Survey (UK 3rd)
The data is in the public sector.
The public sector is in turn made up of several sub-sectors:
- CG (mainly the Cabinet Office for policy + BIS for the core infrastructure assets)
- Cross cutting (mainly Government Statistical Service)
- Non-departmental public bodies – Arms length bodies, quangos
- Public corporations
- one of its 7 “priorities” is “promoting the release of government data, and making the way government worksmore transparent” which leads to:
a page headed: “government transparency and accountability”. The “issue” is described fairly enough but the “actions” are a disparate selection.
It lists a number of initiatives:
- Set up http://data.gov.uk/
- Published specific departmental open data startegies https://www.gov.uk/government/policies/improving-the-transparency-and-accountability-of-government-and-its-services/supporting-pages/commitment-to-publishing-data and http://data.gov.uk/open-data-strategies These are not in consistent formats, of variable quality, sometimes expired and rarely updated.
- introduced legislation on the right to public data http://www.legislation.gov.uk/ukpga/2000/36/contents “to make sure that all the government data that can be published, is published in an accessible format” – but it is no obvious what has changed and to find the new elements.
- published the Open data White Paper in June 2012 https://www.gov.uk/government/publications/open-data-white-paper-unleashing-the-potential following “Making open data real” consultation, published January 2012: https://www.gov.uk/government/consultations/making-open-data-real
- published the NII v1 in October 2013 https://www.gov.uk/government/publications/national-information-infrastructure
- “Release of data fund” (CO) and Open Data “Breakthrough fund” (BIS) launched https://www.gov.uk/government/publications/breakthrough-fund-and-release-of-data-fund
- open data charter https://www.gov.uk/government/publications/open-data-charter/g8-open-data-charter-and-technical-annex
- hosted OGP event in 2013 http://www.opengovpartnership.org/ and G8 open data charter published June 2013 https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/207772/Open_Data_Charter.pdf
- PM letter May 2010 https://www.gov.uk/government/news/letter-to-government-departments-on-opening-up-data
- PM letter July 2011 https://www.gov.uk/government/news/letter-to-cabinet-ministers-on-transparency-and-open-data
- Shakespeare review announced in October 2012, publish May 2013 https://www.gov.uk/government/publications/shakespeare-review-of-public-sector-information and ToRs https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/34663/12-1233-independent-public-sector-information-review-draft-terms.pdf Govt response: https://www.gov.uk/government/publications/government-response-to-shakespeare-review YouGov survey as part of review – http://datahub.io/dataset/shakespeare-review
- Established ODUG https://www.gov.uk/government/groups/129
ODI – http://theodi.org/
“The Open Data Institute is catalysing the evolution of open data culture to create economic, environmental, and social value. It helps unlock supply, generates demand, creates and disseminates knowledge to address local and global issues.” http://theodi.org/about-us “independent, non-profit, non-partisan, limited by guarantee company.” The information might well be on the ODI site but it’s hard to see exactly what has been done with what money.
Business Plan 2012-2017
Open data roadmap, published December 2014 – http://theodi.org/roadmap-uk-2015
The Shakespeare Review was announced in October 2012 and published in May 2013:
He was: “ …invited by government to lead an independent review of Public Sector Information (PSI) to explore the growth opportunities of, and how to widen access to, the wealth of information held by the public sector.”
Quote from Shakespeare review, page 5:
“This next phase of the digital revolution has PSI at the very foundation. Therefore Britain enjoys significant advantages: the size and coherence of our public sector (who else has critically important data of the range and depth of the NHS?) combined with government’s strong commitment to a visionary open data policy means that we have the opportunity to be world leaders in the enlightened use of data. If we play it right we can break free of the shackles of a low-growth economy in which government and the public sector are seen as a resource drag and an obstacle, and they instead become key drivers of a transforming process.”
And page 6 ……..
“This review does not call for any significant increase in spending on a national data strategy, nor any additional administrative complexity; rather, it calls for a broadening of objectives together with a sharpening of planning and controls. We should remain firm in the principle that publicly-funded data belongs to the public; recognise that we cannot always predict where the greatest value lies but know there are huge opportunities across the whole spectrum of PSI; appreciate that value is in discovery (understanding what works), better management (tracking effectiveness of public administration), and commercialisation (making data practically useful to citizens and clients); create faster and more predictable routes to access; and be bold in making it happen.”
It had 9 recommendations (pages 11-19) and these have not been pursued hard ………..
Cabinet Office produces quarterly progress reports:
The first was published in December 2012, relating to Q3 2012:
“Report on Departmental Open Data Commitments and adherence to Public Data Principles”
Relating to April to June 2014, the 8th and latest published in March 2015:
They are generally bland documents with few links and finding the key news in among the waffle is hard.
Data Strategy Board
ToR for PDG and DSB (March 2012):
(See PDG entry below)
The board’s remit is to create maximum value, for businesses and people across the UK, from data held by the Public Data Group (PDG) members (Ordnance Survey, Met Office, Land Registry, Companies House) and beyond.”
The Data Strategy Board closed on 28 June 2013.
Various transparency boards were set up:
- Public sector transparency board https://www.gov.uk/government/groups/public-sector-transparency-board no minutes since June 2013
- International Development Sector Transparency Panel – three meetings, last in Jan 2014 – https://www.gov.uk/government/groups/international-development-sector-transparency-panel
- Crime, Justice transparency panel – https://www.gov.uk/government/policies/creating-a-transparent-justice-system – and meetings?
- DCMS open data forum – https://www.gov.uk/government/collections/dcms-open-data-forum-collections-page – Two meetings? Last in April 2014
- Defra Transparency Board – no minutes found on official sites so try this: http://www.owenboswarva.com/opendata/DNTP/DNTP.htm
- Health & Social Care Transparency panel. No meeting since April 2013 – https://www.gov.uk/government/groups/health-and-social-care-transparency-panel ToRs – https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/259827/HSCTP_Terms_of_reference_FINAL_March_2012.pdf
- Local Public Data Panel. Last meeting January 2012 – http://data.gov.uk/users/local-data-panel DCLG support – https://www.gov.uk/government/publications/dclg-public-data-and-transparency-programme-board
- Research Sector Transparency Board. More life – https://www.gov.uk/government/groups/research-sector-transparency-board last meeting in May 2014
- Social Mobility Transparency Board. No minutes – refused under FoI. ToRs – http://www.owenboswarva.com/opendata/SMTB_ToR.pdf
- Tax Transparency Sector Board – https://www.gov.uk/government/groups/tax-transparency-sector-board – one of the more active
- Transport Transparency Board https://www.gov.uk/government/groups/dft-transport-sector-transparency-board
- Various other more or less official groups including “Environment Agency open data group” http://www.owenboswarva.com/opendata/EA/EAdata.htm#eadag Full (possibly?) list at https://docs.google.com/spreadsheet/ccc?key=0AkAX7sP1au8YdENDa3pQaExlem9uVU5PYUdUVm5SVmc#gid=0
Open Government Partnership UK National Action Plan 2013 to 2015, published October 2013
How many people work on open data and transparency in CO? It’s not obvious from:
If you know where to look you can find something.
Enthusiastic group of unpaid enthusiasts but has limited influence over CO decisions:
and its own unofficial site: http://odug.org.uk/
ODUG data requests http://data.gov.uk/data-request
Public Data Group:
ToR for PDG and DSB (March 2012):
(PDG replaced the proposal for a Public Data Corporation)
….. and still not revised to allow for demise of DSB:
Public Data Group – https://www.gov.uk/government/groups/public-data-group#terms-of-reference
“Open Data Statement”, July 2014, explaining what the four public sector bodies – Companies House, Land Registry, Met Office and Ordnance Survey publish:
“The Met Office is a government Trading Fund Agency owned by the Department for Business Innovation & Skills. It provides services to the public and the commercial sector.”
“The Public Weather Service Customer Group (PWSCG) acts as the customer on behalf of the public for free at the point of use weather services and on behalf of Public Sector users of PWS outputs.”
Geographic Information Group
Head announced in April 2011:
Administrative Data Taskforce report, December 2012
Government response, June 2013:
Called “Improving access for research and policy” – ie not open to all
The consumer empowerment strategy “Better Choices: Better Deals”, published in April 2011, with progress report in December 2012. Launch of miData:
“We fund, support and connect innovative businesses to accelerate sustainable economic growth.”
….. an executive non-departmental public body, sponsored by BIS
Six meetings of the board each year:
Minutes published only after FoI request in 2015.
APPSI’s terms of reference apply to England, Scotland, Wales and Northern Ireland. APPSI’s role is:
- to advise Ministers on how to encourage and create opportunities in the information industry for greater re-use of public sector information
- to advise the Director of the Office of Public Sector Information and Controller of Her Majesty’s Stationery Office about changes and opportunities in the information industry, so that the licensing of Crown copyright and public sector information is aligned with current and emerging developments
- to review and consider complaints under theRe-use of Public Sector Information Regulations 2005 and advise on the impact of the complaints procedures under those regulations
Report: “What is the value of open data?”, January 2014
Last annual report in 2009/10:
http://www.opsi.gov.uk/psi/ – site no longer updated?
It is part of National Archives since 2006
“The National Archives is an executive agency of the Ministry of Justice and a government department in its own right.”
OPSI and HMSO are one and the same and both merged into NArchives.
Department of Health
Information Governance review, Caldicott report, April 2013:
“independent review of information sharing to ensure that there is an appropriate balance between the protection of patient information and the use and sharing of information to improve patient care”
Dame Fiona Caldicott appointed as National Data Guardian, November 2014
data, including “open data” :
It is “the national provider of information, data and IT systems for commissioners, analysts and clinicians in health and social care.” And “HSCIC is an executive non-departmental public body, sponsored by DH.”
What is NAO:
“The National Audit Office (NAO) scrutinises public spending on behalf of Parliament.
Our audit of central government has two main aims. By reporting the results of our audits to Parliament, we hold government departments and bodies to account for the way they use public money, thereby safeguarding the interests of taxpayers. In addition, our work aims to help public service managers improve performance and service delivery. The Audit and inspection rights are vested in the head of the National Audit Office, the Comptroller and Auditor General (C&AG). The staff of the NAO carry out these tasks on his behalf.”
“Implementing transparency” April 2012
“The report recognises the strength of the strategic case for greater transparency, and highlights progress across government in fulfilling most of its initial commitments. However, government needs a better understanding of costs, benefits and use to assess whether transparency is meeting its objectives of increasing accountability, supporting service improvement and stimulating economic growth.”
“A lack of structure in the government’s transparency initiatives could mislead potential users of its open data, according to the National Audit Office (NAO)”
“The ICO is an independent public body and the Ministry of Justice is the ICO’s sponsoring department within Government.”
“The ICO’s mission is to uphold information rights in the public interest, promoting openness by public bodies and data privacy for individuals”
Blog – “Anonymisation – opportunities and risks”, November 2012
Data protection “duck out” is false, September 2013
Next generation FoI, August 2013:
ICO annual report June 2013, the year for companies to get data handling right:
New guidance for FoI, April 2013:
Departmental Open Data Strategies
Most were published in June 2012 and now a bit dated. Most are formulaic. DH is not. Most ran 2012-2015, some have expired.
Cabinet Office, 2012. Open Data Strategy. London: Cabinet Office.
Defra, 2012. Open Data Strategy June 2012 – March 2014. London: Defra.
Department for Business, Innovation & Skills, 2012. Open Data Strategy 2012-14. London: Department for Business, Innovation & Skills.
Department for Communities & Local Government, 2012. Open Data Strategy April 2012-April 2014. London: Department for Communities & Local Government.
Department for Culture, Media and Sport, 2012. Open Data Strategy 2012-15. London: Department for Culture, Media and Sport.
Department for Education, 2012. Open Data Strategy. London: Department for Education.
Department for International Development, 2012. Open Data Strategy April 2012 – March 2014. London: Department for International Development.
Department for Transport, 2012. Open Data Strategy. London: Department for Transport.
Department for Works and Pensions, 2012. Open Data Strategy. London: Department for Works and Pensions.
Department of Energy and Climate Change, 2012. DECC’s Open Data Strategy. London: Department of Energy and Climate Change.
Department of Health, 2012. “The Power of Information: putting all of us in control of the health and care information we need.” London: Department of Health. Very specific/broad and not really open data.
Foreign and Commonwealth Office, 2012. The FCO’s Open Data Strategy. London: Foreign and Commonwealth Office.
HM Revenue & Customs, 2012. Open Data Strategy. London: HM Revenue & Customs.
HM Treasury, 2012. Open Data Strategy 2012-2015. London: HM Treasury.
Whole of Government Accounts:
and PAC inquiry
Home Office, 2012. Open Data Strategy for the Home Office 2012-13. London: Home Office.
and revised in April 2013
Ministry of Defence, 2012. Open Data Strategy 2012-2014. London: Ministry of Defence.
Ministry of Justice, 2012. Open Data Strategy 2012-15. London: Ministry of Justice.
Separately the GSS/National Statistician published this thin document in 2012:
which can be found alongside more material here: https://gss.civilservice.gov.uk/statistics/open-data/
The ONS puts lots of data on data.gov.uk but only repeating what is on its own site:
RCUK data principles
http://www.rcuk.ac.uk/research/datapolicy/ (open to academics or all?)
Evaluation of public dialogue on open data, report, June 2012:
Finch report on making research open, June 2012:
Charities, learned bodies
Wellcome Trust, “Incentives and Culture Change for Data Access”, report May 2014 (lots to do)
“UK open research data forum”, January 2014
“Science as an open enterprise”, June 2012
British Library (Datacite)
Royal Statistical Society
Data manifesto calls for more open data
PASC report on open data:
including government response, September 2014:
One in a series of 11 statistics reports:
Report on Finch report, open access for academic data
Nothing on the various open data bodies under BIS.
Other reports and groups:
Supporting local area initiatives:
Nesta work on Open Data Scotland:
“The Department for Business, Innovation and Skills (BIS) commissioned Deloitte to undertake an independent assessment of the market for public sector information (PSI), as part of the Shakespeare Review. Public sector information covers the wide range of information that public sector bodies collect, produce, reproduce and disseminate in their many areas of activity while accomplishing their public tasks. It is the first UK-wide market assessment of PSI, and spans the use and re-use of PSI at UK, regional and local levels, by a wide range of businesses, civil society groups, government and members of the general public.”
Appendix 9 contains a large bibliography.
The “Information Economy Strategy” sets out a shared vision for a thriving UK information economy that enhances national competitiveness – See more at:
A Right to Data – Policy Exchange
Updates from civil society engagement with the UK Government on data sharing
INSPIRE is “an EU initiative to establish an infrastructure for spatial information in Europe that is geared to help to make spatial or geographical information more accessible and interoperable for a wide range of purposes supporting sustainable development”
The INSPIRE directive lays down a general framework for a Spatial Data Infrastructure (SDI) for the purposes of European Community environmental policies and policies or activities which may have an impact on the environment. The INSPIRE Directive entered into force on 15 May 2007.
Open Rights Group
Global open data index
virtually all eurostat so of limited value
Socrata – 2014 Open Data Benchmark Report