Tag Archives: Big data

Open data and journalism

The relationship between open data and journalism is a complex one. No one questions the desirability of good journalism and the value of open data but both are under threat. British print circulation is falling, and for many titles by over 10% in a year and traditional revenue streams are under pressure. At the same time the momentum behind the British government’s open data movement shows signs of weakening as its impact is being questioned. They could help each other out. Harder stories based on real evidence (including open data) could make a contribution in halting the decline in journalism and open data. Sadly, in the UK, this seems unlikely to happen. If the private sector won’t seize the opportunity we should look to public sector broadcasting to set standards in quality and exciting journalism.  Continue reading Open data and journalism

Advertisements

Bean: “I’m queasy”

The interim Bean review was mainly about setting a course for the future. It did that well and will prove to be a landmark report. In setting out his concerns about the recent past and present Prof Bean exposed some revealing facts. Of course the report is about the data but half of it was about ONS “effectiveness”. As he said: “users noted that this Review would not have been commissioned if all had been well”. (Para 3.25) It’s a line I had used and it is now clear that all was indeed not well. This blog picks out some soundbites from the report that are worthy of note. The second report (in March) will be about governance. Reading what follows makes it hard to imagine that things are going to stay the same.  Continue reading Bean: “I’m queasy”

Bean: initial findings

The eagerly awaited review of government statistics by Professor Sir Charles Bean, published this morning, is an landmark report that should help the ONS refocus on core activities and do them more professionally. It advocates access to new data sources and the use of state of the art technologies.

The report sets out the problems that the ONS has had in recent years – some elements of which are shocking – and says that a cultural shift is required for change. It is an opportunity for all stakeholders and users to debate ONS work priorities priorities in advance of Bean’s final report in March. That report will tackle governance arrangements. Given the state the stats are in and the task list now facing the ONS, UKSA is going to have to change if it is to survive in its current format.   Continue reading Bean: initial findings

Survey response rates – look down!

Response rates to surveys are declining. Fast. New sources, such as big data or administrative data, or making survey response compulsory, are no longer an optional extra. They need to be discussed now as a way to boost the quality of and restore credibility to the nation’s key figures. The rise in employment and the fall in earnings defined the economy of the last Parliament. As survey response rates fall below 50%, there is a chance that those trends which never gelled to give a coherent picture are some way from reality.

Continue reading Survey response rates – look down!

More on big data ….. data linkage

Here’s a bit more on the potential of big data and administrative data, in particular data linkage, in the work of national statistical agencies. I am prompted by a journal landing on the door step in which leading statisticians set out the case for the use of big data. It has given me a renewed sense of optimism that there could be innovative and better statistics to come even if budgets are under threat and traditional methods are suffering.  Continue reading More on big data ….. data linkage

Data scientist or statistician?

What’s the difference between statisticians and data scientists in the context of the government statistical service (GSS)? There are differences in technical skills between the two but they can be bridged. The important difference might be more to do with the individual mindset of most data scientists and the organisational culture they seek – that might be the problem faced by the civil service when it comes to recruitment and retention.  Continue reading Data scientist or statistician?

Big, open and admin – what’s what for government statistics?

The Bean Review of government statistics will assess what the public sector data machine needs to make it fit for purpose for the next decade or two. The regulatory framework and outputs are part of that but at the core is a question about sources: how can big data, open data and administrative data help deliver more and new accurate statistics, in a more timely fashion and for less money. This note tries to unravel what these terms might mean for the Government Statistical Service (GSS). It concludes that there is an imperative to investigate the possibilities and that the Bean review can ensure that the required development work is supported at the highest level in government.  Continue reading Big, open and admin – what’s what for government statistics?

FoI Commission – Good for Stats?

The statistics community must shout loud in the debate about Freedom of Information reform or access to data could be set back a generation. There are accepted rules and practices relating to transparency and openness of data and the Government and civil service are on a path to fulfilling them.

The Government can achieve what it wants from its review of the Freedom of Information Act – to make a “safe space” and keep secret some advice it receives – while making the foundations on which policy is made more transparent.

The news of the review was, understandably, received negatively but there is hope for some positive outcomes. It is time for the UK Statistics Authority to lead.   Continue reading FoI Commission – Good for Stats?

Met Office – best in the world?

The rain was pouring down outside so Bank Holiday Monday was a good time to read the blog from Met Office Chief Scientist Professor Dame Julia Slingo OBE FRS in which she reflects on this summer’s weather. The title – “So what happened to our summer?” – gives away the sense of mild embarrassment about the poor predictions for July and August. The Met Office has lost the confidence of the BBC which has terminated the weather presenting contract. (This Met Office blog covers that and criticism of its apps.) Beyond the accuracy of its forecasting, I see evidence that the Met Office has confused governance, is not being very open with its data, not focusing on the public’s needs and opinions, setting itself lower targets for forecasting accuracy, and is becoming less transparent in its operations. Perhaps it’s no surprise that the organisation is being reviewed by Government, not least as they were given the go-ahead last year to spend £100m on a new computer.   Continue reading Met Office – best in the world?

Treasury review of government statistics

At first it seemed a bit like an episode of “The thick of it“. An independent review of an independent body “while fully protecting” that independence. You can almost hear the words coming from the head DoSAC, Nicola Murray. But this is real life, it’s under a Conservative majority government and it’s about something really important – the statistical underpinning of our democracy. Apart from showing wonderfully how the word “independent” has been devalued in politics, this is a stunningly important review that will help to set the course for policy-making and democracy for years to come.   Continue reading Treasury review of government statistics