Here’s a bit more on the potential of big data and administrative data, in particular data linkage, in the work of national statistical agencies. I am prompted by a journal landing on the door step in which leading statisticians set out the case for the use of big data. It has given me a renewed sense of optimism that there could be innovative and better statistics to come even if budgets are under threat and traditional methods are suffering. Continue reading More on big data ….. data linkage
What’s the difference between statisticians and data scientists in the context of the government statistical service (GSS)? There are differences in technical skills between the two but they can be bridged. The important difference might be more to do with the individual mindset of most data scientists and the organisational culture they seek – that might be the problem faced by the civil service when it comes to recruitment and retention. Continue reading Data scientist or statistician?
The Bean Review of government statistics will assess what the public sector data machine needs to make it fit for purpose for the next decade or two. The regulatory framework and outputs are part of that but at the core is a question about sources: how can big data, open data and administrative data help deliver more and new accurate statistics, in a more timely fashion and for less money. This note tries to unravel what these terms might mean for the Government Statistical Service (GSS). It concludes that there is an imperative to investigate the possibilities and that the Bean review can ensure that the required development work is supported at the highest level in government. Continue reading Big, open and admin – what’s what for government statistics?
The statistics community must shout loud in the debate about Freedom of Information reform or access to data could be set back a generation. There are accepted rules and practices relating to transparency and openness of data and the Government and civil service are on a path to fulfilling them.
The Government can achieve what it wants from its review of the Freedom of Information Act – to make a “safe space” and keep secret some advice it receives – while making the foundations on which policy is made more transparent.
The news of the review was, understandably, received negatively but there is hope for some positive outcomes. It is time for the UK Statistics Authority to lead. Continue reading FoI Commission – Good for Stats?
The rain was pouring down outside so Bank Holiday Monday was a good time to read the blog from Met Office Chief Scientist Professor Dame Julia Slingo OBE FRS in which she reflects on this summer’s weather. The title – “So what happened to our summer?” – gives away the sense of mild embarrassment about the poor predictions for July and August. The Met Office has lost the confidence of the BBC which has terminated the weather presenting contract. (This Met Office blog covers that and criticism of its apps.) Beyond the accuracy of its forecasting, I see evidence that the Met Office has confused governance, is not being very open with its data, not focusing on the public’s needs and opinions, setting itself lower targets for forecasting accuracy, and is becoming less transparent in its operations. Perhaps it’s no surprise that the organisation is being reviewed by Government, not least as they were given the go-ahead last year to spend £100m on a new computer. Continue reading Met Office – best in the world?
At first it seemed a bit like an episode of “The thick of it“. An independent review of an independent body “while fully protecting” that independence. You can almost hear the words coming from the head DoSAC, Nicola Murray. But this is real life, it’s under a Conservative majority government and it’s about something really important – the statistical underpinning of our democracy. Apart from showing wonderfully how the word “independent” has been devalued in politics, this is a stunningly important review that will help to set the course for policy-making and democracy for years to come. Continue reading Treasury review of government statistics
The mood music around the government’s announcement in mid-July that a commission will report to it in November on how to reform the FoIA has been greeted with almost universal concern and fear. But in the world of data and statistics – that is all the evidence used by policy makers and those in and out of government assessing policy – there is a great opportunity. Making more data open, leads to more accurate and useful data, improves policy making, enhances transparency, boosts the democratic process and reduces compliance costs associated with FoIA. Surely the commission and the government will see this virtuous circle and encourage more openness and greater efficiency from public sector bodies. Or does the government want to hide the basic data from us? Continue reading Open data and freedom of information – an example
If you are hoping for an open data revolution in government that could deliver new knowledge, statistics and understanding of our society and economy, one place that you might hope to see it appear is in the Office for National Statistics. The progress to date has been very disappointing but future prospects are more rosy. Continue reading Open data, the ONS and government statistics
The Prime Minister gave his flagship speech on migration a few days ago and, perhaps inadvertently, issued a stark warning about the quality of official statistics. He said that the official estimates of HMOs (homes of multiple occupation) in one part of London were out by a factor of sixty! Continue reading Migration – the reality and the official statistics