Bean: “I’m queasy”

The interim Bean review was mainly about setting a course for the future. It did that well and will prove to be a landmark report. In setting out his concerns about the recent past and present Prof Bean exposed some revealing facts. Of course the report is about the data but half of it was about ONS “effectiveness”. As he said: “users noted that this Review would not have been commissioned if all had been well”. (Para 3.25) It’s a line I had used and it is now clear that all was indeed not well. This blog picks out some soundbites from the report that are worthy of note. The second report (in March) will be about governance. Reading what follows makes it hard to imagine that things are going to stay the same. 

I was quoted in yesterday’s FT (“ONS must change culture to protect data quality“) as saying that “elements of the Bean report” made me a bit “queasy”. A few people asked about this. If you want to know, read on. This is a very painful list. It reads as if the last decade has been a lost decade. The good news – and it is very good news – is that these issues are now in the public domain and the ONS is setting about tackling them. The past is the past, no more denial, now is the time to fix them. Users should offer their help.

The list:

Relocation. In 2004, Sir Peter Gershon and Sir Michael Lyons carried out reviews into public sector efficiency and relocation, respectively. Together they tried to reduce costs, in particular by moving most London-based ONS functions to Newport. “Many staff members were, however, unwilling to move and instead left the organisation. The resultant loss of expertise is widely believed to have had a significant adverse impact on the subsequent production and development of UK economic statistics and the National Accounts in particular.” (Para 3.9) “It is the judgement of this Review that the loss of statistical expertise which resulted from the relocation decision has had a significant – though not necessarily permanent – detrimental effect on the capability of ONS and the quality of its outputs over the past decade.” (Para 3.62)

Budget. We had to wait for this review to get an assessment of what money has been spent on ONS. Summarised in the chart below, it shows stable to gently declining funding for a decade. (Para 3.13) It might not be perfect but it’s better than nothing and a great start.

Screen Shot 2015-12-03 at 22.37.02.png

ONS spending. It’s never been clear how much any of the families of data cost to produce. The report noted that “the core functions of National Accounts and Economic Statistics only account for direct gross expenditure of £24 million out of a total of £180 million for the whole organisation – in other words, just 13%.” (See page 91 of the latest annual report.) Efficiencies in other data areas could give rise to large proportionate increases in these sums. (Para 3.19)

Financial management. In 2013, the Chartered Institute of Public Finance and Accountancy (CIPFA) was asked by ONS to assess ONS financial management. It found “significant failings, including: a lack of appropriate financial management capability, ownership and accountability beyond the central finance team; an absence of basic financial discipline in programme management; inadequate medium-term financial planning; limited integration between financial and business planning; insufficient focus on securing value for money; and a culture that militated against the finance function supporting transformational change.” (Para 3.20)

Common sense. “A number of users commented on the failure to sense-check some statistics before release, arguing that greater use of economic expertise could help prevent embarrassing errors.” (Para 3.31)

Website. “An effective website is a pre-requisite for an effectively functioning ONS. There was a lot of criticism of the current website, however, with one user describing it as “almost unusable”. Others commented that it was too difficult to find statistics. In sum, using the website was both laborious and frustrating. Such criticism of the ONS website is by no means new.” (Box 3B) Repair has been painfully slow – after four years a beta is being tested.

Errors. “Since March 2012, ONS has issued on average close to two corrections a month to its data and has also been criticised for its handling of erroneous statistics.” In 2014 both the trade figures and travel and tourism figures were found to contain errors and were de-designated as National Statistics. (Para 3.34) “While it is an unrealistic aim to completely eradicate errors within ONS, much greater effort is needed to quality assure the production of statistics. It is the role of ONS to present the most accurate and reliable picture possible of the economy. Failing to do so raises serious challenges for decision makers.” (Para 3.42)

Culture. “ONS should know more about its statistics and their sources than anyone else, so it should be in the vanguard of understanding the limitations of its data and explaining those limitations to users, rather than instinctively defending its statistics when users raise questions about them.” (para 3.49)

Staffing. “Review team discussions with staff at Newport highlighted a number of management issues – including career planning, progression prospects and lack of transparency in internal recruitment – that could be hampering ONS’s ability to recruit and retain staff.” (Para 3.55)

Professional expertise. “It seems clear that ONS is not only much less dominant as a centre of statistical expertise than it once was, but that it is also lagging as a centre of economic expertise.” (Para 3.58) As the table shows, the rise in economists and statisticians employed in the ONS has been more modest than across the civil service.

Screen Shot 2015-12-03 at 23.08.21.png

Planning. “ONS cannot afford to chase each topical question as it arises. Rather, it needs to make greater use of one-off exploratory and horizon-scanning projects, informed by policy needs and carried out in collaboration with experts in academia and other entities, to identify areas that warrant a sustained commitment to build-up expertise or the development of new regular statistical outputs.” (Para 3.61)

Collaboration. “ONS was at times overly cautious when it came to sharing work-in-progress, testing new methods, and drawing on data from elsewhere. An important part of developing a space for collaboration with other users of economic statistics is to enable ONS to be more experimental in its approach. This requires a different approach to dissemination, which allows ONS to be open with the user community when discussing experimental approaches, initial results still subject to change and methods that will likely evolve as research progresses. One area that is clearly suited to such a collaborative approach is the application of data science.” (Para 3.71)

Surveys. Discussing the large number of surveys conducted by ONS (at considerable expense), the report says: “Given the ubiquity of electronic data today, it is incongruous that the production of ONS economic statistics still relies so heavily on the posting of paper forms and knocking on doors.” (Para 3.100)

Admin data failure. “The 2007 Statistics and Registration Service Act (SRSA) was in part designed to facilitate increased access to departmental administrative microdata in order to support statistical production. Yet just two microdata sets have been shared with ONS for the purpose of statistics production under the Act’s provisions.” (Para 3.105) Admin data can produce data with greater granularity and more cheaply. “Although there are legal and cultural obstacles, ONS also appears to have been slow to grasp the opportunities presented by administrative and other data.” (Para 3.110)

Technology. “Different statistical outputs are produced in isolation and the supporting IT systems are poorly interconnected. There are hundreds of applications, on 25 different platforms. Many of these are outdated or bespoke and costly to maintain. This complexity of the technology estate has impeded improvements to the core statistical and analytical functions.” (Para 3.123) “ONS must avoid repeating past mistakes, such as those that afflicted the Statistical Modernisation Programme, which sought to revolutionise the ONS technology estate in the 2000s.” (Para 3.126)

Data science. “The UKSA strategy ‘Better Statistics, Better Decisions’ explicitly recognises the need to build greater data science capability in ONS. However, only a few high-level actions were set out in its recent business plan, covering the period out to March 2018.” (Para 3.134) “In contrast, some private-sector custodians of big data have embraced the opportunities offered by data science far more enthusiastically. Bloomberg L.P., for instance, told the Review team that their employment of data scientists, coders and engineers was in the thousands.” (Para 1.135)

For discussion another time: Did UKSA know about all this? If not, why not? If they did, what did they do?

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s