Response rates to surveys are declining. Fast. New sources, such as big data or administrative data, or making survey response compulsory, are no longer an optional extra. They need to be discussed now as a way to boost the quality of and restore credibility to the nation’s key figures. The rise in employment and the fall in earnings defined the economy of the last Parliament. As survey response rates fall below 50%, there is a chance that those trends which never gelled to give a coherent picture are some way from reality.
It’s a few months old now but the ONS response to my Freedom of Information request earlier this year shed some interesting light on declining survey response rates. For many the rates were lower than might be expected – only three of the thirteen surveys noted had response rates over 65%. They were also declining – nine had lower response rates than the year before. The table below shows the rates using data that was retrieved by ONS in December 2014.
This would seem to be a long-term issue reflecting all the major surveys from the ONS as the trend is unambiguously down.
Perhaps one of the most shocking declines in response rates is the Labour Force Survey. It is the biggest and perhaps most important survey that the ONS conducts. Labour market data – unemployment and earnings for example – are key to the setting of economic policy. The chart below shows the decline: in twenty years the response has fallen from 79% to 49%. The refusal rate has gone from 15% to 35% and the proportion of chosen respondents who could not be contacted has risen from 4% to 11%. The UK’s refusal rate is the highest in the EU (after Luxembourg).
The 1993 rate was perhaps OK but, when fewer than half of those approached respond, it is time to stop and reflect. We used to think in terms of surveys having trouble getting to the “hard to reach” groups. Forget the minorities, the surveys are quite probably not even representing the majority!
The decline in response rate has been steady and over many years. There has not been – and is unlikely to be – a moment when the response will suddenly cease to be adequate. As one of the documents in the FoI says: “Response rate is the single value often used to measure success in data collection, but representativeness is equally important. Falling response rates increase the potential for non-response bias, but do not necessarily mean that non-response bias has become more of a problem.”
This might well be a recipe for inaction – the system never breaks it just gets less accurate and slowly loses support. The risk is acknowledged among government statisticians. In February 2014, ONS published a National Statistics Quality Review of the Labour Force Survey, which concluded that if response rates were to continue to decline this would raise concern for both accuracy and the precision of the survey estimates. It identified several proposals to address response rate concerns covering IT, survey design, capability and engagement with respondents. Early days but it seems that they have not yet had an impact.
The ONS might be forced to consider making survey response compulsory. A briefing released as part of the FoI request gave some figures for other countries. It said that, “of the 33 member states of the EU, EFTA and EU candidate countries, participation in the LFS is compulsory in 14”, adding that the average reported response is higher in those countries, at 85%. ONS has shied away from this route until now, saying that: “the implications of this have previously been explored and the steps that would need to be taken discussed with the Treasury Solicitors Department.” But can that resistance to change last for long?
There is even a risk to ONS revenues if this issue is not addressed – falling response rates affect ONS capacity to win new data collection work or to maintain its current portfolio of externally funded surveys.
Finally, presenting statistics with sampling errors and ranges is not the answer. The sampling errors will get ever wider as sample size falls and in any case they do not account for bias which is more than likely increasing as response rates fall. The ONS has done some thoughtful work on this but it does not deal with the source of the problem.
We need admin and big data, plus compulsory response, as part of the plan to get the statistics back on course.