Survey response rates – look down!

Response rates to surveys are declining. Fast. New sources, such as big data or administrative data, or making survey response compulsory, are no longer an optional extra. They need to be discussed now as a way to boost the quality of and restore credibility to the nation’s key figures. The rise in employment and the fall in earnings defined the economy of the last Parliament. As survey response rates fall below 50%, there is a chance that those trends which never gelled to give a coherent picture are some way from reality.

It’s a few months old now but the ONS response to my Freedom of Information request earlier this year shed some interesting light on declining survey response rates. For many the rates were lower than might be expected – only three of the thirteen surveys noted had response rates over 65%. They were also declining – nine had lower response rates than the year before. The table below shows the rates using data that was retrieved by ONS in December 2014.

Screen Shot 2015-11-16 at 12.52.59

This would seem to be a long-term issue reflecting all the major surveys from the ONS as the trend is unambiguously down.

Screen Shot 2015-11-16 at 13.53.01

Perhaps one of the most shocking declines in response rates is the Labour Force Survey. It is the biggest and perhaps most important survey that the ONS conducts. Labour market data – unemployment and earnings for example – are key to the setting of economic policy. The chart below shows the decline: in twenty years the response has fallen from 79% to 49%. The refusal rate has gone from 15% to 35% and the proportion of chosen respondents who could not be contacted has risen from 4% to 11%. The UK’s refusal rate is the highest in the EU (after Luxembourg).

The 1993 rate was perhaps OK but, when fewer than half of those approached respond, it is time to stop and reflect. We used to think in terms of surveys having trouble getting to the “hard to reach” groups. Forget the minorities, the surveys are quite probably not even representing the majority!

Screen Shot 2015-11-16 at 13.16.05

The decline in response rate has been steady and over many years. There has not been – and is unlikely to be – a moment when the response will suddenly cease to be adequate. As one of the documents in the FoI says: “Response rate is the single value often used to measure success in data collection, but representativeness is equally important. Falling response rates increase the potential for non-response bias, but do not necessarily mean that non-response bias has become more of a problem.”

This might well be a recipe for inaction – the system never breaks it just gets less accurate and slowly loses support. The risk is acknowledged among government statisticians. In February 2014, ONS published a National Statistics Quality Review of the Labour Force Survey, which concluded that if response rates were to continue to decline this would raise concern for both accuracy and the precision of the survey estimates. It identified several proposals to address response rate concerns covering IT, survey design, capability and engagement with respondents. Early days but it seems that they have not yet had an impact.

The ONS might be forced to consider making survey response compulsory. A briefing released as part of the FoI request gave some figures for other countries. It said that, “of the 33 member states of the EU, EFTA and EU candidate countries, participation in the LFS is compulsory in 14”, adding that the average reported response is higher in those countries, at 85%. ONS has shied away from this route until now, saying that: “the implications of this have previously been explored and the steps that would need to be taken discussed with the Treasury Solicitors Department.” But can that resistance to change last for long?

There is even a risk to ONS revenues if this issue is not addressed – falling response rates affect ONS capacity to win new data collection work or to maintain its current portfolio of externally funded surveys.

Finally, presenting statistics with sampling errors and ranges is not the answer. The sampling errors will get ever wider as sample size falls and in any case they do not account for bias which is more than likely increasing as response rates fall. The ONS has done some thoughtful work on this but it does not deal with the source of the problem.

We need admin and big data, plus compulsory response, as part of the plan to get the statistics back on course.

Advertisements

4 thoughts on “Survey response rates – look down!

  1. Tragically, I can spot the flaw in your quoted table as the month you have chosen to illustrate for LFS is December. December is the lowest response rate of any month in the survey year due to factors such as Christmas break, where people either go to stay with loved ones or go on holiday, also weather conditions such as deep snow etc can affect response. This is not typical.

    Like

  2. I am involved in the collection of the data being discussed. One of the biggest obstacles we face in data collection is that the general public are, in a lot of cases, almost completely unaware of who we are, what we do and how important their help is. The Office for National Statistics are mentioned almost daily in the media. Highlighting the fact that the statistics are gained from the ‘Labour Force Survey’, for example, would go a long way to raise our profile and the awareness of what we do. With the huge rise in mobile phone use, phone calls can be instantly rejected. It is extremely difficult to gain cooperation from the public when you can’t even get to speak to them. It is often assumed that we are ‘PPI’ or ‘Compensation Claim’ businesses. The public are very aware of the increase in cyber crime and rightly so but again it is an obstacle we have to overcome when trying to gain cooperation. Making our surveys compulsory is definitely something that needs to be considered when it comes to increasing response rates.

    Like

    1. They seem very good points. I can’t help feeling anything but positive about the Bean process. It leaves difficult questions to be answered but they need to be answered. It doesn’t look as if senior management took these issues seriously for some time but will now have to – and we’ll all win.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s