Friday, July 31, 2015

Surveys in Crisis

In "Household Surveys in Crisis," Bruce D. Meyer, Wallace K.C. Mok, and James X. Sullivan describe household surveys as "one of the main innovations in social science research of the last century." Large, nationally representative household surveys are the source of official rates of unemployment, poverty, and health insurance coverage, and are used to allocate government funds. But the quality of survey data is declining on at least three counts.

The first and most commonly studied problem is the rise in unit nonresponse, meaning fewer people are willing to take a survey when asked. Two other growing problems are item nonresponse-- when someone agrees to take the survey but refuses to answer particular questions-- and inaccurate responses. Of course, the three problems can be related. For example, attempts to reduce unit nonresponse by persuading reluctant households to take a survey could raise item nonresponse and inaccurate responses if these reluctant participants rush through a survey they didn't really want to take in the first place.

Unit nonresponse, item nonresponse, and inaccurate responses would not be too troublesome if they were random enough that survey statistics were unbiased, but that is unlikely to be the case. Nonresponse and misreporting may be systematically correlated with relevant characteristics such as income or receipt of government funds. Meyer, Mok, and Sullivan look at survey data about government transfer programs for which corresponding administrative data is also available, so they can compare survey results to presumably more accurate administrative data. In this case, the survey data understates incomes at the bottom of the distribution, understates the rate of program receipt and the poverty reducing effects of government programs, and overstates measures of poverty and of inequality. For other surveys that cannot be linked to administrative data, it is difficult to say which direction biases will go.

Why has survey quality declined? The authors discuss many of the traditional explanations:
"Among the traditional reasons proposed include increasing urbanization, a decline in public spirit, increasing time pressure, rising crime (this pattern reversed long ago), increasing concerns about privacy and confidentiality, and declining cooperation due to 'over-surveyed' households (Groves and Couper 1998; Presser and McCullogh 2011; Brick and Williams 2013). The continuing increase in survey nonresponse as urbanization has slowed and crime has fallen make these less likely explanations for present trends. Tests of the remaining hypotheses are weak, based largely on national time-series analyses with a handful of observations. Several of the hypotheses require measuring societal conditions that can be difficult to capture: the degree of public spirit, concern about confidentiality, and time pressure...We are unaware of strong evidence to support or refute a steady decline in public spirit or a rise in confidentiality concerns as a cause for declines in survey quality."
They find it most likely that the sharp rise in the number of government surveys administered in the US since 1984 has resulted in declining cooperation by "over-surveyed" households. "We suspect that talking with an interviewer, which once was a rare chance to tell someone about your life, now is crowded out by an annoying press of telemarketers and commercial surveyors."

Personally, I have not received any requests to participate in government surveys and rarely receive commercial survey requests. Is this just because I moved around so much as a student? Am I about to be flooded with requests? I think I would actually find it fun to take some surveys after working with the data so much. Please leave a comment about your experience with taking (or declining to take) surveys.

The authors also note that since there is a trend toward greater leisure time, it is unlikely that increased time pressure is resulting in declining survey quality. However, while people have more leisure time, they may also have more things to do with their leisure time (I'm looking at you, Internet) that they prefer to taking surveys. Intuitively I would guess that as people have grown more accustomed to doing everything online, they are less comfortable talking to an interviewer in person or on the phone. Since I almost never have occasion to go to the post office, I can imagine forgetting to mail in a paper survey. Switching surveys to online format could result in a new set of biases, but may eventually be the way to go.

I would also guess that the Internet has changed people's relationship with information, even information about themselves. When you can look up anything easily, that can change what you decide to remember and what facts you feel comfortable reporting off the top of your head to an interviewer.


  1. They find it most likely that the sharp rise in the number of government surveys administered in the US since 1984 has resulted in declining cooperation by "over-surveyed" households. "We suspect that talking with an interviewer, which once was a rare chance to tell someone about your life, now is crowded out by an annoying press of telemarketers and commercial surveyors."

    [Spot on! I hang up on everyone nowadays. The do not call list has too many loopholes. We are targeted so much probably because our household income is in the 92%-ile regionally and probably near the apex in our local community.]

    1. OK, I hang up on everyone but my in-laws and doctors and other service firms that I have business with. Robo-calls were a big turn-off. But I don't talk to any solicitors on that call regardless of what they are offering. Surveys are mostly about targeting me not responding to me.

  2. I tend to think of this particularly in relation to the Current Population Survey (I'm a retired, 67-year-old labor economist). When I taught labor economics, and when I taught intro to macro, I always asked whether anyone in the class had ever been a participant in the CPS. In 40 years, I never had a student who had been in the CPS )or would admit to it, or knew that someone in the student's household was responding for the household). By the end, this struck me as pretty amazing. (I have also never been in the CPS.) I have never been asked to participate in any other government surveys.

    For the rest...I almost never (maybe once or twice a year) receive calls for private surveys. I almost always decline to participate in political surveys, and almost always participate in marketing surveys. I have fairly often cut the interview off if I felt that the questioning had become inappropriate.

  3. .We receive so many political and other fund raising calls that we do not answer any calls with a caller ID we do not recognize ( and even some we do recognize!) So no chance to take any kind of survey.

  4. You folks really receive few survey requests?

    I get at *least* three a week (None from the government, though.) Even though I'm on the Do Not Call list.

    That may be because I still have a landline, though. Polling firms and telemarketers are not supposed to call cell phones (though some do anyway). I'd get rid of the landline, but the cell reception in my apartment is almost nonexistent.

  5. I've solicited for the ACS once and had my mom also solicited for the ACS. Since the Census Bureau doesn't translate the ACS into anything near the number of languages they translate the decennial census in, that means I got the task of filling out the form for her (after getting it mailed a second time), which I did based on my knowledge of the household and guessing.

    When I filled out the ACS, I just took whatever number popped in my head at the time and mailed it in, and did not do any data control. Basically I filled out during the commercial breaks when I was watching TV, and I suspect most people do that and do not attempt to dig out their electric bill or property tax statement. Some of the questions, like employment, are asked too many times, and I remember putting down "paper pusher" to describe the tasks that I do, since there is literally no way to describe all that I do in the small space provided, especially since the plurality might be 20% or so. The same goes for "usual" travel mode and time going to work since those times fluctuate during a week. According to the Census Bureau, they claim that measurement error is minimal - - and I guess it averages out at the end, but how do they even know that?

  6. Carola, interesting post.
    We have have heard the same thing anecdotally, good to see some research into this. In the UK there has been widespread dis-satisfaction with polling companies and their results after our recent election in May 2015. Despite endless daily polling and very carefully constructed nationally representative panels no one really called the correct result. That is, until the exit poll on the night after polling finished. The exit poll had the benefit of being a simple "Who did you vote for?" kind of survey and had a much larger sample size (22,000) than the polling companies had been relying on (~1,000).
    We have the impression that the same applies to more generic household surveys too.
    We experimenting by building a polling mobile app to see whether this method of surveying households can reduce the non-response rate. Using the app also allows us to do much larger sample size surveys at a reasonable cost.
    Initial results are showing a dramatically high response rate (people seem to respond almost instantly to notifications on their cell phones and tablets). We are trying to reduce the item non-response rate by only doing micro surveys that are one question long. Further demographic information can be sought after the responder has actually answered the key question in the survey. Responses can be encouraged by offering a small reward or voucher to the responder for their opinions.
    It's early days, but it looks like a promising alternative (or addition) to the traditional paper-based / phone-based surveys.

  7. This is very informative and interesting to have. The quantitative data analysis is now a days becoming popular among youngsters because it helps a lot.

  8. Valuable commentary . Incidentally , others are interested in a USCIS I-485 , my colleagues filled out a fillable document here


Comments appreciated!