Friday, July 31, 2015

Surveys in Crisis

In "Household Surveys in Crisis," Bruce D. Meyer, Wallace K.C. Mok, and James X. Sullivan describe household surveys as "one of the main innovations in social science research of the last century." Large, nationally representative household surveys are the source of official rates of unemployment, poverty, and health insurance coverage, and are used to allocate government funds. But the quality of survey data is declining on at least three counts.

The first and most commonly studied problem is the rise in unit nonresponse, meaning fewer people are willing to take a survey when asked. Two other growing problems are item nonresponse-- when someone agrees to take the survey but refuses to answer particular questions-- and inaccurate responses. Of course, the three problems can be related. For example, attempts to reduce unit nonresponse by persuading reluctant households to take a survey could raise item nonresponse and inaccurate responses if these reluctant participants rush through a survey they didn't really want to take in the first place.

Unit nonresponse, item nonresponse, and inaccurate responses would not be too troublesome if they were random enough that survey statistics were unbiased, but that is unlikely to be the case. Nonresponse and misreporting may be systematically correlated with relevant characteristics such as income or receipt of government funds. Meyer, Mok, and Sullivan look at survey data about government transfer programs for which corresponding administrative data is also available, so they can compare survey results to presumably more accurate administrative data. In this case, the survey data understates incomes at the bottom of the distribution, understates the rate of program receipt and the poverty reducing effects of government programs, and overstates measures of poverty and of inequality. For other surveys that cannot be linked to administrative data, it is difficult to say which direction biases will go.

Why has survey quality declined? The authors discuss many of the traditional explanations:
"Among the traditional reasons proposed include increasing urbanization, a decline in public spirit, increasing time pressure, rising crime (this pattern reversed long ago), increasing concerns about privacy and confidentiality, and declining cooperation due to 'over-surveyed' households (Groves and Couper 1998; Presser and McCullogh 2011; Brick and Williams 2013). The continuing increase in survey nonresponse as urbanization has slowed and crime has fallen make these less likely explanations for present trends. Tests of the remaining hypotheses are weak, based largely on national time-series analyses with a handful of observations. Several of the hypotheses require measuring societal conditions that can be difficult to capture: the degree of public spirit, concern about confidentiality, and time pressure...We are unaware of strong evidence to support or refute a steady decline in public spirit or a rise in confidentiality concerns as a cause for declines in survey quality."
They find it most likely that the sharp rise in the number of government surveys administered in the US since 1984 has resulted in declining cooperation by "over-surveyed" households. "We suspect that talking with an interviewer, which once was a rare chance to tell someone about your life, now is crowded out by an annoying press of telemarketers and commercial surveyors."

Personally, I have not received any requests to participate in government surveys and rarely receive commercial survey requests. Is this just because I moved around so much as a student? Am I about to be flooded with requests? I think I would actually find it fun to take some surveys after working with the data so much. Please leave a comment about your experience with taking (or declining to take) surveys.

The authors also note that since there is a trend toward greater leisure time, it is unlikely that increased time pressure is resulting in declining survey quality. However, while people have more leisure time, they may also have more things to do with their leisure time (I'm looking at you, Internet) that they prefer to taking surveys. Intuitively I would guess that as people have grown more accustomed to doing everything online, they are less comfortable talking to an interviewer in person or on the phone. Since I almost never have occasion to go to the post office, I can imagine forgetting to mail in a paper survey. Switching surveys to online format could result in a new set of biases, but may eventually be the way to go.

I would also guess that the Internet has changed people's relationship with information, even information about themselves. When you can look up anything easily, that can change what you decide to remember and what facts you feel comfortable reporting off the top of your head to an interviewer.

49 comments:

  1. They find it most likely that the sharp rise in the number of government surveys administered in the US since 1984 has resulted in declining cooperation by "over-surveyed" households. "We suspect that talking with an interviewer, which once was a rare chance to tell someone about your life, now is crowded out by an annoying press of telemarketers and commercial surveyors."

    [Spot on! I hang up on everyone nowadays. The do not call list has too many loopholes. We are targeted so much probably because our household income is in the 92%-ile regionally and probably near the apex in our local community.]

    ReplyDelete
    Replies
    1. OK, I hang up on everyone but my in-laws and doctors and other service firms that I have business with. Robo-calls were a big turn-off. But I don't talk to any solicitors on that call regardless of what they are offering. Surveys are mostly about targeting me not responding to me.

      Delete
  2. I tend to think of this particularly in relation to the Current Population Survey (I'm a retired, 67-year-old labor economist). When I taught labor economics, and when I taught intro to macro, I always asked whether anyone in the class had ever been a participant in the CPS. In 40 years, I never had a student who had been in the CPS )or would admit to it, or knew that someone in the student's household was responding for the household). By the end, this struck me as pretty amazing. (I have also never been in the CPS.) I have never been asked to participate in any other government surveys.

    For the rest...I almost never (maybe once or twice a year) receive calls for private surveys. I almost always decline to participate in political surveys, and almost always participate in marketing surveys. I have fairly often cut the interview off if I felt that the questioning had become inappropriate.

    ReplyDelete
  3. .We receive so many political and other fund raising calls that we do not answer any calls with a caller ID we do not recognize ( and even some we do recognize!) So no chance to take any kind of survey.

    ReplyDelete
  4. You folks really receive few survey requests?

    I get at *least* three a week (None from the government, though.) Even though I'm on the Do Not Call list.

    That may be because I still have a landline, though. Polling firms and telemarketers are not supposed to call cell phones (though some do anyway). I'd get rid of the landline, but the cell reception in my apartment is almost nonexistent.

    ReplyDelete
  5. I've solicited for the ACS once and had my mom also solicited for the ACS. Since the Census Bureau doesn't translate the ACS into anything near the number of languages they translate the decennial census in, that means I got the task of filling out the form for her (after getting it mailed a second time), which I did based on my knowledge of the household and guessing.

    When I filled out the ACS, I just took whatever number popped in my head at the time and mailed it in, and did not do any data control. Basically I filled out during the commercial breaks when I was watching TV, and I suspect most people do that and do not attempt to dig out their electric bill or property tax statement. Some of the questions, like employment, are asked too many times, and I remember putting down "paper pusher" to describe the tasks that I do, since there is literally no way to describe all that I do in the small space provided, especially since the plurality might be 20% or so. The same goes for "usual" travel mode and time going to work since those times fluctuate during a week. According to the Census Bureau, they claim that measurement error is minimal - http://www2.census.gov/programs-surveys/acs/methodology/design_and_methodology/acs_design_methodology_ch15_2014.pdf - and I guess it averages out at the end, but how do they even know that?

    ReplyDelete
  6. Carola, interesting post.
    We have have heard the same thing anecdotally, good to see some research into this. In the UK there has been widespread dis-satisfaction with polling companies and their results after our recent election in May 2015. Despite endless daily polling and very carefully constructed nationally representative panels no one really called the correct result. That is, until the exit poll on the night after polling finished. The exit poll had the benefit of being a simple "Who did you vote for?" kind of survey and had a much larger sample size (22,000) than the polling companies had been relying on (~1,000).
    We have the impression that the same applies to more generic household surveys too.
    We experimenting by building a polling mobile app to see whether this method of surveying households can reduce the non-response rate. Using the app also allows us to do much larger sample size surveys at a reasonable cost.
    Initial results are showing a dramatically high response rate (people seem to respond almost instantly to notifications on their cell phones and tablets). We are trying to reduce the item non-response rate by only doing micro surveys that are one question long. Further demographic information can be sought after the responder has actually answered the key question in the survey. Responses can be encouraged by offering a small reward or voucher to the responder for their opinions.
    It's early days, but it looks like a promising alternative (or addition) to the traditional paper-based / phone-based surveys.

    ReplyDelete
  7. This is very informative and interesting to have. The quantitative data analysis is now a days becoming popular among youngsters because it helps a lot.

    ReplyDelete
  8. Valuable commentary . Incidentally , others are interested in a USCIS I-485 , my colleagues filled out a fillable document here http://pdf.ac/163NiK.

    ReplyDelete
  9. We are prepared to deliver reliable and trusted Custom Research Paper Writing Services in all the outlined levels of study due to the many experts that we have in our company.Our Affordable Writing Services are affordable and cheap as well. Students have very limited cash flows, and their budgets are usually constrained.

    ReplyDelete
  10. The federal system ensures that the commoners are heard as it is easier because it is nearer the people as compared to a centralized government. US constitution essay can inform more about this issue.

    ReplyDelete
  11. Are you looking for Unmatched Cheap Dissertation Writing Services? Composing Nursing Writing Services is never simple, not even for the most knowledgeable or experienced learner is able to complete quality Cheap Term Papers for Sale.

    ReplyDelete
  12. Are you a student and searching for Write My Research Paper Online Services? Do you wish to acquire high-quality services from Legitimate Custom Research Paper Services? As Custom Research Writing Services can always bring a headache to scholars.

    ReplyDelete
  13. [URL="https://www.goappx.com/"]clone scripts[/URL]

    ReplyDelete
  14. Build an on-demand taxi-hailing app like UBER with Goappx’s Uber clone script. Uber Clone Script

    ReplyDelete
  15. This comment has been removed by the author.

    ReplyDelete
  16. Once in a while printer mistakes become bothering to us since we need a critical printout and our printer isn't working. In the event that you are getting Canon Printer In Error State issue over and over then contact specialists to get lasting answer for this issue.

    ReplyDelete
  17. How to Solve canon printer not printing properly? Try not to Be Panic, Here Are The Best approaches to tackle this question, read more to fix this mistake.

    ReplyDelete
  18. http://www.igetsolutions.in/packers-and-movers-ambala
    http://www.igetsolutions.in/packers-and-movers-bhubaneswar
    http://www.igetsolutions.in/packers-and-movers-kanpur
    http://www.igetsolutions.in/packers-and-movers-jodhpur
    http://www.igetsolutions.in/packers-and-movers-noida

    ReplyDelete
  19. Hi Thank you for sharing this wonderful article. Do well to visit Latest School News to apply for Jamb Registration Form. You can always check your Jamb Result and Neco Result Checker here.
    Do well to check your Admission Status using Jamb Caps. Check Best University that offer Medicine and Surgery

    ReplyDelete
  20. are you using outlook email and sometimes facing issue with it? Then we you should follow the steps to fix outlook email not working.

    ReplyDelete
  21. Growing number of mobile app companies tapping the proximity solutions market, along with increasing investments in the beacon technology, is expected to propel the growth of the bluetooth beacons market globally.

    ReplyDelete
  22. If you are using Malware bytes in your laptop, you can relax and breathe easy. It is because of the efficacy of this antimalware program that makes it even more popular than some of the established antivirus products.

    ReplyDelete
  23. Do you need Finance? Are you looking for Finance? Are you looking for finance to enlarge your business? We help individuals and companies to obtain finance for business expanding and to setup a new business ranging any amount. Get finance at affordable interest rate of 3%, Do you need this finance for business and to clear your bills? Then send us an email now for more information contact us now via (financialserviceoffer876@gmail.com) whats-App +918929509036 Dr James Eric Finance Pvt Ltd Thanks

    ReplyDelete
  24. Hallo, teman-teman semua nya kami dari agen indowins ingin menawarkan berbagai jenis permainan dan bonus. whatshapp : +85587536868
    judi online

    ReplyDelete
  25. The growing popularity of Bluetooth speakers among the millennial population coupled with high product compatibility with other consumer electronic devices, has propelled the Bluetooth Speaker market growth.

    ReplyDelete
  26. Amazon.com/mytv is a registration portal of Amazon Prime from which members can watch thousands of movies and TV shows for free from their device some features require a premium subscription. The instant video service can only be watched after registering your device with a simple subscription code. The website for registering on Amazon prime is www.amazon.com/mytv. Amazon Prime Video is one of the most amazing and interesting platforms to watch. With Amazon Prime you can stream unlimited movies and TV episodes on Amazon prime video app.

    Amazon.com/mytv
    www.amazon.com/mytv

    ReplyDelete
  27. Tubi.tv is likewise accessible on an assortment of devices like Amazon Fire TV, ROKU, Playstations, and Smart TV or you can straightforwardly get to it through any of your internet browsers. As you are wondering that you have perused so a lot yet you knew nothing about how to get tubi.tv/activate then don’t you stress at all reason directly underneath we have gone to your guide with the Tubi Tv Activation measure.

    tubi.tv/activate
    tubi tv activate lg smart tv


    ReplyDelete
  28. Now you can Easily Activate Amazon by Amazon.com/mytv. It is a registration portal of Amazon Prime from which members can watch thousands of movies and TV shows for free from their device some features require a premium subscription. The instant video service can only be watched after registering your device with a simple subscription code. The website for registering on Amazon prime is amazon.com/mytv. Amazon Prime Video is one of the most amazing and interesting platforms to watch. With Amazon Prime you can stream unlimited movies and TV episodes on Amazon prime video app. You can Easily Fix Troubleshoot by visiting here.

    amazon com mytv

    ReplyDelete
  29. Activate amazon by visiting our website. Amazon Prime Video allows you to watch on-demand movies and TV shows via an app that you must install on your Samsung TV . You will also get original films, television series, and other entertaining content. To begin with any use of the advantages that come with your www.amazon.com/mytv nomination, it is important that you create a record on a similar platform. However, customers are likewise presented with the option to create a record in the hours of actuating from their membership. Visit here amazon.com/mytv

    www.amazon.com/mytv
    amazon.com/mytv

    ReplyDelete
  30. It is an Application It is very useful for clients in installing all the software and guide updates. At whatever point new updates are accessible, it gives you warnings. Clients can save their ideal guides and areas with Garmin Express and can see the information on the navigation device. Thus, garmin.com/express guarantees fast and exact navigation. This Software gives you Warnings, Offers and item Support. For Setup Gramin Express you need to Visit our website.

    garmin.com/express


    ReplyDelete
  31. Lahore Smart City is going to be the best choice for commercial, investment and residential point of view. The scheme will have everything to attract national and international investors. In return, investors will get high revenue. On the other hand, the housing society is equipped with state of the art facilities. The facilities are just dream of come true for the people of Lahore. Peace, safety and eco-friendly behaviors. Future Development Holding hires and corporate’s with world-class developers, architectures and planners. This Smart City Lahore will have golf clubs and fields designed by experienced and world-recognized designers.

    ReplyDelete
  32. Nice post, you give the clear idea of the topic and I subscribed your blog, I have some links to share here

    buy modalert online

    buy artvigil online

    buy waklert online

    buy modvigil online

    buy vilafinil online

    ReplyDelete
  33. amazon.com/mytv Most importantly, switch on your Smart TV. Then visit amazon.com/mytv enter code Simple process to activate your Amazon prime video on your smart device or phone. Enter code mytv and click on the enter button. Then you see your Amazon prime account activate and you can easily watch your web shows, movies and videos. This purpose of this is to provide information about prime video activation. Users who are facing any problem to activate prime video in their smart TV, can read this and Visit our website.

    amazon.com/mytv
    www.amazon.com/mytv

    ReplyDelete
  34. I am Alice Addy, a Corporate Sales Specialist operating in the Latin American region. We are experts in Market Research (Mercado Latinoamericano de Trifluoruro de Cloro). We provide market reports across various industry verticals and also help with procurement and supply chain intelligence. By giving you better insights, we help you make informed business decisions.

    ReplyDelete
  35. I am Alice Addy, a Corporate Sales Specialist operating in the Latin American region. We are experts in Market Research (Mercado Mexicano de LED). We provide market reports across various industry verticals and also help with procurement and supply chain intelligence. By giving you better insights, we help you make informed business decisions.

    ReplyDelete
  36. The global Polydextrose Market size was valued at USD 225 million in 2015 and is anticipated to grow at a CAGR of 5.6% over the forecast period. Increasing health awareness and growing consumption of functional and nutritional foods is expected to drive the market growth. It promotes the usage in weight management, fat reduction, rising satiety, and prebiotics.

    ReplyDelete
  37. Expert Market Research is a leading business intelligence firm, providing custom and syndicated market reports along with consultancy services for our clients. We serve a wide client base ranging from Fortune 1000 companies to small and medium enterprises. Our reports cover over 100 industries across established and emerging markets researched by our skilled analysts who track the latest economic, demographic, trade and market data globally.

    ReplyDelete
  38. Global Non-Gmo Seeds Market: Non-GMO seeds are being used by farmers in increasing numbers as they are able to save money when growing the size of their yields by using this product. Due to their lower cost and more sustainable produce, consumers are preferring cleaner and better-produced food products with non-GMO Project Certified certifications over GMO food products.

    ReplyDelete

Comments appreciated!