Friday, July 31, 2015

Surveys in Crisis

In "Household Surveys in Crisis," Bruce D. Meyer, Wallace K.C. Mok, and James X. Sullivan describe household surveys as "one of the main innovations in social science research of the last century." Large, nationally representative household surveys are the source of official rates of unemployment, poverty, and health insurance coverage, and are used to allocate government funds. But the quality of survey data is declining on at least three counts.

The first and most commonly studied problem is the rise in unit nonresponse, meaning fewer people are willing to take a survey when asked. Two other growing problems are item nonresponse-- when someone agrees to take the survey but refuses to answer particular questions-- and inaccurate responses. Of course, the three problems can be related. For example, attempts to reduce unit nonresponse by persuading reluctant households to take a survey could raise item nonresponse and inaccurate responses if these reluctant participants rush through a survey they didn't really want to take in the first place.

Unit nonresponse, item nonresponse, and inaccurate responses would not be too troublesome if they were random enough that survey statistics were unbiased, but that is unlikely to be the case. Nonresponse and misreporting may be systematically correlated with relevant characteristics such as income or receipt of government funds. Meyer, Mok, and Sullivan look at survey data about government transfer programs for which corresponding administrative data is also available, so they can compare survey results to presumably more accurate administrative data. In this case, the survey data understates incomes at the bottom of the distribution, understates the rate of program receipt and the poverty reducing effects of government programs, and overstates measures of poverty and of inequality. For other surveys that cannot be linked to administrative data, it is difficult to say which direction biases will go.

Why has survey quality declined? The authors discuss many of the traditional explanations:
"Among the traditional reasons proposed include increasing urbanization, a decline in public spirit, increasing time pressure, rising crime (this pattern reversed long ago), increasing concerns about privacy and confidentiality, and declining cooperation due to 'over-surveyed' households (Groves and Couper 1998; Presser and McCullogh 2011; Brick and Williams 2013). The continuing increase in survey nonresponse as urbanization has slowed and crime has fallen make these less likely explanations for present trends. Tests of the remaining hypotheses are weak, based largely on national time-series analyses with a handful of observations. Several of the hypotheses require measuring societal conditions that can be difficult to capture: the degree of public spirit, concern about confidentiality, and time pressure...We are unaware of strong evidence to support or refute a steady decline in public spirit or a rise in confidentiality concerns as a cause for declines in survey quality."
They find it most likely that the sharp rise in the number of government surveys administered in the US since 1984 has resulted in declining cooperation by "over-surveyed" households. "We suspect that talking with an interviewer, which once was a rare chance to tell someone about your life, now is crowded out by an annoying press of telemarketers and commercial surveyors."

Personally, I have not received any requests to participate in government surveys and rarely receive commercial survey requests. Is this just because I moved around so much as a student? Am I about to be flooded with requests? I think I would actually find it fun to take some surveys after working with the data so much. Please leave a comment about your experience with taking (or declining to take) surveys.

The authors also note that since there is a trend toward greater leisure time, it is unlikely that increased time pressure is resulting in declining survey quality. However, while people have more leisure time, they may also have more things to do with their leisure time (I'm looking at you, Internet) that they prefer to taking surveys. Intuitively I would guess that as people have grown more accustomed to doing everything online, they are less comfortable talking to an interviewer in person or on the phone. Since I almost never have occasion to go to the post office, I can imagine forgetting to mail in a paper survey. Switching surveys to online format could result in a new set of biases, but may eventually be the way to go.

I would also guess that the Internet has changed people's relationship with information, even information about themselves. When you can look up anything easily, that can change what you decide to remember and what facts you feel comfortable reporting off the top of your head to an interviewer.

63 comments:

  1. They find it most likely that the sharp rise in the number of government surveys administered in the US since 1984 has resulted in declining cooperation by "over-surveyed" households. "We suspect that talking with an interviewer, which once was a rare chance to tell someone about your life, now is crowded out by an annoying press of telemarketers and commercial surveyors."

    [Spot on! I hang up on everyone nowadays. The do not call list has too many loopholes. We are targeted so much probably because our household income is in the 92%-ile regionally and probably near the apex in our local community.]

    ReplyDelete
    Replies
    1. OK, I hang up on everyone but my in-laws and doctors and other service firms that I have business with. Robo-calls were a big turn-off. But I don't talk to any solicitors on that call regardless of what they are offering. Surveys are mostly about targeting me not responding to me.

      Delete
  2. I tend to think of this particularly in relation to the Current Population Survey (I'm a retired, 67-year-old labor economist). When I taught labor economics, and when I taught intro to macro, I always asked whether anyone in the class had ever been a participant in the CPS. In 40 years, I never had a student who had been in the CPS )or would admit to it, or knew that someone in the student's household was responding for the household). By the end, this struck me as pretty amazing. (I have also never been in the CPS.) I have never been asked to participate in any other government surveys.

    For the rest...I almost never (maybe once or twice a year) receive calls for private surveys. I almost always decline to participate in political surveys, and almost always participate in marketing surveys. I have fairly often cut the interview off if I felt that the questioning had become inappropriate.

    ReplyDelete
  3. .We receive so many political and other fund raising calls that we do not answer any calls with a caller ID we do not recognize ( and even some we do recognize!) So no chance to take any kind of survey.

    ReplyDelete
  4. You folks really receive few survey requests?

    I get at *least* three a week (None from the government, though.) Even though I'm on the Do Not Call list.

    That may be because I still have a landline, though. Polling firms and telemarketers are not supposed to call cell phones (though some do anyway). I'd get rid of the landline, but the cell reception in my apartment is almost nonexistent.

    ReplyDelete
  5. I've solicited for the ACS once and had my mom also solicited for the ACS. Since the Census Bureau doesn't translate the ACS into anything near the number of languages they translate the decennial census in, that means I got the task of filling out the form for her (after getting it mailed a second time), which I did based on my knowledge of the household and guessing.

    When I filled out the ACS, I just took whatever number popped in my head at the time and mailed it in, and did not do any data control. Basically I filled out during the commercial breaks when I was watching TV, and I suspect most people do that and do not attempt to dig out their electric bill or property tax statement. Some of the questions, like employment, are asked too many times, and I remember putting down "paper pusher" to describe the tasks that I do, since there is literally no way to describe all that I do in the small space provided, especially since the plurality might be 20% or so. The same goes for "usual" travel mode and time going to work since those times fluctuate during a week. According to the Census Bureau, they claim that measurement error is minimal - http://www2.census.gov/programs-surveys/acs/methodology/design_and_methodology/acs_design_methodology_ch15_2014.pdf - and I guess it averages out at the end, but how do they even know that?

    ReplyDelete
  6. Carola, interesting post.
    We have have heard the same thing anecdotally, good to see some research into this. In the UK there has been widespread dis-satisfaction with polling companies and their results after our recent election in May 2015. Despite endless daily polling and very carefully constructed nationally representative panels no one really called the correct result. That is, until the exit poll on the night after polling finished. The exit poll had the benefit of being a simple "Who did you vote for?" kind of survey and had a much larger sample size (22,000) than the polling companies had been relying on (~1,000).
    We have the impression that the same applies to more generic household surveys too.
    We experimenting by building a polling mobile app to see whether this method of surveying households can reduce the non-response rate. Using the app also allows us to do much larger sample size surveys at a reasonable cost.
    Initial results are showing a dramatically high response rate (people seem to respond almost instantly to notifications on their cell phones and tablets). We are trying to reduce the item non-response rate by only doing micro surveys that are one question long. Further demographic information can be sought after the responder has actually answered the key question in the survey. Responses can be encouraged by offering a small reward or voucher to the responder for their opinions.
    It's early days, but it looks like a promising alternative (or addition) to the traditional paper-based / phone-based surveys.

    ReplyDelete
  7. This is very informative and interesting to have. The quantitative data analysis is now a days becoming popular among youngsters because it helps a lot.

    ReplyDelete
  8. Valuable commentary . Incidentally , others are interested in a USCIS I-485 , my colleagues filled out a fillable document here http://pdf.ac/163NiK.

    ReplyDelete
  9. The federal system ensures that the commoners are heard as it is easier because it is nearer the people as compared to a centralized government. US constitution essay can inform more about this issue.

    ReplyDelete
  10. [URL="https://www.goappx.com/"]clone scripts[/URL]

    ReplyDelete
  11. Build an on-demand taxi-hailing app like UBER with Goappx’s Uber clone script. Uber Clone Script

    ReplyDelete
  12. This comment has been removed by the author.

    ReplyDelete
  13. How to Solve canon printer not printing properly? Try not to Be Panic, Here Are The Best approaches to tackle this question, read more to fix this mistake.

    ReplyDelete
  14. Growing number of mobile app companies tapping the proximity solutions market, along with increasing investments in the beacon technology, is expected to propel the growth of the bluetooth beacons market globally.

    ReplyDelete
  15. If you are using Malware bytes in your laptop, you can relax and breathe easy. It is because of the efficacy of this antimalware program that makes it even more popular than some of the established antivirus products.

    ReplyDelete
  16. Do you need Finance? Are you looking for Finance? Are you looking for finance to enlarge your business? We help individuals and companies to obtain finance for business expanding and to setup a new business ranging any amount. Get finance at affordable interest rate of 3%, Do you need this finance for business and to clear your bills? Then send us an email now for more information contact us now via (financialserviceoffer876@gmail.com) whats-App +918929509036 Dr James Eric Finance Pvt Ltd Thanks

    ReplyDelete
  17. The growing popularity of Bluetooth speakers among the millennial population coupled with high product compatibility with other consumer electronic devices, has propelled the Bluetooth Speaker market growth.

    ReplyDelete
  18. Now you can Easily Activate Amazon by Amazon.com/mytv. It is a registration portal of Amazon Prime from which members can watch thousands of movies and TV shows for free from their device some features require a premium subscription. The instant video service can only be watched after registering your device with a simple subscription code. The website for registering on Amazon prime is amazon.com/mytv. Amazon Prime Video is one of the most amazing and interesting platforms to watch. With Amazon Prime you can stream unlimited movies and TV episodes on Amazon prime video app. You can Easily Fix Troubleshoot by visiting here.

    amazon com mytv

    ReplyDelete
  19. I am Alice Addy, a Corporate Sales Specialist operating in the Latin American region. We are experts in Market Research (Mercado Latinoamericano de Trifluoruro de Cloro). We provide market reports across various industry verticals and also help with procurement and supply chain intelligence. By giving you better insights, we help you make informed business decisions.

    ReplyDelete
  20. Polycarbonate refers to a group of thermoplastic polymers that holds the carbonate groups in their chemical structures. Polycarbonates are often used in engineering because they are strong, tough, and some grades are optically transparent.

    polycarbonate manufacturing process

    ReplyDelete
  21. Nitric acid is better known as aqua fortis and spirit of niter. It is a highly corrosive mineral acid. In appearance, the pure compound is colourless, but older samples tend to acquire a yellow cast due to the decomposition of oxides of nitrogen and water.

    nitric acid industry

    ReplyDelete
  22. Are you facing any difficulty while trying Garmin GPS update free lifetime? You are concerned about any possible Garmin GPS Update for free, this piece of blog will provide must-have information. This kind of issues only generates out when you download a different region of the map. The recent Garmin express updates help in renewing the crucial data inside GPS.

    ReplyDelete
  23. If your are facing problem to activate Youtube on device or TV, Go to youtube.com/activate site and enter 8- digit verification confirmation code & proceed.
    Youtube.com/activate
    youtube activate
    youtube.com/activate code

    ReplyDelete
  24. This is enough to make anyone happy. Because this makes our mind fresh I feel happy to come to here.

    ReplyDelete
  25. https://www.realpharmaneeds.com alias realpharmaneeds is a reputed online pharmacy that deals in all pain relief products and provides overnight shipping all over United States and Canada. You can pay by credit/debit card, Paypal or COD, call or text +1 8584617138 or write to us at realpharmaneeds@gmail.com
    order morphine 30mg online overnight delivery
    order opana 20mg online overnight delivery
    order opana 40mg online overnight delivery
    order oxycodone 15mg online overnight delivery

    ReplyDelete
  26. https://www.realpharmaneeds.com alias realpharmaneeds is a reputed online pharmacy that deals in all pain relief products and provides overnight shipping all over United States and Canada. You can pay by credit/debit card, Paypal or COD, call or text +1 8584617138 or write to us at realpharmaneeds@gmail.com
    order phentermine 15mg online overnight delivery
    order phentermine 30mg online overnight delivery
    order phentermine 37.5mg online overnight delivery
    order roxicodone 15mg online overnight delivery
    order roxicodone 30mg online overnight delivery


    ReplyDelete
  27. I would love to visit this website every day. As this promotes eternal knowledge. 바카라사이트

    ReplyDelete
  28. Great blog here! Additionally your website quite a bit up very fast! What web host are you the use of? Can I am getting your affiliate hyperlink on your host? Feel free to visit my website; 일본야동

    ReplyDelete
  29. Thanks for sharing this with us. The content of your article has proved your dedication to the field and your experience. Outstanding. I enjoy reading. You can use the Online Keyboard Counter blog to check the left button of your laptop's keyboard or mouse pad.

    ReplyDelete
  30. Go to video.foxnews.com. Follow the steps below to sign in and start watching Fox News Channel and Fox Business Network. Click Watch Live above the video player. Under the video player, it will say "On Air Now". Click on the network you want to watch (Fox News or Fox Business). Select your cable or satellite operator from the list of participating TV service providers (note: this is not necessarily your Internet Service Provider). Foxnews.com/connectBelow the main provider list, there is a drop-down list with more options.

    ReplyDelete
  31. Introduction to amazon.com/mytv and amazon com mytv. At present, there is almost no one who ignores the e-commerce giant, Amazon. According to statistics, about 90% of people in the world buy their products from amazon.com. Amazon is currently one of the largest e-commerce sites in the world. But their services are not limited to online shopping. The e-commerce giant also dominates the world of entertainment. Amazon Prime Video is an online streaming service offered by the company. It allows people to stream movies, TV shows, and web series at a very minimal cost.amazon.com/mytv
    With the Covid-19 pandemic still present, people are not yet allowed to visit cinemas. This has forced many filmmakers to partner with Amazon Prime Video to stream their movies on the platform.

    ReplyDelete
  32. Connect Disney Plus on your device at Disney Plus is one of the most popular streaming services in the world. You can find a full list of Disney classics as well as your favourite new Disney movies. Disneyplus.com/begin Code, which is also a subset of Hotstar, is an example of Hotstar. Disneyplus.com/begin
    Hotstar offers a variety of TV shows, movies, and news, as well as sports. It is also famous for its video streaming software. This service allows you to stream live TV shows, sports, and videos.

    ReplyDelete
  33. Go to video.foxnews.com. Follow the steps below to sign in and start watching Fox News Channel and Fox Business Network. Click Watch Live above the video player. Under the video player, it will say "On Air Now". Click on the network you want to watch (Fox News or Fox Business). Select your cable or satellite operator from the list of participating TV service providers (note: this is not necessarily your Internet Service Provider). Below the main provider list, there is a drop-down list with more options. Enter your email address (or username) and password. To login to your account, go to foxnews.com/connect and click the login foxnews.com/connect in the top right of the screen. The screen will redirect you to the login page. Enter your email address and password.

    ReplyDelete
  34. Disney Plus is one of the most popular streaming services in the world. You can find a complete list of Disney classics as well as your new favorite Disney films. Disney offers a variety of TV series, movies, and news, as well as sports. Disney plus is a one-stop shop for all of your favorite Disney shows and movies. To watch the disneyplus.com/start on your device, you need to activate it or install it on that device. There's plenty of programming from Pixar, Marvel Studios, and the Star Wars world, in addition to Disney is the most popular subscription channels in the world. This service allows you to stream live TV shows, sports, and videos. Disney also will enable you to purchase new movies.

    ReplyDelete
  35. Fox News is one of the most amazing platforms to keep up to date and connected to the whole world. This streaming app allows its users to activate their devices at foxnews.com/connect and then watch minute-by-minute updates anytime, anywhere. you need to find the activation code or login code and then you are ready to watch Fox News

    ReplyDelete
  36. Erex 100 is an oral cure proficient in upgrading Erectile Dysfunction in men. With a working ingredient of Sildenafil citrate 100mg.

    Email : sales@rsmenterprises.in
    Chat on WhatsApp with +91 92163 25377

    Sildenafil 100mg
    Erex 100mg Tablet
    Buy Men Sexual Tablet

    ReplyDelete
  37. I like reading an article that will make people think. Also, thank you for permitting me to comment.

    ReplyDelete
  38. I really like it when people come together and share thoughts. Great site, stick with it !

    ReplyDelete
  39. This has been an incredibly wonderful article. Many thanks for supplying this info.

    ReplyDelete
  40. Hey There. I found your blog. This is a really well written article.

    ReplyDelete

Comments appreciated!