Sunday, August 30, 2015

False Discoveries and the ROC Curves of Social Science

Diagnostic tests for diseases can suffer from two types of errors. A type I error is a false positive, and a type II error is a false negative. The sensitivity or true positive rate is the probability that a test result will be positive when the disease is actually present. The specificity or true negative rate is the probability that a test result will be negative when the disease is not actually present. Different choices of diagnostic criteria correspond to different combinations of sensitivity and specificity. A more sensitive diagnostic test could reduce false negatives, but might increase the false positive rate. Receiver operating characteristic (ROC) curves are a way to visually present this tradeoff by plotting true positive rates or sensitivity on the y-axis and false positive rates (100%-specificity) on the x-axis.

Source: https://www.medcalc.org/manual/roc-curves.php

As the figure shows, ROC curves are upward sloping-- diagnosing more true positives typically means also increasing the rate of false positives. The curve goes through (0,0) and (100,100), because it is possible to either diagnose nobody as having the disease and get a 0% true positive rate and 0% false positive rate, or to diagnose everyone as having the disease and get a 100% true positive rate and 100% false positive rate. The further an ROC is above the 45 degree line, the better the diagnostic test is, because for any level of false positives, you get a higher level of true positives.

Rafa Irizarry at the Simply Statistics blog makes a really interesting analogy between diagnosing disease and making scientific discoveries. Scientific findings can be true or false, and if we imagine that increasing the rate of important true discoveries also increases the rate of false positive discoveries, we can plot ROC curves for scientific disciplines. Irizarry imagines the ROC curves for biomedical science and physics (see the figure below). Different fields of research vary in the position and shape of the ROC curve--what you can think of as the production possibilities frontier for knowledge in that discipline-- and in the position on the curve.

In Irizarry's opinion, physicists make fewer important discoveries per decade and also fewer false positives per decade than biomedical scientists. Given the slopes of the curves he has drawn, biomedical scientists could make fewer false positives, but at a cost of far fewer important discoveries.

Source: Rafa Irizarry
A particular scientific field could move along its ROC curve by changing the field's standards regarding peer review and replication, changing norms regarding significance testing, etc. More critical review standards for publication would be represented by a shift down and to the left along the ROC curve, reducing the number of false findings that would be published, but also potentially reducing the number of true discoveries being published. A field could shift its ROC curve outward (good) or inward (bad) by changing the "discovery production technology" of the field.

The importance of discoveries is subjective, and we don't really know numbers of  "false positives" in any field of science. Some never go detected. But lately, evidence of fraudulent or otherwise irreplicable findings in political science and psychology point to potentially high false positive rates in the social sciences. A few days ago, Science published an article on "Estimating the Reproducibility of Psychological Science." From the abstract:
We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects.
As studies of this type hint that the social sciences may be far to the right along an ROC curve, it is interesting to try to visualize the shape of the curve. The physics ROC curve that Irizarry drew is very steep near the origin, so an attempt to reduce false positives further would, in his view, sharply reduce the number of important discoveries. Contrast that to his curve for biomedical science. He indicates that biomedical scientists are on a relatively flat portion of the curve, so reducing the false positive rate would not reduce the number of important discoveries by very much.

What does the shape of the economics ROC curve look like in comparison to those of other sciences, and where along the curve are we? What about macroeconomics in particular? Hypothetically, if we have one study that discovers that the fiscal multiplier is smaller than one, and another study that discovers that the fiscal multiplier is greater than one, then one study is an "important discovery" and one is a false positive. If these were our only two macroeconomic studies, we would be exactly on the 45 degree line with perfect sensitivity but zero specificity.


28 comments:

  1. Kui nõukogu algatab soovituse esitamisega ülemäärase tasakaalunihke menetluse, peab selle menetluse lõpetama samalaadse õigusaktiga, st soovituse või otsusega, mis on kehtiv tervikuna/kehtiv selle adressaatidele.
    Agario Spielen Elsa Spelletjes Giochi Minions Gry Agario Jeux De Agario
    --------
    Tänan Teid väga mõistva suhtumise eest ning jään ootama Teie seisukohta eespool nimetatud ühist huvi pakkuva teema suhtes.
    Jeux De Elsa Jeux De Happy Wheels Juegos De Happy Wheels Juegos De Monkey Go Happy Juegos De Wheely

    ReplyDelete
  2. Muitos de nós estamos preocupados que o que resultou de Nice não seja o suficiente para fazer face ao alargamento iminente, e é por essa razão - ainda que agradecendo as declarações da Presidência do Conselho
    gry agario jogos de five nights at freddy's jogos de five nights at freddy's 4 jogos de happy wheels juegos agario
    Mariana, danke für das Teilen, danke in und bei uns zu sein; danke für deinen Unterricht, deine Bücher, deine Erinnerungen


    Muito obrigado Dr. Lab por compartilhar suas experiências de laboratório, técnicas secretas e paixão analógica no em sua primeira edição de Dicas de Aberrações de Filmes.
    juegos de five nights at freddy's juegos de five nights at freddy's 4 juegos de strike force heroes jeux agario jocuri agario
    Unser Dank geht an Norma und Ruth für das Teilen ihrer Herzen mit uns.

    ReplyDelete
  3. Für das entgegengebrachte Vertrauen danke ich der Fraunhofer-Gesellschaft, allen industriellen und institutionellen Projektpartnern und Kunden, den Beratern und Auditoren, den Mitgliedern des
    juegosdeariel.com, juegosdebadicecream.com, juegosdeelsa.es, juegosderapunzel.es, juegosfivenightsatfreddys.com,
    Kuratoriums sowie dem Bundesministerium für Bildung und Forschung und dem Bayerischen Staatsministerium für Wirtschaft, Infrastruktur, Verkehr und Technologie.


    Ich weiß, dass Sie diesen Standpunkt teilen, und ich danke Ihnen für Ihre Unterstützung.
    puppetgames.net, puppetsoccer2016.com, stickmanspiele.com, yoob5.com, yooboyunlari.com,
    Danke für diesen eindrucksvollen Bericht und das ehrliche Teilen, danke, dass wir zusammen stehen.

    ReplyDelete
  4. Eu acho que você fez isso muito bem. Consultei alguns posts no facebook e todos vocês são os melhores. Você pode se orgulhar disso. Além disso, vou tentar ser como você. Eu tenho acompanhado você no facebook para ser atualizado com a mais recente artigo facebook baixar , facebook baixar , facebook movel baixar , baixar facebook , facebook baixar

    ReplyDelete
  5. Ihnen allen vielen Dank dafur, dass Sie diesen glücklichen Tag mit uns teilen. Fofy Games Friv 2018 Friv 2019 wir danken euch fur das bisherige Teilnehmen lassen, mochten aber den Rundbrief nicht mehr erhalten Friv 2020 Friv4school 2019 Gogy 3 Ich mochte Kommissar Fischler fur seine Freimutigkeit und Offenheit danken, Gogy Games Juegos De Gogy Juegos Twizy n der er uns jeden unternommenen Schritt erläutert und die verschiedenen vom Wissenschaftlichen Lenkungsausschu.

    ReplyDelete
  6. Wir danken Ihnen Friv4school Gry Friv Juegos Friv dass Sie diese Hoffnung mit uns teilen und diesen Schritt auf dem Pilgerweg des Vertrauens Juegos Friv Gry Friv mit uns gegangen sind. Juegos Friv Juegos De Friv Ihnen hiermit die gewünschten Informationen vermitteln zu können. Yepi Juegos Friv wir danken Ihnen für das Interesse an unseren Produkten und hoffe

    ReplyDelete
  7. Great presentation of ROC Curve Data Science form of blog and ROC Curve Data Science tutorial. Very helpful for beginners like us to understand ROC Curve Data Science course. if you're interested to have an insight on ROC Curve Data Science training do watch this amazing tutorial.:-https://www.youtube.com/watch?v=G_pvQYUm8Ik

    ReplyDelete
  8. Thanks really valuable. Will certainly share website with my buddies.
    Y88
    Dress Up Games
    Minecraft Games
    Girl Games

    ReplyDelete
  9. You did an awesome posting. I am grateful for your offer here. I like your posting and got some thought regarding this post. Best wishes from
    Hooda Math
    Didi Games
    Girlsgogames

    ReplyDelete
  10. Our Company prides itself as the best providers of Buy Non-Plagiarized Research Papers in the industry due to the numerous years we have provided our clients with reliable and quality services.We ensure that the experts offering the client with writing services have the skills and qualifications in the specific subject area requested by the client thus ensuring they get the best Write My Research Paper services.

    ReplyDelete
  11. Are you seeking for top-quality written Business Writing Services for sale? Proofreading your College Research Papers for Sale ensures that your final Cheap Research Papers is free from grammatical errors before delivering it to you.

    ReplyDelete
  12. Do you ardently require Essay Writing Help Writing Services from a writing firm with a distinguished track record? Are you unsure of which Custom Essay Writing Service Provider to contract for your essays? Do not let a burdensome Custom Essay Writing Services task drag you down.

    ReplyDelete

  13. This is a very informative article. I also agree with the title of your post and you explain well your point of view. I am very happy to see this post. Thank you for sharing with us.
    Maintain and share more related posts.
    tikiqq
    vipbandarq
    walipoker
    wargaqq
    zeuspoker
    zoyaqq
    daftar poker ip pro
    kumpulan situs judi terbaik
    daftar situs judi terpercaya
    daftar poker

    ReplyDelete
  14. Thank you for sharing this blog , It is an amazing blog , I am really impressed by your blog, It’s really useful norton.com/setup

    ReplyDelete
  15. Very nice article, I found it very useful .Even I have this wonderful website norton.com/setup
    www.norton.com/setup

    ReplyDelete

  16. This is an informative blog. Keep it up. I am looking forward to this kind of blog. Thanks for sharing it with us www.avg.com/retail

    avg.com/retail

    www.avg.com/activate

    avg.com/activate

    ReplyDelete
  17. Dear Author, you have written a mind-blowing blog all content was very informative and useable information I would like to say thank you so much for sharing such useful information with us and wish you all the best for upcoming comment. I have also something to share here I hope you will like my comment. norton.com/setup
    www.norton.com/setup, Norton product key

    ReplyDelete
  18. This is one of the best articles I have ever read all information is very useful for me I would love to read this article and subscribe to it. Thank you for sharing this with us. I also want to share some useful links here. www.office.com/setup, Office product key, office.com/setup

    ReplyDelete
  19. Do you need Finance? Are you looking for Finance? Are you looking for finance to enlarge your business? We help individuals and companies to obtain finance for business expanding and to setup a new business ranging any amount. Get finance at affordable interest rate of 3%, Do you need this finance for business and to clear your bills? Then send us an email now for more information contact us now via (financialserviceoffer876@gmail.com) whats-App +918929509036 Dr James Eric Finance Pvt Ltd Thanks

    ReplyDelete
  20. The global Construction Market size reached USD 12,639.41 Billion in 2020 and is expected to register a CAGR of 7.4% during the forecast period. Increasing urbanisation due to growing population globally with rise in disposable income is expected to drive global construction market revenue growth over the forecast period. In addition, rapid technological advancements in infrastructure development with growing demand for environment-friendly buildings with low carbon footprint are other key factors augmenting growth of the global construction market. Another factor expected to propel construction market growth is improving economic conditions in developing countries thus resulting in rising disposable income among consumers and increasing government investment on public infrastructure.

    ReplyDelete

Comments appreciated!