Sunday, August 30, 2015

False Discoveries and the ROC Curves of Social Science

Diagnostic tests for diseases can suffer from two types of errors. A type I error is a false positive, and a type II error is a false negative. The sensitivity or true positive rate is the probability that a test result will be positive when the disease is actually present. The specificity or true negative rate is the probability that a test result will be negative when the disease is not actually present. Different choices of diagnostic criteria correspond to different combinations of sensitivity and specificity. A more sensitive diagnostic test could reduce false negatives, but might increase the false positive rate. Receiver operating characteristic (ROC) curves are a way to visually present this tradeoff by plotting true positive rates or sensitivity on the y-axis and false positive rates (100%-specificity) on the x-axis.

Source: https://www.medcalc.org/manual/roc-curves.php

As the figure shows, ROC curves are upward sloping-- diagnosing more true positives typically means also increasing the rate of false positives. The curve goes through (0,0) and (100,100), because it is possible to either diagnose nobody as having the disease and get a 0% true positive rate and 0% false positive rate, or to diagnose everyone as having the disease and get a 100% true positive rate and 100% false positive rate. The further an ROC is above the 45 degree line, the better the diagnostic test is, because for any level of false positives, you get a higher level of true positives.

Rafa Irizarry at the Simply Statistics blog makes a really interesting analogy between diagnosing disease and making scientific discoveries. Scientific findings can be true or false, and if we imagine that increasing the rate of important true discoveries also increases the rate of false positive discoveries, we can plot ROC curves for scientific disciplines. Irizarry imagines the ROC curves for biomedical science and physics (see the figure below). Different fields of research vary in the position and shape of the ROC curve--what you can think of as the production possibilities frontier for knowledge in that discipline-- and in the position on the curve.

In Irizarry's opinion, physicists make fewer important discoveries per decade and also fewer false positives per decade than biomedical scientists. Given the slopes of the curves he has drawn, biomedical scientists could make fewer false positives, but at a cost of far fewer important discoveries.

Source: Rafa Irizarry
A particular scientific field could move along its ROC curve by changing the field's standards regarding peer review and replication, changing norms regarding significance testing, etc. More critical review standards for publication would be represented by a shift down and to the left along the ROC curve, reducing the number of false findings that would be published, but also potentially reducing the number of true discoveries being published. A field could shift its ROC curve outward (good) or inward (bad) by changing the "discovery production technology" of the field.

The importance of discoveries is subjective, and we don't really know numbers of  "false positives" in any field of science. Some never go detected. But lately, evidence of fraudulent or otherwise irreplicable findings in political science and psychology point to potentially high false positive rates in the social sciences. A few days ago, Science published an article on "Estimating the Reproducibility of Psychological Science." From the abstract:
We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects.
As studies of this type hint that the social sciences may be far to the right along an ROC curve, it is interesting to try to visualize the shape of the curve. The physics ROC curve that Irizarry drew is very steep near the origin, so an attempt to reduce false positives further would, in his view, sharply reduce the number of important discoveries. Contrast that to his curve for biomedical science. He indicates that biomedical scientists are on a relatively flat portion of the curve, so reducing the false positive rate would not reduce the number of important discoveries by very much.

What does the shape of the economics ROC curve look like in comparison to those of other sciences, and where along the curve are we? What about macroeconomics in particular? Hypothetically, if we have one study that discovers that the fiscal multiplier is smaller than one, and another study that discovers that the fiscal multiplier is greater than one, then one study is an "important discovery" and one is a false positive. If these were our only two macroeconomic studies, we would be exactly on the 45 degree line with perfect sensitivity but zero specificity.


10 comments:

  1. Kui nõukogu algatab soovituse esitamisega ülemäärase tasakaalunihke menetluse, peab selle menetluse lõpetama samalaadse õigusaktiga, st soovituse või otsusega, mis on kehtiv tervikuna/kehtiv selle adressaatidele.
    Agario Spielen Elsa Spelletjes Giochi Minions Gry Agario Jeux De Agario
    --------
    Tänan Teid väga mõistva suhtumise eest ning jään ootama Teie seisukohta eespool nimetatud ühist huvi pakkuva teema suhtes.
    Jeux De Elsa Jeux De Happy Wheels Juegos De Happy Wheels Juegos De Monkey Go Happy Juegos De Wheely

    ReplyDelete
  2. Muitos de nós estamos preocupados que o que resultou de Nice não seja o suficiente para fazer face ao alargamento iminente, e é por essa razão - ainda que agradecendo as declarações da Presidência do Conselho
    gry agario jogos de five nights at freddy's jogos de five nights at freddy's 4 jogos de happy wheels juegos agario
    Mariana, danke für das Teilen, danke in und bei uns zu sein; danke für deinen Unterricht, deine Bücher, deine Erinnerungen


    Muito obrigado Dr. Lab por compartilhar suas experiências de laboratório, técnicas secretas e paixão analógica no em sua primeira edição de Dicas de Aberrações de Filmes.
    juegos de five nights at freddy's juegos de five nights at freddy's 4 juegos de strike force heroes jeux agario jocuri agario
    Unser Dank geht an Norma und Ruth für das Teilen ihrer Herzen mit uns.

    ReplyDelete
  3. Für das entgegengebrachte Vertrauen danke ich der Fraunhofer-Gesellschaft, allen industriellen und institutionellen Projektpartnern und Kunden, den Beratern und Auditoren, den Mitgliedern des
    juegosdeariel.com, juegosdebadicecream.com, juegosdeelsa.es, juegosderapunzel.es, juegosfivenightsatfreddys.com,
    Kuratoriums sowie dem Bundesministerium für Bildung und Forschung und dem Bayerischen Staatsministerium für Wirtschaft, Infrastruktur, Verkehr und Technologie.


    Ich weiß, dass Sie diesen Standpunkt teilen, und ich danke Ihnen für Ihre Unterstützung.
    puppetgames.net, puppetsoccer2016.com, stickmanspiele.com, yoob5.com, yooboyunlari.com,
    Danke für diesen eindrucksvollen Bericht und das ehrliche Teilen, danke, dass wir zusammen stehen.

    ReplyDelete
  4. Merci au groupe scout de la Molière pour la mise à disposition du matériel et merci à Philippe Mooser (président du groupe) pour le partage de ses connaissances durant cette journée. juegosfriv2019.com fivenightsatcandys.com fivenightsatfreddys5.net fnaf6.com friv10friv100.com
    Je vous remercie, au nom également des autres membres du Conseil de fondation Merci pour les deux pièces reçues la semaine dernière, et en particulier pour celle avec les contre-dépouilles. friv2009.com friv2017.us friv2018.info gryfriv4.com juegosdekidbutowski.com yobb.info
    C'est notre façon de vous remercier de partager votre connexion!

    ReplyDelete
  5. Eu acho que você fez isso muito bem. Consultei alguns posts no facebook e todos vocês são os melhores. Você pode se orgulhar disso. Além disso, vou tentar ser como você. Eu tenho acompanhado você no facebook para ser atualizado com a mais recente artigo facebook baixar , facebook baixar , facebook movel baixar , baixar facebook , facebook baixar

    ReplyDelete
  6. Ihnen allen vielen Dank dafur, dass Sie diesen glücklichen Tag mit uns teilen. Fofy Games Friv 2018 Friv 2019 wir danken euch fur das bisherige Teilnehmen lassen, mochten aber den Rundbrief nicht mehr erhalten Friv 2020 Friv4school 2019 Gogy 3 Ich mochte Kommissar Fischler fur seine Freimutigkeit und Offenheit danken, Gogy Games Juegos De Gogy Juegos Twizy n der er uns jeden unternommenen Schritt erläutert und die verschiedenen vom Wissenschaftlichen Lenkungsausschu.

    ReplyDelete
  7. Wir danken Ihnen Friv4school Gry Friv Juegos Friv dass Sie diese Hoffnung mit uns teilen und diesen Schritt auf dem Pilgerweg des Vertrauens Juegos Friv Gry Friv mit uns gegangen sind. Juegos Friv Juegos De Friv Ihnen hiermit die gewünschten Informationen vermitteln zu können. Yepi Juegos Friv wir danken Ihnen für das Interesse an unseren Produkten und hoffe

    ReplyDelete
  8. Sr. Cashman, le doy las gracias por esta información. Friv Games Gry Friv Gry Friv n todo momento puede Jeux Friv Friv Games optar por compartir su información personal con terceros. Juegos Friv 2019 Juegos Friv Gry Friv Juegos Friv Gracias por compartir su pasión por el mejoramiento de las vidas de los niños en el Estado de Nueva York

    ReplyDelete

Comments appreciated!