Sunday, August 30, 2015

False Discoveries and the ROC Curves of Social Science

Diagnostic tests for diseases can suffer from two types of errors. A type I error is a false positive, and a type II error is a false negative. The sensitivity or true positive rate is the probability that a test result will be positive when the disease is actually present. The specificity or true negative rate is the probability that a test result will be negative when the disease is not actually present. Different choices of diagnostic criteria correspond to different combinations of sensitivity and specificity. A more sensitive diagnostic test could reduce false negatives, but might increase the false positive rate. Receiver operating characteristic (ROC) curves are a way to visually present this tradeoff by plotting true positive rates or sensitivity on the y-axis and false positive rates (100%-specificity) on the x-axis.

Source: https://www.medcalc.org/manual/roc-curves.php

As the figure shows, ROC curves are upward sloping-- diagnosing more true positives typically means also increasing the rate of false positives. The curve goes through (0,0) and (100,100), because it is possible to either diagnose nobody as having the disease and get a 0% true positive rate and 0% false positive rate, or to diagnose everyone as having the disease and get a 100% true positive rate and 100% false positive rate. The further an ROC is above the 45 degree line, the better the diagnostic test is, because for any level of false positives, you get a higher level of true positives.

Rafa Irizarry at the Simply Statistics blog makes a really interesting analogy between diagnosing disease and making scientific discoveries. Scientific findings can be true or false, and if we imagine that increasing the rate of important true discoveries also increases the rate of false positive discoveries, we can plot ROC curves for scientific disciplines. Irizarry imagines the ROC curves for biomedical science and physics (see the figure below). Different fields of research vary in the position and shape of the ROC curve--what you can think of as the production possibilities frontier for knowledge in that discipline-- and in the position on the curve.

In Irizarry's opinion, physicists make fewer important discoveries per decade and also fewer false positives per decade than biomedical scientists. Given the slopes of the curves he has drawn, biomedical scientists could make fewer false positives, but at a cost of far fewer important discoveries.

Source: Rafa Irizarry
A particular scientific field could move along its ROC curve by changing the field's standards regarding peer review and replication, changing norms regarding significance testing, etc. More critical review standards for publication would be represented by a shift down and to the left along the ROC curve, reducing the number of false findings that would be published, but also potentially reducing the number of true discoveries being published. A field could shift its ROC curve outward (good) or inward (bad) by changing the "discovery production technology" of the field.

The importance of discoveries is subjective, and we don't really know numbers of  "false positives" in any field of science. Some never go detected. But lately, evidence of fraudulent or otherwise irreplicable findings in political science and psychology point to potentially high false positive rates in the social sciences. A few days ago, Science published an article on "Estimating the Reproducibility of Psychological Science." From the abstract:
We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects.
As studies of this type hint that the social sciences may be far to the right along an ROC curve, it is interesting to try to visualize the shape of the curve. The physics ROC curve that Irizarry drew is very steep near the origin, so an attempt to reduce false positives further would, in his view, sharply reduce the number of important discoveries. Contrast that to his curve for biomedical science. He indicates that biomedical scientists are on a relatively flat portion of the curve, so reducing the false positive rate would not reduce the number of important discoveries by very much.

What does the shape of the economics ROC curve look like in comparison to those of other sciences, and where along the curve are we? What about macroeconomics in particular? Hypothetically, if we have one study that discovers that the fiscal multiplier is smaller than one, and another study that discovers that the fiscal multiplier is greater than one, then one study is an "important discovery" and one is a false positive. If these were our only two macroeconomic studies, we would be exactly on the 45 degree line with perfect sensitivity but zero specificity.


14 comments:

  1. Für das entgegengebrachte Vertrauen danke ich der Fraunhofer-Gesellschaft, allen industriellen und institutionellen Projektpartnern und Kunden, den Beratern und Auditoren, den Mitgliedern des
    juegosdeariel.com, juegosdebadicecream.com, juegosdeelsa.es, juegosderapunzel.es, juegosfivenightsatfreddys.com,
    Kuratoriums sowie dem Bundesministerium für Bildung und Forschung und dem Bayerischen Staatsministerium für Wirtschaft, Infrastruktur, Verkehr und Technologie.


    Ich weiß, dass Sie diesen Standpunkt teilen, und ich danke Ihnen für Ihre Unterstützung.
    puppetgames.net, puppetsoccer2016.com, stickmanspiele.com, yoob5.com, yooboyunlari.com,
    Danke für diesen eindrucksvollen Bericht und das ehrliche Teilen, danke, dass wir zusammen stehen.

    ReplyDelete
  2. Wir danken Ihnen Friv4school Gry Friv Juegos Friv dass Sie diese Hoffnung mit uns teilen und diesen Schritt auf dem Pilgerweg des Vertrauens Juegos Friv Gry Friv mit uns gegangen sind. Juegos Friv Juegos De Friv Ihnen hiermit die gewünschten Informationen vermitteln zu können. Yepi Juegos Friv wir danken Ihnen für das Interesse an unseren Produkten und hoffe

    ReplyDelete
  3. Great presentation of ROC Curve Data Science form of blog and ROC Curve Data Science tutorial. Very helpful for beginners like us to understand ROC Curve Data Science course. if you're interested to have an insight on ROC Curve Data Science training do watch this amazing tutorial.:-https://www.youtube.com/watch?v=G_pvQYUm8Ik

    ReplyDelete

  4. This is a very informative article. I also agree with the title of your post and you explain well your point of view. I am very happy to see this post. Thank you for sharing with us.
    Maintain and share more related posts.
    tikiqq
    vipbandarq
    walipoker
    wargaqq
    zeuspoker
    zoyaqq
    daftar poker ip pro
    kumpulan situs judi terbaik
    daftar situs judi terpercaya
    daftar poker

    ReplyDelete

  5. This is an informative blog. Keep it up. I am looking forward to this kind of blog. Thanks for sharing it with us www.avg.com/retail

    avg.com/retail

    www.avg.com/activate

    avg.com/activate

    ReplyDelete
  6. Do you need Finance? Are you looking for Finance? Are you looking for finance to enlarge your business? We help individuals and companies to obtain finance for business expanding and to setup a new business ranging any amount. Get finance at affordable interest rate of 3%, Do you need this finance for business and to clear your bills? Then send us an email now for more information contact us now via (financialserviceoffer876@gmail.com) whats-App +918929509036 Dr James Eric Finance Pvt Ltd Thanks

    ReplyDelete
  7. The global Construction Market size reached USD 12,639.41 Billion in 2020 and is expected to register a CAGR of 7.4% during the forecast period. Increasing urbanisation due to growing population globally with rise in disposable income is expected to drive global construction market revenue growth over the forecast period. In addition, rapid technological advancements in infrastructure development with growing demand for environment-friendly buildings with low carbon footprint are other key factors augmenting growth of the global construction market. Another factor expected to propel construction market growth is improving economic conditions in developing countries thus resulting in rising disposable income among consumers and increasing government investment on public infrastructure.

    ReplyDelete
  8. Everything is very open with a really clear description of the issues.

    ReplyDelete
  9. That's why it's very hard to find, but I'm very fortunate to read your writing.

    ReplyDelete
  10. Thank you for letting me know the good content.

    ReplyDelete

Comments appreciated!