Statistical errors in research are quite common in research, and not always detected. As economists are well aware, when an error with important policy implications is revealed, it may prompt a media frenzy. I was surprised to learn recently of a major statistical error with potentially huge public health implications, yet with seemingly sparse media coverage when it was revealed.
The error concerns the Recommended Dietary Allowance (RDA) of Vitamin D. A 2014 paper found a statistical error in a study used by the Institute of Medicine (IOM) to determine the RDA, resulting in a recommendation that was about an order of magnitude too low.
I am neither a public health expert nor medically trained, but (following Miles Kimball's lead) have developed an interest in public health, and especially nutrition, research, largely due to its parallels with macroeconomic research. What little press coverage I did find about this Vitamin D study omitted technical discussion of the statistical error--"We'll spare you the gritty mathematical details," said one article. But I wanted these details, and so might you, so I dove in to what turned out to be a fascinating story. You may want to share it with your econometrics students: Correct interpretation of confidence intervals can truly be a matter of life and death.
First, some background. The human body can make Vitamin D (unlike other vitamins) when exposed to sunlight. It can also be attained by nutritional sources and supplements. Upon activation by the liver and kidneys, it acts as a hormone that plays a role in calcium metabolism. Sufficient Vitamin D is critical for bone health and a plethora of other health outcomes (more on that later). Research on the health effects of Vitamin D typically looks at health outcomes associated with different serum 25-hydroxyvitamin D (25(OH)D) levels (a measure of concentration in the blood).
The IOM issues dietary recommendations, including RDAs, for the US and Canada. The RDA is supposed to designate the nutrient intake sufficient to meet the needs of 97.5% of healthy individuals. For Vitamin D, issuing this guideline requires first deciding what 26(OH)D level is desirable, then deciding how much supplemental Vitamin D should be taken so that most people have the desired 26(OH)D level. Based on associations between 26(OH)D levels and various health outcomes, the IOM aimed to recommend an RDA that would result in 25(OH)D levels of 50 nmol/L or more.
The IOM then had to determine how much supplemental Vitamin D to recommend based on this goal. They looked at 10 studies of the dose response relationship of vitamin D intake and 25(OH)D. Some of these studies examined 25(OH)D levels for multiple different doses, so in total there were 32 estimates (the green diamonds in Figure 1). They fitted a dose response relationship curve to these points, with 95% confidence interval. The IOM came up with an RDA for individuals 1 to 70 years of age of 600 IU per day. You can see the vertical line at 600 in Figure 1. It intersects the fitted dose response curve at 63 nmol/L and the lower bound of the 95% confidence interval at 56 nmol/L. Remember, this was the amount that was supposed to achieve 25(OH)D levels of at least 50 nmol/L in at least 97.5% of healthy individuals.
In October 2014, Paul J. Veugelers and John Paul Ekwaru explained in a paper in Nutrients that the IOM's interpretation of these confidence intervals was incorrect. They thought that 2.5% of individuals would have serum levels below the lower 95% confidence interval, but in this meta-analysis, the unit of observation was not the individual, but the study average. In the authors' words:
In the figure, you can see that at 600 IU per day, 97.5% of individuals will have serum levels above around 27 nmol/L, not 50 nmol/L. To get 97.5% of individuals with serum levels above 50 nmol/L, you would actually need a higher dose than any of the studies examined. Out-of-sample extrapolation led them to estimate that 8895 IU of vitamin D per day would actually be required. Veugelers and Ekwaru also pointed to two studies in which 10% or 15% of Canadian subjects had serum 25(OH)D levels of less than 50 nmol/L despite vitamin D supplementation at the RDA level. They wrote, "If the RDA had been adequate, these percentages should not have exceeded 2.5%. Herewith these studies show that the current public health target is not being met."
Veugelers and Ekwaru did caution that as t8895 IU of vitamin D per day "is far beyond the range of studied doses, caution is warranted when interpreting this estimate. Regardless, the very high estimate illustrates that the dose is well in excess of the current RDA of 600 IU per day and the tolerable upper intake of 4000 IU per day."
In March 2015, in the same journal, Robert Heaney, Cedric Garland, Carole Baggerly, Christine French, and Edward Gorham published a letter in the same journal that alleviated some of the concern about extrapolating beyond the available data. They presented entirely different data on individuals with daily vitamin D intakes from zero to over 10,000 IU. They came up with an estimate that was slightly lower than Veugelers and Ekwaru's, but confirming the finding that the IOM recommendation was around an order of magnitude too low, and wrote:
"Worldwide reports have highlighted a variety of vitamin D insufficiency and deficiency diseases. Despite many publications and scientific meetings reporting advances in vitamin D science, a disturbing realization is growing that the newer scientific and clinical knowledge is not being translated into better human health," wrote Andrew Norman in a 2008 issue of the American Journal of Clinical Nutrition. A 2007 article in the same journal, by Reinhold Vieth and many coauthors, describes the situation as a "frustrating and regrettable situation for nutrition researchers."
Vieth et al. summarize the many health benefits attributable to adequate vitamin D, and evidence that the tolerable upper limit is around ten times higher than officially-recommended intakes. But they point to an over-cautious and under-nuanced take by the public media that has kept public supplemental intake too low:
In Finland, public health policy was changed in response to widespread low serum 25(OH)D concentration. Vitamin D fortification of certain dairy products and spreads began in 2002, and fortification levels were increased in 2010. This was successful in raising vitamin D intake, and health benefits are already measurable. In July 2017, Dimitrios Papadimitriou noted in the Journal of Preventive Medicine and Public Health the Type I diabetes, which had been on the rise, leveled off and then declined in Finland after the vitamin D fortification policy was implemented. Papadimitriou discusses vitamin D's role as a "powerful nuclear receptor-activating hormone of critical importance, especially to the immune system," and calls for public health authorities worldwide to modify RDAs in line with now quite substantive scientific research.
Papadimitriou titled his article "The Big Vitamin D Mistake," referring in a narrow sense to the IOM's statistical error. But he really discusses a mistake in a broader sense-- mounting evidence has been too slow to be incorporated into policymaking and practice.
The error concerns the Recommended Dietary Allowance (RDA) of Vitamin D. A 2014 paper found a statistical error in a study used by the Institute of Medicine (IOM) to determine the RDA, resulting in a recommendation that was about an order of magnitude too low.
I am neither a public health expert nor medically trained, but (following Miles Kimball's lead) have developed an interest in public health, and especially nutrition, research, largely due to its parallels with macroeconomic research. What little press coverage I did find about this Vitamin D study omitted technical discussion of the statistical error--"We'll spare you the gritty mathematical details," said one article. But I wanted these details, and so might you, so I dove in to what turned out to be a fascinating story. You may want to share it with your econometrics students: Correct interpretation of confidence intervals can truly be a matter of life and death.
First, some background. The human body can make Vitamin D (unlike other vitamins) when exposed to sunlight. It can also be attained by nutritional sources and supplements. Upon activation by the liver and kidneys, it acts as a hormone that plays a role in calcium metabolism. Sufficient Vitamin D is critical for bone health and a plethora of other health outcomes (more on that later). Research on the health effects of Vitamin D typically looks at health outcomes associated with different serum 25-hydroxyvitamin D (25(OH)D) levels (a measure of concentration in the blood).
The IOM issues dietary recommendations, including RDAs, for the US and Canada. The RDA is supposed to designate the nutrient intake sufficient to meet the needs of 97.5% of healthy individuals. For Vitamin D, issuing this guideline requires first deciding what 26(OH)D level is desirable, then deciding how much supplemental Vitamin D should be taken so that most people have the desired 26(OH)D level. Based on associations between 26(OH)D levels and various health outcomes, the IOM aimed to recommend an RDA that would result in 25(OH)D levels of 50 nmol/L or more.
The IOM then had to determine how much supplemental Vitamin D to recommend based on this goal. They looked at 10 studies of the dose response relationship of vitamin D intake and 25(OH)D. Some of these studies examined 25(OH)D levels for multiple different doses, so in total there were 32 estimates (the green diamonds in Figure 1). They fitted a dose response relationship curve to these points, with 95% confidence interval. The IOM came up with an RDA for individuals 1 to 70 years of age of 600 IU per day. You can see the vertical line at 600 in Figure 1. It intersects the fitted dose response curve at 63 nmol/L and the lower bound of the 95% confidence interval at 56 nmol/L. Remember, this was the amount that was supposed to achieve 25(OH)D levels of at least 50 nmol/L in at least 97.5% of healthy individuals.
The correct interpretation of the lower prediction limit is that 97.5% of study averages are predicted to have values exceeding this limit. This is essentially different from the IOM’s conclusion that 97.5% of individuals will have values exceeding the lower prediction limit.Veugelers and Ekwaru returned to the 10 studies, eight of which reported both average and standard deviation serum level for particular doses of Vitamin D. From these statistics, the authors could calculate the 2.5th percentile at each dose. Then they regressed these 2.5 percentile values (the 23 yellow dots in Figure 2) on vitamin D intake, coming up with the red dashed line in Figure 2. The green dashed lines are the confidence intervals from Figure 1, for the sake of comparison.
In the figure, you can see that at 600 IU per day, 97.5% of individuals will have serum levels above around 27 nmol/L, not 50 nmol/L. To get 97.5% of individuals with serum levels above 50 nmol/L, you would actually need a higher dose than any of the studies examined. Out-of-sample extrapolation led them to estimate that 8895 IU of vitamin D per day would actually be required. Veugelers and Ekwaru also pointed to two studies in which 10% or 15% of Canadian subjects had serum 25(OH)D levels of less than 50 nmol/L despite vitamin D supplementation at the RDA level. They wrote, "If the RDA had been adequate, these percentages should not have exceeded 2.5%. Herewith these studies show that the current public health target is not being met."
In March 2015, in the same journal, Robert Heaney, Cedric Garland, Carole Baggerly, Christine French, and Edward Gorham published a letter in the same journal that alleviated some of the concern about extrapolating beyond the available data. They presented entirely different data on individuals with daily vitamin D intakes from zero to over 10,000 IU. They came up with an estimate that was slightly lower than Veugelers and Ekwaru's, but confirming the finding that the IOM recommendation was around an order of magnitude too low, and wrote:
Thus, we confirm the findings of these investigators with regard to the published RDA for vitamin and we call for the IOM and all public health authorities concerned with transmitting accurate nutritional information to the public to designate, as the RDA, a value of approximately 7000 IU per day from all sources.Like Veugelers and Ekwaru, Heaney et al. remarked upon the safety of such a high recommendation, though their take was more optimistic:
The total, all-source intake of 7000 IU/day is below the no observed adverse effect level (NOAEL) of both the IOM and the Endocrine Society, below the tolerable upper intake level (UL) of the Endocrine Society, and well within the safe range delineated by Hathcock et al., who had generated that range using the IOM’s method of hazard identification.The hormonal role of vitamin D explains why the Endocrine Society also issues guidance about it. Remember, vitamin D is fat-soluble, so excess amounts are stored and can accumulate in body tissues-- hence the concern about safety at higher doses. Overall health benefits may increase with dose up to a point, and then start to decline. Initial guidelines on Vitamin D RDA were based on prevention of rickets. But as scientists have learned more about other health benefits, the cost-benefits calculus of vitamin D recommendations has shifted. This shift, however, was slow to be reflected inn health policy.
"Worldwide reports have highlighted a variety of vitamin D insufficiency and deficiency diseases. Despite many publications and scientific meetings reporting advances in vitamin D science, a disturbing realization is growing that the newer scientific and clinical knowledge is not being translated into better human health," wrote Andrew Norman in a 2008 issue of the American Journal of Clinical Nutrition. A 2007 article in the same journal, by Reinhold Vieth and many coauthors, describes the situation as a "frustrating and regrettable situation for nutrition researchers."
Vieth et al. summarize the many health benefits attributable to adequate vitamin D, and evidence that the tolerable upper limit is around ten times higher than officially-recommended intakes. But they point to an over-cautious and under-nuanced take by the public media that has kept public supplemental intake too low:
Evaluation of most relations of health and disease that involve vitamin D leads to the conclusion that a desirable 25(OH)D concentration is ≥75 nmol/L (30 ng/mL). If a concentration of 75 nmol/L is the goal to be achieved by consumption of vitamin D, then why is it so rare for members of the population to accomplish this? One reason is that almost every time the public media report that vitamin D nutrition status is too low, or that higher vitamin D intakes may improve measures of health, the advice that accompanies the report is outdated and thus misleading. Media reports to the public are typically accompanied by a paragraph that approximates the following: “Current recommendations from the Institute of Medicine call for 200 IU/d from birth through age 50 y, 400 IU for those aged 51–70 y, and 600 IU for those aged >70 y. Some experts say that optimal amounts are closer to 1000 IU daily. Until more is known, it is wise not to overdo it.” The only conclusion that the public can draw from this is to do nothing different from what they have done in the past.The evidence in favor of higher 25(OH)D concentration and a higher RDA continued to grow in subsequent years. "Despite research on the association between low vitamin D status and many diseases, no consensus has emerged on the optimal serum 25(OH)D concentration. The concern is whether it is safe to maintain serum 25(OH)D concentrations in the range high enough to prevent some types of cancers and coronary heart disease," wrote Garland et al. 2014 in the American Journal of Public Health. In a meta-analysis of serum 25(OH)D and age-adjusted all-cause mortality, they showed that overall age-adjusted hazard ratios for mortality decline steeply with 25(OH)D for serum levels below 30 nmol/L, then gradually level off (Figure 3). The hazard ratio is not statistically different from 1 at 36 nmol/L.
In Finland, public health policy was changed in response to widespread low serum 25(OH)D concentration. Vitamin D fortification of certain dairy products and spreads began in 2002, and fortification levels were increased in 2010. This was successful in raising vitamin D intake, and health benefits are already measurable. In July 2017, Dimitrios Papadimitriou noted in the Journal of Preventive Medicine and Public Health the Type I diabetes, which had been on the rise, leveled off and then declined in Finland after the vitamin D fortification policy was implemented. Papadimitriou discusses vitamin D's role as a "powerful nuclear receptor-activating hormone of critical importance, especially to the immune system," and calls for public health authorities worldwide to modify RDAs in line with now quite substantive scientific research.
Papadimitriou titled his article "The Big Vitamin D Mistake," referring in a narrow sense to the IOM's statistical error. But he really discusses a mistake in a broader sense-- mounting evidence has been too slow to be incorporated into policymaking and practice.
