Critique 236. Reasons for apparently diverse results from epidemiologic studies relating alcohol consumption to the risk of breast cancer — 19 February 2020
Chu L Ioannidis J PA, Egilman AC Vasiliou V, Ross JS, Wallach JD. Vibration of effects in epidemiologic studies of alcohol consumption and breast cancer risk. Inter J Epidemiol 2020;10:1–11. doi: 10.1093/ije/dyz271
Authors’ Abstract
Background: Different analytical approaches can influence the associations estimated in observational studies. We assessed the variability of effect estimates reported within and across observational studies evaluating the impact of alcohol on breast cancer.
Methods: We abstracted largest harmful, largest protective and smallest (closest to the null value of 1.0) relative risk estimates in studies included in a recent alcohol–breast cancer meta-analysis, and recorded how they differed based on five model specification characteristics, including exposure definition, exposure contrast levels, study populations, adjustment covariates and/or model approaches. For each study, we approximated vibration of effects by dividing the largest by the smallest effect estimate [i.e. ratio of odds ratio (ROR)].
Results: Among 97 eligible studies, 85 (87.6%) reported both harmful and protective relative effect estimates for an alcohol–breast cancer relationship, which ranged from 1.1 to 17.9 and 0.0 to 1.0, respectively. The RORs comparing the largest and smallest estimates in value ranged from 1.0 to 106.2, with a median of 3.0 [interquartile range (IQR) 2.0–5.2]. One-third (35, 36.1%) of the RORs were based on extreme effect estimates with at least three different model specification characteristics; the vast majority (87, 89.7%) had different exposure definitions or contrast levels. Similar vibrations of effect were observed when only extreme estimates with differences based on study populations and/or adjustment covariates were compared.
Conclusions: Most observational studies evaluating the impact of alcohol on breast cancer report relative effect estimates for the same associations that diverge by >2-fold. Therefore, observational studies should estimate the vibration of effects to provide insight regarding the stability of findings.
Forum Comments
The relation of alcohol consumption to the risk of breast of cancer has been the subject of innumerable epidemiologic studies over many decades. The large majority of studies show a slight increase in risk for the consumption of any alcohol, with an increase in risk with larger amounts. This association has been shown to be attenuated by levels of folate, binge drinking, and use of post-menopausal hormonal therapy. Many studies show further that while the incidence of breast cancer increases with any alcohol consumption, mortality from breast cancer is not increased by pre- or post-diagnosis levels of alcohol consumption, and is even decreased among moderate drinkers in some studies.
The present paper took the results of a recent large-scale meta-analysis (Griswold et al) and sought to try and explain why there is a large variability of results when 97 studies included in that analysis are compared. It compared results by differences between the most harmful and most protective effects within each study when different assumptions were made, categorized into the following: (1) different exposure definitions (varying approaches for estimating level of usual alcohol consumption and/or pattern of drinking); (2) different contrast levels (e.g., finding differences according to type of beverage or for particular age subgroups versus results for other beverages or other subgroups); (3) different study populations; (4) difference in covariates used in the analyses; and (5) use of different statistical models.
Comments by individual Forum members
There were a diversity of comments from members of the International Scientific Forum who reviewed this paper. Stated reviewer de Gaetano, “This paper used the relationship between alcohol consumption and breast cancer as an example of their methodological message: observational studies present results that show a very high degree of variability. Thus it is prudent not to draw any firm conclusion from them. This applies to all studies, showing either protection or harm by alcohol.” Other reviewers commented that while the approach presented in this paper is one way of attempting to judge validity of effect estimates for the relation of alcohol consumption to the risk of breast cancer, some Forum members suggested that other approaches may be preferable.
Persistence of bias in analytic approaches: Reviewer Skovenborg wrote: “The attached study analyses the ‘vibration of effects in epidemiologic studies” and includes comments on “White hat bias” and “Allegiance bias.” White hat bias (WHB) is a phrase coined by public health researchers Cope & Allison in 2010 to describe a purported “bias leading to the distortion of information in the service of what may be perceived to be righteous ends”, which consists of both cherry picking the evidence and publication bias. Allegiance bias (or allegiance effect) in behavioral sciences is a bias resulting from the investigator’s or researcher’s allegiance to a specific school of thought (Dragioti et al).
“I found the conclusion of the authors particularly interesting. The sharing of raw data used for analyses will increase transparency, thereby allowing investigators to better understand the impact of using different exposure definitions. Given that these practices are rarely if ever adopted to date in the field of alcohol and cancer risk assessment, one has to be careful about making strong statements about the validity of the published estimates of risk.
“In the presence of very strong opinions and beliefs in the field of alcohol exposure and cancer research, there is a risk that the literature may be shaped by the opinions of researchers, reviewers, and editors, picking the results of analyses that fit best their preconceived theories. In that case, the published, seemingly objective quantitative data may still reflect mostly subjective expert opinions, and the synthesis of data may really represent a form of expert vote counting instead of rigorous quantitative synthesis.”
Reviewer Finkel noted: “In observational studies, we have seen biased selection of data and of references, twisted interpretations, and even editorial comments dragged in from beyond the left-field fence when they had no relevance. The ‘vibration effect’ described in the present paper is one approach to be considered in helping to reach valid conclusions.”
Forum member Ellison wrote: “This paper discusses important concepts for evaluating scientific research, including judging bias in individual papers. As an example of potential bias in studies, all too often we can guess what the conclusions of an article will be by simply looking at the list of authors! Sometimes, authors continue to use ‘old’ criticisms of statistical approaches that had long been corrected (e.g., separating lifetime abstainers from ‘sick quitters,’ which could not be evaluated in some early studies), but often it appears that the studies that are included in a meta-analysis or discussion are very selectively chosen, and only those that support a pre-conceived premise are included.
“An early example of potential bias is a 2006 paper by Fillmore et al that reviewed 54 epidemiological studies and then found reasons to exclude all but 2 studies upon which to base conclusions regarding the relation of alcohol consumption to coronary heart disease (CHD) and 7 for the relation to total mortality (and those few studies left in their analysis were from unusual populations). Yet the authors concluded that while all of the studies reviewed initially showed a ‘J-shaped’ relation of alcohol consumption with CHD and mortality, their results after excluding all that they deemed having errors in their methods indicated no beneficial effects of light drinking. While this paper has been determined to be very biased (as well described in Panel Discussion I), it continues to be quoted in many papers.”
Questions on the approach used in this paper for determining validity of study results: Forum member Zhang had criticisms of the approach used in this paper. “To me, any epidemiological or statistical measures of either effect estimate (such as odds ratio, relative effect, etc.) or its variability (SE, confidence intervals) have to be scrutinized before further measures (such as vibration of effects) based on these effect estimates can be calculated. For example, in the current analysis, the largest harmful effect of alcohol consumption on the risk of breast cancer can be as high as 17.9 and protective estimate can be as low as 0.0. No one would believe such effect estimates are valid. If they are not valid, these values should not be used in the further analysis. Using these values to calculate vibration of effect estimates only generate a sensational effect but do not help to get to the truth.”
Forum member Djoussé noted: “This is an interesting paper. Vibration of effects is the ratio of extreme odds ratios within a study and can address variability of effect estimates, but does not say anything about causal inference. A causal association could be estimated with variability in effect estimates depending on several factors, including number of events, follow up time, heterogeneity within population, geographic location, accuracy in exposure and outcome assessment, etc. As for this paper, taking all studies summarized in a previous meta-analysis could amplify the variability just by design and yield larger ratio of extreme odds ratios.
“In my opinion, the critical question is whether light-to-moderate alcohol consumption increases the risk of breast cancer based on existing observational and imperfect data, in the absence of a large randomized trial (which won’t be conducted any time soon). Alternative approaches to scrutinize findings from observational studies include Mendelian randomization or discussion centered on E-value [which assesses how strong an unmeasured confounder would have to be to eliminate the observed effect estimate] (VanderWeele & Ding). While such methods are imperfect, their use would be more interesting in informing the public and advancing scientific knowledge.”
Forum member Keil wrote: “I notice that this study combines case control studies and cohort studies. All the key studies about the effect of alcohol drinking on many health outcomes (coronary disease, stroke, diabetes, etc) are, in my opinion, prospective cohort studies, and I think that this may be especially true for cancer. The discussion section is interesting reading and makes some important statements but also repeats old stories like ‘sick quitters’, a topic which I thought had been resolved quite some time ago. I am very much aware of great skepticism of some towards the whole field of nutrition epidemiology in terms of conclusions based mostly on observational studies. So the message of the discussion section is that studies on alcohol consumption and breast cancer are biased in many ways and researchers may, in the mode of cherry picking, select those studies which best fit their prejudices.”
Added reviewer Van Velden: “It is difficult, if not impossible, to isolate alcohol as etiology of breast cancer. There are too many confounders to be taken into consideration, especially genetic factors that may play a role which has not been investigated.”
What can be done to improve validity in study results? Forum member Keil stated: “With regard to a remedy of the present situation there is some, but not much advice: be cautious when interpreting these studies, as bias is abundant. Registration of observational studies should be mandatory (but unrealistic, in my opinion), and raw data of studies should be accessible so that independent researchers could/should work with these data and look for reproducibility. The term reproducibility is in my opinion a key term for all working in the field of ‘reducing waste in biomedical research.’ It is unfortunate that the discussion section does not provide any recommendations on how to advise the public or patients regarding alcohol intake.”
Reviewer Keil also questioned the use of the combination of data from many studies for setting drinking guidelines. “Why do we focus on these meta-analyses, which combine often very heterogeneous studies? Big single studies are much more convincing to me.” Indeed, the authors of the present paper demonstrated that the studies with the most extreme ratios between harmful versus protective estimates of effect were seen primarily because of using different study populations or different adjustment covariates. This strengthens the use of single, large, well-done prospective studies for assessing effects of alcohol on breast cancer risk.
Mixing data from consumers of different types of alcohol: Several Forum members also questioned the mixing of data from subjects who consumed different types of alcohol, and the fact that in this study there was no information on the pattern of drinking (binge drinking versus regular moderate use, which may lead to similar estimates of total or average alcohol use for both types of consumption). Stated reviewer van Velden: “Our studies show that red wine drinkers have lower cardiovascular mortality than consumers of ethanol in isolation (van Velden et al 2006; Mansveldt et al). We concentrated on the metabolic syndrome which is a serious risk factor for cardiovascular disease. But even the effects from wine consumption cannot be isolated from those of a responsible lifestyle, including a Mediterranean-type diet and mild exercise, a lifestyle that is more common among wine drinkers (van Velden et al 2007).”
Forum member Goldfinger added: “From a clinical perspective, I remind the audience that more people die each year from cardiovascular disease that from all cancers put together. The J-shaped association of moderate alcohol, particularly wine, consumption with incidence of cardiovascular disease events and mortality is very consistent, and mostly (allegiance bias) universally accepted. Almost two decades ago, Grønbæk et al published findings from Copenhagen studying more than 20,000 persons with respect to mortality and alcohol type. Compared with nondrinkers, light drinkers who avoided wine had a relative risk for death from all causes of 0.90 (95% CI, 0.82 to 0.99) and those who drank wine had a relative risk of 0.66 (CI, 0. 55 to 0.77). Wine drinkers had significantly lower mortality from both coronary heart disease and cancer than did non-wine drinkers (P = 0.007 and P = 0.004, respectively).
“Shortly after the Grønbæk et al paper, Renaud et al showed similarly that in their studies of 35,000 middle-aged men in France, only wine at moderate intake was associated with a protective effect on all-cause mortality. Renaud et al explained that this differential effect by type of beverage may be due not only to ‘the known effects of alcohol and polyphenols on cardiovascular diseases, but also since a very moderate intake of wine also protected from cancer and other causes (as also shown by Grønbæk in Denmark). Our recent results also indicate that the protective effect of a moderate intake of wine on all-cause mortality is observed at all levels of blood pressure and serum cholesterol.’” Goldfinger concluded: “So, considering all alcohol consumption to be the same and drawing clinical conclusions and, more concerning, policy recommendations, may be misleading and inappropriate.”
References from Forum critique
Cope MB, Allison DB. White hat bias: examples of its presence in obesity research and a call for renewed commitment to faithfulness in research reporting. Inter J Obesity 2010;34:84–88. doi:10.1038/ijo.2009.239
Dragioti E, Dimoliatis I, Evangelos E. Disclosure of researcher allegiance in meta-analyses and randomised controlled trials of psychotherapy: a systematic appraisal.” BMJ Open. 2015;5:e007206. doi:10.1136/bmjopen-2014-007206.
Fillmore KM, Kerr WC, Stockwell T, Chikritzhs T, Bostrom A. Moderate alcohol use and reduced mortality risk: systematic error in prospective studies. Addiction Research and Theory 2006;14:101-132.
Griswold MG, Fullman N, Hawley C, et al. Alcohol use and burden for 195 countries and territories, 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet 2018;392:1015-1035,
Grønbæk M, Becker U, Johansen D, et al. Type of alcohol consumed and mortality from all causes, coronary heart disease, and cancer. Ann Inter Med 2000;133:411-419. DOI: 10.7326/0003-4819-133-6-200009190-00008 ·
Mansvelt EPG, Fourie E, Blackhurst D, Kotze T, Stofberg H, van der Merwe S, Kotze MJ, van Velden DP. The influence of a Mediterranean diet with and without red wine on the haemostatic and inflammatory parameters of subjects with the metabolic syndrome. S.Afr.J Enol Vitic 2007;28:2007.
Panel Discussion I: Does alcohol consumption prevent cardiovascular disease? Ann Epidemiol 2007;17S:S37-S39. Doi: 10:1016/j.anneidem.2007.01.008.
Renaud S, Lanzmann-Petithory D, Gueguen R, Conard P. Alcohol and mortality from all causes. Biol Res 2004;37:183-187.
VanderWeele TJ, Ding P. Sensitivity Analysis in Observational Research: Introducing the E-Value. Ann Intern Med 2017;167:268-274. doi: 10.7326/M16-2607.
van Velden DP, van der Merwe S, Fourie E, Kidd M, Blackhurst DM, Kotze MJ, Mansvelt EPG. The influence of a Mediterranean-like diet with and without red wine on patients with the metabolic syndrome. The South African Journal of Clinical Nutrition 2006. 19;Supplement, S1-S28.
van Velden DP, van der Merwe S, Fourie E, Kidd K, Blackhurst DM, Kotze MJ, Mansvelt EPG. The short-term influence of a Mediterranean-type diet and mild exercise with and without red wine on patients with the metabolic syndrome. S.Afr J Enol Vitic 2007;28:No 1.
Forum Summary
The relation of alcohol consumption to the risk of breast of cancer has been the subject of innumerable epidemiologic studies over many decades, with most showing a slight increase in risk even for light drinking. However, it has been shown that this relation is attenuated when certain potential confounders are adjusted for, including folate levels, frequency of binge drinking, and the use of post-menopausal hormonal therapy. The present paper took the results of a recent large-scale meta-analysis and sought to try and explain why there is a large variability of results when the 97 studies included in that analysis were compared. The authors attempted to judge to what degree differences in exposure definitions, contrast levels, different study populations, different adjustment covariates, and different model approaches affected estimates of the effect of alcohol on cancer risk. The key factor the authors used in their analyses was the ratio between the most harmful and the most protective effects of estimated risk within each study.
All of the contrasts studied (different measures of alcohol consumption, different populations, different analytic approaches, etc.) had some effect on the estimates of effect of alcohol on the risk of cancer. As stated by the authors, “These findings suggest that whereas certain analytical and modelling choices, reflecting different types of alcohol and/or doses, can result in genuine differences, it is possible that many different analytical options, with different results, are pursued and selectively reported. Therefore, individual reported relative risk estimates from observational studies should be interpreted with caution.” Some Forum members disagreed with this conclusion, suggesting that due to marked variation in population, type of alcohol, analytic approaches, adjustment for confounding, etc., considerable error may be built in when combining data from heterogeneous studies. They felt that single large, long-term, well-done individual prospective studies may be preferable for judging the true effect of alcohol on the risk of breast cancer.
Thus, while Forum reviewers appreciated the attempt to explain differences in results of studies to get a more valid estimate of the true relation of alcohol consumption to the risk of breast cancer, and the analyses presented were done appropriately, some considered the approach described as not being a useful or valid one. Finding large differences in effects among sub-groups may not provide a reliable measure. Alternative approaches to scrutinize findings from observational studies, including Mendelian randomization or discussion centered on E-value (which assesses how strong an unmeasured confounder would have be to eliminate observed effect estimates), may be preferable. While such methods are imperfect, their use would be more interesting in informing the public and advancing scientific knowledge.
* * * * * * *
Comments on this critique by the International Scientific Forum on Alcohol Research have been provided by the following members:
Yuqing Zhang, MD, DSc, Clinical Epidemiology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
David van Velden, MD, Dept. of Pathology, Stellenbosch University, Stellenbosch, South Africa
Erik Skovenborg, MD, specialized in family medicine, member of the Scandinavian Medical Alcohol Board, Aarhus, Denmark
Ulrich Keil, MD, PhD, Professor Emeritus, Institute of Epidemiology & Social Medicine, University of Muenster, Germany
Tedd Goldfinger, DO, FACC, Desert Cardiology of Tucson Heart Center, University of Arizona School of Medicine, Tucson, AZ, USA
Harvey Finkel, MD, Hematology/Oncology, Retired (Formerly, Clinical Professor of Medicine, Boston University Medical Center, Boston, MA, USA)
R. Curtis Ellison, MD, Professor of Medicine, Section of Preventive Medicine & Epidemiology, Boston University School of Medicine, Boston, MA, USA
Luc Djoussé, MD, DSc, Dept. of Medicine, Division of Aging, Brigham & Women’s Hospital and Harvard Medical School, Boston, MA, USA
Giovanni de Gaetano, MD, PhD, Department of Epidemiology and Prevention, IRCCS Istituto Neurologico Mediterraneo NEUROMED, Pozzilli, Italy