Potential Bias and Incorrect Results In the Medical Literature
and Why You Shouldn’t Believe Everything You Read
Continuing Medical Education
Eligible for online CME Credit
By Stuart M. Caplen, MD
This article will discuss potential causes of bias in the medical literature and why some clinical decisions based on the medical literature may have incorrect conclusions.
Biases and Incorrect Results Reported in the Medical Literature
There are a number of potential biases that may affect results in the medical literature. There can be bias on the part of the researcher doing the experiment, bias in the experimental design, bias in the way the study is written for publication or bias by journals when deciding which articles to publish. Career advancement in academia is to a significant degree based on publication of articles which in some cases may lead to some data manipulation by authors to achieve that goal.
One type of bias that can affect medical literature is publication bias. It has been found that a trial with positive results is much more likely to be published. Positive result publication bias is common as journals benefit by being noticed and cited more when they publish positive papers.[1,2] The predominance of positive trials can create citation bias which is that the more a research paper is discussed and disseminated, the more its effect may be amplified in future publications and clinical practice.
Authors are much more likely to be published if their experimental results were positive. In one review of 105 antidepressant trials, 98% of the positive trials and only 48% of the negative trials reviewed ended up getting published.[2,3] A Cochrane Review found that trials that were statistically significant, had findings perceived to be important or striking, or showed a positive treatment effect, had nearly four times the odds of being published compared to trials that did not meet those criteria. In a study of Indian medical journals 72% of published articles had positive outcomes compared to a publishing rate of 28% for negative articles.
This positive publication bias in the literature can distort the results of meta-analyses. A meta-analysis is an article that combines the results of many trials, to try to determine with a higher certainty than smaller trials can, if there are statistically significant results. There are a number of possible errors in meta-analysis results. One source of potential error is incorrect weighting of the studies. Trials are given different levels of importance or weighted by precision, validity or risk of bias. It is common for one or just a couple of studies to carry much or most of the weight of the meta-analysis, which may affect the conclusions. Author judgment is involved in the evaluation of study quality and weight.[6,7]
A study on the effect of positive paper publication bias on meta-analyses using simulation models found that if statistically significant positive results on a topic were published four times more often than negative results (80% positive, 20% negative) depending on study size, there was an 11% to 100% chance of the meta-analysis author(s) making a type I statistical error (false positive) and finding a significant result when there actually was none. If statistically significant positive results were published four times more often than negative results and there was large amount of heterogeneity in the studies used in the meta-analysis, there was a more than a 90% chance of finding a statistically significant result when there actually was none (type I error). Heterogeneity is a measure of the variation in trial outcomes. There may be differences between trials in the treatment or population studied, the study design, or the data analysis method that may lead to different results. If the trials are very dissimilar with a large amount of heterogeneity, combining them into one group for a meta-analysis may lead to erroneous conclusions.[7,9]
Outcome reporting bias occurs when authors write up only the positive results in a trial and fail to report those that appear negative. In that previously discussed review of 105 antidepressant trials, ten antidepressant studies which were considered negative by the FDA were reported as positive by the researchers. This was accomplished by switching a secondary outcome with a primary one and reporting it as if it were the original intent of the researchers, or simply by not reporting negative results. The scientific method requires that primary outcomes be used as endpoints, and you cannot switch them once the experiment has started.[2,3]
In a study of 102 trials in the medical literature looking at outcome reporting bias, about half of the outcomes on whether tested drugs were effective, and about two-thirds of the outcomes on whether the treatment caused harm were incompletely reported. Positive outcomes were more likely to be reported. That same study examined experimental protocols and found that 62% of the trials reviewed had at least one primary outcome that was changed, introduced later on, or omitted. The authors of this study concluded that the reporting of trial outcomes is not only frequently incomplete but also biased and inconsistent with the authors’ own protocols. “Published articles, as well as reviews that incorporate them, may therefore be unreliable and overestimate the benefits of an intervention.”
There is also evidence that drug company-funded trials which are negative may never be released. In the case of the drug oseltamivir for influenza, it was discovered that adverse events and negative trials sponsored by a drug company had been presented at meetings but never published. Only when this was discovered, and the company was pressured to release the data could the true efficacy and adverse reactions to the drug be accurately determined.
Data dredging is where researchers start comparing multiple secondary outcomes or data elements that were not part of the original experimental design to try to find some positive correlations that are statistically significant. This is not considered proper scientific method because if authors compare enough data elements they will find some statistical significance just by chance due to a false positive type 1 statistical error.
Authors may spin experimental results, which refers to using language, often in the abstract or summary sections of the study, to make negative results appear positive. In the previously discussed antidepressant study, in 15 of the negative articles, 11 authors used spin to improve their experimental results. Some used non-statistically significant results(trends) as if they were positive, by referring only to the numerical outcomes. Only four of the 15 articles reported negative results without embellishing the results.[2,3]
Observer bias and confirmation bias can occur when either consciously or subconsciously experimenters’ observations or conclusions are slanted towards what they want or expect the results of the trial to be. Blinded studies where the researchers do not know which treatment the patient is getting is the best form of experimentation to avoid observer and confirmation bias.
Trials be may also be inadvertently biased just from the way they were designed or even the way questions are phrased on a questionnaire.
Trial conclusions may be incorrect if there is an unequal loss to follow-up of participants with different outcomes called loss-to follow-up bias. If three people in a trial with adverse events are lost to follow-up, those adverse events may never be reported or be reported as less significant than they truly are.
Volunteer or self-selection bias can occur when individuals who volunteer for a trial differ in clinical characteristics from those who do not.
The completeness of the methods section in the medical literature varies widely from minimal to very complete. The methods section should be as complete as possible to allow reviewers or readers to understand exactly how the experiment was performed. Inadequate methods sections may sometimes be due to authors hiding problematic parts of the trial. A complete methods sections is important to have for reference if someone else wishes to replicate the same trial to confirm the findings. Approximately 30% of rejections by journals are related to issues with the methods section.
Meticulous peer review before publication may help uncover and correct some of the biases or omissions by authors that may not be discovered in a non-peer reviewed journal publication. It has been found that double-blind peer review, where reviewers do not know the names or institutions of the authors, is a less biased way of doing peer review. It appears to lessen the bias of reviewers more favorably reviewing articles for publication from well-known authors or institutions.
As discussed in this article there are a number of issues that may make it challenging to decide if a finding in a published clinical trial is correct. Data dredging, redesign of experimental endpoints after the experiment has started, omitting data, using data trends which are not statistically significant as a positive result, a tendency to publish positive results more than negative ones, and other forms of potential bias all contribute to make some of the medical literature suspect.
To improve the quality of the medical literature, publications should increase the number of negative articles they present and ensure that authors do not change outcomes, omit data, or spin their negative results to appear positive. Authors also need to ensure that the methods sections of trials are complete so reviewers and readers can determine if scientific procedures were followed correctly. Important studies, especially smaller ones, should be repeated to ensure other investigators can achieve the same results and that the first author’s study result was not a type I error or a false positive just due to chance. Accurate methods sections are needed to allow that to happen. Double-blinded peer review can help reduce errors of commission and omission in medical articles and reduce reviewer bias.
Medical decisions for patients are made on the basis of published trials in the medical literature. The medical community as a whole needs to both understand and hopefully strive to continue to reduce the biases that can distort the medical literature.
Author’s note: If you wish to learn more about medical statistics it will be discussed further in an upcoming FibonacciMD blog article, Medical Statistics for the Non-Mathematician, for which AMA PRA Category 1 Credit(s)™ will be available.
FibonacciMD APP Members: Take the CME test
Not an APP member? Membership info.
ABOUT THE AUTHOR
Stuart M. Caplen, MD, FACEP, MSM
Dr. Caplen is a retired emergency medicine physician and former emergency department medical director, who also has a Master of Science in Management degree, and green belt certification in Lean/Six Sigma.
 Fanelli D. Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data. PLOS ONE. April 21, 2010. Retrieved from: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0010271
 Carroll AE, Congratulations. Your Study Went Nowhere. The New York Times. Sept. 24, 2018. Retrieved from: https://www.nytimes.com/2018/09/24/upshot/publication-bias-threat-to-science.html
 De Vries, Y., Roest, A., De Jonge, P., Cuijpers, P., Munafò, M., & Bastiaansen, J. (2018). The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: The case of depression. Psychological Medicine, 48(15), 2453-2455. Retrieved from: https://www.cambridge.org/core/journals/psychological-medicine/article/cumulative-effect-of-reporting-and-citation-biases-on-the-apparent-efficacy-of-treatments-the-case-of-depression/71D73CADE32C0D3D996DABEA3FCDBF57
 Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database of Systematic Reviews 2009, Issue 1. Art. No.: MR000006. Retrieved from: https://www.cochrane.org/MR000006/METHOD_publication-bias-in-clinical-trials-due-to-statistical-significance-or-direction-of-trial-results
 Charan J, Chaudhari M, Jackson R, et al. Comparison of methodological quality of positive versus negative comparative studies published in Indian medical journals: a systematic review. BMJ Open 2015;5:e007853. Retrieved from: https://bmjopen.bmj.com/content/5/6/e007853
 Direct weighting. Cochrane. Retrieved from: https://handbook-5-1.cochrane.org/chapter_8/8_8_4_1_direct_weighting.htm
 Bastian H. 5 Tips for Understanding Data in Meta-Analyses. PLOS Blogs. July 3, 2017. Retrieved from: https://absolutelymaybe.plos.org/2017/07/03/5-tips-for-understanding-data-in-meta-analyses/
 Kicinski M. How does under-reporting of negative and inconclusive results affect the false-positive rate in meta-analysis? A simulation study. BMJ Open 2014;4:e004831. Retrieved from: https://bmjopen.bmj.com/content/bmjopen/4/8/e004831.full.pdf
 von Hippel, P.T. The heterogeneity statistic I2 can be biased in small meta-analyses. BMC Med Res Methodol 15, 35 (2015). Retrieved from: https://doi.org/10.1186/s12874-015-0024-z
 Chan A, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical Evidence for Selective Reporting of Outcomes in Randomized Trials: Comparison of Protocols to Published Articles. JAMA. 2004;291(20):2457–2465. Retrieved from: https://jamanetwork.com/journals/jama/fullarticle/198809
 Yogendra Kumar Gupta YK et al. The Tamiflu fiasco and lessons learnt. Indian J Pharmacol. 2015 Jan-Feb; 47(1): 11–16. Retrieved from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4375804/#
 Teo C C. Types of cognitive biases you need to be aware of as a researcher. UX Collective. Sept 27, 2016. Retrieved from: https://uxdesign.cc/cognitive-biases-you-need-to-be-familiar-with-as-a-researcher-c482c9ee1d49
 Forbes D. Blinding: an essential component in decreasing risk of bias. Evid Based Nurs July 2013 | volume 16 | number 3. Retrieved from: https://ebn.bmj.com/content/ebnurs/16/3/70.full.pdf
 Tripepi G et al. Selection Bias and Information Bias in Clinical Research. Nephron Clin Pract 2010. Retrieved from: https://www.karger.com/Article/Fulltext/312871#
 Ghasemi A, Bahadoran Z, Zadeh-Vakili A, Montazeri SA, Hosseinpanah F. The Principles of Biomedical Scientific Writing: Materials and Methods. Int J Endocrinol Metab. 2019;17(1):e88155. Published 2019 Jan 28. Retrieved from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6413392/
 Tomkins A. Reviewer bias in single- versus double-blind peer review. PNAS. November 14, 2017. Retrieved from: https://www.pnas.org/doi/10.1073/pnas.1707323114
IMIT takes pride in its work, and the information published on the IMIT Platform is believed to be accurate and reliable. The IMIT Platform is provided strictly for informational purposes, and IMIT recommends that any medical, diagnostic, or other advice be obtained from a medical professional. Read full disclaimer.