Why No Reproducibility in Biomedical Research?
“Credibility is everything for science, and it is built over time in both obvious and subtle ways. Within the last 12 months, the reproducibility of science, a lynchpin of credibility, has come under intense scrutiny, both from the NIH and other government funding bodies, as well as in the lay and scientific press.” Dr. Emilie Marcus (Yale University / Salk Institute) CEO of Cell Press and Editor-in-Chief of Cell, from the November 2014 issue of Cell (Scientific Credibility and Reproducibility)
“Unfortunately, in the past few years, many studies have reported that the majority of results within biomedical research cannot be replicated.” From one of Harvard University’s blog posts (Reproduce or Bust: Bringing Reproducibility Back to Center Stage) by Steph Guerra
“It can be proven that most claimed research findings are false.” Dr John Ionnidis from the study mentioned directly below
When Dr. Marcus in the first quote above says that “credibility is everything for science,” she is only partially correct. Credibility is everything for everything. Whether you are running an organic ranch (HERE), building houses, educating our nation’s children, running a medical clinic, or (ahem) serving in government, credibility is paramount. Without it, nothing works properly and nothing or no one can be trusted. A society without credibility is a society in decay. So; what about the credibility of our scientific and biomedical research communities?
What is the one thing that makes “science” scientific? Although you will find about a million slightly different definitions, the one thing — the maxim if you will — the property that makes science scientific is reproducibility. Reproduciblity is essentially the same thing as (or at the very least, intimately related to) falsifiability — the various methods of testing a hypothesis to learn whether or not it is true or false; accurate or inaccurate.
The whole point of the science laboratory is to remove variables so that experiments can be duplicated over and over again to make sure that they are accurate and that further science is built on sound principles. When scientific experiments cannot be reproduced either by other teams of scientists or by the scientists that did the experiment in the first place, it’s not science.
The famous medical doctor, JOHN IONNIDIS, said it best via the title of the study he published in PLoS One a dozen years ago next month — Why Most Published Research Findings Are False. I’ve shown you why this is on many levels, one of the most recent being just a few weeks ago (HERE). Today we are going to tackle yet another aspect of EVIDENCE-BASED MEDICINE that proves exactly what Ionnidis stated well over a decade ago; that for any number of reasons, biomedical science cannot be trusted to be true or accurate.
This problem irreproducibility is so pervasive in the scientific community that Wikipedia actually has an entry for it called “Replication Crisis” that says…
“Scientists have found that the results of many scientific studies are difficult or impossible to replicate on subsequent investigation, either by independent researchers or by the original researchers themselves. The crisis has long-standing roots. Since the reproducibility of experiments is an essential part of the scientific method, the inability to replicate the studies of others has potentially grave consequences for many fields of science in which significant theories are grounded on unreproduceable experimental work.”
In other words, what you are going to learn today is that the VERY FOUNDATIONS OF MODERN PHARMACEUTICAL-BASED MEDICINE are based largely on misinformation. Whether this misinformation is deliberate (outright lies and fraud) or accidental, is something you’ll have to decide for yourself.
For the record, this problem is not confined to medical research. The same thing is happening in other fields as well. For instance, the website of the Federal Reserve contains a study called Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say ‘Usually Not’.
After looking at an awful lot of research, the authors concluded that, “Because we are able to replicate less than half of the papers in our sample even with help from the authors, we assert that economics research is usually not replicable.” We see the same thing in the field of chemistry. Listen to what Dalmeet Singh Chawla said in the March 2017 issue of Chemistry World (Taking on Chemistry’s Reproducibility Problem).
“A survey of over 1,500 scientists conducted by Nature last year revealed that 70% of researchers think that science faces a reproducibility crisis. Over half, however, still have faith in published literature in their field – with chemists being amongst the most confident despite reporting the most difficulty replicating other researchers’ or their own work.”
Why is this such a big deal? For the very reason I mentioned to you earlier — that modern medicine is based on the field of chemistry. And while I can see where some experiments might be tougher to duplicate because they involve living organisms and there are inherent differences in living organisms one to another, chemistry is just that; working with non-living chemicals.
Nothing should be easier than reproducing experiments with non-living chemicals. Not only is this not the case, but the icing on the cake is the fact that even though scientists are often getting it wrong, the quote above shows that they think they are getting it right. Allow me to show you some of the sources revealing just how wrong we’ve been getting it for the past half century.
NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH;
- NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH; 1960’s: A few months ago, NPR reporter, Richard Harris, released his book Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions. In it he says, “The issue of reproducibility in biomedical science has been simmering for many years. As far back as the 1960s, scientists raised the alarm about well-known pitfalls—for instance, warning that human cells widely used in laboratory studies were often not at all what they purported to be. At issue is not simply that scientists are wasting their time and our tax dollars; misleading results in laboratory research are actually slowing progress in the search for treatments and cures. This work is at the very heart of the advances in medicine. if preclinical discoveries are deeply flawed, scientists can spend years (not to mention untold millions of dollars) lost in dead ends.” Stick with me to see just how “flawed” the science really is.
- NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH; 2005: This is the year that the barrage against irreproduciblity in science started in earnest. Besides the study by Ionnidis mentioned earlier, he authored another one that year; this one for the Journal of the American Medical Association (Contradicted and Initially Stronger Effects in Highly Cited Clinical Research) dealt with the number of times studies contradict other studies. After looking at 45 highly cited studies and then looking at research trying to duplicate their results, Ionnidis concluded that, “A third of the most-cited clinical research seems to have replication problems, and this seems to be as large, if not larger, than the vast majority of other, less-cited clinical research.” As you will see, however, either the problem of irreproducibility is growing exponentially, or Dr. Ionnidis dramatically underestimated the severity of the problem 12 years ago.
- NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH; 2008: That January’s issue of Nature Genetics “evaluated the replication of data analyses in 18 articles on microarray-based gene expression profiling published in Nature Genetics in 2005–2006. One table or figure from each article was independently evaluated by two teams of analysts. We reproduced two analyses in principle and six partially or with some discrepancies; ten could not be reproduced. Repeatability of published microarray studies is apparently limited.” In other words, in this study the independent scientists were not even trying to reproduce the entire experiment, just a single table — and could not get it done in over half the cases.
- NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH; 2011: A group of researchers from Bayer (yes, the same company famous for their ASPIRIN) published a study that can be found in the September 2011 issue of Nature Reviews: Drug Discovery (Believe It or Not: How Much Can We Rely on Published Data on Potential Drug Targets?) that looked at “published data from 67 in-house projects.” (forty had to do with cancer research) They concluded that at the very most, they were able to duplicate results that were “completely in line” with the original research a quarter or less of the time. Furthermore, “In almost two-thirds of the projects, there were inconsistencies between published data and in-house data that either considerably prolonged the duration of the target validation process or, in most cases, resulted in termination of the projects because the evidence that was generated for the therapeutic hypothesis was insufficient to justify further investments into these projects. Talking to scientists, both in academia and in industry, there seems to be a general impression that many results that are published are hard to reproduce. However, there is an imbalance between this apparently widespread impression and its public recognition, and the surprisingly few scientific publications dealing with this topic.“
- NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH; 2012: In March of 2012, a team led by Glenn Begley, former head of cancer research at pharmaceutical giant Amgen, revealed just how bad things really were in the field of cancer research. Published by Nature, the study (essentially this was a decade-long whistleblower sort of thing) Drug Development: Raise Standards for Preclinical Cancer Research concluded after trying to reproduce 47 of the industry’s “landmark” cancer studies that, “clinical trials in oncology have the highest failure rate compared with other therapeutic areas. Unquestionably, a significant contributor to failure in oncology trials is the quality of published preclinical data.” How bad was it? “Scientific findings were confirmed in only 6 (11%) cases. Even knowing the limitations of preclinical research, this was a shocking result.” How do we know that BIG PHARMA is talking out of both sides of their collective mouths when dealing with this issue? All of the companies forced Amegen to sign non-disclosure agreements. In other words, they knew in advance that their published research would not stand up to rigorous scrutiny.
- NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH; 2013: The December 2013 issue of Nature published a study (Modelling the Effects of Subjective and Objective Decision Making in Scientific Peer Review) on a phenomenon known as “herding“. “Given the increasing concern surrounding the reproducibility of much published research, it is critical to understand whether peer review is intrinsically susceptible to failure. Here we show that even when scientists are motivated to promote the truth, their behaviour may be influenced, and even dominated, by information gleaned from their peers’ behaviour. This phenomenon, known as herding, subjects the scientific community to an inherent risk of converging on an incorrect answer and raises the possibility that, under certain conditions, science may not be self-correcting.” This is not really news, nor is the propensity to only publish data that makes one’s products (in this case drugs) appear more safe and wonderful than they really are. One of the best examples of herding (probably caused by fear of being blackballed by the research community) has to do with vaccines. Both FLU VACCINES as well as the relationship between VACCINES & AUTISM provide many examples. If you want to understand this concept a bit better, take five minutes to learn who DR HUGH FUDENBERG was.
- NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH; 2015 PART I: Two years ago next month, the journal Science published a study on the field of psychology called Estimating the Reproducibility of Psychological Science. After looking at and trying to reproduce “100 experiments reported in papers published in 2008” using teams of scientists from around the globe, the authors had to conclude that, “A large portion of replications produced weaker evidence for the original findings despite using materials provided by the original authors. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results.” In case you didn’t grasp the sheer magnitude of the scientific deceit going on here, re-read that last sentence.
- NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH; 2015 PART II: April’s issue of The Journal of Cell Biology carried a paper called Reproducibility and Cell Biology, which showed this continued pattern. “Growing concerns about the reproducibility of published research threaten to undermine the scientific enterprise and erode public trust. Research reproducibility is crucial to the scientific enterprise, not only because it underpins the accuracy and integrity of our published literature, but also because basic research increasingly contributes to the development of innovative clinical therapies. Recent accounts describe frustrating experiences of pharmaceutical companies attempting to build upon research studies, notably in cancer biology. These companies encountered surprisingly low reproducibility (less than 25%) of published work.“
- NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH; 2015 PART III: The October 2015 issue of the Journal of Controversies in Biomedical Research ran a study (How Medical Practice Has Gone Wrong: Causes of the Lack-of-Reproducibility Crisis in Medical Research) by Henry Bauer, a professor of chemistry and science at Virginia Polytechnic Institute, that hit the nail squarely on the head. Listen to Dr. Bauer’s shocking conclusions. “Finding solutions to problems requires identifying their causes. Without that, only symptoms are likely to be addressed, leaving the root causes to generate further problems. The lack-of-reproducibility crisis in biomedical research is one of the indications that modern medicine in the most advanced countries has gone wrong in recent decades, as described and documented in many books and articles. Modern medical practice has gone wrong by over-emphasizing drug-based treatment for chronic, constitutional conditions. The failure to distinguish between infectious and innate conditions was exacerbated by misinterpretation of quantitative measures, inappropriate statistical analysis, and inadequate regulation. The drug industry has become too influential as a result of these mis-steps and is a source of many conflicts of interest that are barriers to improving matters.” Let me summarize (after I find Dr. B and give him a hearty fist-bump). Most chronic diseases (INFLAMMATORY and AUTOIMMUNE), while having some degree of GENETIC CULPABILITY, are diseases of lifestyle that will never really respond to what our medical community is trying to do with their drugs-for-everything approach. While there are many, the best example I can think of off the top of my head is DIABETES (HERE is the blistering expose showing how grossly ineffective this class of drugs really is).
- NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH; 2016 PART I: In May of 2016, the journal Nature did a survey of nearly 1,600 scientists called 1,500 Scientists Lift the Lid on Reproducibility, which concluded (cherry-picked for time and space as are most studies I talk about), “More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments. Data on how much of the scientific literature is reproducible are rare and generally bleak. 73% said that they think that at least half of the papers in their field can be trusted. Several respondents who had published a failed replication said that editors and reviewers demanded that they play down comparisons with the original study. The survey asked scientists what led to problems in reproducibility. More than 60% of respondents said that each of two factors — pressure to publish and selective reporting — always or often contributed.” In other words, publish or find a new job, along with the whole “Invisible & Abandoned” thing I mentioned earlier. For those of you who are not aware, this would be a good time to mention that I am the current world record holder in consecutively-made free throws (HERE) — 43,548.
- NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH; 2016 PART II: Drs. Kornfield (professor of psychiatry and special lecturer at Columbia University’s College of Physicians and Surgeons) and Titus (former director at the US Office of Research Integrity) published a scientific paper a year ago next month in Nature called Stop Ignoring Misconduct. Their premise? That the journals and institutions that actually are addressing the issue of reproducibility are not owning up to reality. They are trying to blame this phenomenon on anything but fraud. These authors take a different approach saying, “Irreproducibility is the product of two factors: faulty research practices and fraud. Yet, in our view, current initiatives to improve science dismiss the second factor. To dismiss the role of research misconduct is mistaken and unfortunate. At best, ignoring deliberate misconduct in efforts to reduce irreproducibility is a wasted opportunity, like tilling a field without clearing it of rocks. At worst, it permits destructive behaviour to persist and flourish. Only 10–12 individuals are found guilty by the US Office of Research Integrity each year. That number, which the NIH used to dismiss the role of research misconduct, is misleadingly low, as numerous studies show. For instance, a review of 2,047 life-science papers retracted from 1973 to 2012 found that around 43% were attributed to fraud or suspected fraud. A compilation of anonymous surveys suggests that 2% of scientists and trainees admit that they have fabricated, falsified or modified data. And a 1996 study of more than 1,000 postdocs found that more than one-quarter would select or omit data to improve their chances of receiving grant funding.“
- NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH; 2017: I can see where experiments with animals or people could be harder to duplicate because of the variances in living systems. However, replicating experiments in chemistry should be relatively simpler. In March of this year, Chemistry World published an article called Taking on Chemistry’s Reproducibility Problem by saying that, “Not a week passes without reproducibility in science – or the lack of it – hitting the headlines. Although much of the criticism is directed at the biomedical sciences or psychology, many of the same problems also pervade the chemical sciences.” The author went on to show that the problem of replicating studies in chemistry is essentially as big a deal as it is in the biological sciences.
- NON-REPRODUCIBILITY IN BIOMEDICAL RESEARCH; ONGOING STUDY: The website Psych FileDrawer keeps a running tally of experiments from the field of psychology. Although the ongoing study is called Top-20 List of Studies Users Would Like to See Replicated, it contains more like 35 or 40 studies. I counted 35 successes and 66 failures — a success rate of about 1 in 3 — not very good odds, but as good or better than most of what we have looked at so far.
WHEN IT COMES TO REPRODUCIBILITY IN BIOMEDICAL RESEARCH, IT’S MONEY THAT MATTERS
Although the Nature survey revealed “why” our biggest and most important scientific experiments and trials can’t be replicated / duplicated, the bottom line is money. If you have thumbed titles of my dozens of posts on EVIDENCE-BASED MEDICINE, you are already aware of this. How much money are we talking about? According to the website of our government’s National Institutes of Health (NIH), research is big business.
“The NIH invests nearly $32.3 billion annually (2016 stats) in medical research for the American people. More than 80% of the NIH’s funding is awarded through almost 50,000 competitive grants to more than 300,000 researchers at more than 2,500 universities, medical schools, and other research institutions in every state and around the world.”
Listen to what Dr Danielle Teller (Pulmonologist and intensive care specialist as well as a past faculty member at both Pitt and Harvard) says in the Jan 2016 article, Nearly All of Our Medical Research is Wrong.
“Academic medical research functions as a gargantuan cottage industry, where the government gives money to individual investigators and programs–$30 billion annually in the US alone—and then nobody checks in on the manufacturing process until the final product is delivered. The final product isn’t a widget that can be inspected, but rather a claim by investigators that they ran experiments or combed through data and made whatever observations are described in their paper. The quality inspectors, whose job it is to decide whether the claims are interesting and believable, are peers of the investigators, which means that they can be friends, strangers, competitors, or enemies.”
Dr. Sarah Weil from her Feb 2014 article Why Biomedical Research Has A Reproducibility Problem shows how this whole thing is related to the “Invisible & Abandoned” research problem I have talked about over and over on my site.
“Unfortunately, scientists are typically evaluated based on the number of papers they have published and the quality of the journals in which they have published, but not on whether their findings can be reproduced. The “publish or perish” culture drives researchers to dig for significant results they can publish, and in the process may create subtle biases to report results in a manner that inflates the importance of a study and, by proxy, its authors. Whole sets of experiments that do not fit squarely with a hypothesis may be omitted from the published work to make the findings seem more convincing.”
Why is this such a huge issue for the American taxpayer? Maybe because of the 32 billion being spent by our government, the huge majority is paying for research that cannot be replicated. The June 2015 issue of PLoS One (The Economics of Reproducibility in Preclinical Research) showed just how bad things really are.
“Low reproducibility rates within life science research undermine cumulative knowledge production and contribute to both delays and costs of therapeutic drug development. An analysis of past studies indicates that the cumulative (total) prevalence of irreproducible preclinical research exceeds 50%, resulting in approximately US$28,000,000,000 (US$28B)/year spent on preclinical research that is not reproducible—in the United States alone.”
Holy cash-cow batman! That’s a lot of dough! As you might of guessed from what you’ve seen thus far, nowhere is this research a bigger money-maker, not to mention a bigger producer of unduplicatable studies, than is the field of CANCER RESEARCH. Think about this when you hear ‘BROTHER JOE‘ or anyone else for that matter calling for yet another cancer-curing “MOONSHOT“.
The problem is, despite all the rhetoric about stopping this problem — or for that matter, even slowing it down, it continues picking up steam and crushing everything in its path; sort of like a cartoon snowball rolling downhill.
IS ANYTHING BEING DONE ABOUT REPRODUCIBILITY IN BIOMEDICAL RESEARCH; AND IS THE PROBLEM GETTING BETTER?
“There may be no more important issue for authors and reviewers than the question of reproducibility, a bedrock principle in the conduct and validation of experimental science. For most types of experiment, there is an unstated requirement that the work be reproducible, at least once, in an independent experiment, with a strong preference for reproducibility in at least three experiments. Most experimental results reported in the literature will not be subjected to the test of precise replication unless the results are challenged.” From a 2010 issue of Infection and Immunity (Reproducible Science)
On the evening of April 27, 2015, past president of Cal Irvine, atmospheric scientist, expert on global warming, and then president of the National Academy of Sciences, Dr. Ralph J. Cicerone gave an applause-filled speech to NAS members called Research, Reproducibility, Replicability, Reliability.
It was to be sure, feel-good material that left those that heard it all warm and fuzzy inside. Cicerone stated, “I have learned that there is much good work underway…. I am very encouraged by reproducibility experiments that I have learned about, like those in cancer biology and in psychology.” Dr. C might be encouraged, but if you’ve been following along, you can see that others aren’t necessarily feeling that same vibe.
Just two short weeks ago, PLoS One published a study whose title asked a question, Can Cancer Researchers Accurately Judge Whether Preclinical Reports Will Reproduce? Although the short answer is no, let me show you what they actually said. “Whether scientists can accurately assess which experiments will reproduce original findings is important to determining the pace at which science self-corrects. Science is supposed to be self-correcting. However, the efficiency with which science self-corrects depends in part on how well scientists can anticipate whether particular findings will hold up over time.”
After looking at how well scientists predicted the ability of six mouse studies in the field of cancer would hold up over time, the authors of this study concluded that…
“Experts generally overestimated the likelihood that replication studies would reproduce the effects observed in original studies. Experts… did not consistently perform better than trainees, and topic-specific expertise did not improve forecast skill. Our findings suggest that experts tend to overestimate the reproducibility of original studies and/or they underappreciate the difficulty of independently repeating laboratory experiments from original protocols.”
If we can’t replicate studies in mice, what makes us think that we are getting it right in people? In many cases, we’re not. Despite what you are led to believe by industry, this field is not advancing like we are being told it is. Just a few days ago, Wired ran a story called Biology’s Roiling Debate Over Publishing Research Early. The author, one Megan Molteni, concluded that…
“Posting scientific papers online before peer review—in so-called preprint archives—isn’t a new idea. Preprints could solve these issues by decoupling distribution of results from their certification via peer review. But publishers and some scientists worry preprints will only further dilute the research literature and endanger fields already struggling with reproducibility failures….
One of the concerns with preprints is that scientists will sacrifice accuracy for speed—that in the rush to be first on the scientific record, they’ll wind up filling the internet with crap. Traditional peer review is supposed to catch mistakes and make sure a paper’s scientific reasoning is sound, and uploading a virgin paper means people will see work that could be wrong.”
We’ve seen just how wrong much of what we today call science really is.
This is why I believe that when it comes to your health, you need to be about he business of taking matters into your own hands. There are some simple rules to live by if you want to get healthy and stay that way. The MORE OF THESE POINTS you understand, the better the chances of regaining your health and living a life without pain and dysfunction. For instance, when “science” tells us that WHOLE FOODS and organic foods are no better than their processed, genetically altered, and chemically saturated counterparts, we should all start to realize just how bought and paid for science really is.
