EVIDENCE-BASED MEDICINE? RESEARCH IS RARELY AS “EVIDENCE-BASED” AS WE’RE LED TO BELIEVE
A year or so ago I was asked to be part of a mastermind group of diverse people and professions. Honestly, it’s a group that frequently leaves me feeling like the village idiot or class dunce. There are world class athletes as well as the people who train them, take care of their injuries, and create their diets / supplements. There are experts in FUNCTIONAL MEDICINE and BIOMECHANICS. There are Ph.D researchers, biochemists, nutrition experts, FUNCTIONAL NEUROLOGISTS, and scientists of all kinds (Dr. Marrs did a rocking guest post on MITOCHONDRIAL DYSFUNCTION for me awhile back). And then there’s Dr. X.
Dr. X wears many of the hats mentioned above. With a Ph.D in Exercise Physiology from XXXXXXX, he serves on numerous boards and university-based organizations and is widely considered a leader in his field. He’s worked with numerous elite athletes, including many from MLB, the NFL, and the NHL, written more than 125 research articles, abstracts, and books, not to mention he is an expert reviewer for numerous journals and book publishers.
The point is that unlike myself, Dr. X is a bonafide expert at looking at research and determining whether it’s good or bad, valid or non, well-done or shoddy. In other words, the guy is a walking, breathing, nuclear-powered scientific BS detector (reminder to self; block him from my site). I bring all this up because he’s allowed me to share something he wrote off-the-cuff for the group last week. (a board discussion on MBI — a topic you can read about in THIS ARTICLE published just a few days ago — is what triggered him writing the piece below.)
Why am I so interested in scientific BS? I’m interested in showing my readers that when it comes to science, everything is not always what it seems. There is a category on my site titled EVIDENCE-BASED MEDICINE that shows how true this really is. Some of my favorite posts from it include ANNUAL FLU VACCINATIONS and an article called “WHO DECIDES WHAT’S EVIDENCE?”
Oh; and it’s always interesting discussing the way the sugar industry was bribing Ivy League schools back in the 50’s and 60’s in order to “prove” that fat was the chief cause of America’s burgeoning health woes, and not sugar — HERE or HERE, or the way that MEDICAL GUIDELINES frequently have little to do with science or best-evidence. The bottom line is that all too often, science is whatever the group with the most money and influence says it is. Think I’m being too harsh?
In several of my posts (HERE, HERE, HERE and HERE are a few of them), I’ve shown you some of the common tricks researchers use to make their findings say whatever they want them to say (my all-time favorite is probably ‘INVISIBLE & ABANDONED‘). All of this is verified on some level by Dr. X, who is what I might refer to as a scientific purist. “Most research is a racket, but at its best, research is pure and beautiful, like most things in life I think.” This next section is quoted verbatim and not CHERRY-PICKED as is the case with many of my posts.
1.) I have often had reviewers ask for main effects when no interactions were present, thus the reviewers we asking us to violate statistical standards and protocols. This is now a common practice and authors submit main effects without interactions and they are published all the time.
2.) A lot of researchers in every field of science are sloppy, I have seen so many poor methods published in papers that had no findings and yet they are published and make claims that the product, drug, diet, program, etc. don’t work. Research is really hard when done correctly.
- Not only are the methods sloppy, but the data collection is sloppy, uncontrolled, and unreliable. I have worked with a few major pharmaceutical companies in the last year and am blown away by how careless some are when collecting patient data. I also see this when I visit labs and audit research studies. Simple things that are important are often overlooked due to time and ability. They are more interested in getting the study done than conducting a clean study. Thus an increase in the chances of non-significant findings and the need to run stats like MBI or effect size.
3.) Most reviewers are not experts in statistics or experts in the field related to your paper topic. Reviewers and editors simply don’t have the time or care about the quality of work anymore (if they even ever did) and are more interested in a high-impact article being published. History has clearly shown that major journals would rather publish high-impact falsified data then do the actual work vetting and reviewing the papers in detail. The non-major journals tend to not care and allow for much lower quality work to be published. And then there are the pay-to-publish journals that just don’t care what they publish.
- Nine times out of ten at least one of the reviewer comments clearly suggest they don’t understand the content of the paper or the methods and should not be considered an “expert” in the area. There are also almost never any comments about statistics because most reviewers are intimidated by them and scared to comment and look like a fool. There are some good reviewers though, not all are bad, but usually you don’t get two out of two reviewers that are any good.
4.) It’s about who you know. Often if there are coauthors or a lead author that are leaders (or perceived leaders) in a field, funded a lot by the NIH, on the editorial board, etc. get an easy review process when their work is often shoddy. The general rule if you are having trouble getting a paper published is to find a coauthor on the editorial board of a journal.
- There is no real requirement to becoming a journal editor or paper reviewer. All you have to have done is publish a few papers in the area and you may be asked to review manuscript. I have been asked dozens of times to review papers that I have had no business reviewing, I have no idea how they even got my name in most instances. I decline the review every time stating that I am not an expert in this field.
5.) Most people think a published scientific paper is fact and they don’t care to break down the details. People need to understand that just because its published research doesn’t mean it is quality research. There are some new standards that are being applied to medical research in medical school that help doctors determine the quality of the research study, but that’s just a start and is only scratching the surface. Most graduate schools don’t teach enough of this in their research methods courses and focus more on types of research than how to identify good quality research.
The bottom line is that if someone wants to publish something it will eventually get published and there is no way to stop it. There will always be ethical issues in research regardless of funding, people have their own agendas and want their ideas to work so they are biased and this will come through in their methods, statistics, and conclusions.
I was fortunate enough to be taught that all research should be unbiased regardless of the data and the results. Just because one study found one effect in one sample of one population using one protocol and one treatment doesn’t mean it’s now a scientific fact. There are way too many variables when dealing with humans/mammals and diet, drugs, exercise, health, etc.
However, many people just use the data that supports their ideas even when the study is questionable. That’s why you can pick up two books with opposing views and both are highly referenced… Of course this brings up another issue with authors misquoting and mis-citing data from other sources. I have had many of my papers cited in sentences that have nothing to do with what was in my papers or was the opposite of what was found/stated in the paper.
The best quote from that article was: ‘The most serious fault of the medical literature rests with acceptance of such questionable statistical analysis in the face of contradictory evidence from experiment or even common sense.’
I think the only way to combat this issue is to hold the reviewers accountable for their work. I think every published manuscript should have the reviewers and editors’ names, affiliations, and contact information published as large as the authors information. Why does this information need to be secret? The only reason something needs to be secretive in the field of science and education is to either promote someone’s agenda or protect against someone’s agenda. All reviews should be blinded to who the authors are from the editors and reviewers and when published the reviewers should be identified. I think this is the only way to increase the quality of publications in every field.
The more certain you are, the further from the truth you become. ‘A fool has no delight in understanding, but only in revealing his own opinion’ Proverbs 18:2
To quickly summarize….
- Adverse events are frequently not counted or published (HERE).
- Research, statistics, and analysis are often sloppy or mischaracterized (HERE is an example of statistical manipulation that affects the majority of you reading this).
- Peer-review — the process of research being looked at by experts in the field prior to publication — is broken in numerous ways.
- Getting published is easy — as long as you know the right people. On the flip side of that coin, choose the wrong topic to study, and you probably just destroyed your career (HERE or HERE).
- People (most doctors and professionals included) assume that since it’s published in a peer-reviewed scientific / medical journal, it must be true.
- Most professionals don’t know how to tell the difference between good research (evidence) and bad research (evidence).
- When trying to get to the bottom of all this, secrecy rules the day (the only logical reason for “protecting or promoting agendas” would be covering up FINANCIAL CONFLICTS).
I asked Dr. X what resources I could use to develop a better eye for separating the wheat from the tares when it comes to scientific research. He gave me two, both of which after reading, I feel can largely be understood by the general public. The first was the EVIDENCE-BASED PRACTICE TOOL KIT from Winona University, and the second, GRADING QUALITY OF EVIDENCE AND STRENGTH OF RECOMMENDATIONS from the British Medical Journal. I’ve also found that it’s quick and easy to follow the money via a Google search (HERE is an example of me doing just that).
If you find yourself spending more time on our site than you originally intended, be sure to spread the wealth by liking, sharing, or following on FACEBOOK — it’s a great way to reach those you love and care about most. And for those of you struggling with chronic conditions, be sure and check out THIS POST as well. For the record, Dr. X allowing me to use his quote in today’s post is in no ways an endorsement of my site or the ideas found within.
