The Center for Scientific Integrity, the parent nonprofit of Retraction Watch, has launched a new initiative to investigate and rapidly disseminate problems in the medical literature that directly affect human health.
Thanks to a $900,000 grant from Open Philanthropy, the Medical Evidence Project will leverage the tools of forensic metascience — using visual and computational methods to determine a paper’s trustworthiness — to rapidly identify problems in scientific articles, combined with the experience and platform of Retraction Watch to disseminate those findings.
“We originally set up the Center for Scientific Integrity as a home for Retraction Watch, but we always hoped we would be able to do more in the research accountability space,” said Ivan Oransky, executive director of the Center and cofounder of Retraction Watch. “The Medical Evidence Project allows us to support critical analysis and disseminate the findings.”
Several high-profile cases illustrate the need to scrutinize the medical literature. For example, European guidelines for giving beta blockers to patients before major surgery were based on results from a set of clinical trials, the DECREASE trials, later discredited due to misconduct. Dozens of systematic reviews and clinical guidelines on treatments for osteoporosis included clinical trials by Japanese bone-health researcher Yoshihiro Sato, who, it turned out, fabricated clinical trial results.
Beyond these high-profile cases are papers with what might appear to be small errors or mistakes that have an outsized impact on clinical results. The Medical Evidence Project aims to take a systematic look at these articles.
Scientific sleuth James Heathers will serve as the project’s director. “When we publish papers or change policies, we often get estimates of how many people would be positively affected by changes in treatment guidelines,” Heathers said. “I would like to produce that same estimate for the re-evaluation of the existing evidence. I strongly suspect that we’re about to do something seriously cost-effective, but I’ll only sleep well when I’ve proved it.”
Grantmaking organization Open Philanthropy has generously funded the Medical Evidence Project for $450,000 per year for two years. Staff for the project will also include part-time analysts and an editor to do investigations and write and publish results.
Investigations by universities, organizations and publishers often take months or years to complete, and findings are only communicated at the end of the process. The Medical Evidence Project hopes to accelerate that, swiftly disseminating the results of its investigations.
The project will take a broad look at all disciplines of medicine, Heathers said.
“There have been some unpleasant and relevant cases in anesthesiology, obstetrics and gynecology, surgery, trauma, and different elements of alternative and complementary medicine over the last few years,” he said. “I’d like to see if there are any more. But I’m quite agnostic. There isn’t much ‘patient-neutral’ medicine. It all matters.”