15 August 2012 by William

highlighting research  Heres one solution for the reproducibility crisis in scientific research.We talk a lot about open access here, but one thing we haven’t delved into as much as we could is the quality of the research. We have plenty of data on the attention the world’s academics are paying to research outputs (watch this space for more on this) but we haven’t done as much to address the quality aspect as we have done to address the quantity of freely available research via open access. Today, that’s all going to change.

We’re working with Science Exchange, a startup that helps researchers access scientific expertise that they may not have available locally, to launch the Reproducibility Initiative, a program to help researchers, institutions, and funding agencies validate their critical research findings.

“In the last year, problems in reproducing academic research have drawn a lot of public attention, particularly in the context of translating research into medical advances. Recent studies indicate that up to 70% of research from academic labs cannot be reproduced, representing an enormous waste of money and effort,” said Dr. Elizabeth Iorns, Science Exchange’s co-founder and CEO. “In my experience as a researcher, I found that the problem lay primarily in the lack of incentives and opportunities for validation — the Reproducibility Initiative directly tackles these missing pieces.”

The Reproducibility Initiative provides both a mechanism for scientists to independently replicate their findings and a reward for doing so. Scientists who apply to have their studies replicated are matched with experimental service providers based on the expertise required. The Initiative leverages Science Exchange’s existing marketplace for scientific services, which contains a network of over 1000 expert providers at core facilities and contract research organizations (CROs). “Core facilities and commercial scientific service providers are the solution to this problem,” said Dr. Iorns. “They are experts at specific experimental techniques, and operate outside the current academic incentive structure.”

Scientists will receive the results of their validation studies and have the opportunity to publish them in the journal PLoS ONE as part of a Special Collection highlighting the importance of reproducibility in scientific research. Replications published in PLOS ONE will link back to the original publications upon which they are based. Prominent publishers such as Nature Publishing Group and Rockefeller University Press have also expressed their support for this acknowledgement of reproducibility. The data from the studies will be hosted on Figshare and we’ve set up a special Mendeley Reproducibility Initiative Group which will host the papers, provide collection and article level analytics, and facilitate sharing and discussion of the papers.

The Reproducibility Initiative also has the support of prominent scientists. Dr. Lee Ellis of MD Anderson and Dr. John Ioannidis of Stanford University, are both Advisors to the Reproducibility Initiative. “It is critically important to independently validate preclinical data before moving to the clinic,” said Dr. Ellis, who co-authored a widely read article in Nature earlier this year on the need to improve the reliability of preclinical cancer studies. Dr. Ioannidis, who published a study in PLOS Medicine titled “Why Most Published Research Findings Are False” (and who’s one of the all time highest readership authors on Mendeley), also offered his support, stating “The Reproducibility Initiative is a very important pilot effort, offering valuable insights on how reproducibility checks can work in real life.”

The Reproducibility Initiative is initially accepting 40-50 studies for validation. Studies will be selected on the basis of potential clinical impact and the scope of the experiments required and, in aggregate, may eventually serve as a proof-of-concept for this mechanism of validation to funding agencies and patient groups.

Reproducible science incentives could build a foundation for robust translational research and improved therapies. “As awareness of irreproducibility grows, we wanted to provide a way for top quality researchers to distinguish themselves. This is truly a great opportunity for scientists with potentially groundbreaking results to garner direct validation of their work, and to lead the charge for reproducible science” said Dr. Iorns.

Scientists can submit their study for validation at: http://reproducibilityinitiative.org and if you’re at a core facilities or CROs and you’d like to participate as a provider, you can join at: http://scienceexchange.com/provider

Tags: , , , , , , ,

One Response to “Here’s one solution for the reproducibility crisis in scientific research.”

  1. Mack Says:

    Nice!

    A detail in reproducibility efforts could also be publicly available documentations and walk troughs of methods. For instance guided screen recordings of the used statistical methods would be very helpful sometimes. For instance reproducing/following a “12 step” description of multivariate statistics is much easier when one can see the data handled on screen (“Here we have the raw data N=4567. Here we leave out the outliers. Here are the transformed data. Here we do the multivariate regression and from these we then select… and here we have the unedited figure that you can see in our paper” and so forth). When having publicly available data one could do the exact same steps at home (given usage of common software such as R, excel etc). Now wouldn’t that be a winner for educational purposes too?