Can Lab Management Software Help Solve the Reproducibility Crisis?

One of the basic premises of scientific research is that repeating the same experiment under the same conditions should give the same result. However, anyone who’s ever spent any meaningful time in the lab will recognise— with no small amount of frustration— that no matter how good a scientist you are, this isn’t always true of even your own experiments, let alone trying to recreate somebody else’s. This systemic inability to replicate experimental results, even when following published methods, is the reproducibility crisis in a nutshell. 

The term “reproducibility crisis” first appeared between 2010 and 2012, after researchers at Amgen and Bayer published papers that rocked the scientific  establishment when they demonstrated that their in-house R&D teams could only replicate the results of landmark preclinical studies in 11% of cases (Amgen) and 20-25% of cases (Bayer)^(1,2). This, then, was real-world confirmation of Ioannidis’ 2005 mathematical simulation, which identified “that for most study designs and settings, it is more likely for a research claim to be false than true”⁽³⁾.

Why good scientists get different results

While the reproducibility crisis is a real problem, this is absolutely not a result of any moral or capability failure on the part of scientists. Scientists are not failing science. Instead, the systemic architecture is failing scientists. What does that mean? Well, let’s consider where issues with reproducibility come from. 

Biological Noise

Biology is an inherently “noisy” science. Cell lines from different sources have “drifted” genetically in different ways, the number of times a cell line has been passaged can change how it responds to certain stimuli, different seeding densities, environmental controls, batches of antibodies—even brand of reagents—can all make a difference to the result of an experiment. 

Pragmatism vs Perfection

All scientists would like to believe that they’ve tested their hypotheses ad infinitum before publishing their results, but the reality is much messier. Time is money, after all, and academic grant funding or a biotech company’s runway, only stretches so far and for so long. At some point, perfection becomes the enemy of progress, and research has to move forward to secure the next tranche of funding. 

The three / four year staff cycle

In academia in particular, a tenured laboratory might have been there for decades, but its membership will—for the most part—be a revolving door of PhD students and postdocs. Each time somebody leaves the lab, it’s inevitable that some of their “scientific magic” goes with them. If they’re lucky, the PhD student whose project relies on following up on this research will be left with a meticulously completed lab-book and a freezer full of perfectly organised and carefully labelled samples. I would put money on the fact that few “follow on” students or postdocs are so lucky! While this is more obviously problematic in academia, the same issues of staff turnover and loss of institutional knowledge affect biotechs too. 

Publish or perish

Publish or perish is the unofficial motto of academic career plans. Academic researchers experience intense pressure to continuously publish high impact publications to secure funding, tenure, and professional longevity. But scientific journals prioritise novelty over rigour; a meticulous replication of somebody else’s data is unlikely to be published. And of course, the same race to patent exists in industry too. Actively disincentivizing reproduction of previous findings only adds fuel to the fire of the reproducibility crisis. 

Missing metadata vs FAIR data principles

While every peer reviewed publication includes a materials and methods section, this is really just the bare bones of the protocol. The true story behind the result lies in the experimental metadata; when, how and under exactly what conditions was this data recorded? If this metadata exists, it’s often lost in the pages of a lab book, or a scribbled calculation on the back of a glove. It’s not searchable, shareable or structured. 

In 2016, a consortium of scientists and organisations introduced the concept of FAIR data principles to combat the reproducibility crisis⁽⁴⁾. This is a set of guidelines designed to ensure that all scientific research is findable, accessible, interoperable and reproducible. Demonstrating a plan to ensure that your data will meet this criteria is now a mandatory requirement for research funding from many major UK organisations, including The Wellcome Trust, The Medical Research Council, and Cancer Research UK. 

However, despite the best intentions, without the “missing metadata” that rarely forms part of a research publication, it’s nearly impossible for another scientist to faithfully recreate an experiment without costly and labour intensive optimisation to find the “secret formula” to success. 

Could Lab Management Software be the solution?

“Digital transformation” has been a biotech buzzword for years now, and almost all companies use some level of Electronic Lab Notebook (ELN) to record data and timestamp results, or a LIMS (Laboratory Information Management System) to track their samples and inventory. Adoption of first generation lab management software is far less common in academia, but even within fully digitised biotechs with multiple software subscriptions and a clear eye on 21 CFR Part 11 compliance, experimental metadata can slip through the cracks of these siloed systems. 

Perhaps the solution could lie in a unified digital thread; a single continuous datastream that links experimental design, at-the-bench actions, recorded results and physical sample tracking? 

This is the concept behind Lab Thread, a single lab management software solution designed to connect a laboratory’s entire workflow from DNA construct design, through in-lab dilution calculations and buffer recipes, to recording results and data analysis in the ELN, and then tracking the location of the final physical sample with the built-in LIMS. By connecting every step of the workflow (and combining scientific functionality with full project management orchestration), it becomes much harder for that all-important experimental metadata to slip through the cracks. 

Once all this metadata is recorded, it’s only a small step further to take advantage of advances in AI or other analytical tools to conduct “meta-analysis at scale”. If five scientists do the same experiment, and it works 3 times out of the five; it’s no longer a question of whether this data is really valid, but rather just identifying that in the two failed replications, the incubator temperature had dipped by 2 degrees. 

Digital transformation doesn’t have to be expensive. It doesn’t have to involve robotics and whole system overhauls. Instead, it can mean quietly implementing the right  lab management software to ensure that every discovery you make is based on a foundation of traceable, verifiable and reproducible data.

‍

Are you ready to connect your lab? Try Lab Thread FREE for 30 days.

‍

Read this next: Press Release: Lab Thread announces free version for academics.

‍

References:

1. Begley, C., Ellis, L. Raise standards for preclinical cancer research. Nature 483, 531–533 (2012). https://doi.org/10.1038/483531a

2. Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets?. Nat Rev Drug Discov 10, 712 (2011). https://doi.org/10.1038/nrd3439-c1 

3. Ioannidis, J.P.A., Why most published research findings are false. PLOS Medicine 2(8): e124 (2005). https://doi.org/10.1371/journal.pmed.0020124   

4. Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016). https://doi.org/10.1038/sdata.2016.18