Making Science More Reproducible
... or why we founded Briefly
Yesterday we announced our pre-seed fundraise of $1.2m to help tackle the reproducibility crisis in science. That sounds very grand. So I wanted to break it down a little – explain why we think this company should exist, and why now is the right time to start it.
What do you think about when you read the words ‘reproducibility crisis’? Most think about academia. Papers showing that 80% of preclinical cancer papers that can’t be reproduced; publish or perish, the paper mill crisis, p-value hacking and photoshopped gel images.
A system of incentives that rewards Impact to the exclusion of everything else. A peer review process that despite being ‘core’ to the scientific process, goes unpaid and doesn’t receive the attention it deserves.
The pharma industry echoes these sentiments. A team at Amgen described ‘fundamental problems’ and ‘systemic issues’. To make things better, we’ll ‘require tremendous commitment and a desire to change the prevalent culture’.
When you start reading about reproducibility, it’s easy to get sucked into this doom spiral. That our system of science is broken and getting out of it will require some huge restructure. A revolution.
Much of this criticism is valid; even vital. But it can make attempts to solve the problem feel a little, well… hopeless. And the situation isn’t hopeless - really it isn’t! But with many big hairy problems like this, to start making progress, it helps to zoom in and look practically at what is actually happening day-to-day.
To illustrate, let’s run through an example of some everyday experiment reproducing – common to both industry and academia alike and stripped of any questions of publishing or incentive structures.
Scientist A has been developing an assay for a few months and has now become the de facto lab expert. They know all the quirks and tricks of the trade for running it and for producing those tight error bars. If someone wants that assay running, they ask Scientist A.
What happens then, when Scientist A hands in their notice to move on to their next gig? Do we expect that Scientist B could just pick up their notebook and keep the assay running? Not likely.
To hand over a scientific workflow of any significant complexity, a detailed protocol will be required – all that knowledge spelled out completely. Alongside the protocol, Scientist A and B will meet up for a few run-throughs in the lab, to catch any of those important nuances that were forgotten in the written docs. Even then, with our expert hovering over, successful reproduction can still prove elusive.
Bear in mind that in this scenario, we’re talking about the same lab. With the same reagents, the same equipment, in the same conditions. The only change being the hand on the pipette. There are no misaligned incentives here – everyone wants the experiment to work as quickly as possible. And yet… It's still hard.
We started Briefly off the back of this fairly simple realisation: progress in science is being stunted because so few of us really know what the hell each other is doing. Junior scientists are struggling to figure out how their supervisor prepped that cell culture. Data scientists are analysing and modelling datasets, but don’t know precisely how they were generated. Projects are planned and funded based on snazzy data from a published paper, but with only a few sentences describing where and how that snazzy data came from.
The exciting part is that this is a problem that we can actually do something about. Thoughtful implementation of LLMs offer us a route to structured and standardised documentation of our workflows – in a way that doesn’t place a burden on the scientists doing the work. With our experiments documented completely and consistently, we’ll reduce the friction at every handover. When our experiments don’t work, we’ll have a solid grounding upon which to ask questions of our reagents, or of our experiment design, rather than digging back through the steps that were taken.
In the longer term, this standardisation could take us even further. These protocols could directly define workflows for lab automation. Or be aggregated to help us make better decisions about how experiments should be run. Or help us track how each and every experiment evolves over time.
Fundamentally though, we’re convinced that we need to start with real transparency over how our experiments are being performed. This can serve as a foundation for tackling those other, messier problems of reproducibility. Because until we’re clear on what happened, how can we hope to understand why it happened?
If you feel the pain in communicating consistently about your protocols and experiments – we want to hear from you. Academics can use an early access version of our product for free. In a company? We’d love to partner with you to grease the wheels of your research. Either way, click the button below and we’ll be in touch.