Methods need a makeover
Maybe notebooks aren’t the best way to document science
Last time, I discussed that our lack of focus on the how of experimental biology is slowing our progress – blocking us from getting value from the mountains of biological data we’re generating, and adding frustration and inconsistency to life in the lab.
This ‘how’ contains a lot of things – experimental design, lab process, automation, statistics… but here I’m going to focus on the sexiest of them all, and the one nearest and dearest to our hearts at Briefly: documentation.
How exactly did you run that experiment?
In science, but especially in biology, every change to our experimental process can result in changes to our precious data. If we don’t document these changes fully, how can we be sure our experimental tweaks aren’t responsible for what we’re seeing in the data? And how can we expect someone else picking up this method to get to the same result?
I appreciate this is rather, erm, basic. In science, it is important to control variables and write down what you did. Welcome to high school! And yet, glance over at the methods section of any Shiny High Impact Journal and discover that, well, we don’t.
Complex 1000-word protocols are routinely boiled down to a single sentence. "The cells were transfected.” We’re asked to either fill in the gaps ourselves, or send the author an email.
Take a look at scientists’ lab notebooks and you’ll find the completeness and consistency of documentation varies wildly, depending on the individual, the lab, the company, whether the experiment worked or it didn’t…
Challenges like these have pushed some companies to standardise and structure their lab processes – this currently being the only way to get complete control and visibility over how their experiments are being run.
So why don’t we just document more?
It’d be easy to stop here. To conclude that we have a cultural problem in science. To solve it, we need to establish standards and structure, improve education, and change incentives… But we’re not so sure.
Completely and consistently documenting every experimental detail is excruciatingly difficult. There is the pure volume of stuff to write, sure, but the scientist must also figure out where to stop - information that is obvious to them may not be to someone from a different lab or a different background.
Not only is creating this documentation difficult, but truly complete protocols generally aren’t helpful for the scientist planning and running their own experiment.
Let’s look at Nature Protocols, the gold standard. Each and every step is painstakingly recorded. Thousands of words. Marvels of methodology. This is what we should all be striving for… Isn’t it?
These documents may be helpful for a scientist coming at a technique completely blind, but they simply aren’t to the scientist who wrote them – quite the opposite. Critical details and variables are buried in an avalanche of text. Difficult to summarise, difficult to follow, frustrating to adapt and edit. Scientists simply don't have time to create docs with such limited value to their day-to-day work.
Naturally then, both in industry and academia, when faced with a deadline and another experiment to run, the vast majority will write up what they reckon is important, not everything. And next year, many will find themselves re-running their experiments and struggling to remember exactly how they did it last time.
Pestering scientists will not solve this problem – they already understand it, but they have to make tradeoffs.
Structured science
So how do we get ourselves out of this mess? As I touched on earlier, the approach many Techbio companies are taking is to build structure around their science. They take their critical lab workflows, then standardise, track, and automate the hell out of them.
This means constructing rigid databases around wet lab processes. Experimental variables are predefined and tracked, while the rest of the workflow locked down in standard operating procedures and automation.
This approach can work, but it’s limited:
It’s expensive to set up: Companies must build digital infrastructure around their workflows. This means hiring data engineers, integrating deeply with wet lab and computational biologists, and building culture and systems from the ground up.
It’s difficult to maintain: These systems can be brittle. Wet lab workflows in biology are rarely actually standardised. We’re not building cars here. When a workflow is adapted to a new project, all those databases, protocols and automation need updating too.
It’s inappropriate for early R&D: Less established workflows are off limits – they change too much for this kind of standardisation and infrastructure to be worthwhile or practicable.
These problems make this approach a non-starter for academia and smaller companies, and extremely tough to implement at older companies where systems and culture are already established.
A common language
We think there is a better way – a common, structured language to describe lab work. This language should make intuitive sense to scientists; it should be flexible. It should accommodate their latest experimental design while keeping track of all those pesky details.
To get there, we need software that translates what we have today into this new language. Taking care of the formatting automatically, suggesting forgotten details, keeping track of all those calculations and dependencies, while helping find relevant methods to learn from that others have run before.
By introducing structure, we unlock different ways of viewing our experiments. When we’re setting up a new method from scratch, iterating on it for the 27th time, calculating a master mix composition, or running it in the lab – we care about different things. We should see the information we actually need to help us get our work done.
Where this can take us
By creating this common language, we can begin to truly build on top of, adapt and directly compare our experiments with others.
Imagine running your experiment, and getting an alert that someone in your department already ran it 2 years ago. Or even that someone is running something similar right now in Kyoto – maybe you could share some ideas and moral support?
How about having access to the combined wisdom of everyone who has run your method before – choosing the enzyme they used, and avoiding that dodgy one that would have sent you down another troubleshooting rabbit hole.
Imagine how much more efficient and reproducible our science could be if we stopped having to constantly reinvent the wheel.
This is the future we’re building towards at Briefly.
Are you also frustrated by how inefficient and messy doing science is today? We’re creating a place to meet and figure out how we can make things better. We can’t wait to meet you.