I’m going to have to eat a bit of humble pie. When I was a grad student, I may have just slightly looked down on “pipeline bioinformatics”, thinking it was a subject that was boring. It clearly wasn’t as glamorous as designing new algorithms or plucking hidden bits of information out of giant data sets… I may have even thought it was something you just did as an after thought.
I was wrong.
I have to admit, now that I’ve had a taste of it, I’m enjoying it for exactly the opposite reasons: It’s a fascinating game of balancing everything you know about computers and biology all at the same time, while making sure you get the right answer consistently. It’s a cross between doing jigsaw puzzles and playing jeopardy… and I’m kinda liking it.
In order to build a good pipeline, you need infrastructure that glues all the parts together, you need planning to make sure that it has room for growth, and you need to know what constraints the pipeline will face… And, you need to be able to understand how everything from the bits of data you’re pushing through it will interact with all of the hardware on all of the machines and wires it’s going to run on. That’s no small feat – but it’s an exhilarating challenge.
While I may have thought algorithm design was the cat’s pyjamas, building a pipeline is the same resource management challenge scaled up to include a whole lot more moving parts. And, to those who manage all of those working parts, I finally grok what it is that drives you – and I am only working on a pipeline that was assembled by others, not even one of my own creation – which just increases my respect for those who have built pipelines out of nothing:
The thrill of watching data cascade through the waterfall that is the pipeline.
The excitement of having each individual piece operating in harmony, squeezing out that last bit of performance.
The fun of adding in three more pieces you thought would never fit, but making it work.
The satisfaction of knowing you managed to tame the mangy electrons that seemed so unruly before they entered your pipeline.
The reward of having someone look at the data afterwords, and learning something new from it.
Yes, pipeline bioinformaticians, I owe you an apology, your product is a magnificent work of art in it’s own right – and it is only truly completed when people are able to forget that it’s there. Cheers to you!