6. Reproducible pipelines¶
6.1 The idea¶
If you copy a number from your analysis into a paper by hand, sooner or later they will differ. So store each thing once, and generate the rest.
data/figures.json ---> analysis code ---> numbers.tex ---> paper.pdf
\--> tables and figures
\-> web pages and slides
6.2 Rules that prevent most errors¶
-
Raw data is read-only. Scripts read it and write elsewhere.
-
Numbers in text are generated. A script writes them as LaTeX commands, for example.
-
Equations live in one tested function. Everything calls it.
-
One command rebuilds everything. It might be
python build_all.pyormake. -
Each output records its source, such as a commit or a checksum.
-
Generated files say so, in a comment at the top, so that nobody edits them by hand.
-
Paths are relative.
C:\Users\me\dataworks only for you. -
Random steps have a fixed seed, so that results repeat exactly.
-
The environment is recorded, in a
requirements.txt,environment.ymlor container.
6.3 A worked example¶
My pamphlet project has one LaTeX source and one data file. One command builds the website, the PDF, a report, slides and a calculator from them, then checks that they all agree.
When I switched the pamphlet to shared numbers, a check confirmed that the PDF's text hadn't changed at all. That kind of proof is what makes a large change safe.
6.4 Tools for larger projects¶
| Tool | Good for |
|---|---|
make, Snakemake, Nextflow |
Rebuilding only what changed |
Conda, venv, uv |
Python environments |
| Docker, Apptainer | Whole environments, including on HPC |
| DVC, DataLad, Git LFS | Versioning large data |
Start simple. A short script that runs each step in order goes a long way.