Every semester I teach best practices around build pipelines, and every semester someone mentions SolarWinds, and if I, as a company, wanted to set up an entirely independent build pipeline that was entirely independent of the rest of my infrastructure and was managed by different people so I could build in two places and verify binary outputs were identical, how would I do that today? (Assume my build is already reproducible, let's not complicate things)
@mjg59 This triggered a lot of thoughts from me, and more than would fit in a post here, so I finally fired up the blog: https://bmitch.net/blog/2025-03-20-reproducible-builds/
@bmitch Interesting writeup, which I largely agree with.
But if I may join in with @bjh21 's near-shitposting, Debian has solved these problems.
Source code is signed, admittedly mostly in tarball form; git based approaches are disappointingly not yet widely used.
Debian package builds are 100% hermetic - the dependencies are those installed in the build environment. For modern curl|bash
-ware, Debian have gone to war against, and defeated, the language-specific package managers. (The battlefields are a cratered charnel house.)
As Ben notes, there are already multiple rebuilders of Debian packages. The rebuilders don't need to worry so much about malicious inputs: there are no external inputs.
Debian's QA systems already check reproducibility.
FTOAD, this probably isn't a very useful answer for @mjg59.
@Diziet @bjh21 @mjg59
As a Debian user for several decades now, I continue to admire the work done by them. What they have now is a good detection system for compromised builders.
I propose moving from detection to prevention. The verification of a reproducible build should be run as part of the package installer at each endpoint. That will require 100% of packages to be reproducible (they're getting close) and rebuilders to publish their attestations to package repositories.