> Builds being slow is one of my pet peeves also. Modern "best practices" are absurdly wasteful of the available computer power, but because everyone does it the same way, nobody seems to accept that it can be done differently.
A lot of what you describe originates from lessons learned in "classic" build environments:
- broken (or in some cases, regular) build attempts leaving files behind that confuse later build attempts (e.g. because someone forgot to do a git clean before checkout step)
- someone hot-fixing something on a build server which never got documented and/or impacted other builds, leading to long and weird debugging efforts when setting up more build servers
- its worse counterpart, someone setting up the environment on a build server and never documenting it, leading to serious issues when that person inevitably left and then something broke
- OS package upgrades breaking things (e.g. Chrome/FF upgrades and puppeteer using it), and a resulting reluctancy in upgrading build servers' software
- attackers hacking build systems because of vulnerabilities, in the worst case embedding malware into deliverables or stealing (powerful) credentials
- colliding versions of software stacks / libraries / other dependencies leading to issues when new projects are to be built on old build servers
In contrast to that, my current favourite system of running GitLab CI on AWS EKS with ephemeral runner pods is orders of magnitude better:
- every build gets its own fresh checkout of everything, so no chance of leftover build files or an attacker persisting malware without being noticed (remember, everything comes out of git)
- no SSH or other access to the k8s nodes possible
- every build gets a reproducible environment, so when something fails in a build, it's trivial to replicate locally, and all changes are documented
Right but build environment are awfully stupid about it. Re-downloading deps when they did not change is utterly wasteful that achieves nothing, same as re-compiling stuff that did not change.
> broken (or in some cases, regular) build attempts leaving files behind that confuse later build attempts (e.g. because someone forgot to do a git clean before checkout step)
But thanks to CI you will never fix such broken build system!.
Not a fan of it, but I have some experience behind why this happens!
Download gets corrupted (randomly) at some point. Corrupted file gets stuffed into the cache. Now, since everyone is using the cache, everything/everyone is broken because the cache is ‘always good’, and the key didn’t change!
So then, someone figured it out and turned off caching - and it fixed it.
And of course everyone has time to setup a reproducible test case for this random data corruption bug, and can wait for the language tooling to get fixing while all their builds break….
Or they just turn off caching and forget about it.
Not if your pipeline is decent. Parallelize and cache as much as you can.
The only stack that routinely throws wrenches into pipeline optimization is Maven. I'd love to run, say, Sonarqube, OWASP Dependency Checker, regular unit tests and end-to-end tests in parallel in different containers, but Maven - even if you pass the entire target / */target folders through - insists on running all steps prior to the goal you attempt to run. It's not just dumb and slow, it makes the runner images large AF because they have to carry everything needed for all steps in one image, including resources like RAM and CPU.
Heh. docker.io package on Ubuntu did this recently, whereby it stopped honouring the "USER someuser" clause in Dockerfiles. Completely breaks docker builds.
No idea if it's fixed yet, we just updated our systems to not pull in docker.io 20.10.25-0ubuntu1~20.04. or newer.
Indeed: Optimize to reduce developer time spent on bad builds first.
One of the rules of this approach is to filter all extraneous variable input likely to disrupt the build results, especially and including artifacts from previous failed builds.
You’re the perfect example of the cranky experienced developer stuck in their ways and fighting against better solutions.
Most of the issues you’ve described are consequences of yet more issues such as not caching with the correct cache key.
I argue that all of the problems are eminently solvable. You’re arguing for leaving massive issues in the system because of… other massive issues. Only one of these two approaches to problem solving gets to a solution without massive problems.
A lot of what you describe originates from lessons learned in "classic" build environments:
- broken (or in some cases, regular) build attempts leaving files behind that confuse later build attempts (e.g. because someone forgot to do a git clean before checkout step)
- someone hot-fixing something on a build server which never got documented and/or impacted other builds, leading to long and weird debugging efforts when setting up more build servers
- its worse counterpart, someone setting up the environment on a build server and never documenting it, leading to serious issues when that person inevitably left and then something broke
- OS package upgrades breaking things (e.g. Chrome/FF upgrades and puppeteer using it), and a resulting reluctancy in upgrading build servers' software
- attackers hacking build systems because of vulnerabilities, in the worst case embedding malware into deliverables or stealing (powerful) credentials
- colliding versions of software stacks / libraries / other dependencies leading to issues when new projects are to be built on old build servers
In contrast to that, my current favourite system of running GitLab CI on AWS EKS with ephemeral runner pods is orders of magnitude better:
- every build gets its own fresh checkout of everything, so no chance of leftover build files or an attacker persisting malware without being noticed (remember, everything comes out of git)
- no SSH or other access to the k8s nodes possible
- every build gets a reproducible environment, so when something fails in a build, it's trivial to replicate locally, and all changes are documented