I was reminded of a lesson I learnt the hard way a number of years ago. The importance of using build artifacts in deployment.
When I first learnt this lesson, I ended up with multiple production servers failing and a supposedly highly-available, redundant website offline. One of the major package/dependency repositories had gone offline; after scaling down overnight the cluster started scaling up for the day; the new servers were passing healthchecks; but the servers were returning errors. As the problem persisted the last working servers were cycled out of production and terminated. At the time there was nothing I could do until the dependency repository came back online. Thankfully I no longer remember how long the site was offline for, but at the time it was extremely stressful.
The solution to this issue turns out to be quite easy, and in compiled languages the issue is far less likely to occur as developers are more used to using build artifacts. The simple solution is to create a deployment package with no dependencies. For scripted languages such as Python, PHP or JavaScript the deployment process (either manual or automated) only needs minor modifications to create build artifacts.
As development teams adopt best practices and stop editing code on production servers, a testing process inevitably gets implemented. If you’re not yet at this stage of development maturity then I suggest you stop reading this article and start asking your fellow developers why your current practices need to change; seriously, do it now! For the rest of you, prior to testing I’m sure you install and/or update any required dependencies. Once testing has been completed, you likely upload your code to a central location. Then you either deploy it to the servers, or trigger your cluster to perform a deployment. If you’re relying on your package manager during a deployment it is at this point that everything can go wrong. When the package repository is offline or experiencing intermittent errors your servers may fail to deploy.
To avoid this, when the deployment package is created, include all the dependencies; then, when the package is deployed, don’t run the package manager. Yes, it’s that easy. Keep your deployments completely under your own control, as long as the package manager is working during the build process, then the upgrades can get deployed; but if it’s down, the build will fail and won’t be promoted to production, saving you from the headache of your site starting to fail. When the package manager comes back online, trigger a rebuild and the deployments can resume.