Blixhavn.dev

Continuous Deployment - the holy grail of DevOps?

Øystein Blixhavn -

Continuous Deployment is a development strategy central in the DevOps world, and I have for a long time considered it the holy grail of DevOps and agile development. Originally coined by Timothy Fitz in 2009 (and not to be confused with Continuous Delivery), it describes the process of having committed code moved directly to production through an automated process. It has been implemented with success in several large organizations, such as Facebook, Microsoft and IBM.

To clarify: The very similar term Continuous Delivery is a broader strategy involving automation of the pipeline from committing code, to production deployment. It may include Continuous Deployment, but most often, the last step of deployment is manual, and only done after manual acceptance testing in a staging environment.

While a lot of the tools and workflows commonly associated with DevOps are relatively straight forward to implement and set up - at least in theory - Continuous Deployment seems to be another story. A simple concept in and by itself, it requires a lot from the surrounding tools and the development team, and I have long struggled with even considering it to be feasible. This blogpost is my travel towards a better understanding of what Continuous Deployment entails, and whether it actually is the holy grail of DevOps.

What is needed?

I am beginning to understand that the reason why Continuous Deployment seems unreachable to me, is that its requirements are hard to define. It is not achievable simply by following a checklist, and spending minimal effort to implement the items on it. Instead, you need to focus on reaching the capabilities that are provided with each improvement you add to your workflow or pipeline. This will help you focus on the outcomes - why you do what you do, which will help you see the next steps towards a more productive development process.

Automation

As Continuous Deployment is to be done automatically as the last step in a long pipeline, all of the subsequent tools need to be automated. In essence, you need a solid foundation of the whole toolchain supporting automation of every step of deployment:

  • Infrastructure and configuration - tools like Puppet, Ansible and SaltSack sets up execution environments based on configuration files that can be maintained in version control.
  • Reliable tests - you need a test suite that you can trust. This topic should have its own blog post (and probably has plenty), but a good way to get started is in my opinion: focus on tests for critical paths, and write regression tests when fixing bugs. For testing legacy software, consider the "Golden Master" technique.
  • Continuous Integration - building your software, running automated tests, and making sure it compiles and plays nice with your production environment.
  • Deployment - Even without Continuous Deployment, tools and/or scripts should be used to ensure that the whole deployment process is deterministic and automatic.
  • Comprehensive production monitoring - while this should be part of any DevOps setup, it is a crucial component for Continuous Deployment - as a replacement for manual acceptance testing in a staging environment. No matter how comprehensive your test suite is, production metrics will always be valuable to pick up additional errors and performance issues in your code, making it possible to rectify issues without involving users in the feedback loop.

The main idea is that all of these bullet points need to be of a sufficient quality, so that you can confidently say you trust the system to tell you if the quality of your software deteriorates, or if there are any critical failures.

Feature Flags

It baffles me that I have found very little discussion on feature flags with regard to Continuous Deployment, when it seems to me one of the biggest hurdles to overcome. I am convinced that for software above a certain size, you cannot have Continuous Deployment without using feature flags. You will inevitably end up in a situation where some of your code should not be exposed or used until a later date when the remaining part of your feature is ready for release. This introduces its own suite of challenges with regards to cooperation, testing, branching strategies, and increased software complexity. However, it does also give you some very interesting options, such as randomized A/B testing, early access, kill switches, opt-ins, and incremental rollouts. To read more, visit http://featureflags.io/.

Conclusion

While Continuous Deployment seems like a great milestone for your automation efforts, it might be better to keep your focus on lower hanging fruits. Research indicates that in order to stick, Agile Software Development (ASD) or DevOps capabilites need to grow organically and iteratively, so that it is properly anchored in the culture of the team(s) and company implementing it. Place one brick after the other, and don't rush towards Continuous Deployment as a sort of trophy. If your deployment and operations automation is robust and dependable, consider testing how feature flags could be implemented in a way that fits with your development style.
With this in place, Continuous Deployment might be a good step forward.