Continuous Deployment is the pinnacle of high-performance software development. Continuous deployment teams deploy every commit that passes tests to production, and there’s nothing faster than that. Even though you’ll see the “CD” term thrown around the internet, continuous deployment and continuous delivery are not the same. Continuous delivery means that every commit is deployable to production, not that it is deployed. Continuous deployment sits above continuous delivery, since code may only be continually deployed if it’s continually delivered.
I’ve written in my previous blog posts that achieving continuous delivery is no small task. And achieving continuous deployment is also no small task. Working with continuous deployment is more difficult because it requires more discipline and forward thinking. I’ll put it in other words: it’s hard. So why put in the effort into moving from continuous delivery to continuous deployment? Three answers: velocity, quality, and reliability. All three are positive, so where are the trade-offs?
Continuous deployment is like all things in software: There are good and bad aspects that teams must internalize before deciding one way or another, but there’s far more good than bad in this case.
Continuous Deployment is Great for Value Stream Metrics
My last post covered four IT performance metrics. They are:
- Lead Time – time from commit to production
- Deployment Frequency – how often deploys happen
- Mean Time to Repair (MTTR) – time to resolve production issues
- Change Failure Rate – percentage of deploys that cause production issues
I recommend you read the post for more context. Continuous deployment maxes out those metrics in two aspects: technical and organizational. First, let’s consider lead time.
Continuous deployment requires extremely fast deployment pipelines. It would be infeasible if deploys took hours. This requires teams to do everything technically possible to improve their deployment pipeline’s efficiency with activities such as parallelizing test suites and improving infrastructure ramp-up times. On the organizational side, this means the team has truly internalized working in small batches. Small batches are beneficial for a multitude of reasons, primarily that they are easier to develop, test, deploy, verify, and ultimately iterate on.
Adopting small batches naturally brings down MTTR since they are less likely to fail in production due to their simplicity. If they do fail, then it’s far easier to identify the problem. This fact combined with fast lead times yields fast MTTR.
Continuous deployment also implies a low failure rate— it would be impossible otherwise. This metric also correlates with the project’s overall quality. Over time, the team would have patched regressions in the deployment pipeline and built stronger telemetry to detect increasingly faint failure signals. The team is also more likely to leverage feature flags to further mitigate risk in production.
Continuous deployment guarantees top-tier performance metrics. This provides the business with faster cycle times, which is passed onto their customers as more frequent releases. That covers velocity. The relevant associated technical and organizational practices ensure issues are quickly detected and resolved (if they enter production at all). That covers reliability. Deployment pipeline optimizations and continuous learning and experimentation bake quality into the process. That covers quality. Again, every software team wants to max these out.
There’s more to it. Continuous deployment increases an organization’s scaling capacity via automation. New hires may be given their environment, credentials, and deploy a commit to production on their first day. That would not be possible in an environment with more manual steps. Accelerate also finds that increased deployment frequency correlates with happier employees, higher job satisfaction, and lower burnout rates. Moreover, The DevOps Handbook and Accelerate find that businesses with high IT performance metrics generate more revenue and profit. Continuous deployment make workers happier and businesses more profit. Can anything else claim that?
So where’s the trade-off?
Changing Culture is Hard Work
Continuous deployment’s trade-off off is that it’s a global optimization — not local optimization.
Imagine your company for a moment. Where do you fit in the picture? Are you a manager? CTO? Engineer? CEO? Support Engineer? Product manager? Now think of the artifacts you interact with and produce as part of your daily work. You may take in requirements to write user stories or translate user stories into code using input from the product manager, artifacts from the design team, and operations requirements from the SRE team. Each of these artifacts has their own lifecycle. Also, each of those artifacts has a different business priority and they’re all flowing through business at the same time. Trace out the lines in your head. What do you see? Do you see a straight line or something more interconnected?
My guess is that you’ll see something more interconnected like an air traffic map. Software delivery is a networked activity. Software teams will find work-arounds and reroute their work to account for bottlenecks or blockages. This doesn’t mean bottlenecks are non-existent. It means that each member in the network must adopt principles and practices that optimize for the previously stated four metrics. This is only possible through culture and propagating the culture to everyone in the organization. This is a difficult problem and the biggest trade-off with continuous deployment. It’s a cultural outcome more than a technical achievement because replicating culture outcomes is extremely difficult.
Changes must occur within the individuals, management, and leadership. Focusing on continuous improvement requires a new awareness. Here’s a wonderful quote from The DevOps Handbook:
“Doing this further demonstrates that we value improvement of our daily work more than the daily work itself.”
This perceptual shift is a difficult pill to swallow since it impacts every step in the value stream. It’s not enough to release a bug fix. The next step is to identify why this type of regression occurred, and then invent a new type of testing to ensure it never happens again. Then the process repeats. This is a hard sale especially if automated testing is not well established. The same thinking applies in more contexts, and it does not stop with testing.
A cultural shift may also create fear. Consider two cases. Chaos Engineering introduces faults into production systems to learn why the system fails and how to repair it. It’s safe to imagine the first manager had one of two reactions: “Whoa! That’s a crazy idea. Let’s try.” Or “You want to do WHAT?!” You may have encountered one or both people in your career. Continuous deployment prefers the former.
Here’s another scenario. An engineer proposes that the team deploys twice a week instead of each month. The manager rejects the proposal because “it would impact stability and take away time from bug fixing.” Unfortunately the manager in this case has fallen into bi-modal IT trap. The trap assumes that speed and stability are mutually exclusive. That’s wrong. Continuous deployment provides both, assuming teams put in the work.
These two scenarios are presented from an engineering-centric world view. Continuous improvement doesn’t respect borders. It expands to encompass the entire value stream. Product managers will have to think differently about how to measure and verify the results from their decisions. Continuous deployment is only possible with fine-grained telemetry that automatically identifies negative impacts on business metrics. The first questions facing the organization are:, What’s their current state? Are key features tracked? Does everyone have access to a dashboard that shows green-amber-red status on features and current KPIs? Is anyone hooked up to pager on significant drops on those KPIs? If there is no existing technical infrastructure, then how and when will it be built? What priority is that work compared to new features?
An organization’s capability lies in their leadership. If leadership does not buy into the capabilities required for continuous deployment, then any adoption or transformation is doomed to fail. Sometimes everything hinges on convincing the board. However, I do recommend Accelerate if you need to make a data-backed argument to management. Moreover, leadership must continually back and prioritize technical work and demonstrate good behavior over the long hall. Continuous improvement mandates a no-end state — only the continued experimentation and learning.
Think Local, Act Local
I struggle to find a technical drawback to continuous deployment. However, it’s really hard to achieve due to the sheer scope of the solution. Organizations must adopt the ideas of flow, feedback, and continuous improvement to even have a shot. Culture changes are hard. However, continuous deployment—the outcome of flow, feedback, and learning—is a fantastic way to build software.
The ideas of flow, feedback, and learning are important to me. I wrote a while back in evaluating a new engineering job that absence of these cultural norms is a red flag. It’s an even bigger red flag since learning Accelerate’s stats about burnout and happiness. If these ideas fit your approach to software development, then you don’t have to stick around in a job that doesn’t fit you. Just because your current culture is off doesn’t mean your next job will be too. Continuous delivery requires a large amount of tech-savvy skills, and Cloud Academy has the resources to teach you ideas and technical skills required to land your next job.
Take The DevOps Playbook – Moving to a DevOps Culture learning path if you’re interested in the ideas and cultural changes. On the other hand, The CI/CD playbook learning path will teach the technical skills to build a deployment pipeline.
