All software teams strive to build better software and ship it faster. That’s a competitive edge required to survive in the Age of Software. DevOps is the best methodology to leverage that competitive advantage, ultimately allowing practitioners to accelerate software delivery and raise quality levels. Some are skeptical of this claim or dismiss it outright at their peril. Research demonstrates that a DevOps cultural shift can elevate IT performance to Michael Jordan-esque levels. The first step is understanding that velocity and quality are related in an unexpected way.
This article covers the basics of scaling velocity and increasing quality with DevOps. To dive into DevOps and build skills with the tools that matter — Puppet, Docker, Chef, Ansible, and more — check out Cloud Academy’s DevOps Training Library.
The Death of Bi-Modal IT
A direct relationship between velocity and quality sounds counterintuitive at first. On its face, it seems that shifting time and resources into improving velocity takes time and resources away from quality focused work — like bug fixes — or causes teams to ship poorly written or tested code. I think this stems from framing IT using the project management triangle of scope, cost, time, or the “cheap, fast, good — choose two” variation. Thinking of velocity and quality like this is a logical straight line. This perspective, along with shoddy agile practices, reinforce misconception that velocity and quality are mutually exclusive, or in other words: choose velocity or quality. This is the bi-modal IT worldview. A quick survey of successful companies shows this is not always the case. Companies like GitHub, Netflix, and Amazon have high velocities and stable production environments. Why is that?
Software development leaders set out to answer that question in 2018. They published their findings in Accelerate: The Science of Lean Software and DevOps, and their findings are clear: Continuous delivery and continuous improvement led to higher velocities, lower MTTR (mean time to resolve), less frequent failures, and ultimately more profitable businesses (and even happier employees). Moreover, their research clearly refutes the bi-modal IT worldview. Martin Flower,the author of the bi-modal IT link, published in the foreword1:
“This huge increase in responsiveness does not come at a cost in stability, since these organizations find their updates cause failures at a fraction of the rate of their less-performing peers, and these failures are usually fixed within the hour. Their evidence refutes the bimodal IT notion that you have to choose between speed and stability—instead, speed depends on stability, so good IT practices give you both.”
Adhering to bi-modal IT worldview in today’s landscape is akin to being an IT flat-earther. Fowler is not alone. Jez Humble, co-author of Accelerate and author of the original book on continuous delivery, echoed Flower’s sentiment in his 2016 blog post. DevOps overcomes this limitation and offers a three-legged framework to scale software delivery and raise the quality floor.
The Three Ways of DevOps
The DevOps Handbook, created by some of the same authors as Accelerate, describes the three DevOps practices. Put simply they are: continuous delivery, continuous feedback, and continuous improvement. Each practice establishes a feedback loop that reinforces the others. Some may think DevOps is just about continuous delivery, but that misses the point and hinders one’s ability to achieve top-tier velocity and quality levels. A successful DevOps culture involves all three.
Much has already been written about implementing DevOps practices like continuous delivery and building telemetry systems. If you want the high-level gist, then step through the DevOps Fundamental Learning Path, or focus specifically on continuous delivery or building a DevOps culture. I’ve also written a blog on measuring software development performance with the four metrics provided by Accelerate.
Now assuming you’ve adopted the zen of DevOps, it’s safe to focus on practices that tend to be glossed over that drive velocity and quality.
Continuous Delivery Begins with Trunk-Based Development
Continuous delivery (and the expert level cousin continuous deployment) steal the spotlight. This causes teams to wander in the muck of selecting technologies where everything branded as DevOps requires containers, like Kubernetes, or even microservice architecture. These technologies are simply irrelevant implementation details. Businesses can do continuous delivery for printers or mainframe software (there’s a lovely quote in Accelerate on just this), but it all comes down to mindset.
Many teams think incorrectly about continuous delivery. Thinking that continuous delivery means automated deployments is wrong. Thinking that continuous integration means automated testing is wrong. Thinking that continuous delivery means automated topic-branch deploys is also wrong. These are related to continuous delivery, but they miss the point. This is where many teams fail and miss an opportunity to improve velocity and quality. Continuous delivery begins with trunk-based development.
Here’s how the continuous delivery process should work:
- Code is committed to the master (a.k.a. “trunk”).
- Automated tests ensure the code is fit for production deployment (continuous integration as used colloquially today).
- For continuous deployment: Code is deployed to production.
The high-level idea is that the master (or “trunk”) should be green and deployable to production at all times. This implies that changes to the master should happen in small batches that are easier to develop, review, test, and deploy. This also implies that branches should be small and short lived. Trunk-based development does not eliminate the branches. Instead, it changes a team’s relationship with branches. This practice underpins top-tier velocity but it’s met with friction and controversy.
The resistance comes from moving away from an individual-focused model to a team-focused model. In an individual-focused model, developers work independently on their own branches for days, weeks, or even months. This enables more individual control and productivity (which developers love!) but requires merging everyone’s work to see the smallest part of the complete system leading to increasingly longer feedback cycles. A team-focused model uses a long unbroken straight line of development. There’s not much to understand, so commits are simple, but each commit can bring the entire team to a screeching halt. Conversely, feedback cycles may be much faster — this model is similar to successful one-piece flow manufacturing models used in world leading factories like BMW or Toyota).
In my view, the monorepo development practice paired with trunk-based development is the best way to realize the goal put forth in the DevOps Handbook:
At the end of each development interval, we must have an integrated, tested, working, and potentially shippable code that has been demonstrated in a production-like environment, created from the trunk using a one-click process, and validated with automated tests.
If you are not using trunk-based development — and it’s likely you aren’t — then this a perfect starting point to improve velocity and quality. Don’t simply waive trunk-based development away either because your process is so complicated or there’s some technical curveballs in your pipeline.
The DevOps Handbook tells the story of Gary Gruver at HP. His team adopted trunk-based development and built a virtual printer platform to support automated test system for printer firmware. His team saw massive gains in productivity, quality, and even developer happiness. If his team can achieve these results with printers, then you surely can do it with your web application or mobile app.
Gruver admits trunk-based development requires a big mindset shift. Engineers thought trunk-based development would never work, but once they started using it, they could not imagine going back.
Experimenting with new ways of working falls into the third way of DevOps: continuous improvement. The technical and business landscape is changing continually, and you must change too. More importantly, your competitors are doing it. How quickly you can learn and improve is a key competitive advantage that also directly correlates with improved velocity and quality.
An X-Ray into the Value Stream
IT Revolution Press released Project to Product by Mik Kersten in late 2018. Kersten argues that businesses most adopt a product-based value steam to succeed in the age of software. Kersten applies the DevOps practices of flow, feedback, and improvement to create the Flow Framework that measures a product-based value stream. The Flow Framework unifies the entire business value stream into four metrics (which feel like cousins to those four metrics in Accelerate). The Flow Framework makes direct use of the feedback principle to make key value stream metrics accessible to anyone in the team.
Implementing the Flow Framework requires a bit of technical plumbing, but the results may be worth it. We’ve seen before that aspects of an engineering process cannot be improved without being measured. Once it’s measured, it may be used as a basis of objectives and key results. Plus, these metrics can change how your business approaches IT and its business itself. It’s these type of experiments that move teams from good to great.
No-End Game for Velocity and Stability
DevOps mandates a no-end state. Each practice is a feedback loop, and effective processes will find and identify increasing faint failure signals and experiment with solutions. This will drastically raise the quality floor over time. Large organizations can spread the learnings across other units (similar to what Google did with their site reliability engineering team). If you — like Martin Flower and Jez Humble — accept that good IT practices provide both speed and velocity, then you’ll achieve both with the right attitude and a commitment to experimentation and learning.
