Deployment and release
In enterprise organizations, releases are the final step of a long process that, historically, could take months — or even worse — years. Small companies and startups aren’t immune to this. Minimum viable product (MVP) over MVP and fast iterations could lead to technical depth and longer release life cycles over time.
When deployment and release are highly coupled, deployment operations could be really painful and dangerous. If a software engineer sees his feature running on production for the first time after the deployment, coinciding with the release, it is highly probable that something will go wrong.
Hotfixes and rollbacks will be often part of the job with a lot of pressure and stress for everyone.
Deployment and release at Cloud Academy
At Cloud Academy, we experienced similar issues with large and not completely predictable deployments and long releases lifecycles with stakeholders unable to see the outcomes until after the development was over. Over the years, we implemented DevOps principles to improve our processes. We adopted radical technical changes such as introducing Docker in every component of our environments, improving local and test environments reproducibility as well as portability.
Our team also started migrating from a monolithic architecture to microservices. These actions had a huge impact on the entire organization and also served as the prerequisite for the topic of this article: How to let software engineers deliver value as soon as their code is ready to be published and, at the same time, release functionalities based on business needs without strong dependency.
In this article, we’ll explain why it’s important to decouple deployment from release. If you’re just getting started with DevOps, you’ll want to get started with Cloud Academy’s DevOps Playbook – CI/CD Tools and Services. This learning path is a hands-on guide to DevOps adoption and implementation. It includes video courses, dedicated demo sessions to put DevOps tools into production, and five hands-on labs to apply what you’ve learned in a live AWS console.
On-demand test environments
Fast and early feedback is the key to release a stable feature that meets expectations. Exposing a feature to product owners (POs) and stakeholders only from the stage environment as we used to do, simply doesn’t work because you can’t get valuable feedback a few hours before the release date. In addition to that, the cost of introducing changes at the final stage of the development process frequently is very high and will introduce last-minute fixes that are poorly implemented.
Test environments enable software engineers and POs to host a new feature in a dedicated environment to reduce the feedback loop. This gives you the opportunity to share internal upcoming features in the early stages and test them safely. Test environments are a great tool that can be used for various purposes, including having better feedback at the right time a higher chance to meet everyone’s expectations.
Unfortunately, scary deployments are still there and must be coordinated with the release date. A simple mistake during the final phase could lead to signal delays of the release because you can’t totally predict how the production system will work with the new feature that will be shipped. Fortunately, software engineers will have more opportunities to see how the feature behaves thanks to the test environment, but the production deployment is still a potential and dangerous point of failure.
On-demand test environments at Cloud Academy
To start using effective test environments, we implemented a few automations to have a new environment up and running in minutes. We decided to introduce a few persistent test environments that are shared and available for every software engineer and product team. This included on-demand test environments dedicated to projects that last for longer periods that require frequent integrations between frontend and backend, stakeholders’ feedback, and reviews.
Shared and persistent test environment have been used for different purposes:
- Share features starting from early stages with the project manager and stakeholders to achieve a faster feedback loop relaying on a persistent environment accessible in the corporate network.
- Perform heavy test operations that can affect the service or the database performances in an isolated environment that can fail safely.
- Often frontend and backend teams need an integration environment to test their features in a safe environment that can be built with work in progress branches to perform manual testing to identify immediate improvements and potential bugs.
- Implement the first minimum viable product (MVP) of demo environments on the top of test environments that allow to provision a self-contained environment to play with during demos without affecting production environments.
Feature flags based on the environment
Using feature flags is a technique to wrap features and enable or disable them under certain conditions, such as based on environment variables. We started having the opportunity to test features only on stage for a limited period before releasing them (without completely blocking the continuous delivery pipeline for everyone else in the team). Now we can release safe features continuously and take the opportunity to enable them and test them on stage as long as we wish before releasing them.
Feature flags based on the environment are undoubtedly beneficial. However, the final release is still a point of failure and stress for everyone involved because you can’t monitor and see how features behave on production before the release date. The main concern is clear: Releasing a feature in a never-seen environment and moving from 0 users to 10,000 almost immediately creates a major issue. The feature toggling solution completely avoids this issue.
Feature toggling
Several organizations are using dark launching to release features safely by hiding them completely to the customers. This can be achieved with feature toggle services that allow wrap features and enable them based on multiple conditions: environment, user base segmentation, etc. A feature can be released on production turned on only for internal users and can be tested and monitored without affecting the entire user base. Moreover, you can disable a feature that is affecting performances or causing issues to users almost immediately, without rushing for a dedicated hotfix.
Decoupling deployment from release is an ongoing process. In some cases, the development effort to introduce feature toggling is too high, and you can’t afford it or you’re dealing with non-reversible operations that cannot rollback easily. This is because we need still test environments and other strategies to have smaller and more predictable releases that mitigate the unknowns.
Feature toggling at Cloud Academy
When decoupling is applicable, we noticed that the overhead introduced with feature toggling or the time spent with test environments in the deployment process is definitely worth and acceptable. Here are some of the benefits that we experienced:
- The feedback loop is fast and produces great outcomes even in the early phases.
- Software engineers can deploy features safely as soon as they are ready.
- Deploys are performed multiple times every day and are rarely highly coupled with features releases.
- Feature releases can be scheduled in accordance with customer success, marketing team, and everyone else involved.
Summary
Decouple deployment from release leads to a chain reaction composed of a fast feedback loop, frequent and small deployments, and safe releases. Working with complex systems is difficult and in some cases unpredictable: decoupled deployments let Software Engineer fail fast and safely. The final goal is to reduce unknown variables and give the freedom to innovate without friction.
At Cloud Academy, we’ve discovered that decouple deployment from release following a bottom-up strategy was the best approach for us, we started solving small and very technical issues, and we ended up changing our processes while progressively benefiting from the decoupling.
