At Capital One, DevOps is about delivering high quality, working software, faster. This means software that is reliable, secure, usable, and performant while providing value and accomplishing those important end user goals. Everything is about speed of delivery and getting that feedback. Mark Andersen is the Director of DevOps for Capital One’s Auto Finance group and the Cloud Account Lead for Capital One Auto Lending in Plano, Texas. He recently joined Cloud Academy for the webinar, Recipe for DevOps Success, to talk about some of the lessons learned from their DevOps transformation. In this post, we’ll be sharing Mark’s 10 essential ingredients for making your own move to DevOps a success.
Capital One Before DevOps
Prior to DevOps, Capital One was on-prem. Releases were inconsistent and even the smallest change required many handoffs between many teams. Manual steps were documented in spreadsheets. Setting up a new environment could take several months and, even then, there were lots of inconsistencies.
Capital One initiated their DevOps transformation over several phases. They started out with some configuration management automation tools like Chef, which got them part of the way there.
In phase two, Capital One went all-in on cloud, looking at full infrastructure automation and growing into the delivery pipelines. They started with two legacy applications and assembled a cross-functional SWAT team of developers, infrastructure, production, and support personnel who worked closely together for two months. A few professional services resources were added to address the knowledge gaps. This team focused on automating the build deployment of the infrastructure and then the application deployment. Here, they used tools like Jenkins, AWS CloudFormation, and Chef in the process.
They moved on to address applications of increasing complexity and brought on more applications over time. Eventually, the application teams were able to drive themselves, and they disbanded the SWAT team. Today, Capital One is continuing their DevOps journey by implementing robust pipelines with increased quality checks and automation.
Here is what they learned along the way.
10 ingredients for DevOps Transformation
-
Empower your developers.
The goal of DevOps is to enable developers to be self-sufficient so they can move quickly. It’s not enough that your DevOps engineers know the automation. They need to teach the agile pod and the developers how things work and how things get fixed. Developers who know more will be able to resolve issues on their own. They won’t have to wait for help or create a ticket, etc.
Mark says, “I like the analogy of treating automation like a car. You can teach the application team how to drive and how to change the oil and do some of the general maintenance, but they don’t need to know how to build the engine.” Is the app team going to understand every little nuance around cloud formation and deep networking? That’s probably a little too much for them. However, it is useful for them to understand the pipeline in how things get deployed. This way, when things break, they know where to look and understand how everything works together.
-
Be prepared to get creative.
Automating on-premises and legacy software will be challenging, and you’re going to have to employ some creative solutions along the way. Capital One has a lot of the same infrastructure that other Fortune 500 companies have, and it’s not easily automated. On-premises software usually isn’t built for the dynamic nature of how often you deploy or replace things, and it isn’t necessarily built for the cloud. You will have to make some choices in terms of how far you can get in automating and deploying some of these applications, and it won’t be perfect. That’s OK.
-
Don’t try to do everything at once.
Start out by focusing on just one or two things. You’ll experience many ups and downs on the change curve. By breaking things down into smaller batches, you can enjoy more incremental successes and keep your team motivated along the way. And, when you celebrate, do it like Dude Perfect.
-
Make sure the tools you select or build have APIs.
APIs are the enabler for the DevOps and automation journey and also for the cloud. Tools without an API will be a major barrier to automating everything and delivering faster. Even a bad API is better than no API. If you are API-first, your applications and your users can orchestrate these tools together. You can always build the UI after that.
-
Speed matters.
Long feedback loops are bad feedback loops. If you automate everything the first time, it’s going to take a really long time to get that feedback. You’re going to have to look at parallelizing everything and especially your functional tests. Functional tests that take hours do not make for a good feedback loop.
-
Don’t forget to unit test.
Don’t just focus on functional tests, which tend to be slower and more brittle. With automation and faster delivery times, there is a trend of building more functional tests at the expense of unit tests. Unit tests are fast and stable. They give your developers feedback, and you should be able to run them on your laptop or anywhere.
It’s not that functional tests are bad—just don’t favor them over unit tests. You don’t want to have the aggregate of all your dependencies affect your build and your feedback loops. Consider mocking out some of those, especially the ones that are less stable. This speeds up the process of getting feedback in your tests.
-
Translate what you’re doing into developer language.
Networking and security are difficult, and it’s even more so for developers. If you get into doing DevOps in the cloud, you’re going to get into networking. Your developers and agile pods will need training.
Take centralizing your log collection for example. Let everyone see the information and create things in the developer language so they have the information they need. Developers don’t usually understand CIDR blocks or complex networking, but they can identify, for example, that a given IP address is really the LDAP service versus the business service the app is tied to, Make information available in a language that developers can understand.
-
Don’t create a new silo.
Don’t replace your infrastructure silos with a DevOps silo. If you’ve siloed your DevOps team to just working with tickets, you’re not getting the actual flow of delivery from the agile pods if they’re waiting on tickets.
-
Focus on removing and reducing the handoffs.
How can you create self-service for the things that you can’t give developers direct access to? If you can’t give them access to create LDAP groups or security groups, give them a tool to do it the right way or solve the 80% problem. Putting everything behind a manual process will really slow developers down. Give them the tools and the access they need.
-
Don’t let great get in the way of being very good.
Deliver something, get feedback, and do more. If you’re waiting for something to be absolutely perfect, you’ll never deliver. Unicorns don’t exist (if they do, they’re really tough to find). Strive to be a horse. Accept that you will run into lots of challenges along the way. Work on getting better at first. Don’t let perfection get in the way of that.
Learn more about Capital One’s DevOps transformation in the on-demand Cloud Academy webinar, Recipe for DevOps Success.
To get started on DevOps, you can find a full collection of DevOps content in Cloud Academy Library.
