- [Instructor] Hi, and welcome to lecture two, the core DevOps principles. In this lecture, we will learn to recognize and explain the three ways approach, the theory of constraints, chaos engineering, and the notion of learning organizations. The three ways are introduced in The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win, written by Gene Kim, Kevin Behr, and George Spafford. The novel is based around Bill, who's an IT manager at Parts Unlimited. It's Tuesday morning and on his drive into the office, Bill gets a call from the CEO. The company's new IT initiative codenamed The Phoenix Project, is critical to the future of Parts Unlimited, but the project is massively over-budget and very late. You get the picture. In the three ways, the first is flow, understanding and increasing the flow of work from dev to ops. The second way is feedback, creating short feedback loops that enable continuous improvement from ops to dev. And the third way is continuous experimentation and learning, creating a culture that fosters experimentation, taking risks, and learning from failure. For the third way to succeed, we need to understand and accept that repetition and practice is a prerequisite of mastery.

 

- And the three ways are intended to be a set of principles from which can derive all of the DevOps patterns from. So the first way is all about flow, as you go from left to right in the value stream, from development to operations. And the goal is to have work go quickly from left to right. It shouldn't ping pong back and forth between development and operations, it shouldn't be held up by security. Like manufacturing, the goal is to move work quickly, to have fast flow. And the best-known methods, for examples like Amazon and Google, these organizations are doing tens of thousands of deploys per day, right, and this is in contrast to, I think what is more typical, which is most organizations do one deploy every nine months. So the second way is reciprocal, is how do you create the feedback from right to left. So the first way is slow from left to right of work, and the second way is about feedback of information as we go from right to left. 

So how do we take the key learnings that we can only learn during a production outage, or a service impairment, or a security breach, and make sure that gets fed to the very beginning of the line, so we can ideally prevent bad things from happening again. Or, if we can't prevent it, at least be able to detect and correct. And so those patterns that come from the second way include, having operations be able to hand back fragile code, saying, this service has lots of potentials, but it keeps breaking in production, and just like a puppy who keeps peeing on the floor, eventually the operation should be able to say, please take over managing this until the puppy's house trained and then we will take over custodianship of the puppy again. I think the best adage of the second way came from a gentleman named Patrick Lightbody, he was the founder of a company called BrowserMob, and he said, we found that when we woke up developers at 2:00 a.m., defects got fixed faster than ever. I think this just embodies this notion that in order to have generally shared goals, we have to have shared pain. And then the third way is all about creating a culture of continuous experimentation and learning. The whole notion that we have to have the habits that say every day we're expected to experiment, to take risks, just because we've done it in the same way for the last 10 years doesn't mean that we have to do it the same way today. And this notion that repetition is a prerequisite to mastery, that whether we're learning to play a musical instrument, training a sports team to win, or training special forces in the military, it's always better to practice 15 minutes a day every day than to practice once a week for four hours.

 

- [Instructor] DevOps practices are consistently found in high-performing organizations that are emerging and evolving. The first way focuses on all business value streams enabled by IT. And the first way achieves that by putting emphasis on the performance of the entire system, rather than in just a single silo or department. This is a common pattern in high-performing organizations, where you will often see flow defined in process and value stream maps. 

Another characteristic of high-performing organizations is that people tend to self-organize to improve flow. The theory of constraints was first introduced in The Goal, which is another business novel like The Phoenix Project. Gene Kim outlines that The Phoenix Project was actually written as a homage to The Goal. Gene Kim explains in Beyond the Phoenix Project, our hope was that in The Phoenix Project, we could describe in equal clarity each sort of problem that every functional silo in the technology value stream also faced. 

So The Goal speaks to the fact that every process has a point that has the potential to be a bottleneck. To function more efficiently, organizations need to be able to identify their constraints, and widen or eliminate those bottlenecks. It's important to remember that not all constraints can be eliminated, some are unavoidable, and will always be required, for example, regularity controls. So let's look at some of the common constraints, and how we go about identifying them. Development delays is an obvious one, where we have budget approvals, reviews, or resourcing holding up a development project. Setting up environments, where we're requesting and provisioning test stage or production environments is often a major bottleneck for projects. Code deployment, if not part of an automated process, this can have a number of dependencies and so create hold-ups. And of course, testing, security, and QA assessments, if these include third parties, they may result in delays. So often the waste or bottlenecks are found in the form of these constraints. Examples include inconsistent environments, manual build or deployment processes, or poor quality and testing practices. Often, a lack of communication and understanding between IT silos can slow things down. And there's always frequent outages and failing of SLAs which cause bottlenecks. All of these issues require IT resources to spend significant time and money keeping the systems running. DevOps is the progression of the software development life cycle from waterfall to agile and to lean. So devops leverages lean by removing waste from the software development lifecycle. 

So let's look at the second way. Frequently, when releasing software information flows from left to right, i.e., from dev to QA to ops. It is less common, however, for right to left feedback, and little time is allocated for reacting to feedback when it is received. So the second way stresses the importance of right to left feedback, from ops to dev. And in order for this to be successful, the feedback loops should be short so that information is received in a timely manner. Feedback should also be visible, i.e. via dashboards, and without this feedback teams lack the information needed to continually improve. With shorter feedback loops, sharing and knowledge grow, and so trust grows. And as a result, we get better understanding of our customers, both internal and external We can begin to fix defects faster, and we are able to prevent problems. Processes are improved resulting in better flow and faster delivery speeds. And ultimately, there is less reactive unplanned work. A few examples of feedback loops, my favorite is automated testing, that enables us to get rapid feedback and disperse it around the organization quickly. I think postmortems are a fantastic way of being able to feedback improvements and to act on any suggestions that are made, and having visible change, incident, problem, and knowledge managed data available to all teams. 

The third way requires continual experimentation and learning in equal parts. Experimentation and taking risks are what ensures that we keep pushing to improve. We need to go outside our comfort zones, remove that caveat of failure, and go deeper into those danger zones than we might normally do. We also need mastery of the skills that can help us retreat out of the danger zones when we have gone too far. This is all part of effective planning. Companies that demonstrate a commitment to innovation are highly unlikely to chastise or disempower people who fail on a project in the traditional sense. That doesn't mean it's okay to be careless or lazy. The success point is in the communication around a project and its execution. It is also a very good practice to reward team members for failing successfully. A division of Proctor & Gamble, for example, has a president's fail forward award, and that's given to the team or individual that enabled the organization to significantly learn from a failure. This type of ritual rewards taking risks and sends a message about what the company values, which leads us to chaos engineering. A well recognized example of chaos engineering is the Simian Army. It's a concept that was first adopted by Netflix. Chaos Monkey is a service that randomly terminates a production instance, and the objective of the Simian Army was to create the chaos of an outage. So Netflix teams developed and refined responses to outage and attack situations. This helped to build competencies in how to recover the production environment from the inevitable failures. Chaos Monkey is a service which identifies groups of systems and randomly terminates one of the systems in a group. The service operates at a controlled time, so it does not run on weekends or on holidays, and it runs at controlled intervals, so it only operates during business hours. The Chaos Monkey concept can be used by mature organizations to learn from failure. 

A good example of adopting failure in in Ticketmaster, and over the years Ticketmaster have been extracting legacy technologies by adding APIs to modernize the interfaces with their ticketing engines and platforms. They wanted to get them out quickly, so to do that, they needed to touch a lot of systems, and that has driven them towards DevOps practices. For Ticketmaster, it really started with DevOps, part of their transformation was to focus on delivering business value faster, and delivering more of it. So how do we encourage a learning culture? Creating a learning culture is critical to sustaining growth and innovation. Training plans help to identify necessary competencies and encourage a common vocabulary and understanding. Learning can't be something that only occurs in a classroom, training plans and continual learning are crucial to DevOps success. At the end of Beyond the Phoenix Project, Gene Kim, and John Willis conclude that high-performing organizations practicing DevOps principles can be referred to as dynamic learning organizations. I like Andrew Shafer's quite, you're either a learning organization, or you're losing to someone who is. That brings us to the end of lecture two, I will see you in lecture three.