- [Instructor] Hello, and welcome to the lecture. In this lecture, we will begin by defining DevOps and why DevOps matters. DevOps is built around a collective body of knowledge. DevOps does not have a single source of knowledge like other IT frameworks, for example. It has been built on the collective experiences and insights of multiple thought leaders and ultimately a number of resources, including conferences, books, articles, webinars, videos and enterprise experience. This is particularly important because DevOps touches virtually every aspect of IT. Culture, automation, management, quality, HR, and organizational structure. That is it's just a bit too difficult to encapsulate into a single body of knowledge. Now Damon Edwards outlines a short history of DevOps very succinctly. Let's listen to Damon's view.

 

- [Damon] So, between my work with DTO Solutions or writing with the IT Revolution Press, I get to talk to a lot of folks who are interested in DevOps. These conversations have reminded me that while there is a lot already out there about what DevOps is or what DevOps isn't there really isn't much about where DevOps came from or how the DevOps movement actually got started. So, I've told these stories enough times that I decided to go ahead and record it. So, without further ado, here it is. The short history of DevOps, or, well, at least how I think I saw it happening. 

Our story begins in Belgium back in 2007. There we find a guy name Patrick Debois. Now, Patrick had an interesting personal goal. Patrick wanted to learn IT from every possible angle. So, as a consultant, Patrick had been picking his jobs based on the opportunity to work in every part of an IT organization. Now, Patrick had taken a project with the government ministry, doing a large data center migration. So, specifically for this project, he was in charge of the testing. So that meant he had to spend a lot of time straddling various dev groups and the various ops groups. This contrast between how dev works and how ops works had always been something that was very unsettling to Patrick, but it was really the constant switching and back and forth on this specific project that was particularly frustrating to him. On one day he'd find himself deep in the rhythm of Agile development and the next day he was firefighting and living unpredictability of traditional operations. While Patrick knew there had to be a better way, these two worlds of dev and ops just seem miles apart, and there were conflicts everywhere. 

Okay, so now let's jump forward to the Agile 2008 conference in Toronto, Canada. This guy here, Andrew Shafer, he posted on a wall an idea for birds of a feather or ad hoc session that he calls Agile Infrastructure. And when the time for the session rolls around, only one person showed up. And yep, you guessed it. It was Patrick Debois. And when I say only one person showed up, I literally mean only one person showed up. Even Andrew skipped his own session. You see, Andrew had gotten such poor feedback on his proposed topic that he just figured nobody would wanna talk about how to bridge the gap between development and operations. But it was Patrick who actually was at the conference to give a presentation on using scrum and other Agile practices within an operational context was so energized that someone else out there share his frustration, he went and tracked down Andrew in the conference hallway. There they had a long discussion and realized that there had to be other people out there who wanted to talk about what seemed like such a wide spread and systemic issue. Now, a bit of context. Within the broader Agile community, continues integration had been gaining in popularity and was moving Agile more towards deployment, but there still wasn't much out there that fully crossed the divide into operations. So, Andrew and Patrick decided to get together and form the Agile Systems Administration Group on Google. It got some interesting conversations, but by any measure, the traffic remained pretty small. So now let's jump forward to 2009, specifically June 23, 2009 where John Allspaw and Paul Hammond were both at Flickr at the time were giving their now famous talk at O'Reily's Velocity Conference in San Jose. The talk was Called 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr. So Patrick was back home in Belgium watching this via streaming video, and he recognized that that's it. That's exactly the goal and topics that he was so passionate about. So, on Twitter, Patrick was lamenting how he wished he could've attended Velocity, but it was Paul Nazareth who tweeted back, "Hey Patrick, why don't you start your own Velocity "in Belgium, that way we can all attend. "Wouldn't that be great, ha ha ha." Well, far from being a joke, the idea actually stuck in Patrick's head. So he soon put out a call on Twitter for a gathering to bring developers and systems administrators together in Ghent, Belgium on October 30th and 31st, 2009. So, Patrick knew he needed a name for the event. It was obvious to him that he had include Dev, he had include Ops, and well, it was two days, so let's stick a days on there. And for some strange reason, even today when I asked him, he said he's not sure why but he really liked the DOD acronym, reminded him of dead on delivery which, I know, strange, go figure, but that was it. DevOpsDays was the name, and DevOpsDays it was. So, now, we'll never know if it was the right moment in time fortuitous travel schedules, or something else all together. But this really impressive crowd of forward-thinking systems administrators, developers, managers, and tool smiths that came from all over the world to participate in DevOpsDays. 

Now, of course, soon after the event was over, everybody scattered and went back to their corners of the globe, but the conversation continued onward on Twitter. Now, it quickly became too cumbersome to use the full DevOpsDays name as the hashtag for the conversation, as smaller hashtags would take up less of that valuable 140 character real estate on Twitter. So, the days was dropped, and the DevOps hashtag really stuck. Now, speaking the scattering, Lindsay Holmwood liked the idea of DevOpsDays so much that he took it back to Sydney, Australia with him, and held the first DevOpsDays down under. Myself, Andrew Shafer, John Willis, we got together. And the help of Stefan Apitz and Daniel Francisco at LinkedIn, we held the first DevOpsDays in the US, immediately following the 2010 edition of Velocity. But in the meantime, something far more interesting was starting to happen. A spark by the momentum of these early face-to-face meetings, this community of practitioners suddenly emerged from all over the world to share their experiences and debate ideas under this new DevOps banner. DevOps had become this lighting rod for people who had something to say about how IT was or should be running. Presentations, meet ups, sessions within other conferences, these all started happening all over the place. So, really this was a continuation of this online momentum that built behind this avalanche of tweets and blog posts where people were sharing their experiences and learning from others. And I think when people went as far as to start making their own song parodies, music videos, it became pretty clear to folks that DevOps is tapping into a raw nerve. DevOps had become this full-fledged grassroots movement, something pretty rare in IT. Now, it's still being largely ignored by vendors, by analysts, by most traditional enterprise IT shops, but the movement was growing by being fed from the passions of these practitioners, these folks usually from web operations backgrounds who were largely meeting and writing about these topics on their own free time. It was in these conversations that people really commiserated about the cultural and functional efficiencies and the legacy tools that they had been saddled with. In response to DevOps, communities started to drive a whole new generation of tools that formalized the best practices that they wanted. These tools had fun names like Puppet, and Chef, Vagrant, Juju, Rundeck, Logstash, FPM. In fact, you can only guess what the F in FPM stands for. But these were serious tools. And in most cases, these tools were running circles around what the legacy tools could offer. Now, for the community, these were artifacts that represented a better way of working and a way to formalize their thinking about new ideas and new processes. To outsiders, these were shiny new toys that made you jealous if you didn't have them, and drew you into the conversation. But all around, the enthusiasm for these tools was an early and energizing force. 

So pretty soon some of the more plugged in analysts started to figure out, hey, there's something interesting and brilliant here, maybe we should join the conversation. First in were folks like Michael Cotay, at that point from Red Monk. Jay Liman who was with the 451 Group. After that came folks like Cameron Hague, of Gardener. Now, Cameron's influence is interesting. It's subtle but potentially a pivotal point for DevOps relationship with the enterprise. Now I can't definitively prove this assertion one way or the other, but the timing is definitely curious. In March 2011, Cameron slips this slide into a presentation, and it definitely sent a strong signal to Enterprise IT shops and the vendors who service them. Something that was pretty much unheard of up to that point in the DevOps movement. So, let's see what Cameron said. So he said, "By 2015, DevOps will evolve "from a niche strategy employed by large cloud providers "into a mainstream strategy employed "by 20% of Global 2000 organizations." Now, I don't know if you speak analyst but that roughly translates to DevOps is real, check it out. So, you know, I know some folks might see this and get kind of hung up on the, you know, the 20% prediction whether it's too bolish or it's too conservative. But nonetheless, that's not the point. The point is there's a clear message here which is DevOps is coming to the Enterprise, go long on DevOps. Now, not long after that, I don't know whether this is by coincidence or not, but we started to see almost all the large vendors pick up on the word DevOps and start to really fold it into their messaging. You know, some of them figured out how to use it quite nicely while others, well, I guess you could say they missed the mark all together. But nonetheless, you know, DevOps was appearing all over the place and that's a good thing. Even at DTO we saw a significant spike in Enterprise interest at around that same time. You know, suddenly, large household names, far from what you would consider hip web companies, were interested in DevOps. Now, DevOps had really crossed the chasim and was well on it's way into the mainstream at that point. 

So, you know, why is this history so important? Why do I care so much about the history of DevOps? Well, like the old saying goes, you know I believe that if you don't know where you're from you really aren't going to know where you're going. And this history reminds us of a few important things about DevOps. First, it reminds us that DevOps is really from practitioners, by practitioners. It's not from a vendor, it's not from analyst. And we've already seen if you're a vendor or you're an analyst and you try to highjack the message or push things or products or ideas on people who don't want them the community will just find a way to route around you. You know, next, it also reminds us that DevOps it's not a thing, it's not a product, it's not a spec, it's not a standard, it's not a job title. Other than this collective voice of the community, there really is no one true authority on what DevOps is or isn't, because really, DevOps is an experienced-based movement. It's about practitioners getting together, sharing what works, what doesn't work, and, you know, collaborating on making themselves better and helping the companies that they work for. It also reminds us that DevOps is decentralized and open to all. If you have an experience that you want to share or you have questions for other practitioners, you're always welcome. It's the way this movement started and it's the way it continues on today.

 

- [Instructor] "Imagine a world where product owners, "Development, QA, IT Operations and Infosec work together, "not only to help each other, but also ensure "that the overall organization succeeds. "By working towards a common goal, "they enable the fast flow of planned work into production, "while achieving world-class stability, "reliability, availability, and security." I like this definition as we have not had a balance of stability, reliability, availability, and security in the IT industry. I like the concepts of fast flow and planned work. Both quite new concepts in the realm of traditional infrastructure management. Here's another interesting view from Mark Schwartz in his book, The Art of Business Value. "DevOps, in a sense, is about setting up "a value delivery factory - a streamlined, "waste-free pipeline through which value "can be delivered to the business "with a predictably fast cycle time." Now, Mark's book puts particular focus on how business value is measured during Agile software projects. We hear Mark refer to cycle time and this might be a metric that matters to you. For me, the real metric should always be business value. So, first, let's be clear on what DevOps is not so we can help ourselves better understand what DevOps is. While there are increasing job postings for DevOps engineers and the number of DevOps teams is always on the rise, DevOps is not a title or a team. DevOps is not one person's job but everyone's job. Both of those two concepts deviate from the intent of DevOps which is to improve communication and collaboration by avoiding creating another silo. It's certainly not a tool and it's not only a culture and it's not only automation, it's certainly not anarchy and it's not a one size fits all strategy. So, why is DevOps important now? Well, with the advent of Cloud computing the business environment is going through it's biggest transformation ever. More organizations are migrating to the Cloud and Agile software development and Cloud infrastructure is increasing. So, IT can no longer operate in a silo culture. The reality is Enterprises now have to be nimble because they have nimble startup competitors. Consumers have become app consumers. They have an app mentality and expectations. And there is more data available to the business than ever before. So, to meet these changing conditions, IT needs to adapt it's culture, it's practices, and automation processes to become more continuous to shorten and accelerate the time to value for the business. So, let's just pause for a minute and evaluate what is that makes DevOps so unique. 

There are a number of frameworks and programs available to IT teams today and each of these has it's own merits and value points. Each of these frameworks and approaches has delivered some degree of benefit. But none has delivered a full end-to-end IT improvement to the same degree that DevOps practices have. A key benefit of DevOps is the way DevOps applies system thinking across the entire IT spectrum. It influences people and culture, foremost. It's inherent through the processes such as Lean, Agile, ITSM, CI/CD, and it enables automation across business processes to bring the highest amount of value to the business. This is more relevant than ever as modern IT is a system of systems. And DevOps brings consistent systems thinking across all of these systems. If done effectively, this system thinking enables real value for the business. Now, this does require recognition of some of the myths and anti-patents that we can see around DevOps; the idea the DevOps means chaos. DevOps is not the wild west where developers can do whatever they want, whenever they want. DevOps doesn't mean that you can't do operations, it's not NoOps. The term NoOps was initially coined by Adrian Cockcroft, Director of Cloud System Architecture for Netflix, when he described his IT and organization has having little need for operation staff. Particularly because the company had shifted to the Cloud where it could automate many former functions that the staff looked after. So, a lot of organizations strive to be entirely no operations or NoOps and to ideally replace Ops with automation. But there's a real recognition that while the role of Ops is evolving there is still a need for operational roles. So, another common misconception is that DevOps can't be done with legacy systems or that we can't do DevOps if we have outsourced some of our IT capabilities also, the notion that we need to automate everything. Now, this is likely to create almost the opposite effect of what DevOps set out to enable in the first place. So, let's review the goals of DevOps and the perceived value that these goals can bring: the smaller more frequent releases. This can reduce the effort required and the risk of getting something wrong. So, with smaller, more frequent releases we also reduce the cost and potential delay of production iterations. Now, this in turn means developer services are more integrated with the business and frequent iterations mean we stay better aligned with the goals of the business, we have more visibility, and so more consistency. And many of these goals are inter-related. Time to market is the amount of time between an idea for a product and it's availability and as a DevOps practitioner we need to always be asking, "What is more important, time to market or time to value?" Business value needs to be the ultimate end goal of the DevOps practice and process. So, more than anything else DevOps is a cultural movement based on human and technical interactions to improve relationships and results. After the first US based DevOpsdays in Mountain View in 2010, John Willis and Damon Edwards coined the acronym, CAMS, which stands for culture, automation, measurements, and sharing. Now, Jess Humble later added an L to this standing for lean to form CALMS. Culture, culture relates to the people and processes and the aspects of DevOps and the need to improve communication and collaboration. Without the right culture, automation attempts will be fruitless. Automation, tools such as release management, configuration magnet management, monitoring and control tools enable automation so we need to maximize customer value while minimizing waste and improving flow. Measurement, if you can't measure, you can't improve. 

So a successful DevOps implementation will measure everything: people, process, and technology performances. Sharing, sharing is the feedback loop of the CALM cycle creating a culture where people share ideas and problems. It's critical not only because it enables improved communication and collaboration but also because it helps organizations to improve. Automation is by far the most essential element but it should not be the be all and end all result. Automation and tool chains can help increase the speed and consistency of IT delivery. One of the greatest challenges for the Enterprises moving towards DevOps is how to diverse the IT environments in large organizations tend to become. And it's one of the reasons Enterprises have a hard time achieving all of the items at the top of the screen. There is so much variance between teams in terms of the tools they use, as well as, the varying frameworks, methodologies, and processes. And this inconsistency across the Enterprise is the enemy of broad Enterprise wide DevOps success. For practices such as continuous integration and continuous delivery and DevOps itself to thrive, architects, developers, QA, testers, and other teams all need to be using shared systems. Where appropriate, these need to tie back to ITSM and roles such as Agile Service Manager and Agile Process Owner since these roles will help adapt underpinning processors to increase the number and assistance of automation. DevOps includes all the people involved in developing software products and services and the Ops component includes all the people involved in delivering and managing software products and services. But DevOps is not just about Dev and Ops, it is about people in other parts of the organization and even outside of the organization working together to improve service delivery. "You never change things by fighting the existing reality. "To change something, build a new model "which makes the existing model obsolete." Now, that's a quote by Buckminster Fuller who codified the myth behind the geodesic dome. And I think the sentiment behind his quote is very relevant to DevOps. So why does DevOps matter? Businesses want to move quickly and we are all familiar with the mantra of time is money. But technology is only a competitive advantage for a brief period of time and companies must constantly innovate to succeed. Traditionally, software and application development projects were massive efforts that took 18 to 24 months and the traditional Waterfall projects demanded a linear approach that included requirements definition, design, development, testing, and implementation activities. Wherein one activity can't begin until the previous activity is complete. Now, this linear approach meant that customer feedback is delayed until the project is complete. Operations, particularly in heavily regulated environments, can be just as slow often due to over rigorous methodologies, frameworks, and standards coupled with the bureaucracy that goes with larger organizations. Many organizations struggle to do releases every nine months. And another factor that slows work into Ops is the limited release windows often over weekends that are time boxed which don't really work or create problems because it takes too long to fix things if there's issues. All projects must fit into these release windows and if a change failed the project is further delayed until the next release window. So, pushing all projects into the limited release windows resulted in a complex and turbulent install. And that brings us to the difference between Cadence and Velocity. So, if a higher business expectations mean there has never been a greater need for agility in both Dev and Ops. Both parts of the organization need to look for ways to reduce waste and to streamline existing processes. Both parts of the organization need to move more quickly where it is appropriate. 

That is where the risks can be mitigated. Continuous delivery ensures that software is always in a releasable state which allows organizations to introduce incremental changes in smaller batches. And there will still be some complex projects that require greater rigor and will need to move more slowly. Now, typically those projects are where the requirements are extremely well defined and there is little value in using an incremental approach. But even when traditional Waterfall practices are used projects cannot move so slow that it cripples the business. IT service management must be able to accommodate either approach by using, for example, change models and more standard changes that can be implemented during normal business hours without disruption. For six years from 2011 to 2017, IT Revolution, DORA, and Puppet Labs conducted an extensive survey for practitioners and others on emerging practices in DevOps. 

In the State of DevOps Report teaches us a wealth of things including that going faster can sometimes result in a reduction of quality and the fact that organizations that adapt DevOps practices can go both fast and achieve more reliable services. The key metrics that we see are throughput and stability and part of this stems from efforts to capture quality at the source and to ensure that errors aren't knowingly passed downstream. Now these efforts are accomplished via practices such as version control, continuous integration, continuous delivery, and automated testing. 

Another consideration is the fact that organizations are introducing more sophisticated monitoring early in their development cycles. ING Bank is a good example of how adopting DevOps has improved the business and business flow. So, the team at ING wanted to establish a culture and environment where building and testing and releasing software can happen rapidly, frequently, and more reliably. When beginning this journey they started with what matters most, the people and the culture. IT has become the beating heart of the bank. Sources vary in their interpretation in gathering of DevOps adoption data but most organizations are now at least talking about DevOps. And if not talking about it, implementing capabilities that they describe as DevOps. In the GitLab 2018 Developer Survey, for example, 65% of respondents say that DevOps is a tremendous time saver. Now, that alone is a very, very high number but even more poignant is that 81% of managers agreed with that statement. In the words of Damon Edwards, "The completely achievable goal aligns IT goals "with business goals by removing "all of the bottlenecks, inefficiencies, and risks "between a business idea, the ah-ha moment, and a measurable customer outcome, the ka-ching moment." 

Now, let's look at the business perspective. The reality today is that every business has become a tech business, IoT is rapidly increasing, consumers have developed an app mentality, and customers value outcomes not products. So, time to value is replacing time to market and how people think. Intelligent data must shape direction quickly and customer insight is more important than customer satisfaction. So, from the 2017 State of DevOps Survey Report, organizational culture is one of the strongest predictors of both IT performance and overall performance of the organization. High performers were more than twice as likely to achieve or exceed the following objectives: a quantity of products or services delivered, operating efficiency, customer satisfaction, a quality of products or services provided, achieving organizational and mission goals, and measures that demonstrate to external parties whether or not an organization is achieving intended results. So, it's clear the business benefits are there. So, how do we start applying these? It's worth going back to the basics and asking why when considering what DevOps can do for you. Simon Sinek's Golden Circle is a much referred to model that could help focus and validate what sets you apart. The why at the center is the purpose, the reason that organization exists. In my opinion, if you get the why worked out first every other decision becomes easy. The how is what sets you apart, how to deliver the benefit or services to your customers. The what is the products and services, and of course, this is the outer valence of the shell as the products and services can change often to adapt and align with customer needs.

 

- This little idea explains why some organizations and some leaders are able to inspire while others aren't. Let me define the terms really quickly. Every single person, every single organization on the planet knows what they do, 100%. Some, know how they do it whether you call it your differentiating value proposition or your propietary process or a USP. But very, very few people or organizations know why they do what they do. And by why I don't mean to make a profit, that's a result, it's always a result. By why, I mean what's your purpose, what's your cause, what's your belief?

 

- [Instructor] A few examples of why statements that really work: Helping Britain Prosper by Lloyds Banking Group. We exist to make the lives of retired people better from Saga. We connect people and strengthen relationships from a greeting card manufacturer. Nourishing families so they can flourish and thrive which is a Kellogg's why statement from Karen Martin's Clarity First. And it was Peter Drucker who said, "The purpose of a business is to create a customer." Another interesting business prospective is when IT is seen to be a hindrance rather than a benefit. And in the words of Clyde Logue, the founder of StreamStep, "Agile was instrumental in development "regaining the trust of the business, "but it unintentionally left IT operations behind. "DevOps is a way for the business to regain trust "in the entire IT organization as a whole." Let's examine the IT perspective. What is driving DevOps? Essentially, the same drivers we see driving the business perspective. The question and challenge is often does both the business and technology teams recognize all the statements to be true or is there divergence? And which areas may not be seen as a priority for the IT teams and how can we insure convergence between these two? One of the biggest challenges is the changing focus of IT. IT no longer needs to align or integrate with the business, IT is the business. IT has been taking steps to improve, however, often activities occur in silos and they can hinder the overall effectiveness of initiatives. As a result, each initiative gains some benefits but perhaps did not show an improvement in the end. DevOps must continuously deliver outcomes by bridging and improving almost every aspect of IT. Often failed or misinformed initiatives can create negative perceptions: an Agile software project that did not complete due to blocks in the process chain or misalignment with ITSM procedures for example. Or perhaps a Cloud project slowed or stalled due to lack of support for pay as you go services or billing or billing procedures. The traditional IT mindset tends to view failure as a negative. A DevOps culture accepts and learns from experiments and trials. Thinking outside of silos is often the first challenge for IT teams and this can create, what we call, a wall of confusion. DevOps is a response to the growing awareness that there is a disconnect between what is traditionally considered development activity and what is traditionally considered operations activity. The two can be perceived as being polar opposites. Dev by it's nature is required and incentivized to generate change. Ops is required and incentivized to maintain stability. So, a wall of confusion often occurs as a result of these two conflicting goals. From the outside, and often in the eyes of the business at least, this disconnect manifest itself as conflict and inefficiency. This inherent conflict can create a downward spiral. Slower feature time to market, longer deployment cycles, increasing numbers of altitudes, and an ever increasing amount of technical debt. The business has a very low tolerance for unavailability and IT systems are just meant to work. Any sets of confusions around roles and processes can signal some kind of issue. 

An extreme focus on either end of the spectrum is not any better. Some businesses have regularity goals in place that don't let them take risks and so they can't always change as quickly as they would like. Other organizations, for example startups, are willing to take risks and it's more important to them to demonstrate innovation and to be first to market. Regardless of an organization's risk tolerance, stability always remains important; the business wants both. The good news is that DevOps done well breaks that traditional iron triangle. With DevOps you can have it all: software delivered at high speed, high quality, and low cost. A key start point is often tackling stereotypes and breaking out of silo thinking. Stereotypes and perceptions can have a negative, cultural impact on the ability for DevOps security and support to interact and collaborate. The 2014 State of DevOps Report identified five predictors for IT performance. Peer-reviewed change approval process. When the technical team held itself accountable for the quality of it's code through peer review, performance increased. Version control for all production artifacts. So version control provides a single source of truth for all changes; that means when a change fails it's easy to pinpoint the cause of failure and roll back to the last good state reducing the time to recover. And it also shows that organizations using version control for both system and application configurations have higher IT performance. Proactive monitoring, teams that practice proactive monitoring are able to diagnose and solve problems faster and have a high degree of accountability. High trust organization of culture, organizational culture was highly predictive of both IT performance and overall organizational performance. High trust cultures led to greater performance while bureaucratic and fear based cultures tended to be distractive to performance. So ultimately, what patents do we see being the most successful? Well, it's not just Dev versus Ops, it's Dev plus Ops and sometimes win win requires concessions on both parts. That brings us to the end of lecture one. I will see you in lecture two.