Welcome back to Introduction to Operations. I'm Ben Lambert, and I'll be your instructor for this lesson.

In this lesson, we're going to talk about performance. And why it's something that operations engineers have to think about. What is performance in the context of a web application? It's the speed and efficiency of each component and the system as a whole. Okay, what does that mean? It basically boils down to mean, how quickly does the system respond to user requests. Now, there's more to it than that. But if we were to try and make a definition that's as simple as possible, I think that would work.

So, we're saying, system performance is how quickly the system responds to user requests. With this definition in mind, where does operations begin when trying to measure and improve system performance? Remember, in the lesson on monitoring, we said, that once all the data was in place, teams could use it as they need it. Well, this is one of those times. Having quantifiable data, showing where performance problems might exist will help us give a starting place when it comes time to make system changes. Having actual data is very important.

If we don't know how long requests are taking, then we need to first implement some sort of monitoring solution. The reason monitoring is important is that perceived performance is subjective. What does that mean? It means that two people observing the same thing may have differing opinions of what they're seeing. As an example, if two people are both watching a car pass by and one person says, wow that car was moving fast, and the other replies, not really. Both are correct based on their perception of what fast means. So, if two users browse to your webpage and it takes three seconds to load, one person may find that to be just fine. And the other may be thinking, wow, anything over a second is so slow.

So we need to be working with quantifiable numbers when we think about system performance. But so far, we've only talked about measuring actual performance. We haven't talked about performance goals for a system. And that's because, the acceptable threshold for performance needs to be a business decision. There needs to be some sort of technical limitation, sure. For example, making sure every page loads in under one millisecond is probably not going to happen. But, it's a business decision to set the ideal performance numbers. Once you have a target for that range, let's say it's under one second, then you have something to shoot for. And with the data you've collected from you monitoring, you can go to identifying the slowest constraints and start to remove them.

We've talked about measuring system performance and setting a performance target. However, we haven't talked about why performance is important in this lesson. We mentioned it previously that users these days are spoiled by sites with impressive levels of performance. Google, Netflix, Amazon, among others have instilled in a lot of us a subconscious expectation of high performance sites. So, when we go to a site that takes five seconds to load, we often just lose interest in that site. Users have high expectations for how quickly sites and services should load. Okay, hopefully by this point, the value in having both a performance target and actual system metrics is clear. Without a target, we don't have anything to aim for with any system changes we make. And if, we don't have actual metrics, we can't quantify the impact, either good or bad of any of the changes we make.

So, once you have these numbers, what's next? This is where we start to go through the application and architecture and make sure we find and remove the bottle necks. The more complex the software, the more places there are for bottle necks to hide. Every time a web server runs some code or makes a call to a database, or some rest API, we introduce latency. If things are running well, then that latency shouldn't be too noticeable.

Let's talk through some of the common causes for system performing issues. Here we have a load balancer in front of some web application servers. The load balancer needs to be fast. They need to handle a lot of traffic and if they're not fast enough, then the entire application will be slow. High quality load balancers both in and out of the cloud, shouldn't be adding any substantial amounts of time to the request. If they are, you need to test and verify that this is in fact the load balancer and not something such as the network. If it is the load balancer, look into the documentation for that particular vendor's product and make sure that the settings are correct. When it comes to cloud based load balancers, if you're seeing latency, test the back end servers and see how quickly they're responding.

It's not uncommon to see system performance related issues related to application code having some bit of logic that causes latency. This could be your code or some third party code. Also, slow database queries are very common. Also not enough CPU for your running processes or CPU being consumed by some other process which is starving your application is fairly common. And the same applies for memory.

Often times in web application, you'll be serving up a lot of static files. Images, Jave Script, CSS, PDF's, among other assets. For high traffic sites, this tends to cause performance issues if you're not using a content delivery network. Services such as CloudFront on AWS, allow you to serve up files to the users from cache locations that are geographically close to the user. This makes it faster for the users and frees up your servers to handle more important things than serving static assets.

These are just a few of the areas where performance issues are often found. When it comes to locating the source or sources of latency, I like to start by reviewing the APM tools data and by running the sites through something like Google's PageSpeed Insights. I find these tools to be a great place to start for identifying performance problems. Tools like PageSpeed are a great way to understand what kind of things could be causing applications to run slowly. I recommend that you check it out, it's free to use. Enter the URL for your site and click analyze. Go through all of the items, even the ones that pass.

It'll help you recognize some common causes for slow performance. There are a lot of mechanisms that will help you improve the systems performance. Things like caching, edge locations, content compression, among other things. And they're great to know about. Though, we won't be getting into that level of detail today. Throughout this course, we've talked about a lot of the things that could go wrong and some ways to help ensure that when things break, we have a fall back.

However, how do you know when those things break if the fall backs are working. This is where planning and practicing failure comes in handy. What exactly does that mean? That's what we'll find out in our next lesson. So, if you're ready, let's get started.