Introduction to Amazon CloudWatch
Amazon CloudWatch Operations
AWS Cost Management Tools
This section of the SysOps Administrator - Associate learning path introduces you to the different monitoring and reporting services and tools that are relevant to the SOA-C02 exam. We look at both the monitoring of your infrastructure, in addition to the reporting of your bills.
- Understand how Amazon CloudWatch is used to monitoring the performance of your infrastructure
- Learn how to identify anomalies in your infrastructure using Amazon CloudWatch
- Learn how Amazon EventBridge makes it easier to build event-driven applications at scale
- Learn about the different methods of logging that are available
- Understand how to review your costs and optimize them going forward
In this section, we will take a look on how to use the AWS Cost Explorer, one of the most essential tools for our Cloud Financial Management within AWS. Moreover, the Cost Explorer is the very main tool for you to gather information and analyze all the costs and expenses in your environment. So let me introduce you to the tool itself.
To get there, you can go over to your Billing Dashboard. From here, go to the Cost Explorer, and just launch it. See, you're getting guided to a whole new menu where you can start to analyze your costs. The Cost Explorer is a built-in tool that lets you get deeper insights into the costs and usage of your cloud environment.
With the help of the Cost Explorer, you will be able to identify trends, hunt down the biggest cost drivers and detect anomalies. And the best part, the Cost Explorer, in its base variation, does not cost anything. You can use it for free. However, take note that there can be additional costs when you use tools to request the API of the Cost Explorer.
Every API request costs about one cent, which could drive up your costs very fast when you fire up a lot of API requests. With the Cost Explorer, you can visualize your usage patterns over time and identify underlying cost drivers.
What we see here is the very first view that you get when you open the Cost Explorer. It is grouped by the different services that you use. You get a six-month overview, and the visualization type is bar, which means that the costs are stacked to each other in these kind of bars here. You can also change this to stacks or to lines, what doesn't make too much sense in this case. So let's stick to the bar view.
The legend here below shows you which service is illustrated by which color. In this example, we can see over the last six months our EC2 usage has increased quite a lot. If you scroll down a little bit you can see the detailed data table. Below the Cost Explorer here is the data table, which provides a deeper insight into the data seen in your chart. And you can also export this as a CSV file for further processing with a table tool like Excel if you wanna make forecasts or like further analysis.
You can also like, like I said we have the last six months here, you can also like scroll vertically to see all the services and cost drivers and you can scroll horizontally to see each month. It's basically just a table.
You can choose the exact timeframe you want to see within Cost Explorer. By default, it's on six months. So you can either select day by day that you wanna see or you can choose to auto select here to see for example like just last seven days, current month, three months, six months, one year, months to date, year to date, or three months forecast or 12 months forecast.
Let's go for the last seven days here. So we can see how much we spend for the day different services day by day. We can also delete the group by filter here and get like monthly view. It makes no sense because we wanna see days. So you're gonna select daily here and then we can see how much we spend day by day, if there's, for example, an anomaly. We can see, oh wow, like way more than the other days. Why is that?
You can see we are here January 1st the first day of the month. And on the first day of the months, you usually have to pay for tags, reserved instances, and all that stuff that incurred like more costs than over the other days. And that could be the explanation for this anomally here.
And you may have noticed you can also go from a daily to an hourly view, but it is deactivated. I can't select it. Why is that? The reason for that is super easy. You have to pay for the hourly view because you need like much more data records. So you have to enable it first. If you really wanna go in that much detail. I will show you how you can do this.
Hourly view, you have to go to preferences. And this is the point we are looking for, the hourly and resource level data. We can select this and save it. Here you can also see the cost that would occur. And we can see here one cent per 1000 usage records per month.
It depends how much EC2 instances because this just applies to EC2. It depends how much EC2 instances you're running. Of course, the more instances you're running the more usage records you are going to record and the more costs you incur, but 1 cent per 1000 records, in this case, it's not that much.
So we wanna save it here, go back to the Cost Explorer and we can see, oh, the hourly view is available. We can only see last 14 days of usage was the hourly view enabled. In this case because I just enabled it, I won't see anything because the data needs to be recorded first. So it will take a minimum of four days for Cost Explorer to collect enough data that the hourly view is available here.
About data grouping and filter. We started with the group function, with the grouped by functionality, you can segment your data based on a particular cost or usage. For example, you can choose a region to see in what region you spent the most money on which regions you spent money.
Let's go with a bar so we can see, let's take December. We spent the most money in Ireland followed by others because others mean we have like a lot of data here. Well, let's go to December. We've used many different regions here and all the regions that we use that are not shown here are combined into the others bar. If you want to stripe your chart down to make it more granular, you should use the filter function located on the right.
Using the filter panel. You can refine your data set to include or exclude specific filtering dimensions and values. For example, let's say we wanna only see costs that occurred in Frankfurt. We go to the filter section here search for the Frankfurt region, select it, applied it, then we can just see the costs that applied for the Frankfurt region. And we can use like many combinations here to dive deep within our costs.
For example, if we just wanna see the costs for our EC2 instances, just filter here for EC2, for our compute instances. Nothing else just compute instances.
So just our literal service and we filter it by instance type and then we can see what instances occurred the most. So for example, here, we have a G4DN which is like a graphical, a big graphical instance that is quite expensive in this case.
As you can see here on the right, there are quite a lot of filters that you can use and no worries, we won't look at all of them, but some of them are quite interesting. Let's for example talk about usage type, usage types are the unit that each service uses to measure the usage of a specific type of resource.
For example, if I wanna know how long my t2 micro instances we are running in the past, I would just type in here, t2 micro, oops micro, select a filter here. And I can see my t2 micro instances were running 370 hours in July and about 200 hours in August. And you can use this for many other services as well.
Usage type groups also is pretty cool, these are the kind of type of filters that collect a specific category of usage type filter and put it into one. For example, if I wanna know how long all my EC2 instances were running in the past, I would just search for EC2 running hours and I could see the amount all my EC2 instances were running over the last six months.
Another one that is quite useful if you wanna go in more detail is the filter for API operations. So let's take S3 as an example, we can see here. We don't have that much S3 costs here, just in December $188. But if I wanna know more about these costs, how these costs are structured, I can look for specific API operations that apply to S3.
For example, if I want to know how much my costs were for like reading files from my S3 three buckets, I would look up GetObject because this is the API operation for reading objects from S3. I apply this filter here and it can see the costs for GetObject from S3. That's it, that's how you're gonna use API operations.
As you can imagine, there are a lot of API operations that are used within AWS and I can highly recommend to have a look at API reference documentation for each service because you can see here, this is like just for S3, all the API operations that you can use for S3. And just by looking at them, you can, for most of them, you can already like kind of guess what they're doing. Like for example, here, create object, create bucket and you can like just look forward for the API operation that fits for your kind of analysis.
A little bit deeper into Cost Explorer and the advanced options that can also be quite useful. For example, show only untagged resources. This is a pretty important filter because we talk about tags already a lot and little spoiler here we will talk about them way more because this is such a super, super, super important topic. And if you set this filter here you will see all the resources that have no tags attached, super important to see that.
What is also quite cool is the show costs as here. The unblended filter here will be the best for the vast majority of AWS customers. This is the cost dataset presented you on the bill page like for the bills and for the invoices. And it's also the default option for analyzing costs using AWS Cost Explorer or setting the custom budgets using AWS budgets.
The unblended costs represent your usage costs on the day that they are charged to you or in finance terms they represent your costs on a cash basis of accounting. For most of you this is the only data set that you will ever need. Okay?
Amortized costs are also quite interesting because the amortized cost is useful in cases in which it doesn't make sense to view your costs on the day that they were charged or as many of finance owners say it's useful to view costs on an actual basis rather than a cash basis.
This cost dataset is especially useful for those of you who have purchased AWS or reservations such as reserved instances or savings plans. Savings plans and reservations often have upfront or recurring monthly fees associated with them.
Recurrent fees for reservations are charged on the first day of the month that can lead to a spike on one day if you're using unblended costs as your cost dataset. When you toggle over to amortized costs, these recurring costs, as well as any upfront costs, are distributed evenly across the month.
Armotised costs are a powerful tool if you seek to gain insight into the effective daily costs associated with your reservation portfolio, or when you are looking for an easy way to normalize costs and usage information when operating at scale.
And then there are also like, two more that can be quite helpful. And these are the net unblended costs and the net amortized costs. These are basically the same as the two that I just explained here but they also include discounts like the reserved instance volume discounts. Like these discounts are calculated into these costs.
Stuart has been working within the IT industry for two decades covering a huge range of topic areas and technologies, from data center and network infrastructure design, to cloud architecture and implementation.
To date, Stuart has created 150+ courses relating to Cloud reaching over 180,000 students, mostly within the AWS category and with a heavy focus on security and compliance.
Stuart is a member of the AWS Community Builders Program for his contributions towards AWS.
He is AWS certified and accredited in addition to being a published author covering topics across the AWS landscape.
In January 2016 Stuart was awarded ‘Expert of the Year Award 2015’ from Experts Exchange for his knowledge share within cloud services to the community.
Stuart enjoys writing about cloud technologies and you will find many of his articles within our blog pages.