Amazon Web Services offers some great tools for managing, analyzing, forecasting, and optimizing infrastructure costs in the cloud. In this three-part blog series, we will explain how you and your organization can take advantage of them.
One of the major factors driving companies to adopt cloud solutions like Amazon Web Services is the minimal upfront cost of resource provisioning followed by an easy, pay-as-you-go operating model. These two factors, along with the disposable nature of the wide variety of available computing resources, and the ease of resizing those resources, makes AWS an attractive alternative to large data centers.
However, companies moving their workloads to the cloud will sooner or later start thinking about optimizing costs. Medium-to-large organizations making a transition to the cloud or who have completed a transition will want to see the benefits of their investment.
These companies – when running large server fleets in their own data centers – were accustomed to seeing computing and storage resources sitting idle and incurring costs even after a project or application’s life-cycle ended. Unfortunately, the practice often remains unchanged after they have migrated to the cloud. The result is unnecessary but easily avoidable expenditure.
Even small companies and cloud-native startups sometimes fall into this trap. Typically, these companies try to minimize their resource footprint and maximize the return on investment in infrastructure. But as the R&D phase ends and products enter the market, keeping infrastructure highly-available takes higher priority than running them optimally. Empty servers, underutilized RDS instances or unattached storage volumes are often overlooked in favor of day-to-day operations.
The outcome is the same: costs are often comparable, or even greater than running an on-premise network. After all, it’s not possible for cloud providers to know when a customer stops using a resource and shut it down for them. That responsibility still lies with the customer.
AWS Billing and Cost Management Tools
Fortunately, Amazon Web Services (AWS) and third-party vendors offer an array of tools for managing and monitoring cloud operating costs. Infrastructure and operation managers will find these tools extremely useful because they can help find:
- What resource types are incurring the largest share of total cost (EC2, RDS, Redshift etc.).
- The trend in monthly bills (is it the same, is it increasing or decreasing?).
- A forecast for the current monthly bill.
- A means to send alerts when costs exceed some threshold.
- Potential areas of savings.
- Estimated costs before a resource is provisioned.
Using the insight from these tools, infrastructure managers can decide:
- Which AWS resources or service can be shut down or retired.
- Which AWS resources can be resized based on their usage pattern.
- How to apportion the total bill to various customers based on their resource usage.
In this three-part series, we will introduce some of these tools. We will start with how to look at the cost reports. We will then introduce some tools that can help better manage the resource footprint. Finally, we will talk about tools that can forecast costs.
Back to Basics
Let’s start with the basics of AWS billing management.
When you choose the Billing and Cost Management option from the AWS console drop down menu, the screen that appears gives an overall picture of the account’s current spending for the month. It also shows the costs incurred for last month’s usage, and based on current usage pattern this month’s billing forecast. Here is an example:
Another widget on the screen breaks down the current month’s spending by service type. As we can see from the image below, the main cost component for this billing cycle is the EC2 service:
Making Sense of AWS Bills
To see the actual billing information, you can either choose to access it from the Bills console or you can refer to the PDF copy sent via e-mail. The PDF version is available if you have chosen the first option from billing preferences screen. Both options are shown below:
We find the latter method useful because it allows you to make the information accessible to relevant people without giving them access to the console. One other hand, the console version allows you to drill down the billing components by service and region, something the PDF version does not offer.
Also, it’s worthwhile to enable the last option (“Receive Billing Reports”). This allows billing reports to be delivered to an S3 bucket. The billing reports are delivered once or more daily, which allows you to refer to them whenever you need rather than waiting for the monthly invoice to come.
In the following image, we have accessed the latest bill from the Billing Management console. We can see that the majority of the costs is associated with Elastic Compute Cloud (EC2).
Drilling down on the EC2 component by expanding the arrow beside it, we can see the where the money has been spent. It shows we have been running our EC2 instances in three regions (ap-southeast-2, us-east and us-west). It also shows individual costs incurred by EBS volumes, data transfers, and Elastic IPs.
Going through these individual regions shows us North Virginia has the highest cost component for EC2 service. This is meaningful information, but as we will see later, there are better ways to find it.
AWS Cost Explorer
AWS Cost Explorer is a graphical cost analysis tool available from the Billing and Cost Management console’s menu. It uses the data from each month’s spending for up to the last thirteen months and can forecast a spend pattern for the next three months. The spend graphs can be tailored for certain time periods as well. With cost explorer, AWS customers can easily visualize where their money is being spent. It can show spending by:
- AWS regions: This can be helpful for large, global companies running workload from multiple regions.
- Availability Zone: Can be useful when you are using an AZ as a “backup” location and not expecting too much cost from there.
- Service: Which type of service is costing most? Typically it’s EC2, but then there can be other resources like EMR or Redshift clusters.
- EC2 instance types: Which EC2 instances are costing the most? Are they supposed to run with large configuration?
- Linked account: If multiple AWS accounts are linked to a “parent” account paying the bill, this report can show which linked account is accruing highest cost. The customer can then be billed back if necessary.
- API calls: What type of API calls are expensive?
- Purchase options: How much is being spent for reserved instances? How much is spent for on-demand instances? If on-demand instances are using larger portion of the spending, it may be worthwhile to think about reserved instances.
- Cost allocation tags: You can attach “tags” to AWS resources. Costs analysis can be mapped against each of the tags.
The following image shows the cost explorer dashboard once it has been enabled. As you can see, AWS cost explorer comes with a few “canned” reports:
In the image below, we are creating a custom view. This is for our RDS instance spending over the last three months and we want to predict how much we would be spending in the next three months. If you click on the image, it will open in a new browser tab.
From the image, you can see we are providing the following details:
- A name for the report.
- A time range of the last three months as an analysis period.
- A time range of the next three months for forecasting.
- Filter the spend by service (in this case, RDS).
- If we did not use a forecast window, we could further drill down by grouping. For example, we could group our last three months’ RDS spending by region.
- A view of monthly expense instead of a daily breakdown.
From this particular report, we can see our RDS spending has been fairly steady in the last three months – and AWS also predicts a similar trend in the coming three months.
Once the report is ready, we can save it to use later. To access a cost explorer report you can choose it from the drop-down list in the Launch Cost Explorer console. In the same drop down list, there is an option for “View/Manage all reports.” Clicking on that option will show a list of both user-defined and pre-configured reports. You can delete your existing reports from that list:
Cost and Usage Reports
AWS customers can also create custom reports based on their usage pattern. These reports are not like cost explorer reports with graphs and trends, but they offer greater granularity of information. The feature is accessible from the Reports tab of the Billing and Cost Management console.
Also, cost and usage reports are automatically generated once defined and periodically saved in an S3 bucket. The frequency of reports is also customizable (hourly or daily). Cost and usage reports are comma separated files (CSV) with a large number of columns. With large AWS footprints, these documents can be fairly large and almost unreadable. You can load them into third-party tools to further analyze, roll-up etc.
To get started, you can create a custom report by clicking on the “Create report” button. This will start a wizard where you can specify various options as shown below:
Note how we have to set proper permissions on the S3 bucket. The tool helps us by providing a sample bucket policy:
Once you review and complete the Wizard, the report will be available:
From the image, you can see a report’s definition can’t be modified. For example, if you want the report to go to a separate S3 bucket, you have to delete and recreate it.
There are some pre-built reports wizards like the EC2 Instance Usage Report. Clicking on this link from the console will open another window where you can provide custom search criteria. In the image below, we are trying to see how much we are spending every day for our m3.large instance types. We have divided the report by platform. From this report, you can see:
- Most of our m3.large instance are using Red Hat Enterprise Linux, while some are using other forms of Linux and the remainder are Windows boxes.
- The average daily spend is $500 per day, but it has fallen slightly in the last two days.
Similarly, we can go beyond the last 14 days and see if we are achieving any savings on EC2 over the last year. From the image below, it seems we are: in August 2015, this AWS account was spending $85,000 for its EC2 instances. As of June 2016, it is spending an average of $65,000, a cost reduction of $20,000:
We will wrap up this post with an introduction to Billing Reports. Billing reports are yet another type of report and like cost and usage reports, they too have to be saved in an S3 bucket in CSV format. Billing Reports can be enabled from the AWS Billing preferences screen. Once the S3 bucket has been verified, you can choose to receive the following types of reports:
- Monthly report
- Detailed report
- Cost allocation report
- Detailed billing report with resources and tags
As you can see, the AWS bill at the end of the month gives only a high level picture of the last billing cycle. You can take advantage of different types of reports and analyze the historical data to better understand your spending pattern. Understanding the spending pattern is only half the job though. The next step should be using that information to optimize the costs. And this is what we will cover in our next article.
New Content: Featuring CKAD Labs in Consumable Sizes
New Content in the Training Library In May, our team of expert certification specialists released seven new or updated learning paths, 18 courses, 28 hands-on labs, and two lab challenges! Three very large CKAD hands-on labs (average duration over two hours each) have been refactore...
New Content: AWS VPC & CloudFormation Playgrounds, Alibaba Lab Challenges and more
New Content in the Training Library In April, our Content Team released three new or updated learning paths, 15 courses, 18 hands-on labs, and six lab challenges! You can always find the latest content additions, as well as insight into what content we’re working on next, on our Conte...
New Content: Platforms, Programming, and DevOps – Something for Everyone
This month our team of expert certification specialists released three new or updated learning paths, 16 courses, 13 hands-on labs, and four lab challenges! New content on Cloud Academy You can always visit our Content Roadmap to see what’s just released as well as what’s coming soon....
Mastering AWS Organizations Service Control Policies
Service Control Policies (SCPs) are IAM-like policies to manage permissions in AWS Organizations. SCPs restrict the actions allowed for accounts within the organization making each one of them compliant with your guidelines. SCPs are not meant to grant permissions; you should consider ...
New Content: Focus on DevOps and Programming Content this Month
This month our team of expert certification specialists released 12 new or updated learning paths, 15 courses, 25 hands-on labs, and four lab challenges! New content on Cloud Academy You can always visit our Content Roadmap to see what’s just released as well as what’s coming soon. Ja...
New Content: Get Ready for the CISM Cert Exam & Learn About Alibaba, Plus All the AWS, GCP, and Azure Courses You Know You Can Count On
This month our team of intrepid certification specialists released five learning paths, seven courses, 19 hands-on labs, and three lab challenges! One particularly interesting new learning path is Certified Information Security Manager (CISM) Foundations. After completing this learn...
Which Certifications Should I Get?
The old AWS slogan, “Cloud is the new normal” is indeed a reality today. Really, cloud has been the new normal for a while now and getting credentials has become an increasingly effective way to quickly showcase your abilities to recruiters and companies. With all that in mind, the s...
The 12 AWS Certifications: Which is Right for You and Your Team?
As companies increasingly shift workloads to the public cloud, cloud computing has moved from a nice-to-have to a core competency in the enterprise. This shift requires a new set of skills to design, deploy, and manage applications in cloud computing. As the market leader and most ma...
AWS Certified Solutions Architect Associate: A Study Guide
Want to take a really impactful step in your technical career? Explore the AWS Solutions Architect Associate certificate. Its new version (SAA-C02) was released on March 23, 2020. The AWS Solutions Architect - Associate Certification (or Sol Arch Associate for short) offers some ...
New Content: AWS Terraform, Java Programming Lab Challenges, Azure DP-900 & DP-300 Certification Exam Prep, Plus Plenty More Amazon, Google, Microsoft, and Big Data Courses
This month our Content Team continues building the catalog of courses for everyone learning about AWS, GCP, and Microsoft Azure. In addition, this month’s updates include several Java programming lab challenges and a couple of courses on big data. In total, we released five new learning...
Where Should You Be Focusing Your AWS Security Efforts?
Another day, another re:Invent session! This time I listened to Stephen Schmidt’s session, “AWS Security: Where we've been, where we're going.” Amongst covering the highlights of AWS security during 2020, a number of newly added AWS features/services were discussed, including: AWS Audit...
AWS re:Invent: 2020 Keynote Top Highlights and More
We’ve gotten through the first five days of the special all-virtual 2020 edition of AWS re:Invent. It’s always a really exciting time for practitioners in the field to see what features and services AWS has cooked up for the year ahead. This year’s conference is a marathon and not a...