Amazon Web Services offers some great tools for managing, analyzing, forecasting, and optimizing infrastructure costs in the cloud. In this three-part blog series, we will explain how you and your organization can take advantage of them.
One of the major factors driving companies to adopt cloud solutions like Amazon Web Services is the minimal upfront cost of resource provisioning followed by an easy, pay-as-you-go operating model. These two factors, along with the disposable nature of the wide variety of available computing resources, and the ease of resizing those resources, makes AWS an attractive alternative to large data centers.
However, companies moving their workloads to the cloud will sooner or later start thinking about optimizing costs. Medium-to-large organizations making a transition to the cloud or who have completed a transition will want to see the benefits of their investment.
These companies – when running large server fleets in their own data centers – were accustomed to seeing computing and storage resources sitting idle and incurring costs even after a project or application’s life-cycle ended. Unfortunately, the practice often remains unchanged after they have migrated to the cloud. The result is unnecessary but easily avoidable expenditure.
Even small companies and cloud-native startups sometimes fall into this trap. Typically, these companies try to minimize their resource footprint and maximize the return on investment in infrastructure. But as the R&D phase ends and products enter the market, keeping infrastructure highly-available takes higher priority than running them optimally. Empty servers, underutilized RDS instances or unattached storage volumes are often overlooked in favor of day-to-day operations.
The outcome is the same: costs are often comparable, or even greater than running an on-premise network. After all, it’s not possible for cloud providers to know when a customer stops using a resource and shut it down for them. That responsibility still lies with the customer.
AWS Billing and Cost Management Tools
Fortunately, Amazon Web Services (AWS) and third-party vendors offer an array of tools for managing and monitoring cloud operating costs. Infrastructure and operation managers will find these tools extremely useful because they can help find:
- What resource types are incurring the largest share of total cost (EC2, RDS, Redshift etc.).
- The trend in monthly bills (is it the same, is it increasing or decreasing?).
- A forecast for the current monthly bill.
- A means to send alerts when costs exceed some threshold.
- Potential areas of savings.
- Estimated costs before a resource is provisioned.
Using the insight from these tools, infrastructure managers can decide:
- Which AWS resources or service can be shut down or retired.
- Which AWS resources can be resized based on their usage pattern.
- How to apportion the total bill to various customers based on their resource usage.
In this three-part series, we will introduce some of these tools. We will start with how to look at the cost reports. We will then introduce some tools that can help better manage the resource footprint. Finally, we will talk about tools that can forecast costs.
Back to Basics
Let’s start with the basics of AWS billing management.
When you choose the Billing and Cost Management option from the AWS console drop down menu, the screen that appears gives an overall picture of the account’s current spending for the month. It also shows the costs incurred for last month’s usage, and based on current usage pattern this month’s billing forecast. Here is an example:
Another widget on the screen breaks down the current month’s spending by service type. As we can see from the image below, the main cost component for this billing cycle is the EC2 service:
Making Sense of AWS Bills
To see the actual billing information, you can either choose to access it from the Bills console or you can refer to the PDF copy sent via e-mail. The PDF version is available if you have chosen the first option from billing preferences screen. Both options are shown below:
We find the latter method useful because it allows you to make the information accessible to relevant people without giving them access to the console. One other hand, the console version allows you to drill down the billing components by service and region, something the PDF version does not offer.
Also, it’s worthwhile to enable the last option (“Receive Billing Reports”). This allows billing reports to be delivered to an S3 bucket. The billing reports are delivered once or more daily, which allows you to refer to them whenever you need rather than waiting for the monthly invoice to come.
In the following image, we have accessed the latest bill from the Billing Management console. We can see that the majority of the costs is associated with Elastic Compute Cloud (EC2).
Drilling down on the EC2 component by expanding the arrow beside it, we can see the where the money has been spent. It shows we have been running our EC2 instances in three regions (ap-southeast-2, us-east and us-west). It also shows individual costs incurred by EBS volumes, data transfers, and Elastic IPs.
Going through these individual regions shows us North Virginia has the highest cost component for EC2 service. This is meaningful information, but as we will see later, there are better ways to find it.
AWS Cost Explorer
AWS Cost Explorer is a graphical cost analysis tool available from the Billing and Cost Management console’s menu. It uses the data from each month’s spending for up to the last thirteen months and can forecast a spend pattern for the next three months. The spend graphs can be tailored for certain time periods as well. With cost explorer, AWS customers can easily visualize where their money is being spent. It can show spending by:
- AWS regions: This can be helpful for large, global companies running workload from multiple regions.
- Availability Zone: Can be useful when you are using an AZ as a “backup” location and not expecting too much cost from there.
- Service: Which type of service is costing most? Typically it’s EC2, but then there can be other resources like EMR or Redshift clusters.
- EC2 instance types: Which EC2 instances are costing the most? Are they supposed to run with large configuration?
- Linked account: If multiple AWS accounts are linked to a “parent” account paying the bill, this report can show which linked account is accruing highest cost. The customer can then be billed back if necessary.
- API calls: What type of API calls are expensive?
- Purchase options: How much is being spent for reserved instances? How much is spent for on-demand instances? If on-demand instances are using larger portion of the spending, it may be worthwhile to think about reserved instances.
- Cost allocation tags: You can attach “tags” to AWS resources. Costs analysis can be mapped against each of the tags.
The following image shows the cost explorer dashboard once it has been enabled. As you can see, AWS cost explorer comes with a few “canned” reports:
In the image below, we are creating a custom view. This is for our RDS instance spending over the last three months and we want to predict how much we would be spending in the next three months. If you click on the image, it will open in a new browser tab.
From the image, you can see we are providing the following details:
- A name for the report.
- A time range of the last three months as an analysis period.
- A time range of the next three months for forecasting.
- Filter the spend by service (in this case, RDS).
- If we did not use a forecast window, we could further drill down by grouping. For example, we could group our last three months’ RDS spending by region.
- A view of monthly expense instead of a daily breakdown.
From this particular report, we can see our RDS spending has been fairly steady in the last three months – and AWS also predicts a similar trend in the coming three months.
Once the report is ready, we can save it to use later. To access a cost explorer report you can choose it from the drop-down list in the Launch Cost Explorer console. In the same drop down list, there is an option for “View/Manage all reports.” Clicking on that option will show a list of both user-defined and pre-configured reports. You can delete your existing reports from that list:
Cost and Usage Reports
AWS customers can also create custom reports based on their usage pattern. These reports are not like cost explorer reports with graphs and trends, but they offer greater granularity of information. The feature is accessible from the Reports tab of the Billing and Cost Management console.
Also, cost and usage reports are automatically generated once defined and periodically saved in an S3 bucket. The frequency of reports is also customizable (hourly or daily). Cost and usage reports are comma separated files (CSV) with a large number of columns. With large AWS footprints, these documents can be fairly large and almost unreadable. You can load them into third-party tools to further analyze, roll-up etc.
To get started, you can create a custom report by clicking on the “Create report” button. This will start a wizard where you can specify various options as shown below:
Note how we have to set proper permissions on the S3 bucket. The tool helps us by providing a sample bucket policy:
Once you review and complete the Wizard, the report will be available:
From the image, you can see a report’s definition can’t be modified. For example, if you want the report to go to a separate S3 bucket, you have to delete and recreate it.
There are some pre-built reports wizards like the EC2 Instance Usage Report. Clicking on this link from the console will open another window where you can provide custom search criteria. In the image below, we are trying to see how much we are spending every day for our m3.large instance types. We have divided the report by platform. From this report, you can see:
- Most of our m3.large instance are using Red Hat Enterprise Linux, while some are using other forms of Linux and the remainder are Windows boxes.
- The average daily spend is $500 per day, but it has fallen slightly in the last two days.
Similarly, we can go beyond the last 14 days and see if we are achieving any savings on EC2 over the last year. From the image below, it seems we are: in August 2015, this AWS account was spending $85,000 for its EC2 instances. As of June 2016, it is spending an average of $65,000, a cost reduction of $20,000:
We will wrap up this post with an introduction to Billing Reports. Billing reports are yet another type of report and like cost and usage reports, they too have to be saved in an S3 bucket in CSV format. Billing Reports can be enabled from the AWS Billing preferences screen. Once the S3 bucket has been verified, you can choose to receive the following types of reports:
- Monthly report
- Detailed report
- Cost allocation report
- Detailed billing report with resources and tags
As you can see, the AWS bill at the end of the month gives only a high level picture of the last billing cycle. You can take advantage of different types of reports and analyze the historical data to better understand your spending pattern. Understanding the spending pattern is only half the job though. The next step should be using that information to optimize the costs. And this is what we will cover in our next article.
AWS Security: Bastion Host, NAT instances and VPC Peering
Effective security requires close control over your data and resources. Bastion hosts, NAT instances, and VPC peering can help you secure your AWS infrastructure. Welcome to part four of my AWS Security overview. In part three, we looked at network security at the subnet level. This ti...
Top 13 Amazon Virtual Private Cloud (VPC) Best Practices
Amazon Virtual Private Cloud (VPC) brings a host of advantages to the table, including static private IP addresses, Elastic Network Interfaces, secure bastion host setup, DHCP options, Advanced Network Access Control, predictable internal IP ranges, VPN connectivity, movement of interna...
Big Changes to the AWS Certification Exams
With AWS re:Invent 2019 just around the corner, we can expect some early announcements to trickle through with upcoming features and services. However, AWS has just announced some big changes to their certification exams. So what’s changing and what’s new? There is a brand NEW ...
New on Cloud Academy: ITIL® 4, Microsoft 365 Tenant, Jenkins, TOGAF® 9.1, and more
At Cloud Academy, we're always striving to make improvements to our training platform. Based on your feedback, we released some new features to help make it easier for you to continue studying. These new features allow you to: Remove content from “Continue Studying” section Disc...
AWS Security Groups: Instance Level Security
Instance security requires that you fully understand AWS security groups, along with patching responsibility, key pairs, and various tenancy options. As a precursor to this post, you should have a thorough understanding of the AWS Shared Responsibility Model before moving onto discussi...
Cloud Migration Risks & Benefits
If you’re like most businesses, you already have at least one workload running in the cloud. However, that doesn’t mean that cloud migration is right for everyone. While cloud environments are generally scalable, reliable, and highly available, those won’t be the only considerations dri...
Real-Time Application Monitoring with Amazon Kinesis
Amazon Kinesis is a real-time data streaming service that makes it easy to collect, process, and analyze data so you can get quick insights and react as fast as possible to new information. With Amazon Kinesis you can ingest real-time data such as application logs, website clickstre...
Google Cloud Functions vs. AWS Lambda: The Fight for Serverless Cloud Domination
Serverless computing: What is it and why is it important? A quick background The general concept of serverless computing was introduced to the market by Amazon Web Services (AWS) around 2014 with the release of AWS Lambda. As we know, cloud computing has made it possible for users to ...
Google Vision vs. Amazon Rekognition: A Vendor-Neutral Comparison
Google Cloud Vision and Amazon Rekognition offer a broad spectrum of solutions, some of which are comparable in terms of functional details, quality, performance, and costs. This post is a fact-based comparative analysis on Google Vision vs. Amazon Rekognition and will focus on the tech...
New on Cloud Academy: CISSP, AWS, Azure, & DevOps Labs, Python for Beginners, and more…
As Hurricane Dorian intensifies, it looks like Floridians across the entire state might have to hunker down for another big one. If you've gone through a hurricane, you know that preparing for one is no joke. You'll need a survival kit with plenty of water, flashlights, batteries, and n...
Amazon Route 53: Why You Should Consider DNS Migration
What Amazon Route 53 brings to the DNS table Amazon Route 53 is a highly available and scalable Domain Name System (DNS) service offered by AWS. It is named by the TCP or UDP port 53, which is where DNS server requests are addressed. Like any DNS service, Route 53 handles domain regist...
How to Unlock Complimentary Access to Cloud Academy
Are you looking to get trained or certified on AWS, Azure, Google Cloud Platform, DevOps, Cloud Security, Python, Java, or another technical skill? Then you'll want to mark your calendars for August 23, 2019. Starting Friday at 12:00 a.m. PDT (3:00 a.m. EDT), Cloud Academy is offering c...