Amazon Web Services offers some great tools for managing, analyzing, forecasting, and optimizing infrastructure costs in the cloud. In this three-part blog series, we will explain how you and your organization can take advantage of them.
One of the major factors driving companies to adopt cloud solutions like Amazon Web Services is the minimal upfront cost of resource provisioning followed by an easy, pay-as-you-go operating model. These two factors, along with the disposable nature of the wide variety of available computing resources, and the ease of resizing those resources, makes AWS an attractive alternative to large data centers.
However, companies moving their workloads to the cloud will sooner or later start thinking about optimizing costs. Medium-to-large organizations making a transition to the cloud or who have completed a transition will want to see the benefits of their investment.
These companies – when running large server fleets in their own data centers – were accustomed to seeing computing and storage resources sitting idle and incurring costs even after a project or application’s life-cycle ended. Unfortunately, the practice often remains unchanged after they have migrated to the cloud. The result is unnecessary but easily avoidable expenditure.
Even small companies and cloud-native startups sometimes fall into this trap. Typically, these companies try to minimize their resource footprint and maximize the return on investment in infrastructure. But as the R&D phase ends and products enter the market, keeping infrastructure highly-available takes higher priority than running them optimally. Empty servers, underutilized RDS instances or unattached storage volumes are often overlooked in favor of day-to-day operations.
The outcome is the same: costs are often comparable, or even greater than running an on-premise network. After all, it’s not possible for cloud providers to know when a customer stops using a resource and shut it down for them. That responsibility still lies with the customer.
AWS Billing and Cost Management Tools
Fortunately, Amazon Web Services (AWS) and third-party vendors offer an array of tools for managing and monitoring cloud operating costs. Infrastructure and operation managers will find these tools extremely useful because they can help find:
- What resource types are incurring the largest share of total cost (EC2, RDS, Redshift etc.).
- The trend in monthly bills (is it the same, is it increasing or decreasing?).
- A forecast for the current monthly bill.
- A means to send alerts when costs exceed some threshold.
- Potential areas of savings.
- Estimated costs before a resource is provisioned.
Using the insight from these tools, infrastructure managers can decide:
- Which AWS resources or service can be shut down or retired.
- Which AWS resources can be resized based on their usage pattern.
- How to apportion the total bill to various customers based on their resource usage.
In this three-part series, we will introduce some of these tools. We will start with how to look at the cost reports. We will then introduce some tools that can help better manage the resource footprint. Finally, we will talk about tools that can forecast costs.
Back to Basics
Let’s start with the basics of AWS billing management.
When you choose the Billing and Cost Management option from the AWS console drop down menu, the screen that appears gives an overall picture of the account’s current spending for the month. It also shows the costs incurred for last month’s usage, and based on current usage pattern this month’s billing forecast. Here is an example:
Another widget on the screen breaks down the current month’s spending by service type. As we can see from the image below, the main cost component for this billing cycle is the EC2 service:
Making Sense of AWS Bills
To see the actual billing information, you can either choose to access it from the Bills console or you can refer to the PDF copy sent via e-mail. The PDF version is available if you have chosen the first option from billing preferences screen. Both options are shown below:
We find the latter method useful because it allows you to make the information accessible to relevant people without giving them access to the console. One other hand, the console version allows you to drill down the billing components by service and region, something the PDF version does not offer.
Also, it’s worthwhile to enable the last option (“Receive Billing Reports”). This allows billing reports to be delivered to an S3 bucket. The billing reports are delivered once or more daily, which allows you to refer to them whenever you need rather than waiting for the monthly invoice to come.
In the following image, we have accessed the latest bill from the Billing Management console. We can see that the majority of the costs is associated with Elastic Compute Cloud (EC2).
Drilling down on the EC2 component by expanding the arrow beside it, we can see the where the money has been spent. It shows we have been running our EC2 instances in three regions (ap-southeast-2, us-east and us-west). It also shows individual costs incurred by EBS volumes, data transfers, and Elastic IPs.
Going through these individual regions shows us North Virginia has the highest cost component for EC2 service. This is meaningful information, but as we will see later, there are better ways to find it.
AWS Cost Explorer
AWS Cost Explorer is a graphical cost analysis tool available from the Billing and Cost Management console’s menu. It uses the data from each month’s spending for up to the last thirteen months and can forecast a spend pattern for the next three months. The spend graphs can be tailored for certain time periods as well. With cost explorer, AWS customers can easily visualize where their money is being spent. It can show spending by:
- AWS regions: This can be helpful for large, global companies running workload from multiple regions.
- Availability Zone: Can be useful when you are using an AZ as a “backup” location and not expecting too much cost from there.
- Service: Which type of service is costing most? Typically it’s EC2, but then there can be other resources like EMR or Redshift clusters.
- EC2 instance types: Which EC2 instances are costing the most? Are they supposed to run with large configuration?
- Linked account: If multiple AWS accounts are linked to a “parent” account paying the bill, this report can show which linked account is accruing highest cost. The customer can then be billed back if necessary.
- API calls: What type of API calls are expensive?
- Purchase options: How much is being spent for reserved instances? How much is spent for on-demand instances? If on-demand instances are using larger portion of the spending, it may be worthwhile to think about reserved instances.
- Cost allocation tags: You can attach “tags” to AWS resources. Costs analysis can be mapped against each of the tags.
The following image shows the cost explorer dashboard once it has been enabled. As you can see, AWS cost explorer comes with a few “canned” reports:
In the image below, we are creating a custom view. This is for our RDS instance spending over the last three months and we want to predict how much we would be spending in the next three months. If you click on the image, it will open in a new browser tab.
From the image, you can see we are providing the following details:
- A name for the report.
- A time range of the last three months as an analysis period.
- A time range of the next three months for forecasting.
- Filter the spend by service (in this case, RDS).
- If we did not use a forecast window, we could further drill down by grouping. For example, we could group our last three months’ RDS spending by region.
- A view of monthly expense instead of a daily breakdown.
From this particular report, we can see our RDS spending has been fairly steady in the last three months – and AWS also predicts a similar trend in the coming three months.
Once the report is ready, we can save it to use later. To access a cost explorer report you can choose it from the drop-down list in the Launch Cost Explorer console. In the same drop down list, there is an option for “View/Manage all reports.” Clicking on that option will show a list of both user-defined and pre-configured reports. You can delete your existing reports from that list:
Cost and Usage Reports
AWS customers can also create custom reports based on their usage pattern. These reports are not like cost explorer reports with graphs and trends, but they offer greater granularity of information. The feature is accessible from the Reports tab of the Billing and Cost Management console.
Also, cost and usage reports are automatically generated once defined and periodically saved in an S3 bucket. The frequency of reports is also customizable (hourly or daily). Cost and usage reports are comma separated files (CSV) with a large number of columns. With large AWS footprints, these documents can be fairly large and almost unreadable. You can load them into third-party tools to further analyze, roll-up etc.
To get started, you can create a custom report by clicking on the “Create report” button. This will start a wizard where you can specify various options as shown below:
Note how we have to set proper permissions on the S3 bucket. The tool helps us by providing a sample bucket policy:
Once you review and complete the Wizard, the report will be available:
From the image, you can see a report’s definition can’t be modified. For example, if you want the report to go to a separate S3 bucket, you have to delete and recreate it.
There are some pre-built reports wizards like the EC2 Instance Usage Report. Clicking on this link from the console will open another window where you can provide custom search criteria. In the image below, we are trying to see how much we are spending every day for our m3.large instance types. We have divided the report by platform. From this report, you can see:
- Most of our m3.large instance are using Red Hat Enterprise Linux, while some are using other forms of Linux and the remainder are Windows boxes.
- The average daily spend is $500 per day, but it has fallen slightly in the last two days.
Similarly, we can go beyond the last 14 days and see if we are achieving any savings on EC2 over the last year. From the image below, it seems we are: in August 2015, this AWS account was spending $85,000 for its EC2 instances. As of June 2016, it is spending an average of $65,000, a cost reduction of $20,000:
We will wrap up this post with an introduction to Billing Reports. Billing reports are yet another type of report and like cost and usage reports, they too have to be saved in an S3 bucket in CSV format. Billing Reports can be enabled from the AWS Billing preferences screen. Once the S3 bucket has been verified, you can choose to receive the following types of reports:
- Monthly report
- Detailed report
- Cost allocation report
- Detailed billing report with resources and tags
As you can see, the AWS bill at the end of the month gives only a high level picture of the last billing cycle. You can take advantage of different types of reports and analyze the historical data to better understand your spending pattern. Understanding the spending pattern is only half the job though. The next step should be using that information to optimize the costs. And this is what we will cover in our next article.
WaitCondition Controls the Pace of AWS CloudFormation Templates
AWS's WaitCondition can be used with CloudFormation templates to ensure required resources are running.As you may already be aware, AWS CloudFormation is used for infrastructure automation by allowing you to write JSON templates to automatically install, configure, and bootstrap your ...
The 9 AWS Certifications: Which is Right for You and Your Team?
As companies increasingly shift workloads to the public cloud, cloud computing has moved from a nice-to-have to a core competency in the enterprise. This shift requires a new set of skills to design, deploy, and manage applications in the cloud.As the market leader and most mature p...
Two New EC2 Instance Types Announced at AWS re:Invent 2018 – Monday Night Live
The announcements at re:Invent just keep on coming! Let’s look at what benefits these two new EC2 instance types offer and how these two new instances could be of benefit to you. If you're not too familiar with Amazon EC2, you might want to familiarize yourself by creating your first Am...
Google Cloud Certification: Preparation and Prerequisites
Google Cloud Platform (GCP) has evolved from being a niche player to a serious competitor to Amazon Web Services and Microsoft Azure. In 2018, research firm Gartner placed Google in the Leaders quadrant in its Magic Quadrant for Cloud Infrastructure as a Service for the first time. In t...
Understanding AWS VPC Egress Filtering Methods
In order to understand AWS VPC egress filtering methods, you first need to understand that security on AWS is governed by a shared responsibility model where both vendor and subscriber have various operational responsibilities. AWS assumes responsibility for the underlying infrastructur...
S3 FTP: Build a Reliable and Inexpensive FTP Server Using Amazon’s S3
Is it possible to create an S3 FTP file backup/transfer solution, minimizing associated file storage and capacity planning administration headache?FTP (File Transfer Protocol) is a fast and convenient way to transfer large files over the Internet. You might, at some point, have conf...
Microservices Architecture: Advantages and Drawbacks
Microservices are a way of breaking large software projects into loosely coupled modules, which communicate with each other through simple Application Programming Interfaces (APIs).Microservices have become increasingly popular over the past few years. The modular architectural style,...
What Are Best Practices for Tagging AWS Resources?
There are many use cases for tags, but what are the best practices for tagging AWS resources? In order for your organization to effectively manage resources (and your monthly AWS bill), you need to implement and adopt a thoughtful tagging strategy that makes sense for your business. The...
How to Optimize Amazon S3 Performance
Amazon S3 is the most common storage options for many organizations, being object storage it is used for a wide variety of data types, from the smallest objects to huge datasets. All in all, Amazon S3 is a great service to store a wide scope of data types in a highly available and resil...
How to Optimize Cloud Costs with Spot Instances: New on Cloud Academy
One of the main promises of cloud computing is access to nearly endless capacity. However, it doesn’t come cheap. With the introduction of Spot Instances for Amazon Web Services’ Elastic Compute Cloud (AWS EC2) in 2009, spot instances have been a way for major cloud providers to sell sp...
What are the Benefits of Machine Learning in the Cloud?
A Comparison of Machine Learning Services on AWS, Azure, and Google CloudArtificial intelligence and machine learning are steadily making their way into enterprise applications in areas such as customer support, fraud detection, and business intelligence. There is every reason to beli...
How to Use AWS CLI
The AWS Command Line Interface (CLI) is for managing your AWS services from a terminal session on your own client, allowing you to control and configure multiple AWS services.So you’ve been using AWS for awhile and finally feel comfortable clicking your way through all the services....