Storage and Databases
Services at a glance
In this course we learn to recognize and explain AWS compute and storage fundamentals, and to recognise and explain the family of AWS services relevant to the certified developer exam. This course provides you with snapshots of each service, and covering just what you need to know, gives you a good, high-level starting point for exam preparation. It includes coverage of:
Amazon Simple Queue Service (SQS)
Amazon Simple Notification Service (SNS)
Amazon Simple Workflow Service (SWF)
Amazon Simple Email Service (SES)
Amazon API Gateway
Amazon Data Pipeline
AWS Elastic Beanstalk
Storage and database
Amazon Simple Storage Service (S3)
Amazon Elastic Block Store (EBS)
AWS Relational Database Service (RDS)
Other Database Services
Elastic Cloud Compute (EC2)
Elastic Load Balancing (ELB)
If you have thoughts or suggestions for this course, please contact Cloud Academy at firstname.lastname@example.org.
Hello, and welcome to this lecture where I will explain what the EC2 service is and does and how to configure an EC2 instance, so let's get started.
As EC2 is one of the most common services used throughout AWS, I will discuss this service in greater detail over the other services that I will cover in this course.
EC2 is the most common compute service that AWS offers, as it allows you to deploy virtual servers within your AWS environment; and most people will require a server in one form or another as a part of their solution. There are a number of elements in creating your EC2 instance, which I want to break down and explain. This will hopefully help to define how the service works and answer a number of questions that you may have.
The EC2 service can be broken down into the following sections:
- Amazon Machine Images
- instance types
- instance purchasing options
- user data
- storage options
- and security
The first point I want to cover are AMIs, Amazon Machine Images. These are essentially templates of pre-configured EC2 instances which allow you to quickly launch a new EC2 instance based on the configuration within the AMI. This prevents you from having to install an operating system or any other common applications that you might need to install on a number of other EC2 instances. From a high-level perspective, an AMI will include an operating system and applications, along with any custom configuration.
AWS provides a large number of AMIs covering different operating systems, from Linux to Red Hat to Microsoft Windows, among others. When configuring your EC2 instance, selecting your AMI is the first configuration choice you will need to make. You can also create your own AMI images to help you speed up your own deployments.
For example, you would start with selecting an AWS AMI, let's say a Linux server. Once it is up and running, you may then need to install a number of your own custom applications and make specific configuration changes. Now if you needed another server to perform the same functionality, you could go through the same process of selecting a Linux AWS AMI and, again, manually installing your applications and make your configurations. Or, once you have made those changes on the first server, you can then simply create a brand new AMI of that instance with all the applications installed and configurations already made. Then if you need another server of the same configuration, all you would need to do is to select your custom AMI as the base image for your instance and it will launch the Linux server, your custom applications already installed, and any configurations already made.
As you can see, this has many benefits and certainly comes in useful when we start to look at autoscaling later in this course. In addition to both AWS-managed and your own custom AMIs, you could also select an AMI from the AWS marketplace. The AWS marketplace is basically an online store that allows you to purchase AMIs from trusted vendors like Cisco, Citrix, Alert Logic, etc. These vendor AMIs may have specific applications and configurations already made, such as instances that are optimized with built-in security and monitoring tools or contain database migration systems. Lastly, community AMIs also exist, which are a repository of AMIs that have been created and shared by other AWS members.
Let's now take a look at instance types. Once you have selected your AMI from any of the different sources already discussed, you must then select an instance type. An instance type simply defines the size of the instance from a CPU, memory, storage, and networking perspective. Having this flexibility of varied instances allows you to select the most appropriate size or power of a virtual server that you need for optimal performance with your applications.
These different instance types are categorized into different family types that offer distinct performance benefits which, again, helps you to select the most appropriate instance for your needs. Within each of these instance families, you will have a range of instance types with varied CPU, memory, storage, and networking performance. These instance families can be summarized as follows.
- General purpose; instance types within this family have a balanced mixed of CPU, memory, and storage, making them ideal for small-to-medium databases and test and development servers and backend servers.
- Compute optimized; as the name implies, instance types within this family have a greater focus on compute. They have the highest performing processes installed, allowing them to be used for high-performance front-end servers, web servers, high-performance science and engineering applications, video encoding, and batch processing.
- GPU; GPU stands for graphics processing unit, and so the instances within this family are optimized for graphic-intensive applications.
- Memory optimized; instance types here are primarily used for large-scale, enterprise-class, in-memory applications, such as performing real time processing of unstructured data or for in-memory databases such as SAP HANA.
- Storage optimized; as expected, these are optimized for enhanced storage. Instances in this family use SSD-backed instance storage for low latency and very high input/output performance, including very high IOPS, which stands for input output operations per second. These are great for analytic workloads and no SQL databases, data file systems and lock processing applications.
You can purchase EC2 instances through a variety of different payment plans. These have been designed to help you save costs by selecting the most appropriate option for your deployment. The different EC2 payment options are as follows:
- on-demand instances
- reserved instances
- spot instances
- dedicated instances
- and dedicated hosts
It's good to be aware of these different options, as by having an understanding of these can help you save a considerable amount of money depending on your use case. Let me run through each option to help explain.
Starting with on-demand instances, these are EC2 instances that you can launch at any time and have it provisioned and available to you within minutes. You can use this instance for a shorter time or for as long as you need before terminating the instance. These instances have a flat rate and is determined on the instance type selected and is paid by the hour. On-demand instances are typically used for short-term uses where workloads can be irregular and where the workload can't by interrupted. Many users of AWS use on-demand instances within their testing and development environments.
Reserved instances allow you to purchase an instance type for a set period of time in return for a reduced cost compared to on-demand. This reduction can be as much as 75%. These reservations against instances must be purchased in either one- or three-year timeframes. Further reductions can be achieved with reserved instances depending on which payment methods you select. There are three options available to you.
- All upfront; the complete payment for the one- or three-year reservation is paid. This offers the largest discount and no further payment is required, regardless of the number of hours the instance is used.
- Partial upfront; a smaller upfront payment is made, and then a discount is applied to any hours used by the instance during the term.
- No upfront; no upfront or partial payments are made, and the smallest discount of the three models is applied to any hours used by the instance.
Reserved instances are used for long-term, predictable workloads, allowing you to make full use of the cost savings to be had when using compute resources offered by EC2.
Spot instances allow you to bid for unused EC2 compute resources; however, your resource is not guaranteed for a fixed period of time. To use an EC2 spot instance, you must bid higher than the current spot price which is set by AWS. This spot price fluctuates depending on supply and demand of the unused resource. If your bid price for an instance type is higher than the spot price, then you'll purchase the instance; but as soon your bid price becomes lower than the fluctuating spot price, you will be issued a two-minute warning before the instance is automatically terminated and removed from your environment by AWS. The bonus of spot instances is that you can bid for large EC2 instances at a very low price point, saving a huge amount. But due to the nature of how the instances can suddenly be removed from your environment, spot instances are only useful for processing data and applications that can be suddenly interrupted, such as batch jobs and background processing of data.
One key point to make about pricing with EC2 instances is that you'll only pay for the instance when it is up and running, excluding reserved instances all upfront. You will not pay for the instance if you are stopped or terminated the instance.
Let me now talk to you about EC2 tenancy. This relates to what underlying host your EC2 instance will reside on as a virtual server within the AWS data center. Again, there are different options available to you, with pros and cons to each.
- Shared tenancy; this option will launch your EC2 instance on any available host with the specified resources required for your selected instance type, regardless of which other customers and users also have EC2 instances running on the same host, hence the shared tenancy name. Advanced security mechanisms are in place to prevent one EC2 instance accessing another on the same host. How this security is applied and operated is out of scope of this course and is maintained by AWS themselves.
- Dedicated tenancy includes both dedicated instances and dedicated hosts.
- Dedicated instances are hosted on hardware that no other customer can access. It can only be accessed by your own AWS account. You may be required to launch your instances as a dedicated instance due to internal security policies or external compliance controls. Dedicated instances do incur additional charges due to the fact you are preventing other customers from running EC2 instances on the same hardware, and so there will be unused capacity.
- Dedicated hosts; dedicated hosts are effectively the same as dedicated instances, however, they offer additional visibility and control over how you place your instances on the physical host. This allows you to use the same host for a number of instances that you want to launch. Also, dedicated hosts allow you to align with any compliance and regulatory requirements.
If you do not need to address any compliance issues that require dedicated tenancy, then I would recommend using shared tenancy to reduce your overall costs.
During the configuration of your EC2 instance, there is a section called user data, which allows you to enter commands that will run during the first boot cycle of that instance. This is a great way to automatically perform functions upon boot, such as to pull down any additional software you want installing from any software repositories you may have. You could also download and get the latest OS updates during boot. For example, you could enter yum update /y for a Linux server, which will then update its own software automatically at time of boot.
As a part of the configuration when setting up an EC2 instance, you are asked to select and configure your storage requirements. Selecting storage for your EC2 instance will depend on what you intend to use the instance for and how critical the data is. Storage for EC2 can be classified between two distinct categories:
- persistent storage
- and ephemeral, which is temporary storage
Persistent storage is available by using elastic block storage, EBS volumes; and ephemeral storage is created by some EC2 instances themselves using local storage on the underlying host, known as instance backed storage. Let's look at each of these storage options in greater depth.
EBS volumes are separate devices from the EC2 instance itself, and so it is not physically attached like ephemeral storage is. EBS volumes are considered network attached storage devices, which are then logically attached to the EC2 instance by the AWS network. This principle is not dissimilar to attaching an external hard disk to your home laptop or PC where the external hard disk would be your EBS volume and your PC is your EC2 instance. The external disk is separate from your own computer's in-built hard disk and is instead attached via a cable, whereas this would be reflected as the AWS network between an EBS volume and an EC2 instance. The data on these volumes are automatically replicated to other EBS volumes within the same availability zone for resiliency, which is all managed by AWS. You can disconnect the EBS volume from your EC2 instance and the data will remain intact, allowing you to reattach it to another EC2 instance if required. You can also implement encryption on these volumes and, if required, take backup snapshots of all the data on the volume to the simple storage service, S3. EBS volumes can be created in different sizes, again with different performance capabilities depending on your requirements.
Ephemeral storage, or instance backed storage, is storage that is physically attached to the underlying host on which the EC2 instance resides on. Looking back at our previous example, this would be similar to your own laptop or PCs hard disk. There is a difference here though. With AWS EC2 instances, as soon as the instance is stopped or terminated, all saved data on that disk is lost. If you reboot your instance, then that data will remain; but not if you stop it. Therefore, if you have any data that you need to retain, it is not recommended that you use instance backed storage for this data. Instead use EBS volumes for persistent data storage. Unlike EBS volumes, you are unable to detach instance store volumes from the instance.
Security is fundamental with any deployment, and so I just want to holler a couple of points relating to security around EC2. Firstly, in doing creation of your EC2 instance, you'll be tasked to select a security group for your instance.
A security group is essentially a firewall for your instance, allowing you to restrict what traffic, from both an ingress and egress perspective, is allowed to communicate with it. You can restrict this communication by source, destination, inbound and outbound rules, along with which ports and protocols can be used. More information on security groups can be found in my previous blog found here covering instance level security.
At the very end of your EC2 instance creation, you will need to select an existing key pair or create and download a new one. But what is a key pair, and what is it used for? A key pair, as the name implies, is made up of two components: a public key and a private key. The function of the key pairs is to encrypt the login information for Linux and Windows EC2 instances and then decrypt the same information, allowing you to authenticate onto the instance.
The public key encrypts data such as the user name and password. For Windows instances, the private key is used to decrypt this data, allowing you to gain access to the login credentials, including the password. For Linux instances, the private key is used to remotely connect onto the instance via SSH. The public key is held and kept by AWS. The private key is your responsibility to keep and ensure that it is not lost.
So going back to when you create your EC2 instance, when creating a new key pair, you are given the opportunity to download the key pair. Once you have done this, you must keep that file safe until you are ready to log on to the associated EC2 instance. It's worth noting that you can use the same key pair on multiple instances. Once you have authenticated onto the EC2 instance the first time, you can then set up your own local authentication methods, such as local Windows user accounts, allowing other users to connect and authenticate to it or even use Microsoft active directory.
One final point regarding security on your EC2 instance. It is your responsibility to maintain and install the latest OS patches and security fixes released by the OS vendor. We have now covered the main elements of the EC2 service that should hopefully allow you to get started by creating your first EC2 instance and selecting the most appropriate configuration for your needs; but to reiterate what we have covered and make it all fit together, I will demonstrate how to create a new EC2 instance from within the console, quickly highlighting the elements we have discussed as I go through. Within this demonstration I will be using the Microsoft Remote Desktop Client for Mac to connect to any instances. More information on this software and how to download it can be found with the link on screen so let’s take a look.
Okay, so in this demonstration I'm going to show you how to quickly create an EC2 instance and then connect to it as well. So I'm logged in to the management console. I've clicked on services; and then if we go down to the compute section, we can find EC2. So if we select that, and then to launch our instance we want to click on the big blue button that says launch instance; and now here we can select our AMI, our Amazon Machine Image.
And these are a number of AMIs that are pre-configured by AWS. We have Windows, we have Linux, we have Red Hat, etc. We also have on the left here the AWS marketplace that I mentioned where we have a number of vendor AMIs and then also the community AMIs as well. For this demonstration, I'm just going to select a Windows server and so I shall select that AMI.
Now on the next screen, we get to choose our instance type; and in this column here, we have the different family types of instances. So we have the general purpose, the compute optimized, the memory optimized, and storage, etc. So I'm going to select the general purpose, and then we have different instance types here; and I'm going to select the T2.Micro, and we can see that different instance types have different number of CPUs, amount of memory, and different network performance,etc.
So I'm going to select the T2.Micro on the general purpose, and then click on next configure instance details. I'm going to select one instance to launch. I can select a spot instance if I'd like to. For this, I'm just going to use an on-demand instance. I can then select a number of network options with regards to where I want this EC2 instance to reside in a specific VPC and availability zone and if I want a public IP to be assigned to this EC2 instance, which is currently set to enable. I can then also select an IAM role if I want specific permissions for this EC2 instance to have. I'm just going to leave that as none.
If we take a look at the shutdown behavior; when I shut down the instance, I can either have the instance just stop, which will mean I keep the instance and it's just left in my VPC and I can restart it again at a later date or I can have it terminated, which means it'll be deleted when I shut down that instance. So I'm just going to leave that as stop so I can restart it again at a later date if I need to.
If I move down to tenancy, we can see that we have shared, dedicated instance, or dedicated host, which we discussed earlier; and I'm just going to leave it as a shared tenancy instance.
If we click on advanced details, here we can see the user data. So this is where you'd add any scripting or any code for commands that you wanted to run on first boot of this instance.
Now if we go down to next add storage, we can see that there's a root volume with this EC2 instance, which will be ephemeral. So that will be instance backed storage, which as we know is temporary; and that's a 30-gig, general purpose SSD drive. If we wanted some persistent storage, we could add a new volume; and we could have that set as EBS, for elastic block storage, which is currently set at 8-gig and, again, SSD storage. If we want to encrypt that volume then it's a simple tick box under the encrypted column here. So now we have some persistent storage with this instance as well.
Following the storage, click on add tags; and this is where we can add key value pairs of information. So I'm going to add a couple of tags. I'll add a name and call this Demo for Cloud Academy, and then we can add another tag if we wanted. We can call that Project, and then just call that demo. Once we've added any tags that we require, we can go down to configure security group.
Now we can select an existing security group or create a new one. Let's create a new one. So we'll give this a name; so, Demo Cloud Academy. Demo Cloud Academy, and the current settings for this security group are RDP, using the TCP protocol, using port 3389 from any IP address. So as it stands, if we have this security group assigned to our instance, it means any IP address can initiate an RDP connection to this instance, which obviously isn't very secure. So I'm just going to select that as my IP. So now only my IP address can initiate an RDP connection to any instances associated with this security group; and if you wanted to add any additional rules, simply click on add rule and fill out the various column details as required. So I'm just going to delete that rule, and we're just going to have that as RDP.
So we're going to go down to review and launch, and here we can see a summary of all the settings that we've selected so far. So we can see our AMI details, the instance type, the security group; if we look at the instance details, we can see the availability zone,etc, and then the shutdown behavior that I mentioned before. We can look at our storage and our tags as well. If we need to change any of this information, then we can simply click on the edit buttons for each section and go back and make any changes; but I'm happy with that, so I'm just going to click on launch and now we have to select a key pair.
So this will allow us to connect to our instance. Now we can choose an existing key pair, create a new key pair, or proceed without a key pair. If we proceeded without a key pair, then there will be no way of connecting to this instance; so let's create a new key pair. Let's give it a name, Demo; and then we can download this key pair. So that will be our private key, and AWS will keep the public key; and then we just need to go down to launch instance. And that will go off and launch our instance as per our configuration; and if we click on view instances, we can see that this instance is starting to fire up.
Here's our name, Demo for Cloud Academy. We can see the instance type and which availability zone it's going to be launched in, and it's currently pending. That will take it a couple of minutes to become active, so what I'll do, I'll pause the video here; and then once it's up and running, we shall carry on.
Okay, we can now see that our EC2 instance is up and running with a green circle and the instance status of running. Now we can see a description for our instance at the bottom of the pane here. We can see the instance state, the type of instance it is, what AMI it's using, the launch time, and also public IP address,etc, which we're going to need to connect to our instance. So let's do that now. So I'm going to copy the IP address. I'm going to go across to my remote desktop client.
So if we click on new for a new connection, just call this demo, paste in the IP address; and we now need the user name and password. So if we go back to our management console, and if we click on connect; we can see here that user name is Administrator, and we need to get the password. Now to decrypt the password, we're going to need our key pair file that we downloaded. So let me just select that, and I'll open; and then we click on decrypt password, and there we have our password for our instance.
So you could see how the key pair file was used to decrypt our password for our administrator user. Now if we go back to our client, Administrator, paste in the password; and if we try to connect, you can see that it's now logging on to the instance. It's setting up the personalized settings. This might take a minute or so just because it's Windows.
And there you are, we are now on the instance itself; and you can see the public IP addresses out there that we connected to and the instance type and the availability zone,etc, so some of the same information. And that is how you create an EC2 instance and connect to it using RDP for a Windows box.
Before we move on from EC2, I just want to point out one last item relating to the status checks that you can see from within the EC2 dashboard once you have launched your instance. These status checks are used to check the health and status of your EC2 instance, and understanding what common faults could trigger these checks to fail can help you troubleshoot issues with your EC2 resources. There are two types of status checks:
- system status checks
- and instance status checks
Let's look at system status checks first. If this fails, then is likely to be an issue with the underlying host rather than a configuration issue with your EC2 instance itself. Common issues that trigger system status checks to fail are loss of power, loss of network connectivity, and hardware ad software issues on the underlying host. Basically, a system status check failure is out of your control as the fault lies with the components that AWS are responsible for. The best way to resolve this would be to stop the instance and restart. This is likely to cause the instance to launch on another physical host, resolving the issue. Do not reboot the instance, as this will cause the instance to continue running on the same physical server.
Let's now take a look at the instance status checks. These differ from system status checks; as if this failed, then it would likely require your input to help resolve the issue. This check looks at the EC2 instance itself, rather than focusing on the underlying host. Common issues that trigger this check to fail are incorrect network configuration, corrupt file systems, exhausted memory, or an incompatible kernel. These faults will require you to troubleshoot and resolve the issue. For example, changing the network configuration.
That brings us to the end of this lecture on EC2. Like I mentioned previously, this was going to be the longest and the most in-depth lecture simply due to this being a key compute service.
If you would like some hands-on experience with EC2, then we do offer two labs in which you can practice creating your own EC2 instances for both Linux and Windows. Coming up in the next lecture, I will discuss elastic load balancing and auto scaling.
About the Author
Stuart has been working within the IT industry for two decades covering a huge range of topic areas and technologies, from data centre and network infrastructure design, to cloud architecture and implementation.
To date Stuart has created over 40 courses relating to Cloud, most within the AWS category with a heavy focus on security and compliance
He is AWS certified and accredited in addition to being a published author covering topics across the AWS landscape.
In January 2016 Stuart was awarded ‘Expert of the Year Award 2015’ from Experts Exchange for his knowledge share within cloud services to the community.
Stuart enjoys writing about cloud technologies and you will find many of his articles within our blog pages.