Amazon S3 Replication and Bucket Key Encryption
The course is part of these learning paths
This course explores two different Amazon S3 features: the replication of data between buckets and bucket key encryption when working with SSE-KMS to protect your data. You will learn how Amazon S3 replication works, when to use it, and some of the configurable options. We'll also look at how S3 Bucket Keys can be used to reduce costs when using SSE-KMS.
If you have any feedback relating to this course, please contact us at firstname.lastname@example.org.
The objectives of this course are to explain:
- How Amazon S3 replication works, when you might use it, and some of the configurable options
- How S3 Bucket Keys can be used to reduce costs when using SSE-KMS
This course has been designed for those who support, operate, and architect solutions involving Amazon S3.
As a prerequisite to this course, it would be advantageous to have a working knowledge of Amazon S3, including some basic understanding of S3 security and management features.
Hello and welcome to this lecture which will look at how bucket keys can reduce costs when working with the S3 encryption option of Server-side encryption using KMS managed keys, known as SSE-KMS. If you are unfamiliar with KMS, the Key Management Service, then we do have a course here that goes into detail on the topic:
How to use KMS Key encryption to protect your data: https://cloudacademy.com/course/amazon-web-services-key-management-service-kms/
Before we look at this in more detail, let me provide you with a quick explanation of some of the components mentioned in this lecture, starting with the CMK, the Customer Master Key.
This is the main key type within KMS and it contains the key material that is used to encrypt your data and there are 3 different types of CMK:
- Customer Managed: These keys offer the greatest level of flexibility and control out of the 3 types. You are able to create, disable or delete the key, configure the key policies associated with your key, configure Grants, and also alter and adjust the key rotation periods and view full usage history of the key.
- AWS Managed: So AWS Managed Keys are, as the name suggests, managed by AWS, however you are still able to view these keys within the Management Console, and also audit and track their usage and view their key policies. However, because they are managed by AWS you are not able to modify them, for example, it's not possible to edit the key policy or control their rotation frequency. They can only be used by the service that creates them and can be identified by their alias, for example aws/s3 is an AWS managed key used for S3 encryption.
- and AWS Owned, now these are not visible within the KMS console or anywhere within your account, neither do you have the ability to audit and track their usage, they are essentially abstracted from your AWS account. But of course, some services use this key type to encrypt your data within your account, for example the S3 Master key used for SSE-S3 encryption.
The CMK NEVER leaves KMS, it is created within KMS and remains within KMS at all times, but they can generate Data Encryption Keys and bucket keys and these keys can leave KMS and are used by other AWS services to implement encryption, such as S3.
Next, we have Data Encryption Keys. Data keys are created by CMKs however they are used outside of KMS to perform encryption against your data, either in your own applications or by other AWS services.
When a request to generate a data key is received by KMS, the associated CMK in the request will create 2 identical data encryption keys, one will be a plaintext key, and the other will be an encrypted key
During the encryption process, it’s the plaintext data key that will be used to perform the encryption of your data using an encryption algorithm. Then once the encryption has taken place, this plaintext data key will then be deleted and the encrypted data key will be stored and associated with the newly encrypted data.
To demonstrate an example of DEKs being used within AWS services, let me take a quick walkthrough at how the encryption and decryption process works for Amazon S3 server-side encryption with KMS managed keys, known as SSE-KMS, but this is without the S3 bucket keys enabled.
The encryption process is as follows
- Firstly, a client uploads object data to S3.
- S3 then requests data keys from a KMS-CMK.
- Using the specified CMK, KMS generates two data keys, a plain text data key and an encrypted version of the same data key.
- These two keys are then sent back to S3.
- S3 then combines the object data and the plain text data key to perform the encryption. This creates an encrypted version of the object data which is then stored on S3 along with the encrypted data key. The plaintext data key is then removed from memory.
The decryption process is as follows:
- A request is made by the client to S3 to retrieve the object data.
- S3 sends the associated encrypted data key of the object data to KMS.
- KMS then uses the correct CMK with the encrypted data key to decrypt it and create a plain text data key.
- This plain text data key is then sent back to S3
- The plain text data key is then combined with the encrypted object data to decrypt it and the plain text key is deleted from memory
- This decrypted object data is then sent back to the client
For more information on other S3 encryption mechanisms and how they work, please see our existing course here:
Understanding encryption mechanisms to secure your data: https://cloudacademy.com/course/s3-encryption-mechanisms/
So now we have a good foundation of server-side encryption when using KMS managed keys, let’s take a look at how Bucket keys can be introduced to the mix when using SSE-KMS.
Bucket keys will reduce your overall spend when using SSE-KMS by reducing the amount of requests and traffic from S3 to KMS. So let’s take a look at how it sits within the infrastructure to understand how this is achieved. Firstly, let’s look at the process of enabling bucket keys on a newly created bucket.
- A bucket is created and configured with SSE-KMS encryption with bucket keys enabled
- S3 sends a request to KMS indicating that it requires a new bucket key using a specific CMK
- KMS will respond by generating a new bucket key using the CMK indicated in the request
- KMS will send the bucket key back to S3
- S3 will then associate that bucket key with the newly created bucket
So at this stage, we now have a bucket key associated with your bucket, but what happens when you try to add an object to this bucket using the Bucket encryption key? Let’s take a look
- Object is uploaded to the S3 bucket using SSE-KMS
- S3 uses the associated S3 bucket key to generate 2 data keys, a plaintext data key and an encrypted version of the same data key.
- S3 then combines the object data and the plaintext data key to perform the encryption. This creates an encrypted version of the object data which is then stored on S3 along with the encrypted data key.
- The plaintext data key is then removed from memory.
As you can see, by using the bucket key, there is no need to request the data keys to be generated from the CMK in KMS, this reduces the number of requests to KMS and therefore reduces your encryption costs up to as much as 99%, which can be a lot of money if accessing millions of objects a month that are encrypted. The keys are instead generated by your S3 bucket key and so the encryption process remains within S3.
Let me now show you how the decryption process works:
- A request is made by the client to S3 to retrieve the object data.
- S3 combines the associated encrypted data key of the object data with the Bucket key to generate a plaintext key.
- S3 then uses the plaintext data key with the encrypted object data to decrypt it
- The plaintext key is deleted from memory
- The decrypted object data is then sent back to the client by S3
Stuart has been working within the IT industry for two decades covering a huge range of topic areas and technologies, from data center and network infrastructure design, to cloud architecture and implementation.
To date, Stuart has created 90+ courses relating to Cloud reaching over 100,000 students, mostly within the AWS category and with a heavy focus on security and compliance.
Stuart is a member of the AWS Community Builders Program for his contributions towards AWS.
He is AWS certified and accredited in addition to being a published author covering topics across the AWS landscape.
In January 2016 Stuart was awarded ‘Expert of the Year Award 2015’ from Experts Exchange for his knowledge share within cloud services to the community.
Stuart enjoys writing about cloud technologies and you will find many of his articles within our blog pages.