RDS vs. EC2
Amazon RDS Costs
Data Lakes in AWS
The course is part of this learning path
This section of the Solution Architect Associate learning path introduces you to the AWS database services relevant to the SAA-C03 exam. We then understand the service options available and learn how to select and apply AWS database services to meet specific design scenarios relevant to the Solution Architect Associate exam.
Want more? Try a lab playground or do a Lab Challenge!
- Understand the various database services that can be used when building cloud solutions on AWS
- Learn how to build databases using Amazon RDS, DynamoDB, Redshift, DocumentDB, Keyspaces, and QLDB
- Learn how to create Elasticache and Neptune clusters
- Understand AWS database costs
- Learn about data lakes and how to build a data lake in AWS
Hello and welcome to this lecture, covering Amazon Quantum Ledger Database, which is also known as Amazon QLDB. This was released in September of 2019.
So to start with, what actually is Amazon QLDB? It's yet another fully managed and serverless database service, which has been designed as a ledger database. This has a whole host of use cases. One quick example would be for recording financial data over a period of time. QLDB would allow to maintain a complete history of accounting and transactional data between multiple parties in an immutable, transparent and cryptographic way through the use of the cryptographic algorithm, SHA-256, making it highly secure.
This means you can rest assured that nothing has changed or can be changed through the use of a database journal, which is configured as append-only. Essentially, the immutable transaction log that records all entries in a sequenced manner over time. This service therefore negates the need for an organization to develop and implement their own ledger applications.
This may sound similar to blockchain technology where a ledger is also used. However, in blockchain, that ledger is distributed across multiple hosts in a decentralized environment, whereas QLDB is owned and managed by a central and trusted authority. This removes the requirement of a consensus of everyone across the network, which is required with blockchain.
Often, ledger applications to fulfill these requirements are added to relational databases, and this quickly becomes difficult to manage since they are not immutable, which makes errors difficult to trace, especially during audits.
I mentioned earlier that QLDB is serverless. So again, the administration of having to maintain the underlying infrastructure is removed and all scaling is managed by AWS, which includes any read and write limitations of the database.
Amazon QLDB is great for scenarios where you can maintain an accurate record of changes requiring the utmost integrity assurance. So to help get an understanding of how QLDB is used across different industries, let me take a look at a couple of examples and use cases for the service.
So QLDB would be a great fit within the insurance industry, which a claim by its nature can be a long winded and extensive process involving many different parties and operations over a long time period. You could implement different processes, systems and applications to track, audit and record the claim history via relational databases and custom auditing mechanisms verifying the validity of the records. However, this could all be replaced with Amazon QLDB. Using an immutable append-only framework, prevents the ability to manipulate previous data entries, which helps to prevent fraudulent activity.
Another use case where QLDB would be a good fit is within the human resources field, including payroll processes. Accuracy and auditing is essential when it comes to employee data, which is often confidential, including payroll information. HR tracks and records an employee's history covering their performance, benefits, training, remuneration and much more. Having a clearly defined verifiable employment history that can be encrypted, trusted, and reliable containing all elements of the individual held centrally makes Amazon QLDB a great fit.
So we can see that Amazon QLDB is really about maintaining an immutable ledger with cryptographic abilities to enable the verifiable tracking of changes over time. There are many different use cases where this can be used, and I've just highlighted a couple here to provide more of an understanding of how this would be used.
Let's now take a look at some of the concepts and components that make up the service.
Firstly, and as I've mentioned a few times already during this lecture, QLDB is based upon a ledger. So let's look at the structure of a ledger database.
So going back to the tables, they all effectively compromise of a group of Amazon ion documents and their revisions. As with most documents, when a revision is made, it usually signifies a change, an update, a replacement. Basically, something changes to the document, making a revision. Now, as we know, QLDB by design maintains an audit history of all changes and so that revision is saved in addition to all previous versions of the same ion document. This journal of transactional changes allows you to easily query document history across all document iterations.
Changes to these documents are done so via database transactions. In doing this transaction, Amazon QLDB will read data from its ledger, perform the update as required, and then save the changes to the journal. To be clear, the journal acts as an append-only transactional log and maintains the source of truth for that document and the entire history of changes to that document, ensuring that it remains immutable.
Each time a change is committed to the journal, a sequence number is added to identify its place in the change history. In addition to this, an SHA-256 bit hash is used for verification purposes, which creates a cryptographic digest file of the journal. Now this identifies an encrypted signature of the changes made to your document and the history of your entire document at that point in time. This can then be used to verify the integrity of the changes made in relation to the digest file created. This helps to ensure that the data within your document has not been altered or changed in any way since it was first written to in QLDB.
For a deeper understanding of how this whole process works, please review the following: https://docs.aws.amazon.com/qldb/latest/developerguide/verification.html
Now we have a base understanding of the ledger itself through the use of tables and documents, including the ledger. Let me now take a look at how storage is used for QLDB.
Amazon QLDB uses two different methods of storage, each for very different uses, these being journal storage and index storage.
Journal storage is the storage that is used to hold the history of changes made within the ledger database. So this will hold all of the immutable changes and history to the ion documents within your table.
Index storage on the other hand is the storage that is used to provision the tables and indexes within the ledger database and it's optimized for querying.
With QLDB being a fully managed serverless service, this storage is managed for you and there are no specifications to select or make during the creation of your ledger database. In the next lecture, I will show you how simple it is to create a ledger and load some sample data into the database.
The final point I want to cover with Amazon QLDB is, is integration with Amazon Kinesis through the use of QLDB streams.
Amazon Kinesis makes it easy to collect, process, and analyze real-time streaming data so you can get timely insights and react quickly to new information. With Amazon Kinesis, you can ingest real-time data such as application logs, website clickstreams, IoT telemetry data, and more into your databases, your data lakes and data warehouses, or build your own real-time applications using this data. Kinesis enables you to process and analyze data as it arrives and respond in real-time instead of having to wait until all your data is collected before the processing can begin.
Using QLDB streams, you are able to capture all changes that are made to the journal and feed this information into an Amazon Kinesis data stream in near real-time. This allows you to architect solutions whereby other AWS services could process this data from Kinesis to provide additional benefit. For example, this is a great way to implement event-driven architectures. Event-driven architectures are triggered by events that occur within the infrastructure.
So in this case, suppose your ledger contained financial accounts recording transactions and in this instance, an event could be a Lambda function that triggers an SNS notification to an account owner when a finance balance drops below a certain threshold following an update that has been made to the journal.
Course Introduction - Amazon Redshift - DEMO: Creating an Amazon Redshift Cluster - DEMO: Creating a Ledger using Amazon QLDB - Amazon DocumentDB (With MongoDB Compatibility) - DEMO: Creating an Amazon DocumentDB Cluster - Amazon Keyspaces (for Apache Cassandra) - DEMO: Creating a Keyspace and Table in Amazon Keyspaces (for Apache Cassandra) - Course Summary
Stuart has been working within the IT industry for two decades covering a huge range of topic areas and technologies, from data center and network infrastructure design, to cloud architecture and implementation.
To date, Stuart has created 150+ courses relating to Cloud reaching over 180,000 students, mostly within the AWS category and with a heavy focus on security and compliance.
Stuart is a member of the AWS Community Builders Program for his contributions towards AWS.
He is AWS certified and accredited in addition to being a published author covering topics across the AWS landscape.
In January 2016 Stuart was awarded ‘Expert of the Year Award 2015’ from Experts Exchange for his knowledge share within cloud services to the community.
Stuart enjoys writing about cloud technologies and you will find many of his articles within our blog pages.