So to bring it all together, let's pull it into a real world scenario that's maybe a little on the complex side, but it starts to show how the storage technology can be accomplished for real world uses.

Imagine your company sells computer accessories and peripherals, and it also has an online store. Pretty classic, you can replace this with your own entity model if you've been following along, or just imagine whatever your company sells. Your analytics department has requested that the sales data be fed into the company's cloud data warehouse.

Now the online store uses a relational database, but the data warehouse goes beyond that and stores information from many systems beyond just the sales system. So the data warehouse has a couple of ways of getting information to it. Well, let's walk through how we can start to architect a system, and how these different components can interact with each other.

So your online store probably has an associated database to store sales information. This is a really strong use case for a relational database. Very clear expected data inputs of who is buying what, when, why, and where, and maybe some inventory tracking. This then relates to a data warehouse. Now this goes beyond just what your comparatively simple online stores database has. This will store lots of information and lots of different formats, but very importantly, it's a central repository for information. So the question then becomes, how do these two link up? And the answer is, an API such as a REST API.

Many data warehouses will have a system for gets, puts, and posts to be made in order to upload new information to them. Now, for those of you who have large data movement problems, we're talking in the tens, dozens, hundreds, or terabyte range, we might need to go with some alternative technologies. But for those of you who are getting started on your data journey, this architecture of a relational database within some type of execution engine such as a Lambda function, posting data to a REST API to store it in a larger data warehouse is extremely common and extremely supportable.

If you wanna know more about this subject, we actually have some labs associated with this class that will really help you dig in in a hands on fashion to actually play with, experiment, and interface with databases and data storage formats. Also, if you're interested in your general data engineering development or maybe even data scientists, pay attention to the learning path that this is part of because there's more classes that directly tie into the whole data engineer data professional experience.

As always, please send any of your feedback in, good or bad, it helps us to target this course content more accurately in the future. I look forward to instructing everyone on the next class, and I will see you then.

Lectures

Introduction - Data Sources and Formats - Modeling Your Data - CSV - XML - JSON - Specialized Data Formats - Databases - Exchanging Data Across Applications