So the first question you might have is understanding the difference between a data engineer and a data scientist. These two roles are often mixed together, and certainly they work together a lot. However, data engineer and data scientists have quite separate tasks and skillsets.

A data engineer develops constructs tests and maintains to present data. A data scientist is someone who massages and organizes data to gain insight from it. So basically the data engineer engineers the data for the scientist to work on. A little bit like a lab technician and a scientist.

The lab technician insures all the samples and tests are ready in the laboratory and the scientists runs the tests and makes the evaluation on those results. The lab technician makes sure all the testing and equipment is cleansed and stored, so it can be used again. And the scientist could not do anything without the lab technician. If a sample is not properly stored by a lab technician, the scientist's results could be rendered void.

So the relationship is based on mutual respect of their specific skills and the management of their specific tasks. And it's very much a team effort between scientist and laboratory technician. They're both relying on each other. And so it is with the relationship between data engineer and data scientist.

So the architecture a data engineer will work on, can be made up of many different components. Relational and non-relational databases, the architecture will likely include processing tools and proprietary systems. And the data engineer will often add tools and services to the architecture to ensure that data is always ready for the data scientists to use.

So the role of data engineer is using this data architecture to load extract and transform raw data. And raw data generally means that the data often contains errors and anomalies, right? Duplications, incomplete values, mismatch fields, often human error where someone has mistyped a value or made a spelling mistake, et cetera, or it could be machine error where you have data view that's incomplete or presenting the wrong fields perhaps, or is unformatted and contain system codes or characters that you don't want to have in there.

A data engineer will analyze the data and come up with ways to improve the quality and reliability of that data. And that could be using a data import tool to ignore the suspect rows perhaps and only import rows where the data fields meet a specific criteria. For example, they need to be a character or a numeric value, or they need to be of a certain length. And that could be achieved by using a Python script, for example, to remove and replace specific characters from those fields or convert the fields to a specific data type. And this is where good data engineers get really creative by recognizing a specific data problem to solve and knowing the solutions available to help fix that quickly and efficiently.

The data engineer will create a managed data set processes for data modeling, data mining, and production. And having the architecture and systems in place is the role of the data engineer. Data scientists and business stakeholders rely on the data engineer to ensure the architecture is ready and available, which is why data engineer is a very valued position with a high remuneration.

Lectures

Introduction - What Does a Data Engineer Do? - Data Engineer vs. Database Administrator - Key Skills of a Data Engineer - Key Traits of a Data Engineer - Is This Role Right For Me?