So to dive right into the meat of the discussion rather than who I am and who we are, let's talk about the types of data sources and formats. At the beginning, the one most people started with is unstructured data. This is simply a thing such as a plain text document, an email message, or in some ways, a Word document. This is considered unstructured because they don't inherently have the concept of enforcing a data schema or otherwise structure onto the data. They give you a lot of flexibility because you can jam whatever content you want in there, but on the flip side, they're hard to process and parse because there's no standard, particularly if you're trading between companies, departments or even just yourself on a later day.

Typically, when it comes to storing more than just freeform text, people then jump to semi-structured. These are things like CSVs, XMLs, and to an extent, JSON files. These don't necessarily have a predefined schema, although, many of them have a way of defining a schema, but they do force everybody to put the data into a record or a row, and there's at least a common way to delineate and separate different bits of information, especially with things like JSONs, which are a little more on the structured side versus CSVs. You can really start to refer to data by the field name or within a larger data structure. But very importantly, these semi-structured approaches still allow a great deal of flexibility in how you're storing the data.

Finally, is the structured data formats. Now these are things like relational databases, but it also extends to format such as Avro, Parquet or maybe if you're a Hadoop enthusiast, ORC. We'll discuss these a little bit later, but structured data sources, like you're seeing on the right here, are by far the most efficient and fastest to work with. And remember, when we say it's faster or more performing, that doesn't necessarily mean it's easier. In fact, it's often quite the opposite. The more performative ways of accessing data oftentimes actually require the most upfront setup, so it becomes a balance between flexibility and rapid use versus needing a more high performance planned approach.

Lectures

Introduction - Modeling Your Data - CSV - XML - JSON - Specialized Data Formats - Databases - Exchanging Data Across Applications - Applying What we Have Learnt - Sales Data for an Online