Let’s do a quick review of what you’ve learned.

A slowly changing dimension is a column in a dimension table where the values change infrequently, such as an address or phone number.

There are several ways of handling this situation. Type 1 SCD means that the old values are not saved, so the database only shows the latest values.

With Type 2 SCD, you store a new copy of a record whenever one of its fields changes. To make this work, you have to add StartDate, EndDate, and IsCurrent columns. You also need to create a surrogate key that’s unique for each version of a record.

With Type 3 SCD, instead of creating a new record when a field changes, you have two columns for a field: the old version and the current version. You can either store the original value or the previous value in the old column. You’ll also need a column that says when the value was last modified.

Type 6 is a combination of Type 2 and Type 3. It adds a new record whenever the field changes, and it also has extra columns to keep track of the previous version of the field. With this method, you can get the current version and one of the previous versions by retrieving a single record instead of two records.

In Azure Synapse Pipelines or Azure Data Factory, a data flow provides a graphical way to define data transformations without writing any code.

If you enable “Data flow debug”, it’ll start a Spark cluster that you can use for testing.

Schema drift is when the source dataset’s schema changes after you create the data flow. When you enable “Allow schema drift”, it will read in all of the column names every time the data flow is executed and allow all of the new or changed columns to pass through the data flow.

Some of the transformations you can add to a data flow include:

• Derived Column, which lets you create or modify a column,
• Exists, which checks whether specific data exists or doesn’t exist in a data stream,
• Lookup, which adds columns from a second stream to a data stream,
• Alter Row, which lets you set a policy regarding which data updates are allowed, and
• Sink, which specifies a destination for writing data.

If you’re writing data to a dedicated SQL pool, selecting the “Enable staging” option can improve performance.

To run a data flow, you need to create a pipeline containing the data flow and run the pipeline. Before you can run a pipeline, you need to publish everything related to it, including datasets, scripts, the data flow, and the pipeline itself. Then you can run the pipeline by creating a trigger. You can watch a pipeline run in the Monitor section of the interface. 

To learn more about Azure Synapse Analytics, you can take another one of our courses or labs, or you can read Microsoft’s documentation at this URL. 

Please give this course a rating, and if you have any questions or comments, please let us know. Thanks!