Schema Drift vs. Semantic Drift in Data Pipelines
When a column is removed from a source database or dataset, the next pipeline run usually fails, making the problem easy to detect. But when a field keeps the same name and data type while its meaning changesβa phenomenon known as semantic driftβthe pipeline can continue running normally, allowing the problem to go unnoticed.
This happens because a schema describes the structure of data, not what the values actually represent. It tells us which fields exist, their data types, and whether they can contain null values, but not whether a timestamp uses UTC, whether a durability score should range from 10 to 100, or whether an empty string means something different from null.
As a result, most pipeline checks are designed to catch obvious errors, such as missing columns or incorrect data types, but often fail to detect less obvious problems where the data looks correct but its meaning or values have changed.
I wanted to see how far this limitation reaches in practice, so I built a small lab where I introduced nine types of pipeline failures and tested four detectors against each one. This article examines what each detector caught and what happened when changes went unnoticed and corrupted the data.
You can find the code here and reproduce the experiment with a few commands.
What We'll Cover
- How the Lab Works
- Understanding Data Shape and Meaning
- The Results Matrix
- Detecting Changes in Data Shape
- Semantic Changes: What the Wrong Numbers Looked Like
- Damage vs. Visibility
- Detecting Changes in Data Meaning
(Content continues with these sections.)
via FreeCodeCamp