How to Detect Hidden Target Leakage in Public Datasets with Python and a Dependency Graph

By Kayode Adeniyi | September 19, 2026

Some time ago, I gave a machine learning model five columns from a public CDC dataset and asked it to predict a sixth column from the same file. The model scored an R² of 0.998 — about as close to perfect as a real model gets.

That score looked like a success, but the model had learned very little about the real world. The CDC had calculated the sixth column from the other five, so the model simply reverse-engineered the CDC's formula.

Data scientists call this problem target leakage, and it happens when the inputs you give a model already contain the answer in some form.

Leakage like this hides easily in public data, because a large share of public data is derived from other public data. A government index might be built from survey columns, and a second index might be built from the first. Agencies explain these recipes in their methodology PDFs, yet data catalogues rarely store them in a machine-checkable form.

In this tutorial, you'll write that record yourself and then build a small Python tool that reads it. The tool works like the dependency checker inside a package manager: you tell it what you want to predict and which columns you plan to use, and it refuses any column that sits on a derivation path to or from your target.

By the end, you'll know how to:

Reproduce a real leak using live CDC data and scikit-learn
Describe what a dataset was built from in a small YAML file called a manifest
Walk that graph with breadth-first search (BFS) and depth-first search (DFS)
Build a checking tool that fails loudly on typos, broken files, and empty inputs
Run the check automatically on every push with GitHub Actions

Why Target Leakage Is So Easy to Miss
Reproducing the Leak with CDC Data
Modeling Dataset Provenance as a Graph
Writing a Manifest in YAML
Walking the Graph: BFS and DFS
Building the Leakage Checker
Automating the Check with GitHub Actions
Where This Approach Fits in 2026

Why Target Leakage Is So Easy to Miss

Target leakage is not a modeling bug — it is a provenance problem. The features are legitimate columns of a legitimate dataset; the problem is that someone else already computed the target from them.

With the rise of automated feature stores, LLM-generated feature engineering, and federated public data portals, the surface area for leakage has grown sharply. A 2026-era pipeline can pull hundreds of columns from a dozen open catalogs without any human ever tracing derivation history. Your evaluation metrics will happily report 0.99 R² while your model learns nothing transferable.

The remedy is not more careful modeling. The remedy is a checkable record of how each column was derived and a tool that refuses to run when a planned feature sits on a derivation path to or from the label.

Reproducing the Leak with CDC Data

We'll start by reproducing the failure that motivated this tutorial. Assume a CSV with columns such as bmi, age, sex, physactivity, and a diabetesindex label the CDC computed from them.

import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

df = pd.read_csv("cdc_subset.csv")

features = ["bmi", "age", "sex", "phys_activity", "smoker"]
target = "diabetes_index"

X_train, X_test, y_train, y_test = train_test_split(
    df[features], df[target], test_size=0.2, random_state=0
)

model = RandomForestRegressor(n_estimators=200, random_state=0)
model.fit(X_train, y_train)
print("R²:", r2_score(y_test, model.predict(X_test)))

Run this and you will likely see an R² above 0.99. The model hasn't learned anything about diabetes — it has learned the CDC's arithmetic.

Modeling Dataset Provenance as a Graph

If a column is computed from other columns, we can represent that as a directed graph:

Each node is a column (or an upstream source).
Each edge A → B means B is derived from A (or, equivalently, A feeds B).

Leakage exists when:

There is a directed path from any planned feature to the target (the target is downstream of a feature), or
There is a directed path from the target to a feature (the feature is downstream of the target).

Both cases indicate the feature encodes information about the label that wouldn't exist at inference time. A dependency graph makes this checkable in code.

Writing a Manifest in YAML

Store the derivation rules in a small YAML file next to the dataset. Call it a manifest.

# manifest.yaml
dataset: cdc_subset
columns:
  bmi:
    derived_from: []
  age:
    derived_from: []
  sex:
    derived_from: []
  phys_activity:
    derived_from: []
  smoker:
    derived_from: []
  diabetes_index:
    derived_from: [bmi, age, sex, phys_activity, smoker]

A manifest is intentionally simple: it records only what came from what. From here, a loader builds a graph and a checker walks it.

Walking the Graph: BFS and DFS

We need two related queries:

Reachability from a feature to the target — best answered with BFS, which finds shortest paths and stops as soon as the target is reached.
Full ancestor/descendant sets — best answered with DFS, which is natural for enumerating all paths through a derivation tree.

from collections import defaultdict, deque

def build_graph(manifest: dict) -> dict:
    graph = defaultdict(set)
    for col, meta in manifest["columns"].items():
        for parent in meta.get("derived_from", []):
            graph[parent].add(col)
        graph.setdefault(col, set())
    return graph

def bfs_path(graph, start, goal):
    """Return a shortest derivation path start → ... → goal, or None."""
    if start not in graph:
        return None
    queue = deque([[start]])
    seen = {start}
    while queue:
        path = queue.popleft()
        node = path[-1]
        if node == goal:
            return path
        for nxt in graph.get(node, ()):
            if nxt not in seen:
                seen.add(nxt)
                queue.append(path + [nxt])
    return None

def dfs_descendants(graph, start, _seen=None):
    """Return every node reachable from start."""
    _seen = _seen or set()
    for nxt in graph.get(start, ()):
        if nxt not in _seen:
            _seen.add(nxt)
            dfs_descendants(graph, nxt, _seen)
    return _seen

Both functions will be reused by the checker below.

Building the Leakage Checker

Now we compose the pieces: load the manifest, build the graph, and refuse any planned feature that lies on a derivation path to or from the target.

import yaml
import sys

def load_manifest(path: str) -> dict:
    try:
        with open(path, "r", encoding="utf-8") as f:
            data = yaml.safe_load(f)
    except FileNotFoundError:
        sys.exit(f"manifest not found: {path}")
    except yaml.YAMLError as e:
        sys.exit(f"invalid YAML in {path}: {e}")
    if not data or "columns" not in data or not data["columns"]:
        sys.exit(f"manifest {path} is empty or missing 'columns'")
    return data

def check_leakage(manifest: dict, features: list[str], target: str) -> None:
    if not features:
        sys.exit("no features provided")
    columns = manifest["columns"]
    unknown = [c for c in features + [target] if c not in columns]
    if unknown:
        sys.exit(f"unknown columns in manifest: {unknown}")

    graph = build_graph(manifest)

    for feat in features:
        fwd = bfs_path(graph, feat, target)
        if fwd:
            sys.exit(
                f"LEAKAGE: feature '{feat}' is on a derivation path "
                f"to target '{target}': {' -> '.join(fwd)}"
            )
        back = bfs_path(graph, target, feat)
        if back:
            sys.exit(
                f"LEAKAGE: target '{target}' feeds feature '{feat}': "
                f"{' -> '.join(back)}"
            )

    print(f"OK: {len(features)} features are leak-free relative to '{target}'")

if __name__ == "__main__":
    manifest = load_manifest("manifest.yaml")
    check_leakage(manifest, ["bmi", "age"], "diabetes_index")

Run this on a manifest where diabetes_index is derived from bmi, and the script exits non-zero with a clear path printed. That is exactly what you want in CI.

Automating the Check with GitHub Actions

Wire the checker into a workflow so no pull request can introduce a leaky feature set.

# .github/workflows/leakage.yml
name: leakage-check
on: [push, pull_request]

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pyyaml
      - run: python check_leakage.py

Now any commit that adds a feature descended from the label will fail CI before it reaches a model.

Where This Approach Fits in 2026

In 2026, the leakage problem has moved from notebooks into pipelines. Foundation-model fine-tuning sets, retrieval corpora, and open government data portals all concatenate upstream products, and the derivation chains are getting longer. Feature stores now advertise lineage tracking, but open catalogs still lag behind.

The small idea in this tutorial — a manifest plus a graph walk — is deliberately closer to how a package manager thinks than to how a data catalog thinks. It scales, it's testable, and it catches the failure mode that metrics themselves never will: a model that scores beautifully because the answer was already in the spreadsheet.

When your evaluation looks too good to be true, don't tune hyperparameters. Trace the graph.

via FreeCodeCamp

How to Detect Hidden Target Leakage in Public Datasets with