Machine Learning Data Leakage: How to Spot It Before It Inflates Results

Why leakage matters

Data leakage occurs when training or evaluation uses information that would not legitimately be available at the intended prediction moment. The result can look excellent in a notebook while failing in deployment. Leakage is often a data and study-design problem, not merely a model-selection problem.

Define the prediction target, prediction time, and intended population before preparing features. For each feature, ask when it is created, by whom, and whether it is available at that moment. This simple timeline catches many errors earlier than comparing model scores.

Recognize target and proxy leakage

A target-derived field can reveal the answer directly, such as a final status recorded after the outcome. A proxy may encode the same information indirectly, such as a workflow code assigned only after a decision. Strong predictive performance can be a clue, but it is not proof of leakage.

Review feature definitions with someone who understands the operational process. Inspect correlations and suspiciously predictive variables, then test performance with the suspect feature removed. Document why each feature is available at decision time. Do not keep a leaky field because it improves a portfolio metric.

Respect time and group boundaries

Random splitting can leak information when observations from the same person, device, location, or event appear across training and test sets. It can also let future records inform predictions about the past. The correct split depends on the deployment scenario, not a convenient default.

For future prediction, consider chronological evaluation and ensure preprocessing uses only past training information. For repeated entities, consider group-aware splits. Avoid tuning repeatedly against the final holdout; doing so turns it into an informal training set and weakens its value as an independent check.

Fit preprocessing inside validation

Imputation, scaling, feature selection, and encoding can leak test-set information if fitted before splitting. Fit transformations on training folds only, then apply them to validation data. A pipeline abstraction helps keep these steps together and reduces accidental inconsistency.

Use cross-validation that matches the data structure and use case. A standard random fold strategy may not be appropriate for temporal or grouped data. Keep model selection and final evaluation conceptually separate, and make the chosen procedure reproducible with fixed, documented configuration where useful.

Audit labels, duplicates, and collection process

Labels may be delayed, corrected later, or generated by a process that already used a feature. Duplicate or near-duplicate examples can cross split boundaries. In healthcare, finance, or other high-stakes contexts, historical labels can also encode institutional choices rather than an objective ground truth.

Trace how outcomes are recorded and whether labels exist only for selected cases. Check duplicates using appropriate identifiers and inspect record timing. Leakage checks should include a data lineage review, not only a code review, because the meaning of a field may be invisible in its name.

Report a credible evaluation

Describe the split strategy, prediction time, preprocessing, baseline, metric, and limitations. Compare against a simple baseline and interpret errors in the context of the task. A single score without its evaluation design is not enough to establish generalization.

For a portfolio, include a “leakage audit” section listing suspected risks and how you tested them. State what would need validation before deployment, including population shift and monitoring. Your next step is to rerun an existing project with a time- or group-aware split and explain how the conclusion changes.

• State the assumptions and limitations plainly.
• Link to the official documentation or data source used.
• Give the reader one concrete action they can complete next.