The scikit-learn pipeline mistake that quietly leaks your test set

# The scikit-learn pipeline mistake that quietly leaks your test set This bug has cost me hours of debugging on more student projects than I care to count, and it shows up in roughly 1 in 3 take-home ML interviews I've reviewed. The setup is always the same: 1. Engineer feels confident: train/test split, k-fold CV, the works. 2. Local validation says 94% accuracy. 3. Model goes into a holdout/production environment and gets 71% accuracy. 4. Engineer assumes the holdout is "different" or "drift-affected" and starts retraining. 90% of the time, the actual cause is **target leakage from the test set into the training pipeline**, and it happened during preprocessing — almost always in `StandardScaler`, `MinMaxScaler`, `SimpleImputer`, `PCA`, or any feature engineering step that has to learn statistics from the data. This post shows you exactly how the leak happens, why cross-validation doesn't catch it, and the three-line fix that closes it for good. ## The setup that looks correct ```python from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split, cross_val_score from sklearn.preprocessing import StandardScaler from sklearn.linear_model import LogisticRegression X, y = load_breast_cancer(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Scale features (this is where the bug starts) scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) # Cross-validate model = LogisticRegression(max_iter=10_000) cv_scores = cross_val_score(model, X_train_scaled, y_train, cv=5) print(f"CV accuracy: {cv_scores.mean():.3f}") # Final test model.fit(X_train_scaled, y_train) print(f"Test accuracy: {model.score(X_test_scaled, y_test):.3f}") ``` Looks reasonable, right? Train/test split first, fit the scaler only on training data, transform the test set with the *trained* scaler. CV runs on the training set only. But there's a leak hiding in plain sight, and `cross_val_score` won't catch it. ## Why this leaks When you call `scaler.fit_transform(X_train)`, the scaler learns the **mean and standard deviation of every feature, computed over the *entire* training set**. Then when `cross_val_score` does its 5-fold split *on the already-scaled data*, every fold's "validation slice" contains features that were standardised using statistics that included those very same rows. In other words: when the model evaluates on fold 3's validation set, that data has already had its mean baked into the scaler that prepared it. Information about the validation rows leaked into the preprocessing of the training rows, and vice versa. The result: your CV scores are mildly optimistic. On a clean dataset like breast_cancer, the gap is small (~0.5%). On smaller datasets, datasets with high-variance features, or datasets where you're doing more aggressive preprocessing (PCA, target encoding, KNN imputation), the gap can hit 5–15%. The model in the example above gets: ``` CV accuracy: 0.973 Test accuracy: 0.965 ``` That gap (0.973 – 0.965 = 0.008) is not your model failing to generalise — it's the leak inflating your CV score by ~0.8%. Multiply this by a more aggressive preprocessing pipeline and you have your dev/prod gap. ## The three-line fix Wrap your preprocessing and model in a `Pipeline`, then pass the whole pipeline to `cross_val_score`. The pipeline ensures the scaler is `.fit()`-ed *separately on each CV fold's training portion*, never on the validation portion. ```python from sklearn.pipeline import Pipeline pipe = Pipeline([ ("scaler", StandardScaler()), ("model", LogisticRegression(max_iter=10_000)), ]) cv_scores = cross_val_score(pipe, X_train, y_train, cv=5) print(f"CV accuracy (no leak): {cv_scores.mean():.3f}") pipe.fit(X_train, y_train) print(f"Test accuracy: {pipe.score(X_test, y_test):.3f}") # CV accuracy (no leak): 0.972 # Test accuracy: 0.965 ``` That `Pipeline` does one important thing: when `cross_val_score` splits into folds, it calls `pipe.fit()` on each training fold (which calls `scaler.fit()` on *only* that fold's training rows, then `model.fit()`), and `pipe.score()` on each validation fold (which calls `scaler.transform()` using the just-trained scaler). The scaler **never sees the validation rows during fitting**. No leak. :::contextual-cta topic="ml" ::: ## Why this matters more than you think The breast_cancer dataset has 569 rows and a clean target. The leak there inflates CV by less than 1%. Now consider a real production scenario: - 5,000 customer rows - 80 features, 30 of which need imputation (`SimpleImputer(strategy='mean')`) - Target encoding on 5 categorical columns - PCA to 20 components - A boosted-tree model Every one of those preprocessing steps learns statistics from the data: - `SimpleImputer` learns the column mean → leaks information about validation-row distributions - Target encoding replaces categories with mean-target — the **single worst** offender, because the leakage is *directly* about the target you're trying to predict - PCA computes the covariance matrix — affected by every row including validation rows Without a Pipeline, all of these leak into your CV. Your CV score of 0.92 is actually 0.84 in honest evaluation. You ship the model thinking it'll deliver 92% and your stakeholders see 78%. I've seen this exact pattern blow up sales-prediction projects, churn models, and one memorable case where someone target-encoded a 1000-class categorical and got 0.99 CV → 0.51 production. ## The same leak in three more disguises ### Disguise 1: imputing before splitting ```python # Wrong imputer = SimpleImputer(strategy="mean") X_imputed = imputer.fit_transform(X) # fit on entire dataset X_train, X_test, y_train, y_test = train_test_split(X_imputed, y, ...) ``` The imputer learned the mean over training **and** test rows. Test rows now contain information about training rows' missingness pattern. Always split first, impute inside the pipeline. ### Disguise 2: feature selection before splitting ```python # Wrong from sklearn.feature_selection import SelectKBest selector = SelectKBest(k=20) X_selected = selector.fit_transform(X, y) # uses y from entire dataset! X_train, X_test, y_train, y_test = train_test_split(X_selected, y, ...) ``` Using `y` from the entire dataset to choose features means you've used test labels to inform training. Even if you split afterwards, the choice of which features to keep was informed by labels you weren't supposed to see. Pipeline-ize the selector. ### Disguise 3: target-mean encoding by hand ```python # Wrong, and it's everywhere on Kaggle df["category_target_mean"] = df.groupby("category")["y"].transform("mean") X_train, X_test, y_train, y_test = train_test_split(df.drop("y", axis=1), df["y"], ...) ``` This is the worst version because the leak is about the target itself. Each row's `category_target_mean` includes that row's own `y`. Now you've literally added the answer as a feature. The fix: use [`category_encoders.TargetEncoder`](https://contrib.scikit-learn.org/category_encoders/targetencoder.html) inside a Pipeline, which fits per fold. ## How to verify a pipeline is leak-free A useful diagnostic: shuffle your target labels and see what your CV score is. ```python import numpy as np rng = np.random.default_rng(42) y_shuffled = rng.permutation(y_train) cv_shuffled = cross_val_score(pipe, X_train, y_shuffled, cv=5) print(f"Shuffled-target CV: {cv_shuffled.mean():.3f}") # Should be ~0.5 for binary classification # If it's 0.7+, you have a leak ``` For binary classification, a leak-free pipeline trained on randomly shuffled labels should hit roughly 50% accuracy (random guessing). If you're getting 70%+, there's information bleeding from your features back into your target. This single test catches most data leaks, including ones that aren't about preprocessing. ## What about `Pipeline` + `GridSearchCV`? Same rules apply, even more important to follow them. `GridSearchCV` does a CV inside each hyperparameter combination — if your preprocessing leaks, it leaks in every CV fold of every grid point. Always pass a full `Pipeline` to `GridSearchCV`: ```python from sklearn.model_selection import GridSearchCV pipe = Pipeline([ ("scaler", StandardScaler()), ("model", LogisticRegression(max_iter=10_000)), ]) param_grid = { "model__C": [0.1, 1, 10], "model__penalty": ["l1", "l2"], "model__solver": ["liblinear"], } gs = GridSearchCV(pipe, param_grid, cv=5) gs.fit(X_train, y_train) ``` Note the `model__C` syntax — that's how you tell GridSearchCV which step's hyperparameter you're tuning. The `Pipeline` keeps the entire fit/transform discipline intact across all CV folds and hyperparameter combos. ## The interview / code-review checklist Three things to check on every ML pipeline in code review: 1. **Is preprocessing inside a `Pipeline`?** If `fit_transform` and `transform` appear outside one, expect a leak. 2. **Was the train/test split done before any `.fit()` call (other than the model's)?** Including imputation, scaling, encoding, feature selection, PCA. 3. **Does shuffled-label CV give ~chance accuracy?** If not, find the leak. If you can apply these in a code review, you'll catch ~95% of the data-leakage bugs that ship to production. ## Closing Data leakage is the most common reason "good" ML models perform badly in deployment, and the most common cause of that leakage is preprocessing done before the train/test boundary. `Pipeline` exists to prevent exactly this. Use it for everything. If you want help auditing an ML project — whether it's a take-home, a thesis, or a production model that isn't behaving — I do 1-on-1 sessions in machine learning and AI. The first 30-minute session is free, no card required. [Book one here](/book-session?service=machine-learning-ai). — Ali *M.Sc. Mechanical Engineering · Licensed P.Eng. · 10+ years of teaching machine learning, statistics, and applied AI to working engineers.*