Machine learning tutorial: compare a regression model with a simple baseline

A model is not useful merely because it produces a number. In this exercise, train a regression pipeline and compare its errors with a baseline that always predicts the training-set average. Use synthetic data so the lesson is about experimental design, not a claim about a real business or person.

1. Decide what the score means

Regression predicts a numeric target. Mean absolute error, or MAE, measures the average absolute difference between predictions and observed targets. Lower is better on the same test set. Its units are the target's units, so an error of ten cannot be labelled good or bad without context.

Install scikit-learn with `python -m pip install scikit-learn`. Save the following as `regression_baseline.py` and run it with Python 3.12 or later.

2. Fit both approaches on the same training data

from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_absolute_error

X, y = make_regression(
    n_samples=300, n_features=4, noise=15, random_state=7
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=7
)
models = {
    "mean baseline": DummyRegressor(strategy="mean"),
    "ridge pipeline": make_pipeline(StandardScaler(), Ridge(alpha=1.0)),
}
errors = {}
for name, model in models.items():
    model.fit(X_train, y_train)
    predictions = model.predict(X_test)
    errors[name] = mean_absolute_error(y_test, predictions)
    print(name, "MAE:", round(errors[name], 2))

assert len(y_test) == 75
assert errors["ridge pipeline"] < errors["mean baseline"]

3. Explain the comparison

The ridge pipeline should outperform the mean baseline on this deliberately linear synthetic dataset. That is a useful correctness check for the exercise, not evidence that ridge regression will outperform other approaches on an unrelated problem.

Both models use the same split. Comparing one model on easy examples and another on harder ones would make the error difference difficult to interpret. The baseline also learns its average from the training targets rather than looking at the test answers.

4. Inspect individual mistakes

For an extension, print a few actual targets, predictions and absolute errors. Look for whether the largest errors cluster around a particular feature range. On a real dataset, also investigate missing values, inconsistent units and information that would not be available at prediction time.

If your task concerns future events, a random split may be inappropriate. Train on earlier observations and evaluate on later ones, keeping the time order visible. This synthetic example has no timeline, which is why a random split is reasonable here.

Troubleshooting

An unexpectedly perfect score can indicate that the answer leaked into the input features. A model that loses to the baseline may still be implemented correctly; the inputs might contain little predictive information. Start by checking data definitions and the evaluation design before adding more complexity.

References

Synthetic regression data · DummyRegressor · Mean absolute error