Fix an overfitted model that memorises training examples

Training error keeps falling while validation error rises. Adding more model complexity makes the demonstration look better but worsens performance on unseen records.

Why this happens

The model may be fitting noise or quirks specific to the training sample. A small or unrepresentative validation set can also make the apparent gap unstable.

Step-by-step solution

1. Confirm the split is valid and free of duplicate entities across partitions. Compare training and validation learning curves.

2. Reduce complexity or strengthen regularisation, and compare against a simple baseline. Tune with cross-validation appropriate to the data structure.

3. Investigate data quality and coverage before assuming more parameters will help. Add representative examples where the model fails.

Worked example

For a decision tree, compare constrained depth or minimum leaf sizes with an unconstrained tree on the same folds. Record both training and validation scores for each candidate.

Verify the fix

Choose a model whose held-out performance improves, not merely one with a smaller training-validation gap. An equally poor score on both sets can indicate underfitting.

Prevent a repeat

Keep a compact experiment log and reserve final evaluation data. Stop selecting models based on the best score from repeated test-set inspection.

Reference

Official documentation or standards guidance

An original Gistmax troubleshooting guide. Scenarios and sample values are illustrative; configuration depends on your software version and environment. Cover diagram: Gistmax Tutorials.