Fix unseen-category errors in a machine-learning pipeline

An application receives a new city name and its encoder raises an error. Production categories can change even when the input field itself is valid.

Why this happens

The encoder learned a fixed vocabulary during training. An unknown value needs an explicit policy; pretending it was a known category can distort predictions.

Step-by-step solution

1. Inspect the unexpected value for spelling, case and schema changes. Separate dirty input from a genuinely new category.

2. Choose an encoder policy such as handle_unknown='ignore' when appropriate, and understand its representation of unknown values.

3. Monitor the unknown-category rate and review retraining when it grows. Keep the fitted encoder with the estimator in the same pipeline.

Worked example

from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(handle_unknown='ignore', sparse_output=False)
encoder.fit([['Lagos'], ['Accra']])
unknown = encoder.transform([['Kigali']])
assert unknown.shape == (1, 2)
assert unknown.sum() == 0

Verify the fix

The example produces an all-zero encoded row rather than an exception. Confirm that the downstream model handles this representation acceptably; successful execution alone does not establish prediction quality.

Prevent a repeat

Record category drift and maintain a documented normalisation policy. Do not refit the encoder independently in the serving process.

Reference

Official documentation or standards guidance

An original Gistmax troubleshooting guide. Scenarios and sample values are illustrative; configuration depends on your software version and environment. Cover diagram: Gistmax Tutorials.