Fix UnicodeDecodeError without damaging customer names

A CSV containing accented names fails during loading. Changing the decoder until the file opens can leave corrupted text that is difficult to recognise later.

Why this happens

A text file is a sequence of bytes encoded according to a convention. The decoder must match that convention. Ignoring undecodable bytes can erase meaningful characters without reporting which records changed.

Step-by-step solution

1. Ask the source system for its export encoding or request a fresh UTF-8 export. Keep the original bytes unchanged while investigating.

2. Choose the documented encoding explicitly. Use utf-8-sig when the confirmed UTF-8 file includes a byte-order mark.

3. Read with strict error handling first. If the source is genuinely inconsistent, quarantine affected rows and record the repair rather than silently ignoring errors.

Worked example

raw = 'name\nChloé\n'.encode('utf-8')
text = raw.decode('utf-8', errors='strict')
assert 'Chloé' in text
assert text.encode('utf-8') == raw

Verify the fix

Compare several names with the original export, not just the first ASCII-only row. A successful round trip confirms the illustrated encoding operation; it does not prove an unknown file was labelled correctly.

Prevent a repeat

Agree a UTF-8 export contract with upstream systems and add non-ASCII fixtures to tests. Avoid using errors='ignore' as a general ingestion fix.

Reference

Official documentation or standards guidance

An original Gistmax troubleshooting guide. Scenarios and sample values are illustrative; configuration depends on your software version and environment. Cover diagram: Gistmax Tutorials.