Fix duplicate orders without deleting legitimate transactions

Revenue doubles after a daily file is imported twice. Removing every row with a repeated customer or amount would also erase real repeat purchases.

Why this happens

Deduplication requires a business key. Identical prices and customer names are not unique transaction identifiers. Updated versions of an order need a different policy from repeated copies of the same event.

Step-by-step solution

1. Identify the source order ID or event ID and check whether it is unique within the intended scope, such as store plus order ID.

2. For exact replayed records, deduplicate by the stable transaction key. If versions differ, use a documented update timestamp and resolve conflicts explicitly.

3. Record how many records were removed and reconcile the remaining total. Preserve an audit copy so disputed removals can be investigated.

Worked example

import pandas as pd
df = pd.DataFrame({'id':['A1','A1','A2'], 'amount':[100,100,100]})
clean = df.drop_duplicates(subset=['id'], keep='first')
assert len(clean) == 2
assert clean['amount'].sum() == 200

Verify the fix

The repeated A1 disappears, while A2 remains despite having the same amount. Test conflicting duplicates separately; blindly keeping the first is not a valid conflict-resolution policy.

Prevent a repeat

Make ingestion idempotent with unique transaction keys and an import ledger. Retries should not increase business totals.

Reference

Official documentation or standards guidance

An original Gistmax troubleshooting guide. Scenarios and sample values are illustrative; configuration depends on your software version and environment. Cover diagram: Gistmax Tutorials.