Handle a CSV that is too large to fit in memory

An analytics script works on a sample but crashes while loading the full export. The file may be smaller than available RAM because parsed objects and temporary copies add overhead.

Why this happens

Loading all columns and all rows at once creates unnecessary work when the question only needs a few fields or aggregate totals. Concatenating every chunk recreates the same memory problem.

Step-by-step solution

1. Estimate the required fields and filter early. Use usecols and appropriate dtypes, while preserving identifiers such as phone numbers as strings.

2. Read manageable chunks and update a compact aggregate. Keep only the state needed for the question, rather than collecting every chunk in a list.

3. Measure memory and execution time on progressively larger inputs. For joins and global sorting, consider a database or an out-of-core tool instead of forcing a streaming approach.

Worked example

import pandas as pd
from io import StringIO
source = StringIO('amount,unused\n10,a\n20,b\n30,c\n')
total = 0
for chunk in pd.read_csv(source, usecols=['amount'], chunksize=2):
    total += chunk['amount'].sum()
assert total == 60

Verify the fix

The chunked calculation must match a full calculation on a small fixture. Test the final partial chunk and confirm peak memory no longer grows with the entire row count for this simple sum.

Prevent a repeat

Document which operations are streamable. Exact medians and arbitrary deduplication may require substantially more state than a sum.

Reference

Official documentation or standards guidance

An original Gistmax troubleshooting guide. Scenarios and sample values are illustrative; configuration depends on your software version and environment. Cover diagram: Gistmax Tutorials.