A script tasked with summarizing a payment log ran without incident for a year, before failing on a file that was too large. A tutorial dedicated to Python iterators and generators shows how to build processing pipelines that read data one record at a time, without loading it all into memory.
- Loading a large log file into a list causes memory errors, because every record stays in memory at once.
- Iterators and generators read data one record at a time, which keeps memory use low.
- Calls like list(), sorted(), or DataFrame construction quietly reload the entire stream into memory.
- Global sorts and exact deduplication still need approximate methods or external tools.
In the described case, the script would open the file, build a list of parsed records, filter errors, then display a table. After a change in log rotation, a week’s worth of traffic ended up in a single 9 GB file. The nightly processing halted with a MemoryError before producing any summary. According to the author, the parsing logic was not at fault: the problem came from the list, which forces keeping all records in memory at once.
The mechanism relies on a simple contract. An iterable, like a list, a string, or an open file, can provide an iterator, that is, a cursor that advances element by element and never goes back. The for loop calls these steps automatically, and the end of the sequence is signaled by the StopIteration exception, which is not an error. An exhausted iterator does not reset: a function that returns “nothing” on a second read is not necessarily empty, it has simply already been consumed.
The tutorial recommends starting with the file object, which is already an iterator. Reading a file with f.readlines() creates a list of all lines, whereas for line in f reads only one buffered block at a time. Generator functions, which use the yield keyword, then allow breaking down the analysis into named steps. The yield from statement chains multiple rotated files as a single stream, and generator expressions, written in parentheses, are used for small intermediate transformations. The author notes a pitfall: the leftmost part of a generator expression is evaluated immediately.
Measure Before Trusting
The tutorial also stresses points where the list quietly returns. Calls to list(), sorted(), a dictionary grouping records by request ID, or constructing a DataFrame reintroduce the entire stream into memory. Generators are also not faster than lists: their benefit is bounded memory and reorderable steps. For batch processing, Python 3.12 offers itertools.batched(), and an equivalent recipe using islice is provided for earlier versions.
To verify the actual behavior, the author recommends the tracemalloc module, which measures memory allocated by Python, in current value and peak. On a synthetic log of 200,000 lines, about 17 MB, the “list” version reached a peak of 198 MB on a laptop, compared to 0.4 MB for the streaming version. The author specifies these are the results of a single run, not a benchmark. They also note that tracemalloc does not see C extension allocations nor the system cache.
The Limits of Streaming
Some processes cannot be performed as a stream. A global sort, exact deduplication across the entire file, or a percentile calculated on all latency values require knowing all records. In this case, the tutorial mentions an approximate algorithm based on a sample, or an external tool that sorts on disk.
The tutorial finally notes that this approach remains in pure Python. Pandas, Spark, Dask, or asynchronous streams address related problems, but are not necessary for a log file, an export, or a data dump that fits on disk but not in RAM.
