Python Tutorial: Generators - How to use them and the benefits you receive

Generators in Python trade “build everything first” for “produce values on demand,” using the yield keyword to stream results one at a time. That shift matters because it avoids storing large collections in memory and can improve performance when working with big datasets—while still fitting naturally into familiar loops.

The tutorial starts with a straightforward function, square_numbers, that takes a list of numbers, builds a result list by appending each square, and returns the completed list. Calling it with [1, 2, 3, 4, 5] produces [1, 4, 9, 16, 25]. Converting this into a generator removes the result list and the return statement; instead, the function yields each computed square using yield. Once yield is in place, printing the variable no longer shows the full list. It shows a generator object, because the generator doesn’t compute or store all squares up front. Values are generated only when requested.

To demonstrate on-demand behavior, the tutorial uses next() to pull values from the generator. The first next() call returns 1, the second returns 4, and subsequent calls continue yielding 9, 16, and 25. A further call triggers a StopIteration exception, signaling that the generator is exhausted. For typical usage, a for loop is preferred: for num in mynums iterates through yielded values and stops cleanly without exposing StopIteration to the user.

Readability is presented as an immediate advantage over the list-building approach. The generator version reads like a direct transformation: for each number in the input, yield the square. The tutorial also connects generators to list comprehensions. A list comprehension can compute squares with x * x for x in [1, 2, 3, 4, 5]. Switching from brackets to parentheses creates a generator expression that behaves similarly—still producing values lazily. If all values are needed at once, converting the generator to a list via list(generator) works, but it removes the memory and performance benefits because it forces materialization of every element.

The performance section makes the trade-off concrete using a benchmark that generates dictionaries representing people with random names and majors. One function returns a list of 1 million records; another returns a generator that yields the same dictionaries. The list-based approach increases memory by nearly 300 MB and takes about 1.2 seconds, because it allocates and stores all million items. The generator approach keeps memory nearly flat because it doesn’t generate the million dictionaries immediately; it pauses at the first yield and only produces items as the loop consumes them. Timing reflects that difference: the generator version is effectively near-instant until iteration proceeds. Finally, the tutorial shows that converting the generator back into a list restores the list-like performance and memory cost.

Overall, generators offer a practical way to process large sequences efficiently: clearer “yield as you go” code, lower memory usage, and faster startup—provided the results aren’t immediately forced into a list.