Python Multiprocessing Tutorial: Run Code in Parallel Using the Multiprocessing Module

Multiprocessing speeds up Python workloads by running multiple tasks at the same time across separate processes—often cutting wall-clock time dramatically when work can be parallelized. The tutorial starts with a simple “sleep” function to show the difference between synchronous execution (tasks run back-to-back) and parallel execution (tasks run concurrently on different processes), then scales up to a practical image-processing pipeline where parallelism meaningfully reduces runtime.

In the baseline example, a function prints messages, sleeps for one second, and prints again. Running it once takes about one second; running it twice takes about two seconds—clear evidence of synchronous behavior. That sets up the core motivation: if independent tasks can run in parallel, multiprocessing can reduce total runtime from roughly the sum of task durations toward the duration of the slowest tasks.

The tutorial then contrasts threading and multiprocessing through the lens of CPU-bound vs. I/O-bound work. CPU-bound tasks (heavy number-crunching) benefit from multiprocessing because separate processes can use multiple CPU cores simultaneously. I/O-bound tasks (waiting on file or network operations) may also benefit, though the best choice depends on the workload and hardware. The key takeaway is that multiprocessing spreads work across multiple processors, while threading may not deliver the same speed-up for CPU-heavy tasks.

Next comes the mechanics. The “older way” manually creates multiprocessing.Process objects, starts them with .start(), and waits for completion with .join(). Without join, the main script continues immediately and may report completion before child processes finish—an important pitfall. When join is used correctly, the script waits until all processes complete, producing an accurate runtime measurement.

To run the same function many times, the tutorial shows how to create processes in a loop, store them in a list, and join them afterward so all tasks start before waiting. It also demonstrates passing arguments to worker functions: multiprocessing requires arguments to be pickle-serializable, so values must be convertible for inter-process communication.

For a cleaner approach, the tutorial switches to concurrent.futures.ProcessPoolExecutor. Using executor.submit schedules work and returns Future objects; calling future.result() blocks until completion. A context manager ensures processes are cleaned up and joined automatically. For bulk parallelism, executor.map applies a function across an iterable and returns results in input order, while concurrent.futures.as_completed yields futures as they finish—useful when task durations vary.

The real-world example processes 15 high-resolution images using Pillow: opening each image, applying a Gaussian blur, resizing via thumbnail, and saving to a processed folder. The synchronous version takes about 22 seconds. Rewriting the loop into a single-image worker function and using ProcessPoolExecutor.map to process filenames in parallel reduces runtime to about 7 seconds, with multiple Python processes visible in the system’s activity monitor.

Finally, the tutorial tests the same workload with ThreadPoolExecutor and finds it can be faster here (about 7.2 seconds), reinforcing the rule of thumb: benchmark both threading and multiprocessing for your specific mix of CPU work and disk I/O. The practical conclusion is straightforward—use multiprocessing when parallelism and CPU usage justify it, but validate with measurements because I/O-heavy pipelines can behave differently than expected.