Session 14 - Advanced Numpy | Data Science Mentorship Program (DSMP) 2022-23 | Free Session

Numpy’s edge over Python lists comes down to three practical wins—speed, memory efficiency, and easier computation—and the session then builds on that foundation with advanced indexing, filtering, and shape-handling techniques that power real data work. The class starts by comparing a Python list approach to Numpy arrays using a workload of two lists containing 1 crore items each, then adding them element-wise while timing execution. The list-based loop takes about 3.26 seconds, while the Numpy version completes in a fraction of that time; the instructor estimates Numpy is roughly 50x faster. The reason given is structural: Numpy stores data in contiguous C-type arrays and avoids extra overhead from dynamic resizing and indirect address lookups that Python lists incur.

Memory is the second battleground. A Python list with 1 crore items consumes a large amount of RAM, and the session demonstrates using system utilities to measure memory footprint. The key Numpy advantage is that arrays can choose smaller integer dtypes (e.g., int32 instead of int64), letting users trade numeric range for reduced space. The instructor emphasizes that this flexibility can significantly cut memory usage when working with large datasets.

Convenience rounds out the comparison. Numpy’s built-in arithmetic and vectorized operations make element-wise math as simple as writing expressions like C = A + B, without manual loops. From there, the session shifts into “advanced Numpy” topics that directly affect how analysts extract and transform data.

First comes advanced indexing. Alongside normal indexing and slicing, the class introduces fancy indexing, where a user supplies a list of row/column indices inside square brackets to fetch non-contiguous elements in one step. It also introduces boolean indexing (masking), where comparison operations produce a True/False array used to filter the original data. Examples include selecting values greater than 50, combining conditions with logical operators (e.g., element-wise AND), and filtering divisibility (e.g., values divisible by 7 using modulus checks).

Next is broadcasting, framed as a crucial concept for performing arithmetic on arrays with different shapes. The session demonstrates why adding arrays of shapes like (2,3) and (3,) can work: Numpy automatically “stretches” the smaller array across the larger one when dimensions are compatible. Broadcasting rules are summarized as: align dimensions by prepending 1s to the smaller shape, expand along axes until sizes match, and fail with an error when a dimension is neither equal nor 1.

The session then shows how to define custom mathematical functions in Numpy and apply them efficiently across arrays—using sigmoid as the main example—illustrating how machine learning workflows often require functions not provided as built-ins. It further constructs a mean squared error (MSE) loss function from scratch by computing squared differences between actual and predicted values and averaging them.

Finally, the class tackles practical data issues and visualization. It demonstrates handling missing values using NaN and boolean masks to drop rows/entries where values are NaN. The session closes with plotting using Matplotlib: generating x ranges, computing y values (including polynomial and exponential-like expressions), and plotting curves with plt.plot, while noting that more advanced plotting (like multiple lines and 3D) will come later.

Overall, the core takeaway is that Numpy isn’t just faster—it enables a workflow: vectorized math, shape-aware operations via broadcasting, expressive indexing/filtering, and efficient custom computations that are central to data science and machine learning.