Python Pandas Tutorial (Part 9): Cleaning Data - Casting Datatypes and Handling Missing Values

Cleaning data in pandas often starts with two practical tasks: removing or retaining rows/columns with missing values, and converting columns into the right numeric types so calculations don’t break. The core takeaway is that pandas treats different “missing” markers differently—so you need to both standardize missing values (including custom strings) and then choose the right drop/cast strategy for the analysis you’re trying to run.

For missing values, the tutorial begins with `dropna()`, showing how pandas behaves under its defaults and how to control it. With `axis='index'` and `how='any'`, pandas drops any row that contains at least one missing value. Changing `how` to `all` keeps rows where only some fields are missing, dropping only rows where every value in the row is missing. Switching `axis` to `columns` flips the logic to dropping columns that are entirely missing. The lesson matters because “missing” doesn’t always mean “useless”: an analysis might tolerate a missing email address but not a missing identifier.

To target missingness in specific fields, the tutorial uses `subset` with `dropna()`. By specifying `subset=['email']`, rows are removed only when the email value is missing; other columns can be incomplete without triggering a drop. When multiple columns are listed in `subset`, the `how` parameter determines whether a row is dropped only when all specified fields are missing (`how='all'`) or when any of them are missing (`how='any'`). The tutorial also emphasizes that these operations are not permanent unless `inplace=True` is used.

Next comes the problem of “custom missing values.” Real datasets sometimes encode missingness as strings like `'Na'` or `'missing'` rather than actual `NaN`. The tutorial demonstrates replacing these placeholders with `numpy.nan` using `replace(..., inplace=True)`, so downstream methods like `isna()` and `dropna()` treat them consistently. It then shows how to inspect missingness with `isna()` and how to fill missing values with `fillna()`—either with a domain-appropriate constant (like `0` for numeric grades) or with a placeholder string.

The second major pillar is casting data types. A column that looks numeric may still be stored as `object` (often strings), which breaks numeric operations such as `mean()`. The tutorial uses the `dtypes` attribute to confirm that a column like `age` is stored as strings, then converts it with `astype()`. A key caveat appears when missing values are present: converting to `int` fails because `np.nan` is a float under the hood, so the safer cast is to `float` (or to fill missing values first). Once cast correctly, `mean()` works and returns the expected average.

To tie everything together, the tutorial applies the same cleaning and casting steps to Stack Overflow survey data to compute average years of coding experience. The `years_code` column contains numeric values plus strings like `'less than one year'` and `'more than 50 years'`. The workflow is: convert custom missing markers during CSV loading (via `na_values`), replace `'less than one year'` with `0`, replace `'more than 50 years'` with `51`, cast the column to `float`, and then compute the mean (about 11.5 years). The result underscores why missing-value handling and type casting aren’t optional chores—they’re prerequisites for reliable analysis.