Feature Selection in Python | Machine Learning Basics | Boston Housing Data

Feature selection is used to dramatically improve a K-nearest neighbors regression model on the Boston housing dataset by cutting weak or redundant predictors and then validating the remaining variables with model-based search. Starting from a relatively poor fit (RMSE around 6.5k and R² about 0.5), the workflow targets the “curse of dimensionality” and practical issues like computation time, data-collection cost, and—importantly—model interpretability.

The tutorial first revisits the Boston housing variables and corrects a key preprocessing mistake: the “RAD” feature is an index/label (1–24), so it must be treated as categorical rather than numeric. After encoding categorical levels into dummy variables, the model is re-evaluated. With the corrected encoding in place, the next step is variance-based filtering. Low-variance features are removed because they barely change across samples and therefore contribute little predictive signal. Dropping a couple of low-variance dummy variables yields a modest improvement, but the process is constrained by categorical logic—levels can’t be arbitrarily deleted without breaking the meaning of the encoding.

Next comes correlation filtering. Using heatmaps and correlation thresholds, the approach aims to avoid strong multicollinearity among explanatory variables (values around 0.8+ are flagged as risky) while still keeping features that correlate with the target MEDV (median home value). The tutorial experiments with different cutoffs and finds that correlation-only filtering can help, but not always in a principled way. The best early result comes from retaining a small set of variables (RMSE drops to roughly 4.7k and R² rises to about 0.74), suggesting that some RAD levels may be better handled by grouping rather than treating every level separately.

To move beyond classifier-agnostic filtering, the tutorial switches to wrapper-based feature selection using sequential feature selection from scikit-learn’s feature selection tools. Instead of ranking features by correlation, it searches for subsets that optimize model performance directly. In forward selection, the best single feature is LSTAT, followed by number of rooms, crime, and pupil-teacher ratio; backward selection lands on the same core set. Building a model with these selected predictors substantially improves fit (R² climbs to about 0.7–0.75 range, with the tutorial reporting a strong reduction in RMSE).

An interaction term is then tested—specifically multiplying LSTAT and number of rooms—to capture a relationship that correlation alone might miss. The improvement is small but positive, and the tutorial notes that adjusted R² would penalize adding extra variables, so the gain must justify the added complexity.

Finally, the tutorial addresses outliers. It observes that some extreme MEDV values appear capped at 50, creating suspicious “outlier” behavior. After removing those capped points, R² increases further to around 0.8, with the model fit becoming smoother and more plausible. The overall takeaway is that feature selection works best as an iterative pipeline: correct encoding first, then prune by variance/correlation, then validate with wrapper search, and finally refine with interaction terms and outlier handling.