Intro to ML Monitoring: Data Drift, Quality, Bias and Explainability

ML monitoring is positioned as the practical way to catch “bad data” and model failures early—by tracking data drift, data quality, bias across segments, and explainability signals in production—then triggering alerts or workflows before users feel the impact. The workshop frames monitoring as an end-to-end loop: collect AI telemetry from inputs/outputs, compare it to training references (or rolling windows), measure drift and quality constraints, and use those findings to investigate, retrain, or mitigate.

A core theme is that production models rarely fail in a single dramatic way; instead, they degrade as inputs shift or pipelines break. Common failure modes include data drift (input distributions changing over time), data quality problems (sensor issues, manual entry errors, schema changes), and concept drift (the relationship between inputs and the real-world target changes). The session emphasizes that even strong offline evaluation can’t prevent these issues, because real environments introduce new conditions—seasonal changes, shifting user behavior, or upstream library updates that subtly alter schemas.

To make monitoring concrete, the workshop uses Google Colab with scikit-learn’s Iris dataset and a K-nearest neighbors classifier as a stand-in production model. The open-source library WhyLabs maintains—y logs—generates compact “profiles” (summary statistics rather than raw data) for each batch of inputs. Those profiles are logged to WhyLabs’ platform (y labs) to build time series views, detect drift, and support alerting. In the hands-on portion, the Iris data is split into seven simulated “days” of production batches, and the platform reveals anomalies like batches with extreme feature values, batches of all zeros, and distribution shifts that aren’t obvious from a single snapshot.

Monitoring is demonstrated in layers. First comes input monitoring: a reference profile is created from training data, then a drift monitor compares new batches against that reference using Hellinger distance. Alerts trigger when drift exceeds a chosen threshold, catching cases where medians may look similar but distribution tails shift dramatically. Next comes output monitoring: predicted class distributions and confidence/probability outputs are logged and segmented into an “outputs” view. Even without ground truth, output drift can signal trouble—such as the model collapsing toward a single class when inputs are out-of-range.

When ground truth is added, performance monitoring becomes possible. The platform logs classification metrics (accuracy and related scores) per batch, showing sharp drops on the anomalous days. The workshop also highlights segment-level monitoring for fairness: using an additional “state” attribute (where flowers were grown), it compares metrics like accuracy and false positive rate across segments (Florida, Missouri, Washington). Tracing views make it easier to see that one segment can underperform even when overall metrics look acceptable.

Finally, explainability is introduced through SHAP. Feature importance plots identify which inputs drive predictions (in this simplified setup, petal length and petal width dominate), and those importance signals are stored in the platform’s explainability tab. The session closes with local drift and data validation tooling using y logs constraints—effectively unit tests for data quality (e.g., feature ranges like pedal width must be >0 and <15). If constraints fail, the workflow can block predictions or trigger alerts, turning monitoring into an actionable guardrail rather than a passive dashboard.