Validation layer
Required columns, dtypes, numeric ranges, categorical domains, and missing-value policies.
Skip to content
Proof-of-work — batch monitoring pipeline
ML Monitoring
Batch ML monitoring & data quality pipeline
I validate incoming tabular batches against a JSON schema, compare them to a frozen reference window (numeric PSI/KS, categorical drift), monitor prediction behaviour, and emit machine-readable reports plus a static HTML roll-up. CI exercises the same logic with pytest.
Artifacts are produced by `run_full_monitoring`; the hosted Streamlit viewer is read-only over those files — not a production observability stack.
Python
pandas
pytest
Streamlit
Limitations & scope
- Demonstrates engineering judgement on monitoring layers — not an enterprise APM or feature store.
- Offline batch flow only; no online inference or auto-remediation.
- Synthetic reference/current splits committed for reproducible review.
Pipeline architecture
Each incoming CSV batch runs through validation against data/metadata/schema_definition.json, drift estimation versus the frozen reference window,
and optional prediction-behaviour checks. Outputs land under artifacts/reports/ and artifacts/drift/ for programmatic inspection or dashboards.
Orchestration lives in python -m src.pipeline.run_full_monitoring. Simulation helpers regenerate deterministic datasets when needed.
What gets measured
Drift layer
Population stability (PSI), Kolmogorov–Smirnov on numeric columns, and categorical shift summaries vs reference.
Prediction monitoring
Distribution checks on model scores and derived labels when present — highlighting silent shifts before downstream KPIs react.
Alerting & narratives
Rule metadata surfaces in JSON reports; decision summaries consolidate severity for reviewers.
Generated artifacts
- report_batch.json — validation, drift roll-ups, prediction slice, alerts.
- drift_batch.json — per-column drift metrics for audit trails.
- monitoring_overview.csv — batch-level status matrix across the last run.
- monitoring_run_report.html — static HTML digest produced alongside JSON.
Streamlit viewer
The hosted Streamlit page reads the committed artifacts only — select a batch, inspect alerts, drift tables, and score histograms without recomputing metrics server-side. Treat it as an exploration aid for reviewers, not an online scoring tier.
Reproducibility & tests
Pytest covers drift math, alert thresholds, simulations, and exporters so changes stay grounded. GitHub Actions installs requirements.txt and runs the suite on each push to main.