Can I use Prometheus and Grafana to monitor ML model accuracy?

They work for latency, errors, and input statistics, but poorly for metrics like accuracy that join predictions with later labels. In The Modern ML Monitoring Mess (part 3), we tried to monitor cumulative accuracy for a toy taxi-tip model with Prometheus. It needed high-cardinality joins and convoluted PromQL, results depended on the scrape interval, and accuracy couldn't be computed over sliding windows.

Sources: The Modern ML Monitoring Mess: Failure Modes in Extending Prometheus (3/4)

More questions on this topic

All questions about using AI for data and knowledge work