How do I monitor an ML model in production when labels arrive late or not at all?

Track inputs and predictions right away, and compute accuracy-style metrics as labels arrive, knowing each window is incomplete until then. In our position paper Rethinking Streaming Machine Learning Evaluation, we argue that tracking accuracy alone on unbounded, time-ordered data misses problems, partly because labels arrive late, and we recommend additional metrics. The Modern ML Monitoring Mess series works through why.

Sources: Rethinking Streaming Machine Learning Evaluation (ICLR 2022: Workshop on ML Evaluation Standards); The Modern ML Monitoring Mess: Rethinking Streaming Evaluation (1/4)

More questions on this topic

All questions about using AI for data and knowledge work