My model's accuracy on a rolling window dropped. Is the model actually getting worse?

Not necessarily: a windowed metric can move even when the model's predictive power hasn't changed. The Modern ML Monitoring Mess (part 1) gives three causes: class ratios differ across windows, windows hold different numbers of examples (fewer requests on a Sunday than a Monday), and labels arrive late, so recent predictions lack labels. We recommend checking these before triggering a retrain.

Sources: The Modern ML Monitoring Mess: Rethinking Streaming Evaluation (1/4)

More questions on this topic

All questions about using AI for data and knowledge work