Computer Science > Machine Learning
Title:A Temporal Algebraic Framework for Data Stream Evaluation Under Delayed Labels and Distribution Shifts
Abstract:In data streams, labels often arrive late or only for some instances, class prevalence drifts, and rare classes vanish from short windows. Standard evaluation then confuses a degraded decision boundary with a change in prevalence, and is biased when labels are delayed or selected by the model. We extend an algebraic framework that obtains pooled (micro) and per-class (macro) measures from one confusion matrix to streams, weighting the counts in time by exponential fading or a sliding window; the update costs O(m^2) per step, whatever the stream length, and each pooled measure stays a weighted combination of the per-class ones. Delayed and selective feedback enter as an observation mask: until its label arrives, an instance is counted with the model's calibrated class probabilities, and the bias of this interim estimate is driven by their calibration error. Audited conventions report no value for a class without verified instances. We separate prior shift from concept drift by monitoring prior-free coordinates (recall and specificity) and attribute a change in a measure to the two sources with a first-order Jacobian decomposition. On synthetic streams and four real datasets, a detector on these coordinates with a memory per class raises no excess false alarms under label shift, unlike the Jacobian alarm and error-rate detectors. The correction for delayed labels recovers recall in some real settings and is only partial for precision, specificity and F1. Twenty-five statements are checked in Lean 4, and the code is in the aggmetrics package.
| Comments: | 53 pages, 25 tables, 2 figures. Code: this https URL (experiments), this https URL (Lean 4 formalization), this https URL (aggmetrics >= 0.4.0, subpackage this http URL ) |
|---|---|
| Subjects: | Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Databases (cs.DB) |
| MSC classes: | 68T05, 62H30, 62L10, 68V20 |
| ACM classes: | I.2.6; H.2.8; G.3 |
| Cite as: | arXiv:2611.21914 [cs.LG] |
| (or arXiv:2611.21914v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.21914 arXiv-issued DOI via DataCite (pending registration) |
Submission history
Access Paper:
- HTML (experimental)
Additional Features
- Audio Summary
Current browse context:
References & Citations
- Google Scholar
- Semantic Scholar
BibTeX formatted citation
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.