Computer Science > Computation and Language
Title:A minimal probe of task modeling in language models
Abstract:Language models read long contexts and score well on knowledge and reasoning benchmarks, but neither shows whether a model can do work whose next step depends on everything that came before, as a long-running agent must. We call this ability task modeling and propose counting identical items as its simplest test: every item adds to the total and no item records it. We define counting capacity (CC) as the sequence length up to which a model returns the exact count more often than not, and measure it for 205 models. Every resolved measurement loses count abruptly, at a median of 50 items and at most 627; across release years the median rose 1.6-fold while context windows grew about 80-fold. Changing the counted symbol or the delimiter moves a model's CC severalfold, yet the model accounts for about three quarters of the variance across 16 models and 11 encodings. Reasoning leaves the median CC unchanged across 77 matched pairs. In six open-weight models the count stays readable in early layers past the boundary but does not reach the output, and an activation edit that carries the count forward also carries the state of an object-tracking task. CC tracks the METR task horizon after controlling for general capability, and rises with capacity on grid-execution tasks and, with reasoning off, on object tracking. We do not show that counting is necessary or sufficient for long-horizon work; that remains a hypothesis, which this cheap measurement can test.
| Comments: | for accessing the data and code for reproduction of the study, see this https URL |
|---|---|
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.02028 [cs.CL] |
| (or arXiv:2605.02028v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.02028 arXiv-issued DOI via DataCite |
Submission history
Access Paper:
- HTML (experimental)
Additional Features
- Audio Summary
Current browse context:
References & Citations
- Google Scholar
- Semantic Scholar
BibTeX formatted citation
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.