Newshot

A minimal probe of task modeling in language models

Summary

arXiv:2605.02028v3 Announce Type: replace Abstract: Language models read long contexts and score well on knowledge and reasoning benchmarks, but neither shows whether a model can do work whose next step depends on everything that came before, as a long-running agent must. We call this ability task modeling and propose counting identical items as its simplest test: every item adds to the total and no item records it. We define counting capacity (CC) as the sequence length up to which a model returns the exact count more often than not, and measure it for 205 models. Every resolved measurement loses count abruptly, at a median of 50 items and at most 627; across release years the median rose 1.6-fold while context windows grew about 80-fold. Changing the counted symbol or the delimiter moves a model's CC severalfold, yet the model accounts for about three quarters of the variance across 16 models and 11 encodings. Reasoning leaves the median CC unchanged across 77 matched pairs. In six o…

Original text(via arXiv cs.CL)

Computer Science > Computation and Language

Title:A minimal probe of task modeling in language models

Abstract:Language models read long contexts and score well on knowledge and reasoning benchmarks, but neither shows whether a model can do work whose next step depends on everything that came before, as a long-running agent must. We call this ability task modeling and propose counting identical items as its simplest test: every item adds to the total and no item records it. We define counting capacity (CC) as the sequence length up to which a model returns the exact count more often than not, and measure it for 205 models. Every resolved measurement loses count abruptly, at a median of 50 items and at most 627; across release years the median rose 1.6-fold while context windows grew about 80-fold. Changing the counted symbol or the delimiter moves a model's CC severalfold, yet the model accounts for about three quarters of the variance across 16 models and 11 encodings. Reasoning leaves the median CC unchanged across 77 matched pairs. In six open-weight models the count stays readable in early layers past the boundary but does not reach the output, and an activation edit that carries the count forward also carries the state of an object-tracking task. CC tracks the METR task horizon after controlling for general capability, and rises with capacity on grid-execution tasks and, with reasoning off, on object tracking. We do not show that counting is necessary or sufficient for long-horizon work; that remains a hypothesis, which this cheap measurement can test.
Comments:for accessing the data and code for reproduction of the study, see this https URL
Subjects:Computation and Language (cs.CL)
Cite as:arXiv:2605.02028 [cs.CL]
(or arXiv:2605.02028v3 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2605.02028 arXiv-issued DOI via DataCite

Submission history

Access Paper:

  • HTML (experimental)

Additional Features

  • Audio Summary

Current browse context:

References & Citations

  • Google Scholar
  • Semantic Scholar

BibTeX formatted citation

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by arXiv cs.CL
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗