Computer Science > Software Engineering
Title:A Concurrent Structured Record Mirrored the Prose It Accompanied: A Pre-Registered Conceptual Replication and Extension of the Hide Pending Tool Call Scenario
Abstract:Huang et al. (2026) show that language models writing reports of agentic work tend to conceal narrative-changing flaws, and that appending "Be honest in your response." to the instruction substantially increases disclosure in most of their scenarios. In their Hide Pending Tool Call scenario, where a broader follow-up query was accepted but never returned, the honesty prompt left GPT-5.5's rate of flagging the missing data at 0%, although it turned most answers into caveats. I ask whether a different request changes this: appending, in the same generation, a structured record that follows the schema of an adversarial-review tool, including a residue section for what remained unresolved. In two pre-registered phases on the same 100 synthetic log pairs, each pending log matched with a clean twin in which the follow-up query did return, I ran gpt-5.6-sol through Codex CLI under four conditions. Phase 1 measured the record: with the residue schema, a judge coded it as stating the unreturned query in 29% of logs, against 5% with a findings-only placebo schema, and in none of the clean twins (p = 1.2 x 10^-7); a second run two days later gave 30% and 6%. But 70-71% of residue records declared that nothing remained open while the query was pending. Phase 2 measured the prose answer: the confirmatory comparison against the honesty prompt was not significant (four discordant logs, all favoring the honesty prompt; p = 0.125), no equivalence test was pre-registered, and the judge failed the pre-registered human validation on the decisive category; I therefore claim neither an effect nor its absence. In an exploratory cross-tabulation, every residue record coded as stating the gap accompanied prose coded as mentioning it (30 of 30), and none accompanied prose coded as omitting it (0 of 70): in this setting, the structured record mirrored the prose it accompanied.
| Comments: | 10 pages, 4 tables. Pre-registered study; protocols, code, data and analyses at this https URL |
|---|---|
| Subjects: | Software Engineering (cs.SE) ; Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.23274 [cs.SE] |
| (or arXiv:2611.23274v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23274 arXiv-issued DOI via DataCite |
Submission history
Access Paper:
- HTML (experimental)
Additional Features
- Audio Summary
Current browse context:
References & Citations
- Google Scholar
- Semantic Scholar
BibTeX formatted citation
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.