Newshot

80,000 Group Chat Messages Turned Into an Exam: Tencent Yuanbao and KCL Release the First Agent Memory Benchmark Built on Group Chats

Translation(model translation, not the source's own text)

80,000 Group Chat Messages Turned Into an Exam: Tencent Yuanbao × KCL Release the First Agent Memory Benchmark Built on Group Chats

Tencent Yuanbao × King's College London NLP Team Contributed to Quantum位 | Public Account QbitAI

A travel group of nine people initially agreed to depart on May 1, but later changed it to April 30. When someone asks an AI assistant to send a ticket-purchase reminder, it will most likely still notify them of the old date.

Although it "remembers" the history, it fails to truly serve this group.

In real group chat scenarios, there are often more subtle requirements: who wants a window seat, who has budget constraints, which arrangements are merely proposals, and which have become collective decisions.

For an AI assistant to serve a group of people, the key is not just retrieving memories, but judging whose memory and which memory should be used at this moment.

To address this problem, Tencent Yuanbao and the King's College London NLP team jointly launched GroupAssistBench. Based on 50 real group chats and 80,299 historical messages, the study constructed 2,395 evaluation questions to examine whether Agent memory can truly move from "remembering well" to "using well."

The research team compared 14 configurations, including full context, retrieval augmentation, and external memory. One representative contrast is: EverMemOS scored 0.621 on the L1 memory recall evaluation, higher than the 0.570 of full context using the same task model; but on the L2 open-ended service evaluation, it scored 0.247, lower than the 0.431 of full context.

Recalling better does not necessarily mean serving better.

The paper has been made public on arXiv, and the code and dataset have also been open-sourced.

△Main experimental results of the 14 configurations: L1 memory recall on the left, L2 open-ended service on the right. Comparisons should be made within each metric.

From a Group Chat to Two Kinds of Tests: Memory Must Be Findable and Also Used Correctly

Existing Agent Memory evaluations often test historical recall through factual question answering, and recent work has begun to focus on downstream memory use. But real group chats raise more complex problems: the same piece of information may be supplemented by different members, arrangements update as discussions proceed, and a single reply may be constrained by multiple people at the same time.

GroupAssistBench builds data from interactions with real deployed group assistants, preserving naturally occurring topic shifts, state changes, and interwoven multi-person information, and divides the evaluation into two interrelated task levels.

L1 and L2: From Answering Facts to Completing Services

L1 memory recall has clear reference facts, such as "which day was initially agreed for departure." L2 memory use requires completing a service, such as "send everyone a reminder about the trip and ticket purchase." The answer has no fixed wording, but it must comply with valid historical decisions and member requirements. The two types of questions number 1,374 and 1,021 respectively.

The study further divides them by the memory structure they rely on into M1 single memory, M2 multi-memory integration, and M3 update and conflict handling. M3 especially requires identifying: when an old decision is superseded by a new one, and which members' constraints still apply. These categories describe forms of information dependency rather than a simple difficulty ranking.

△Overview of the evaluation framework: L1 factual recall and L2 open-ended service questions are built from the same group chat history, and dependency types are divided into single memory, multiple memory, and update conflict.

Every Question Must Truly Depend on History

To avoid systems guessing correctly based on common sense alone, the team first compared answers "with history provided" and "without history provided," screening for requests that genuinely require group chat memory; then established Memory Anchors for the necessary facts, locating the source messages and evidence snippets, and set respective history cutoffs to prevent models from reading information after the question was asked.

The data underwent anonymization and privacy review, and the public version does not contain original group chat identity information. The construction process combined programmatic rules, semantic review, and manual verification; in sampling verification, 95.8% of the questions were judged valid by majority vote.

△Data construction process: screen requests that genuinely depend on history, locate memory anchors, construct evaluation questions, and then gatekeep through rules and manual verification.

Services Have No Standard Answers, So How Are Scores Assigned?

When a group assistant helps everyone arrange activities, reminds them to buy tickets, or coordinates disagreements, there are often multiple appropriate replies. Simple string matching cannot determine whether a service truly satisfies historical constraints. Therefore, the team built a customized Rubric bound to the corresponding historical evidence for each L2 question.

This Rubric consists of three parts: Grounding checks whether the answer explicitly reflects the necessary historical facts; Service checks whether the task is completed; and Penalty punishes errors such as violating existing decisions, ignoring member restrictions, or using outdated information. For example, a ticket-purchase reminder must adopt the new date while also accounting for the relevant number of people, transportation arrangements, and individual seat preferences; treating an obsolete date as the final arrangement should be penalized.

The Penalty design also draws on real users' corrective feedback to group assistants. The study conducted manual verification using 1,152 criterion judgments across 60 L2 questions and 180 responses, and the agreement rate between automatic judgment and manual majority vote was 89.3%.

497 Paired Memories: Why Are Facts Found but Still Not Used?

To avoid mistaking gaps between different question types for differences in memory ability, the study tracked 497 paired memories: the same piece of history was both directly asked about in L1 and genuinely depended on by an L2 service. This allows observation of what happens to memories that have "already been recalled" when they reach the service stage.

Among the analyzed external memory systems, for memories that were recalled in L1 but not judged as effectively used in L2, about 36% did not enter the L2 model input, while the remaining about 64% entered the input but were not judged as effectively applied in the response. These percentages only describe the fate of this type of paired memory, not the failure rate of all L2 tasks.

The former case is related to the request lacking clear retrieval cues: when a user says "help everyone arrange it," they do not point out one by one which historical preferences need to be retrieved. The latter shows that evidence having been given to the model does not automatically mean the model knows how it should constrain the current response.

△The fate of 497 paired memories: comparing whether memories successfully recalled in L1 are applied in L2, enter the input but are not judged as applied, or do not enter the input.

Two Interventions: String Together Scattered Evidence, Turn Historical Requirements Into Constraints

Based on the above phenomena, the team explored two lightweight intervention directions. RER (Relational Evidence Refinement) organizes the relationships among people, events, times, and decisions after retrieval, and supplements relevant candidate evidence when needed; ACB (Assistance Constraint Binding) organizes the obtained facts into response constraints with subject, scope, and current valid state.

In the experiments, RER raised LightMem's L2 average score by 6.28 percentage points; ACB raised xMemory and EverMemOS by 3.94 and 3.16 percentage points respectively. When combined, LightMem, xMemory, and EverMemOS improved their L2 scores by 7.03, 4.64, and 4.43 percentage points relative to their respective baselines.

These results show that evidence organization and constraint explicitation are expected to improve memory-driven service performance. The independent contribution of each of the two mechanisms still needs further breakdown, and all improvements cannot simply be attributed to "retrieving a few more memories."

△Effects of RER and ACB interventions: comparing the performance improvements of relational evidence refinement, service constraint binding, and their combination across different memory systems.

From Remembering History to Using History at the Right Time

Real group chats are a testing ground for Personal Agents and multi-user assistants. As agents begin to take on more long-term services, the value of a memory system is not only "how much it stores and how fast it searches," but also whether it can understand when to use information, whom it applies to, and which information has become invalid. GroupAssistBench attempts to make such differences observable, analyzable, and comparable.

Paper link: https://arxiv.org/abs/2611.04867 Hugging Face paper: https://huggingface.co/papers/2611.04867 Code: https://github.com/Tencent/GroupAssistBench Dataset: https://huggingface.co/datasets/tencent/GroupAssistBench Project homepage: https://zhanghao-groupassisbench.github.io

  • AI cranks out an MV on the spot for a band, is this a hackathon or a music festival 2026-11-21
  • Chinese computing embarks on a new path to the summit 2026-11-21
  • Going viral across the internet! Grok's leading lady teaches you how to submit 3,000 PRs a month 2026-11-21
  • After overtaking Gemini, Karpo open-sources another work, browser Agent achieves quick and double wins 2026-11-21

The translation is machine-generated and individual terms may be off. Nothing is rewritten or cut; the original is authoritative.

Published by 量子位
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗