Newshot

Tsinghua's Xu Mengdi: No Matter How Good Pretraining Gets, Robots Need to Learn at Deployment | CoRL 2026

AI summary

Human-robot interaction is not just the goal but also the solution. At the CoRL 2026 "Learn@Deploy" half-day workshop on November 12, 2026, Xu Mengdi, Assistant Professor at Tsinghua University's Institute for Interdisciplinary Information Sciences, gave the opening keynote, distilling her recent work and sharing her views on the field's future direction.

Translation(model translation, not the source's own text)

Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026

Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026
Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026

On November 12, 2026, at the half-day "Learn@Deploy" (learning during robot deployment) workshop at CoRL 2026, 徐梦迪, Assistant Professor at Tsinghua University's Institute for Interdisciplinary Information Sciences, gave the opening talk as a keynote speaker.

After graduating from Tsinghua with a degree in Vehicle Engineering, 徐梦迪 went to the Department of Mechanical Engineering at Carnegie Mellon University for her PhD, where she was advised by Ding Zhao, head of the Safe AI Lab. Her thesis, "Building Adaptable Generalist Robots," won the department's Best PhD Thesis Award. In the thesis, she argued that building generalist robots cannot rely solely on scaling parameters and data and hoping pretraining will generalize on its own; the ability to adapt must be added, so that robots can quickly adjust their policies on site using a small amount of new information.

After her PhD, 徐梦迪 joined the Stanford Vision and Learning Lab as a postdoc, working with 吴佳俊 and 李飞飞. During this time, she worked on ROSETTA, which uses large models to convert the wide variety of human spoken preferences into reward signals that robots can learn from, enabling robots to learn new instructions they had never seen during training.

In 2025, 徐梦迪 joined Tsinghua's Institute for Interdisciplinary Information Sciences and founded her own lab, with her direction further focused on "learning at deployment time." In early 2026, the team released CoVer-VLA, which has the robot generate several candidate action sets before acting, then uses a verifier to pick the one that best matches the instruction, thereby aligning with the instruction. Dex-X learns dexterous manipulation from human videos and uses simulation to fill in the tactile supervision missing from the videos; it was accepted to CoRL 2026.

In this talk, 徐梦迪 distilled her work over the past two years and shared with the academic community her judgment on future technological development. The following is an edited transcript of the talk brought back from the scene by Leiphone (WeChat official account: Leiphone).

▎Speaker: 徐梦迪, Assistant Professor, Institute for Interdisciplinary Information Sciences, Tsinghua University

Talk title: Building Adaptable Generalist Robots for Human-Centered Environments

Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026
Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026

Today I want to discuss how to build adaptable generalist robots, especially robots in scenes where humans are present. The unique challenge of such scenes is that robots must complete navigation and manipulation tasks in the open world, such as clearing a table after lunch; they must also interact with people, from following instructions, preferences, and gestures to providing physical assistance, such as helping a person stand up or walk.

I have some observations that I hope will get everyone thinking about why learning at deployment time is so important in human-centered scenes.

Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026
Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026

First, humans can easily ask robots to complete tasks beyond their skill range. The simplest approach is to have a person do it once, and the robot directly learns the person's behavior or preference. For example, pouring water: the robot initially cannot do it, or encounters a new cup it has never seen, and a person can demonstrate once so the robot understands the object's properties and completes the task.

Second, open-world environments are in many cases observable. For example, when a low-cost robotic arm tries to pick up a kettle filled with a lot of water, after the first attempt it knows the kettle exceeds its payload, and on the second attempt it can learn from experience and switch to lifting the kettle with both arms before pouring.

Third, human preferences vary greatly. When pouring water, different people want different amounts, and even the same person may want different amounts at different times of day. Robots need to interact with people in a streaming fashion, for example by letting a person press a green button to directly control the amount of water.

These observations all lead us to one conclusion: robots must acquire new information at deployment time to adapt their policies.

But robot learning is still a machine learning problem and must have supervision. What supervision signals can we obtain at deployment time? I think three are the most important: agentic systems, human feedback, and simulation.

Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026
Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026

The first is agentic systems, composed of language models, vision-language models, or independent verifiers, which help the robot judge whether behavior is good or bad and provide these good and bad signals so the robot can adapt its policy across different attempts.

Another natural source of supervision is having humans interact directly with the robot. But human instructions come in diverse types, including language, gestures, demonstrations, and joint modalities.

Finally, there is simulation, which can serve both as an evaluation platform and for training, broadly including physics simulators, robot simulators, and world models, helping the robot choose better actions by predicting the outcomes of different actions.

Next I will share three of our recent papers, each combining different resources.

Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026
Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026

Scaling verification is more effective than scaling policy learning

First, let's talk about why scaling verification matters. Our conclusion is that scaling verification can achieve vision-language-action alignment more effectively than scaling policy learning.

We did this work because we observed that even the most advanced vision-language-action models still have limited instruction-following ability. Existing approaches include, first, augmenting training data by rewriting instructions, but this cannot cover all human instructions at deployment time; second, using larger vision-language model backbones, but learning action generation causes severe forgetting.

Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026
Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026

So can we assume a pretrained vision-language model already exists, make only lightweight modifications, and improve the policy through test-time reasoning and verification?

We propose training an independent verifier with contrastive learning and combining it with the vision-language-action model into a contrastive verification framework. During offline training, hard positives and negatives are generated; at deployment time the verifier works at two levels, first verifying good instructions for the current task, then verifying good actions under a given instruction.

Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026
Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026

Paper link: https://arxiv.org/abs/2602.12281

The pipeline is: given a user prompt and the current scene image, first use a vision-language model to generate diverse rewritten instructions, then for each rewritten instruction call the vision-language-action model multiple times to obtain candidate action chunks, then score at the instruction level and the action level respectively, and select the optimal action to deploy.

The verifier is trained purely offline, and at deployment time only multiple parallel queries are needed, with very low latency. For example, with a batch size of 32 it adds only about 8 milliseconds, roughly 1% of the overall latency.

The experiments also compared different sampling mechanisms: adding only Gaussian noise, repeatedly querying the original model, sampling only instructions, and mixed instruction-and-action sampling. The results show that mixed sampling significantly increases action diversity, making it more likely to sample the best action chunk and reducing action errors. The significance of mixed sampling lies precisely in using different language instructions to increase action diversity.

We also compared "using the extra data directly to fine-tune Pi 0" with "using a small amount of data to train a verifier and verifying generated samples at deployment time." The latter improves performance about 7 times faster. The in-distribution improvement is about 20%, and the out-of-distribution improvement about 10%. The framework is also fairly general, can be plugged into different types of vision-language-action models, and repeatedly verifies and selects at the start of each action chunk.

Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026
Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026
Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026
Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026

The second work is closer to human-centered environments, with the goal of making robot policies more adaptable to different human instructions. This work was completed about two years ago, when reinforcement learning was used to learn new skills and fill the gap between the pretrained skill set and the skill set humans need.

Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026
Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026

Paper link: https://openreview.net/forum?id=Ig6goVdtjb

We conducted a large-scale study in which the robot interacted with 20 different people who had little understanding of robots. We found that human instructions are indeed very diverse, and that people switch roles, sometimes putting themselves in the robot's position and sometimes acting like the robot's supervisor.

Adapting robots to these preferences was very difficult at the time. Our approach was to use agentic systems (such as vision-language models) to convert human preferences into learning reward signals, then use these rewards to train a reinforcement learning agent.

Through experiments, we noticed that the most critical component is disambiguating human instructions. Recently, when I used Codex for similar tasks, I also saw that even when the instruction is vague, it first does grounding, associating the instruction with the current robot rollout and the current scene. At the time we tried to make the grounding mechanism more explicit, and now the most advanced language models already have this mechanism.

Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026
Tsinghua's 徐梦迪: No Matter How Good Pretraining Is, It's Not Enough—Robots Need to Learn During Deployment | CoRL 2026

Article title: Tsinghua 徐梦迪: No matter how good pretraining is, it's not enough—robots need to learn during deployment | CoRL 2026

Body: The third work discusses how to learn from human demonstrations. Suppose a robot interacts with people in the scene; the simplest way to let the robot learn a new skill is to have a person demonstrate the task once, and the robot learns useful behavioral signals from the demonstration.

Tsinghua 徐梦迪: No matter how good pretraining is, it's not enough—robots need to learn during deployment | CoRL 2026
Tsinghua 徐梦迪: No matter how good pretraining is, it's not enough—robots need to learn during deployment | CoRL 2026

Paper link: https://arxiv.org/abs/2609.07747

The difficulty is that although human videos contain rich behavioral and preference information, there are embodiment gaps and modality gaps. For example, tactile information is very important to humans, and a task may not be completed without touch, but it is difficult for a robot to directly obtain contact information from human videos.

Our approach uses a simulator as a bridge to fill in contact information, enabling the robot to use tactile information during deployment inference. The pipeline is simple: a teacher–student architecture, and both the teacher and the student are trained in simulation. The results once again prove that tactile information is important: without touch, the success rate drops by at least 10%, and tactile sensing is especially critical in dexterous manipulation scenarios.

Tsinghua 徐梦迪: No matter how good pretraining is, it's not enough—robots need to learn during deployment | CoRL 2026
Tsinghua 徐梦迪: No matter how good pretraining is, it's not enough—robots need to learn during deployment | CoRL 2026

Through these past works, I have formed some ideas. I believe that if we want to study deployment-time learning, the following points are very important:

First, robots should learn like humans, both from agent systems and from humans first. Human-robot interaction is not only a goal but also a solution to make robots more capable and more general.

Second, in-context learning is important. Robots need to interact with humans at high frequency and adapt their policies at high frequency. The lightest-weight way is in-context learning, without updating model parameters; at deployment time, it may also be necessary to update parameters using newly collected interaction data.

Third, robot systems should evolve as a whole system. System 1 may need new robot models to achieve better unconscious learning, but overall the robot system may be a complex system with at least a two-layer structure, with a steering mechanism to leverage information from different language models.

Fourth, simulation is important. It provides an alternative way to explore scenarios and can also save exploration and reward costs.

The top international robotics conference CoRL 2026 is about to be held. Researchers, students, and readers who follow the latest AI progress are welcome to join our Leiphone.com community group.

Here, you can meet peers, exchange research, share admissions information, scientific research collaborations, dinner meetups, submissions, team formation for attending the conference, and more, and find potential partners.

During the conference, we will also continue to bring on-site report dispatches, interpretations of guest talks, summaries of frontier directions, and forum highlights.

? Group entry portal: Scan the QR code to join the group, or add WeChat szhsn610, with the note: CoRL + unit/school + name + direction.

Tsinghua 徐梦迪: No matter how good pretraining is, it's not enough—robots need to learn during deployment | CoRL 2026
Tsinghua 徐梦迪: No matter how good pretraining is, it's not enough—robots need to learn during deployment | CoRL 2026
Tsinghua 徐梦迪: No matter how good pretraining is, it's not enough—robots need to learn during deployment | CoRL 2026
Tsinghua 徐梦迪: No matter how good pretraining is, it's not enough—robots need to learn during deployment | CoRL 2026
Tsinghua 徐梦迪: No matter how good pretraining is, it's not enough—robots need to learn during deployment | CoRL 2026
Tsinghua 徐梦迪: No matter how good pretraining is, it's not enough—robots need to learn during deployment | CoRL 2026

This is an original article by Leiphone.com. Reproduction without authorization is prohibited. For details, see the reprint notice.

  • 李飞飞's latest CoRL2026 speech transcript: From the "cabinet model" to the wor...
  • 李飞飞 in conversation with 苏姿丰: First on the same stage after AMD's acquisition, talking at the CoRL site about...
  • CoRL 2026 | Noitom Robotics opens HiPHI-Hand, supporting Nvid...
  • Six years of CoRL: paper count ×4.5, who is driving it up?

The translation is machine-generated and individual terms may be off. Nothing is rewritten or cut; the original is authoritative.

Published by 雷锋网
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗