Article title: BAAI Wujie·PhysCapBench: The First Cross-Domain "Physical Data Ruler" for World Models
Body: The core of a physical AI foundation model is to learn transferable physical laws from real experience, forming the ability to understand, predict, and act upon the physical world. Real-world data contains rich physical information: internet videos record motion and environmental evolution, egocentric data presents hand-object interaction, real-robot trajectories record actions and state responses, and force/tactile feedback supplements contact forces, slip, and deformation; together, different data sources provide complementary physical experience. How to measure, distill, and couple the physical characteristics in these data is a fundamental problem that urgently needs to be solved in building physical AI foundation models.
Addressing this problem, BAAI, together with Inverse Matrix, Peking University, Tsinghua University, and Beihang University, presents Wujie·PhysCapBench—the first benchmark that covers three major domains—third-person scenes, first-person manipulation, and handheld gripper demonstrations—and uniformly evaluates the physical information in video captions.

Wujie·PhysCapBench: A Cross-Domain Benchmark for Physically Grounded Video Captioning
Starting from video descriptions, it uniformly measures the coverage and fidelity of descriptions with respect to physical information, providing a cross-domain "physical data ruler" for description strategy selection, annotation quality assessment, and training data optimization.

Figure 1: Overall framework of Wujie·PhysCapBench—cross-domain data, physical annotation, description evaluation, and downstream validation
The physical information that world models lack may be hiding in a single caption. In building pre-training data for world models, we have consistently observed that, when faced with videos from different sources such as third-person scenes, first-person manipulation, and handheld gripper demonstrations, existing captions often summarize the scene and actions but fail to adequately preserve motion, contact, state changes, and their temporal relationships; this gap exists not only in short descriptions of raw data but may also appear in longer, more fluent automatically generated descriptions. Completeness of linguistic expression does not equal completeness of physical information.
A soft rubber toy is a typical example (see Figure 7): after being compressed, it gradually recovers its shape as the pressure is released; compression, deformation, unloading, and rebound together constitute the key physical information of this process. In our generation comparison, when the physical description was used, the result better preserved the toy's three-dimensional form and its recovery trend after unloading; when the original description was used, the toy still showed a clearly flattened state after the pressure was released. How a caption expresses the physical process directly determines what conditions and supervision the model receives.
Wujie·PhysCapBench distills the observations and reflections from pre-training data practice into cross-domain reference descriptions, unified evaluation dimensions, and a downstream validation pipeline: it evaluates which physical information a caption preserves and also tests whether this information can serve video generation and robot policy learning, providing a basis for transforming multi-source real data into effective training supervision.
01 Covering three types of complementary data, establishing a cross-domain evaluation framework
Third-person scenes, first-person manipulation, and handheld gripper demonstrations present physical processes from different perspectives, providing complementary learning experience for world models: third-person scenes focus on object motion, deformation, interaction, and environmental change; first-person manipulation focuses on human-object contact, the manipulation process, and state transitions; handheld gripper demonstrations record the interaction between the gripper and objects, the manipulation stages, and the execution results.
We bring the three types of data into a single evaluation framework, with the accuracy and completeness of physical information as the common quality objective. The benchmark contains a total of 1,500 video sequences, 500 for each domain; for multi-view demonstrations, the same interaction process is organized as one video sequence.

Figure 2: Physical processes in third-person scenes, first-person manipulation, and handheld gripper demonstrations

02 Seven evaluation dimensions, locating the physical information gaps in captions
Description strategies place different emphasis on covering task goals, action steps, and state changes. Wujie·PhysCapBench evaluates captions along seven dimensions, breaking overall quality down into specific information types and providing a basis for description strategy selection and annotation optimization. The inspection focus of each of the seven dimensions is as follows.

Figure 4: The seven physical information evaluation dimensions of Wujie·PhysCapBench
What the seven dimensions together must answer is actually the same question: does this description explain the physical process clearly and accurately? Below is the description strategy leaderboard for the first-person manipulation domain, with the overall score and the seven dimension scores laid out side by side, so the information weaknesses of different strategies can be seen directly.

Figure 5: Description strategy leaderboard for the first-person manipulation domain, with overall score and seven dimensions. Physis is an internal test
03 Building verifiable physical descriptions grounded in video evidence
Physical processes usually span multiple temporal stages: motion before contact occurs, deformation after contact, and the state after the action ends often require continuous video evidence to be described accurately. Therefore, high-quality physical captions need to satisfy both information coverage and evidence consistency—preserving key changes while ensuring that the description can be supported by the video.
Wujie·PhysCapBench organizes the reference description construction process around this requirement: starting from global observation and event localization, it gradually works through the participating objects, the action process, contact relationships, and state changes, and then verifies the description content; seven professional annotators participated in annotation and quality checking. For mechanism and causal information, descriptions are grounded in observable evidence; for complex manipulation, the focus is on checking whether the stage division, interaction relationships, and resulting states remain consistent.

Figure 6: Video-evidence-based physical annotation process
Through this process, the reference descriptions both serve unified evaluation and provide a reusable quality basis for subsequent description generation and data annotation.
04 If physical information is written more completely, how much better can downstream models perform?
The evaluation of description quality needs to be further linked to downstream utility: Wujie·PhysCapBench examines, from the two directions of video generation and manipulation policy, whether more complete physical information can improve model performance. For video generation, more complete physical descriptions can provide more effective generation conditions. The soft rubber toy case mentioned at the beginning intuitively demonstrates the generation differences under different description conditions.

Figure 7: Generation comparison for a soft rubber toy under compression and after unloading. Left: real video; middle: generation result using the physical description; right: generation result using the original description
Further, with the generation model, first frame, and generation duration fixed, the original data description and the physical description were respectively used as conditions to compare the perceptual differences between generated videos and real videos.

Figure 8: Comparison of video reconstruction metrics under original description and physical description conditions. LPIPS and PVCS measure perceptual differences; lower values are better. Lines connect the means of the same model under the two description conditions, and green labels indicate the relative decrease
Physical descriptions brought improvements in both domains and on both generation models. Among them, in the first-person manipulation domain, LPIPS decreased by up to 30.88% relatively, and PVCS decreased by up to 28.09% relatively. These results show that the action, interaction, and state information preserved in descriptions can form effective generation conditions, helping models generate evolution processes closer to real videos.
For robot policies, training language supervision is enriched through a Training Instruction Upsampler. On top of the original task instruction, it supplements the observable action process, contact relationships, and state changes in the video, providing more complete language supervision for policy learning; the enhanced instruction is used in the training phase, while the original short instruction is still used at inference time.
The physical interaction example below shows the state changes that need attention during manipulation: the failed trajectory tilts and overturns the bowls during manipulation and placement, failing to form a stable stack; the successful trajectory completes the stacking of three bowls.

Article title: BAAI Wujie·PhysCapBench: The First Cross-Domain "Physical Data Ruler" for World Models
Body: Figure 9: RoboDojo contact and placement examples, failure on the left, success on the right
Whether an object moves stably with the gripper, whether it tilts after contact, and whether it forms a stable support after release are all key state information during manipulation. Incorporating this information into training descriptions helps provide process supervision for policy learning that is more informative than task goals alone.
To evaluate the practical effect of training instruction augmentation, we conducted a controlled experiment on a RoboTwin 2.0 subtask: using the same 50 demonstrations, with augmented instructions applied to half of the training samples, π0.5's number of successes across 40 shared test initial scenes increased from 15 to 18, with the success rate rising from 37.5% to 45.0%, an improvement of 7.5 percentage points. In multi-task evaluations on RoboTwin and RoboDojo, the average success rates on selected tasks improved for all four policy models.
From generation conditions to training instructions, the two types of experiments demonstrate the downstream value of physical descriptions and also provide a basis for data optimization: the quality of captions should be tied to the information required for model prediction and execution.
Data construction for world models needs to focus on what physical experience is present in videos, and whether that experience is completely and accurately transformed into training supervision. Wujie·PhysCapBench distills the data practices from pretraining into a cross-domain "physical data ruler": through reference data from three major domains, seven physical information evaluation dimensions, and downstream validation via video generation and manipulation policies, it provides a unified basis for description strategy selection, annotation quality assessment, and training data optimization.
From evaluating the physical information in captions to validating their downstream value, Wujie·PhysCapBench provides reusable data and evaluation infrastructure for building the foundation of world models.
https://hfwang2001.github.io/physcapbench-site/
https://github.com/hfwang2001/PhysCapBench
Hugging Face
https://huggingface.co/datasets/hfwang01/PhysCapBench
http://arxiv.org/abs/2611.16424