PKU at IROS 2026: VLM + 3D Gaussian Splatting for Dual-Arm Grasping

Grasping for the Task: Peking University Pairs Vision-Language Models with 3D Gaussian Splatting to Guide Dual-Arm Robots

An IROS 2026 paper from Peking University's School of Intelligence Science and Technology links open-vocabulary 3D perception, physics-aware grasp ranking and dual-arm posture optimization into a single pipeline, and reports a 94.5% handover success rate in its tested settings.

Same mug, different jobs: the correct grasp region changes with the task (pour, pass, hold, adjust). Source: Du et al., IROS 2026, Fig. 1.

A robot can hold a hammer securely by its head and still be holding it wrong. To drive a nail, it needs the handle; to pass the hammer safely to a person, the head may be the better choice. Researchers at Peking University have built a system that lets a dual-arm robot work out which grasp a task actually calls for, then carry out the regrasp without twisting either arm into an awkward, near-singular posture.

The paper, "Task-Oriented Dual-Arm Manipulation via Semantic-Physics Grasp Scoring and Kinematic Comfortability," has been accepted to the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026), held in Pittsburgh from September 27 to October 1. Jiawei Du is first author and Sicheng Xiang second author; Associate Professor Dingsheng Luo, Vice Dean of the School of Intelligence Science and Technology, is corresponding author.

Why VLM + 3D Gaussian Splatting Is Getting Attention

3D Gaussian Splatting, introduced in 2023, represents a scene as a large set of small, semi-transparent 3D "blobs" that can be rendered quickly from any viewpoint. Robotics researchers soon began attaching language features from vision-language models (VLMs) such as CLIP to these representations. The result is a 3D map that can be searched with words: ask for "the handle" and the robot gets a location in space, not just a highlighted region in a single camera image.

The idea builds on earlier language-embedded radiance fields: LERF-TOGO (CoRL 2023) showed zero-shot, task-oriented grasping from language queries, and 2024 work such as GaussianGrasper and GraspSplats carried the approach over to Gaussian Splatting, with GraspSplats reporting scene construction in under 60 seconds. Most of this work stops at deciding where a single gripper should grasp. The Peking University paper pushes the same kind of representation into a harder setting: two arms, one handing an object to the other, under physical and kinematic constraints.

The Problem: A Stable Grasp Is Not Always a Useful One

Conventional grasp planners are good at finding places where a gripper can hold an object firmly. They are not designed to know that a mug should be held by its handle for pouring rather than by its rim. Purely language-driven methods have the opposite weakness: they can find the right part but may pick a spot far from the center of mass, where gravity creates enough torque to make the object slip.

Dual-arm robots offer a practical way to resolve the mismatch: one arm supports the object while the other takes a task-appropriate grip. But the two hands must hold a precise relative pose during the exchange, and a pose that is mathematically reachable can still leave an arm near a joint limit or a singularity, with little room left for the next motion.

How the System Works

Pipeline overview, from semantic 3DGS reconstruction to bimanual optimization and execution. Source: Du et al., IROS 2026, Fig. 2.

1. Find the right part. The robot builds its 3D model using what the authors call "Moving Object, Static Camera" (MOSC) collection: the grasping arm rotates the object in front of a fixed RGB-D camera. Segment Anything (SAM) splits each view into whole-object and part masks, and CLIP features for those regions are fused into a hierarchical 3DGS model. A large language model translates a task such as "pour water" into a part description such as "cup handle," which is matched against the stored features. The authors say they chose 3DGS not because it gives better grasp geometry, but because it fuses language features from many views consistently, including across occlusions and part boundaries.

2. Pick a grasp that is both relevant and stable. An off-the-shelf generator, GSNet, proposes candidate 6-DoF grasps. The team does not retrain it. Instead, each candidate is re-scored on three factors: how well it matches the target part, its geometric grasp quality, and how far its line of action passes from an estimate of the object's center of mass. Shorter lever arms mean less gravitational torque during the handover.

3. Keep both arms "comfortable." Dual-arm execution is solved as a constrained optimization that preserves the required pose between the two hands while avoiding joint limits and collisions. Its key ingredient is a joint-weighting schedule that changes with distance to the target: on approach, the shoulder and base joints take on most of the movement while the wrist stays steady; close to the target, the wrist is given more freedom for fine alignment. The optimizer converges in roughly 80 to 100 steps, with a reported solve time under 200 milliseconds.

What the Experiments Show

The team tested the framework in PyBullet simulation with a hammer, a mug and a power drill, and on a physical platform built from commercially available hardware: two Realman RM-65 7-DoF collaborative arms[1] , two Inspire Robotics 6-DoF dexterous hands, an Intel RealSense D435i camera and an NVIDIA RTX 4090 GPU. All grasp-scoring variants drew from the same GSNet candidate pool, so differences come from the scoring step alone.

Grasp selection. The geometry-only baseline found stable grasps (82.5% stability) but often on the wrong part, such as the hammer claw (34.2% semantic accuracy, 28.1% task success). The semantics-only variant found the right part 91.5% of the time but frequently chose an edge with a long lever arm (45.3% stability, 52.4% task success). Combining the two raised results to 95.8% semantic accuracy, 89.2% stability and 86.7% task success.[2] 

Dual-arm coordination. Standard pseudoinverse-Jacobian inverse kinematics produced a 28.5% singularity rate and 61.2% handover success. A fixed-weight version of the optimizer improved this to 12.1% and 78.4%. The full method with scheduled weights cut the singularity rate to 2.4%, raised handover success to 94.5%, and lifted average manipulability from 0.045 to 0.124.

Real-world sequences: (top) hammer handover without optimization; (middle) the same handover with comfortability optimization; (bottom) placing flowers in a vase. Source: Du et al., IROS 2026, Fig. 8.

On the physical robot, the team showed a "pass the hammer to me" handover with and without the optimizer, and a "put the flowers in the vase for display" pick-and-place task that reuses the same pipeline outside a handover scenario.

Why It Matters for Industry

Handing tools to people, regrasping parts for assembly and preparing items for a next step are routine in factories, warehouses and service settings. Today they usually rely on fixed grasp libraries, CAD models or hand-labeled parts for each object. A pipeline that takes a plain-language instruction and works out both where to grasp and how two arms should move points toward systems that are faster to re-task.

Three design choices stand out for practitioners. First, the work is modular: it reuses an existing grasp generator and foundation models rather than training a new end-to-end network, which makes components easier to swap and results easier to trace. Second, the physics step is lightweight and explainable, a single lever-arm penalty rather than a learned black box. Third, the system runs on off-the-shelf arms, hands and sensors, including Chinese-made collaborative arms and dexterous hands of the kind now widely available to integrators.

Limits and What to Watch

The authors are explicit that the results show feasibility on the tested cases, not broad generalization. Several gaps remain before deployment:

• Scale. Testing covers three simulated objects and two real-world tasks, and the reported figures are aggregates without trial-to-trial variance.

• Speed. Building the semantic 3DGS model takes about three minutes per object, so the front end is not yet suited to fast-changing scenes, even though the arm optimization itself runs in under 200 ms.

• Physics. The center-of-mass estimate is a geometric centroid. For uneven objects, such as a hammer with a heavy head, it can be off; the authors describe it as a lever-arm heuristic rather than a calibrated torque model.

• Benchmarks. The paper compares against component baselines. Head-to-head tests against end-to-end systems such as LERF-TOGO and DA-VIL on a shared sensing and control stack are left to future work, along with material-aware center-of-mass estimation and repeated trials.[3] 

About the Researchers

Jiawei Du and Sicheng Xiang are graduate students at the School of Intelligence Science and Technology, Peking University. Their research focuses on robotic perception, task-oriented grasping, intelligent manipulation and coordinated dual-arm systems. Co-authors are Zibo Wang and Yao Geng of Peking University and Yi Wang of the Department of Automation, Tsinghua University.[1] 

Their advisor, Associate Professor Dingsheng Luo, is Vice Dean of the School of Intelligence Science and Technology. His group works on intelligent robotics, embodied intelligence, robot learning and coordinated multi-arm manipulation, and has published earlier work on dual-arm operation-point selection, joint-function-aware arm reaching and holistic control for mobile manipulation. The paper lists affiliations with Peking University's National Key Laboratory of General Artificial Intelligence and Key Laboratory of Machine Perception (Ministry of Education), as well as the PKU-Wuhan Institute for Artificial Intelligence.

Researcher background information was provided by the research team.

Paper Information

Title: Task-Oriented Dual-Arm Manipulation via Semantic-Physics Grasp Scoring and Kinematic Comfortability

Authors: Jiawei Du, Sicheng Xiang, Zibo Wang, Yao Geng, Yi Wang, Dingsheng Luo (corresponding)

Affiliations: Peking University (National Key Laboratory of General Artificial Intelligence; Key Laboratory of Machine Perception, MoE; School of Intelligence Science and Technology); Department of Automation, Tsinghua University; PKU-Wuhan Institute for Artificial Intelligence

Conference: 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2026), Pittsburgh, USA

Funding (as acknowledged): National Natural Science Foundation of China (No. 62176004, No. U1711327), among others

Sources

[1] J. Du, S. Xiang, Z. Wang, Y. Geng, Y. Wang, D. Luo, "Task-Oriented Dual-Arm Manipulation via Semantic-Physics Grasp Scoring and Kinematic Comfortability", IROS 2026 (paper provided by the authors)

[2] B. Kerbl et al., "3D Gaussian Splatting for Real-Time Radiance Field Rendering", ACM Transactions on Graphics, 2023

[3] A. Rashid et al., "Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping", CoRL 2023

[4] Y. Zheng et al., "GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Grasping", arXiv:2403.09637, 2024

[5] M. Ji et al., "GraspSplats: Efficient Manipulation with 3D Feature Splatting", arXiv:2409.02084, 2024