Enabling robots to "understand" the world and learn to manipulate objects like humans—what sounds like science fiction is becoming reality through the work of Shubham Tulsiani, Assistant Professor at Carnegie Mellon University's Robotics Institute. In January 2026, he received the National Science Foundation (NSF) CAREER Award, the most prestigious honor for early-career faculty. The award recognizes his pioneering contributions to developing perception systems that better understand and interact with the 3D physical world.
Core Challenge: Bridging the Gap from 2D Video to 3D Action
Tulsiani's research addresses a fundamental question: How can robots learn 3D world structure and manipulation knowledge from the vast amount of 2D videos available on the internet? Imagine a robot watching a video of a human using a spray bottle—it must not only recognize the "press nozzle" action but also understand the nozzle's motion trajectory in 3D space and how this action changes the environment. When it later encounters a different brand of spray bottle it has never seen before, it should still be able to perform the same operation accurately. This is the 2D-3D gap his work aims to bridge.
His research emphasizes "physical grounding" and "spatial grounding," enabling machines to not just "see" the world but understand objects' 3D shapes, poses, materials, interaction mechanisms, and how actions affect the environment.
Academic Contributions: A Complete Pipeline from Perception to Manipulation
In computer vision, Tulsiani's early work (CVPR 2015 Best Student Paper Award) laid the foundation for 3D reconstruction from single images. He subsequently advanced research using cutting-edge techniques like Neural Radiance Fields (NeRF) and diffusion models, including SparseFusion, NeRS, and Cameras as Rays. These methods significantly reduce supervision requirements, enabling 3D understanding of in-the-wild scenes using internet-scale 2D data.
In robotics, his contributions more directly translate perception into action. Two representative works stand out:
-
Track2Act (ECCV 2024): This work validates a critical assertion—"internet videos contain rich 3D motion information usable for robot manipulation." By predicting point trajectories in videos, robots can achieve zero-shot manipulation in entirely novel scenes without any robot demonstration data.
-
Where2Act (ICCV 2021): This research proposes the core insight that "actionability is a function of object geometry." The system can directly predict actionable parts and optimal interaction directions for articulated objects from pixels, enabling robots to understand "where to push, pull, or rotate."
These works have been recognized multiple times at top conferences like ICRA and CoRL, including as a finalist for Best Paper in Robot Manipulation at ICRA 2024. His approach emphasizes learning generalizable manipulation skills from human demonstrations or videos, reducing dependence on expensive simulation or annotation data, and advancing robots toward the "perceive → learn → execute" paradigm.
Industry Impact: Accelerating Embodied Intelligence Commercialization
Tulsiani's work holds particularly significant potential for industry:
Manufacturing and Logistics: Zero-shot and few-shot manipulation techniques can dramatically reduce programming costs for industrial robots. Traditional robots require tedious teaching for each new task, while video-based learning methods enable robots to quickly adapt to new objects and scenarios.
Human-Robot Collaboration: Hand-object interaction reconstruction technology can improve collaborative robot safety and efficiency while advancing virtual hand tracking in AR/VR applications.
Service Robotics and Autonomous Driving: Physically-grounded 3D perception enhances the robustness of home service robots, drone navigation, and autonomous driving systems in unknown environments, enabling them to understand objects' physical properties and predict interaction outcomes.
Notably, Tulsiani previously worked at Meta AI Research (FAIR) Pittsburgh, collaborating with Abhinav Gupta. His research approaches have influenced multiple tech giants' strategies in robot vision and learning. As diffusion models, large models, and robot learning continue to converge, his framework promises to accelerate embodied AI's transition from laboratory to real-world deployment.
Outlook: The Next Milestone for Embodied Intelligence
The NSF CAREER Award recognizes not only Tulsiani's personal achievements but also signals that the direction of "learning generalizable robot skills from internet videos" has gained significant recognition from academia and funding agencies. His research combines theoretical depth (novel 3D representation and generation methods) with strong application orientation, laying critical technical foundations for the long-term goals of general-purpose robots and embodied AGI.
For RobotToday readers, Tulsiani's work represents the cutting edge of integrating robot perception and manipulation. He has demonstrated an important principle: robots don't need millions of trial-and-error attempts in expensive simulation environments—they can learn by observing others' behaviors, much like human infants. This may be the key breakthrough enabling commercial deployment of general-purpose robots.
Related Links:
- Shubham Tulsiani's Homepage: https://shubhtuls.github.io/
- NSF CAREER Award Details: https://www.nsf.gov/awardsearch/show-award?AWD_ID=2442282
- CMU Official News: https://www.cs.cmu.edu/news/2026/tulsiani-career-award
Leave a comment