WorldOracle Launches Benchmark to Evaluate Action-Conditioned World Models in Robotics
- Published
- Source
- HumanoidsDaily
WorldOracle has introduced a benchmark to assess action-conditioned world models (ACWMs) by comparing their predictions with actual robot executions. In a demonstration, a robot's grasp failed while the model inaccurately predicted success, highlighting discrepancies between simulated and real-world outcomes. The research team evaluated seven models, revealing that while robot arm movements were predicted well, object movements often deviated significantly from reality.
This initiative is crucial as it addresses the reliability of predictions made by ACWMs, which are essential for improving robotic manipulation tasks. The findings indicate that even when robotic arms follow action sequences accurately, the predicted outcomes for manipulated objects can be misleading. This discrepancy emphasizes the need for robust evaluation methods to enhance model accuracy in real-world applications.
WorldOracle officially opened to developers on October 16, allowing them to submit predicted videos for scoring against real execution records. The evaluation process will help identify weaknesses in models, with a focus on improving predictions over complete operational sequences, as evidenced by the significant score drops observed in models like DreamDojo and Cosmos 3 during longer tasks.
Original report
WorldOracle Tests Whether World Models Can Reliably Predict Real-World Executions
HumanoidsDaily · [email protected] (Humanoids Daily Staff)This briefing is an independently written summary based on publicly available reporting and is provided for industry information and news discovery. The original report and source publication are credited and linked where applicable. RobotToday does not claim ownership of third-party source material.
Rights concerns? Contact [email protected] with the relevant URL and details. We will review the matter and take appropriate action where warranted.
More related news
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
Related Suppliers
NVIDIA Robotics
United States
NVIDIA robotics division delivering Isaac simulation and training tools, Jetson edge compute modules, and Omniverse digital twin frameworks for physical AI.
Deci AI (Acquired by NVIDIA)
Israel
Israeli deep learning optimization startup acquired by NVIDIA in May 2024 for ~$300M; known for AutoNAC neural architecture search technology.
Agibot 智元机器人
China
Shanghai humanoid-robot manufacturer founded 2023; builds A-series, X-series and D1 quadruped robots and has produced more than 15,000 units to date.
Tekpoint GmbH
Austria
Vienna B2B tech distributor founded 2009 with 120+ staff; AGIBOT's European humanoid distributor and a reseller of Roborock and Navimow home robots.
Qingdao CHOHO Industrial Co., Ltd. 青岛征和工业股份有限公司
China
Qingdao chain-drive maker (SZSE: 003033) that launched the 17-DoF chain-driven CHOHO Hand and micro-chain systems for humanoid joints.
Aether AI
United States
Seed-stage San Diego startup building causal world models that predict action outcomes, with robotics as its first target market.
Simlat
Israel
Israeli UAS training sim company; platform-agnostic drone operator simulators supporting 80+ programs in 30 countries for military and civil use.
Fatfish Technology Pte Ltd
Singapore
Singapore's FatFish AI builds Visual Action Intelligence software, led by its CAMP One engine, for cross-brand robot perception and action.
Quicktron Robotics 无锡快仓智能科技有限公司
China
Shanghai AMR maker with 45,000+ robots deployed for 1,000+ clients in 20+ countries; M/C/F/A series plus QuickBin goods-to-person software
AMI Labs
France
AMI Labs is a Paris AI startup founded by Yann LeCun in 2025, building JEPA-based world models trained on real-world sensor data, not language.
OpenAI
United States
US AI research lab and product company founded 2015; develops GPT, DALL-E, and robotics/embodied AI foundation models targeting physical-world AGI.
Shenzhen Stella-Robot Technology Co., Ltd. 深圳星际光年机器人技术有限公司
China
Shenzhen embodied intelligence startup founded in 2024, developing full-stack dexterous hand and joint module technology for humanoid robot manipulation.
Robotics needs a service framework.
RSF defines a common language for robot service capability, lifecycle operations, certification pathways, and service-provider networks.
Join the RobotToday community on LinkedIn
Daily robotics news, in-depth analysis, conference highlights, and discussions with professionals worldwide.
