Skip to main content

SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments

The Harvard Computational Robotics Group builds an agentic pipeline that turns 3D and 4D captures into interactive simulation scenes, checking and repairing each step inside the simulator. A simulated Dexmate Vega with Sharpa Wave hands works in several of the converted scenes.

Paper

26 Oct 2026

A simulated Dexmate Vega with two Sharpa Wave hands lifts a plate with a slice of pizza in a kitchen scene that SceneAgent converted from an AI-generated 3D capture.

Luke Hollis1,*, Tianxing Fan1,*, Heng Yang1

  • 1Computational Robotics Group, School of Engineering and Applied Sciences, Harvard University

* Equal contribution

CoRL 2026 Late-Breaking Results · arXiv:2610.34745

Abstract

Visual captures provide detailed scene appearance, but converting them into interactive simulation environments requires object reconstruction, physical modeling, and correction of intermediate errors. We present SceneAgent, an agentic pipeline that coordinates these steps through simulator-based visual inspection and repair. The pipeline processes 3D Gaussian Splatting (3DGS), photogrammetry, LiDAR captures, AI-generated 3DGS scenes, and 4D videos of dynamic scenes, and combines object segmentation, physical parameter estimation, part decomposition, articulation, and deformable simulation. To support policy learning, SceneAgent generates diverse object variants ("digital sisters") and scripted demonstrations. On a cluttered bin scene from GraspClutter6D, SceneAgent raises F1 at 1 cm for every coding agent with scorable outputs both with and without its skills, and its reconstructions are better than SimFoundry: all agent configurations exceed SimFoundry's F1 score, with a best of 60.75% versus 6.9%. On three dynamic scenes from 4DCodeBench, SceneAgent raises the mean dynamic IoU of Gemini 4 Argon from 0.86% to 99.09%. We further demonstrate diverse simulated interactions and preliminary policy transfer to a real robot.

Citation

@misc{hollis2026sceneagentagenticpipeline3d,  title = {SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments},   author = {Luke Hollis and Tianxing Fan and Heng Yang},  year = {2026},  eprint = {2610.34745},  archivePrefix = {arXiv},  primaryClass = {cs.RO},  url = {https://arxiv.org/abs/2610.34745}, }

Start Your Physical AI Journey with Us

Vega 1 Pro reaching forward with both arms