SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments
The Harvard Computational Robotics Group builds an agentic pipeline that turns 3D and 4D captures into interactive simulation scenes, checking and repairing each step inside the simulator. A simulated Dexmate Vega with Sharpa Wave hands works in several of the converted scenes.

- 1Computational Robotics Group, School of Engineering and Applied Sciences, Harvard University
* Equal contribution
CoRL 2026 Late-Breaking Results · arXiv:2610.34745
Abstract
Visual captures provide detailed scene appearance, but converting them into interactive simulation environments requires object reconstruction, physical modeling, and correction of intermediate errors. We present SceneAgent, an agentic pipeline that coordinates these steps through simulator-based visual inspection and repair. The pipeline processes 3D Gaussian Splatting (3DGS), photogrammetry, LiDAR captures, AI-generated 3DGS scenes, and 4D videos of dynamic scenes, and combines object segmentation, physical parameter estimation, part decomposition, articulation, and deformable simulation. To support policy learning, SceneAgent generates diverse object variants ("digital sisters") and scripted demonstrations. On a cluttered bin scene from GraspClutter6D, SceneAgent raises F1 at 1 cm for every coding agent with scorable outputs both with and without its skills, and its reconstructions are better than SimFoundry: all agent configurations exceed SimFoundry's F1 score, with a best of 60.75% versus 6.9%. On three dynamic scenes from 4DCodeBench, SceneAgent raises the mean dynamic IoU of Gemini 4 Argon from 0.86% to 99.09%. We further demonstrate diverse simulated interactions and preliminary policy transfer to a real robot.
Citation
@misc{hollis2026sceneagentagenticpipeline3d, title = {SceneAgent: An Agentic Pipeline from 3D and 4D Captures to Simulation Environments}, author = {Luke Hollis and Tianxing Fan and Heng Yang}, year = {2026}, eprint = {2610.34745}, archivePrefix = {arXiv}, primaryClass = {cs.RO}, url = {https://arxiv.org/abs/2610.34745}, }Start Your Physical AI Journey with Us
