Skip to main content
Visit DexHubOrder Now

MagicSim: A Unified Infrastructure for Executable Embodied Interaction

Northwestern, Peking University and partners build a single simulation runtime where benchmarks, automatic data collection and agent interfaces share one execution loop. A Vega 1 Pro with Sharpa Wave hands is one of its ready-to-run mobile manipulator embodiments.

Paper

16 Jun 2026

MagicSim overview: one deterministic runtime drives long-horizon episodes through a command, skill, planner, robot and record pipeline, exposed as a benchmark, an automatic data collector and an agent interface.

Haoran Lu1,*,†, Songling Liu5,6,*, Yue Chen2,*, Guo Ye1, Mutian Shen1, Shuyang Yu1, Yu Xiao1, Shang Wu1, Jiayi Wang1, Jianshu Zhang1, Jihai Zhao1, Xiangtian Gui4, Chuye Hong3, Yuran Wang2, Maojiang Su1, Ruihai Wu2,3, Zhaoran Wang1, Han Liu1,†

  • 1Northwestern University
  • 2Peking University
  • 3University of California, Berkeley
  • 4ShanghaiTech University
  • 5Institute of Automation, Chinese Academy of Sciences
  • 6University of Chinese Academy of Sciences

* Equal contribution · † Corresponding author

arXiv preprint arXiv:2606.17511

PDFarXivVega IK Configs

Abstract

Robot learning and embodied agents now require simulation to serve as a shared execution substrate linking control, skills, and planning, not only as a renderer, controller testbed, or fixed task environment. Existing pipelines split these layers with "magic" actions, disconnected training environments, or forward-only renders that cannot reproduce, evaluate, and annotate the same episode. We present MagicSim, an embodied interaction infrastructure built around one deterministic batched runtime and a shared Markov decision process (MDP). From YAML-first specifications that decouple contents, placement, behavior, and agent exposure, MagicSim constructs diverse executable worlds spanning task families, interaction regimes, physics, layouts, sensors, avatars, and robot embodiments in one reset-and-step loop. A common execution interface grounds high-level commands through controllers, atomicskills, planner primitives, and asynchronous planning, realizing them as robot actions rather than simulator-side state edits. One task definition supports three capabilities: benchmark and RL evaluation, an autocollect interface that automatically turns commands into grounded trajectories, and agent/VLM-facing interaction. For automatic execution, commands flow through a Command->Skill->Planner->Robot->Record pipeline, while per-environment command, skill, planning, retry, annotation, and episode states advance independently above the shared physics tick. Successful rollouts are saved as structured multimodal trajectories aligning language supervision, action representations, visual/geometric representations, and task-level status with the executed episode. MagicSim thus unifies diverse world construction, embodied execution, task evaluation, automatic rollout generation, and interactive agent interfaces in one planner-in-the-loop runtime.

Citation

@misc{lu2026magicsimunifiedinfrastructureexecutable,  title = {MagicSim: A Unified Infrastructure for Executable Embodied Interaction},   author = {Haoran Lu and Songling Liu and Yue Chen and Guo Ye and Mutian Shen and Shuyang Yu and Yu Xiao and Shang Wu and Jiayi Wang and Jianshu Zhang and Jihai Zhao and Xiangtian Gui and Chuye Hong and Yuran Wang and Maojiang Su and Ruihai Wu and Zhaoran Wang and Han Liu},  year = {2026},  eprint = {2606.17511},  archivePrefix = {arXiv},  primaryClass = {cs.RO},  url = {https://arxiv.org/abs/2606.17511}, }

Start Your Physical AI Journey with Us

Talk to Us
Vega 1 Pro reaching forward with both arms