Replay & Render

SONIC replay drives an external controller process, so rendering Isaac Sim inside the live control loop can starve ZMQ/DDS communication and trip controller heartbeat timeouts. Replay and rendering are therefore split:

Phase 1: run SONIC + MuJoCo replay, save the qpos/qvel state trace
Phase 2: after replay, load the trace into Isaac Sim, sync each state, render

Run

HEADLESS=1 bash scripts/run_replay_wbc.sh \
  simple/G1WholebodyXMoveBendCarryBoxSonic-v0 \
  --sim-mode mujoco_isaac \
  --offline-isaac-render \
  --no-webrtc \
  --data-dir data/teleop_wbc/simple/G1WholebodyXMoveBendCarryBoxSonic-v0/level-0 \
  --num-episodes 1 \
  --replay-dir logs/sonic_replay_isaac_render

--data-dir defaults to data/teleop_wbc/<env_id>/level-0, so it can be omitted when replaying the dataset the launcher just recorded.

Outputs:

logs/sonic_replay_isaac_render/episode_0_mujoco_state_trace.npz   # qpos / qvel
logs/sonic_replay_isaac_render/episode_0/*.mp4                    # Isaac Sim frames

A _failed suffix on an MP4 is the task success label — not a rendering failure.

How it works

During live replay the controller runs against MuJoCo only, and each control frame stores mjData.qpos / mjData.qvel. Afterwards a separate headless mujoco_isaac environment is created and reset from the same episode configuration, then every saved state is pushed in:

for state in states:
    sonic_render_env.mjData.qpos[:] = state["qpos"]
    sonic_render_env.mjData.qvel[:] = state["qvel"]
    mujoco.mj_forward(sonic_render_env.mjModel, sonic_render_env.mjData)
    sonic_render_env.isaac.step(sonic_render_env.mujoco)
    frame_dict = sonic_render_env._render_frame()

Each camera stream in frame_dict is written to its own MP4 with VideoWriter.

Because the trace is a plain .npz, rendering can be rerun from it without launching the controller again — useful for debugging rendering or validating Isaac Sim synchronization.

Verify output

ffprobe -v error -select_streams v:0 \
  -show_entries stream=nb_frames,r_frame_rate,duration,width,height \
  -of default=noprint_wrappers=1 \
  logs/sonic_replay_isaac_render/episode_0/head_stereo_left_failed.mp4

Troubleshooting

• Isaac Sim extension warnings — deprecated extensions, missing material USD files and optional robot-motion extensions log warnings even on success. Treat them as fatal only if Python raises or no video is written.

• No video produced — check that --replay-dir is set, that --offline-isaac-render is paired with an Isaac sim mode (--sim-mode mujoco_isaac), that the replay produced a non-empty state trace, that Isaac Sim starts headless, and that the task layout has cameras.