# When a MuJoCo manipulation sim fails

Find the first physical link whose measurement disagrees with the command.

## The arm does not reach the target

- If the environment already exposes end-effector or operational-space
  control, use it before copying an IK loop. Record the resulting joint targets
  only when the application's action contract requires joint-space data.
- Report the pose residual and saturated joints; do not treat returned joint
  positions as convergence.
- Probe the task footprint and lift heights independently. One reachable point
  does not validate the workspace.
- Remove or move the object and repeat the same target. A result that succeeds
  without the object identifies obstruction rather than kinematics.
- When actuator force is already at its range, more settle steps are unlikely
  to help.

## The arm reaches but the grasp fails

- Measure the world-frame pinch axis and fingertip gap at the attempted pose.
  Position-only IK can leave a wrist rotation that makes the object impossible
  to span.
- Sweep grasp offset and support height together. Recalibrate after changing
  wrist orientation; grasp offset is pose-dependent.
- Measure the fingertip/object contact height relative to the TCP site. The
  physical pinch point may sit millimeters away from the site even when the TCP
  reaches the object's center.
- Inspect the approach trajectory, not only its endpoint. A joint-space servo
  can sweep through the object on the way to a valid pose.
- Resolve gripper geoms by body membership when mesh geoms are unnamed.
- Drive the gripper through both actuator limits and measure which end is open.
- Calibrate a stable hold command as well as open/closed polarity. A
  position-controlled gripper held at its fully closed limit can eject an
  object, so more squeeze time or slower carrying is not automatically safer.

## The scene looks wrong

- Compare the world surface, robot mount, object spawn, and camera against the
  intended real setup. A generic asset can make valid task poses unreachable.
- Check collision geometry separately from visual geometry. A convincing mesh
  can hide an unsuitable contact model.

## Rendering hangs or changes between identical resets

- Confirm the platform render backend was selected before importing MuJoCo.
- Create and use the renderer on the same thread. On macOS CGL, first use from
  another thread can deadlock silently.
- Replace target-tracking cameras with a static camera when repeated calls
  change an otherwise identical frame.
- Warm the renderer with one throwaway full reset, forward, scene update, and
  render before recording comparisons.
- Interactive-viewer failures are separate from offscreen rendering. On macOS,
  check the current `mjpython` requirement and its compatibility with the
  environment runner.

---
name: mujoco
description: Build and debug lightweight robot manipulation simulations with MuJoCo.
---

# MuJoCo

A plausible render proves little by itself. Follow the physical chain from
model through kinematics, actuation, contact, and observation.

## Start from the model

- For a new manipulation app or first policy demo, read
  [architect](../architect/SKILL.md) before creating a scene, controller, or
  viewer. Reuse its compatible reference-app selection if already made.
  Existing-model edits, physics debugging, and explanations stay here.
- Read the MJCF and the pinned asset revision before adding control code. Check
  joint ranges, actuator limits, collision geometry, sites, masses, and the
  intended work surface.
- Prefer a maintained model from
  [MuJoCo Menagerie](https://github.com/google-deepmind/mujoco_menagerie), but
  verify it against the real robot and task envelope.
- Confirm gripper polarity, fingertip gap, and contact geometry empirically.
  Names and documentation can disagree with the model that actually runs.
- Use the current [MuJoCo documentation](https://mujoco.readthedocs.io/) for
  MJCF and Python APIs rather than carrying signatures forward from memory.

## Follow the physical chain

- **Kinematics:** solve only for reachable targets and check the residual;
  damped least-squares can return a poor local solution without raising.
- **Actuation:** compare commanded position or torque with joint state,
  actuator force, range limits, and saturation.
- **Contact:** inspect which geoms belong to the gripper and object. Unnamed
  mesh geoms make name-only contact filters unsafe.
- **Grasp:** calibrate the grasp point, approach path, wrist orientation, and
  lift together. The end-effector site is not automatically the physical pinch
  point.
- **Observation:** make cameras and renderer lifecycle deterministic before
  using frames as training or regression data.
- **Controls:** distinguish model state, actuator limits, and rounded UI
  ranges. Clamp reset values to the actual widget bounds before binding them;
  a physically valid state can still be rejected by a narrower control.

## Go deeper only when needed

- For reachability, collision, grasp, saturation, and rendering symptoms, read
  [FAILURES.md](FAILURES.md).
- For the measured SO-arm and macOS evidence from Robium's manipulation trial,
  read [SO-ARM-MACOS.md](SO-ARM-MACOS.md). Preserve its numbers only with the
  stated model, scene, hardware, and renderer conditions.
- Use LeRobot guidance when the boundary reaches datasets, policies, or
  evaluation; use simulator-selection guidance when MuJoCo itself has not yet
  been chosen.

## Done

- The intended workspace is reachable, commands produce the expected joint and
  contact state, grasps survive a lift across representative poses, and seeded
  resets produce acceptably stable observations.

# SO-arm manipulation on macOS

This card records Robium observations from an SO-arm manipulation build on
Apple Silicon with MuJoCo 3.x, measured on 2026-07-13 through 2026-07-15. They
are diagnostic anchors, not defaults for another robot or scene.

## Runtime and rendering

- `MUJOCO_GL=cgl` was the working macOS offscreen backend and had to be set
  before importing `mujoco`; EGL and OSMesa are Linux paths.
- A 256×256 scene rendered at roughly 84 fps. Physics stepping measured about
  0.012 ms while rendering measured about 11.8 ms, so frame generation, not
  physics, dominated this small scene.
- A target-body camera produced up to 85 least-significant-bit differences from
  call-history-dependent smoothing. A static `xyaxes` look-at camera removed
  that source.
- A new renderer's first complete reset-and-render differed by about 6 LSB from
  later renders. One throwaway full reset, `mj_forward`, scene update, and
  render consumed the cold cycle.
- A renderer created on one thread and first rendered on another could hang in
  CGL `make_current`. Construct the environment and renderer in the thread that
  runs the episode.

## Model and workspace evidence

- MuJoCo Menagerie's current `robotstudio_so101` model states a MuJoCo 3.1.3+
  requirement. Pin and inspect the resolved MuJoCo version when using that
  model; a Gym/Gymnasium environment or another extra may add its own constraint.
- Menagerie provides the SO-101 MJCF and scenes, not a universal Gymnasium task
  or reward contract. In Robium's 2026-07 trial, no suitable off-the-shelf
  SO-101 Gymnasium environment was validated, so the manipulation environment
  was built around the model directly. Re-check the current LeRobot and
  Gymnasium ecosystem before budgeting that work in a new application.
- The trial's default object position near `(0.5, 0, 0.03)` was outside the
  usable mounted-arm workspace.
- Raising the work surface by 6 cm restored the intended envelope and avoided a
  forced roughly 32° downward pitch. This value belongs to that scene geometry.
- With the cube present, one target produced an arm residual near 0.18 with
  joints 1 and 2 saturated; with the cube removed, the residual fell near
  0.0006 with no saturation. This was collision evidence, not an IK tuning
  problem.

## Grasp evidence

- Near-zero wrist roll made the pinch axis about 91% vertical, leaving a 4.2 cm
  aperture unable to span a 6 cm cube. Solving wrist roll separately and then
  re-solving position improved the oracle from 0/10 to 8/10. Adding roll
  directly to that DLS objective diverged.
- A 3 mm grasp-offset change cost 6/10 successes in one sweep. Offset and
  support height were tuned together.
- Raising the first approach waypoint from 0.16 m to 0.20 m prevented a sweep
  through the object and improved that scenario from 2/10 to 10/10.
- The tested gripper was low-control closed and high-control open. Measure both
  actuator endpoints on any other model rather than copying this polarity.

Current upstream starting points:

- [Menagerie `robotstudio_so101`](https://github.com/google-deepmind/mujoco_menagerie/tree/main/robotstudio_so101)
- [The Robot Studio SO-101 simulation model](https://github.com/TheRobotStudio/SO-ARM100/tree/main/Simulation/SO101)