Claude Cursor Skill

lerobot

Build and debug LeRobot datasets, training, and policy evaluation. For a first pretrained robot-arm demo, start with architect's reference-app selection.

LLM Mart · 0 points · 0 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download robium-ai-robium-skills_lerobot-498ea4e.zip · 17 KB
Part of robium-ai/robium — 44 skills

Install

skills CLI npx skills add https://github.com/robium-ai/robium/tree/main/skills/lerobot
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install robium-ai-robium@llmmart
Git git clone https://github.com/robium-ai/robium.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole robium-ai/robium collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

LeRobot

Follow one contract chain through LeRobot: embodiment, dataset, processor, checkpoint, runtime observations and actions, then evaluation. Find the first contract that does not match.

Establish the contract

  • For a new manipulation app or first pretrained-policy demo, read architect before creating an environment or training pipeline. It finds the saved apps checkout and selects a compatible baseline. If selection already happened, continue here. Existing dataset, training, or evaluation work does not need onboarding; an inference-only request does not authorize training.
  • Inspect the installed LeRobot version and current CLI help before writing flags. Dataset formats, policy families, scripts, and extras change quickly.
  • Match the robot's state, action space, cameras, rates, and task to the dataset. A policy adapts to those features; it cannot repair a mismatched embodiment.
  • Inspect every checkpoint's configuration and processor files, not only its weights. Base and fine-tuned checkpoints from one family can expect different camera layouts.
  • Keep Hub identity, transfer, publication, and Jobs lifecycle at the Hugging Face boundary. Keep source-selection strategy in data.

Prove the loop cheaply

  • Start with a small shipped policy and a known dataset/environment pair.
  • Run a short train that writes a checkpoint, then load that exact checkpoint through evaluation. Completion and numeric metrics are the smoke-test result; policy quality is not.
  • Confirm loss, saved processors, input/output shapes, rollout metrics, and video or real-robot behavior before increasing steps or hardware cost.
  • Treat real-hardware rollout as a new safety boundary even when simulation evaluation passed.

Go deeper only when needed

Done

  • The target dataset loads, the checkpoint carries its processors, evaluation exercises matching observations and actions, and measured results justify any longer run or hardware deployment.
Files (robium)
  • examples
    • load-dataset-snippet.py 2.4 KB
      # status: verified 2026-08-02 (deep-verify: pusht-dataset-loads)
      # source: assembled from huggingface/lerobot's docs/source/lerobot-dataset-v3.mdx
      # ("Load a dataset for training" section) and its README's own LeRobotDataset
      # example, both fetched directly via raw GitHub URLs on 2026-07-10 (main
      # branch). The `lerobot/pusht` repo id and its shape (206 episodes, 25,650
      # frames, "observation.image" + "observation.state" + "action" features)
      # were confirmed by fetching that dataset's meta/info.json directly from
      # huggingface.co on 2026-07-10, not assumed from memory.
      #
      # Pairs with examples/train-act-command.md, which trains an ACT policy
      # against this same dataset (lerobot/pusht); the feature names printed
      # below (observation.image, observation.state, action) are exactly what
      # that training command's policy adapts to automatically.
      #
      # Requires: uv add "lerobot[dataset]" (verify the current extra in LeRobot's
      # install docs; references/datasets.md explains the dataset contract.)
      
      from lerobot.datasets import LeRobotDataset
      
      REPO_ID = "lerobot/pusht"
      
      
      def main() -> None:
          # Downloads and caches the dataset under ~/.cache/huggingface/lerobot/
          # the first time it's loaded; subsequent loads reuse the local cache.
          dataset = LeRobotDataset(REPO_ID)
      
          print(f"repo_id: {REPO_ID}")
          print(f"num_episodes: {dataset.num_episodes}")
          print(f"num_frames: {dataset.num_frames}")
          print(f"fps: {dataset.fps}")
          print(f"features: {sorted(dataset.features.keys())}")
      
          # Random access by frame index: returns a dict of PyTorch tensors.
          sample = dataset[0]
          print(f"observation.state shape: {tuple(sample['observation.state'].shape)}")
          print(f"observation.image shape: {tuple(sample['observation.image'].shape)}")
          print(f"action shape: {tuple(sample['action'].shape)}")
      
          # A temporal window instead of a single frame: request the current frame
          # plus the two preceding it (seconds relative to t, so this depends on
          # `dataset.fps`). Useful for policies that condition on recent history.
          delta_timestamps = {"observation.image": [-0.2, -0.1, 0.0]}
          windowed = LeRobotDataset(REPO_ID, delta_timestamps=delta_timestamps)
          windowed_sample = windowed[0]
          # Shape becomes [T, C, H, W] instead of [C, H, W] for the windowed key.
          print(
              "windowed observation.image shape: "
              f"{tuple(windowed_sample['observation.image'].shape)}"
          )
      
      
      if __name__ == "__main__":
          main()
      
    • train-act-command.md 4.2 KB
      # Small-scale ACT training smoke run on `lerobot/pusht`
      
      **Status:** unverified as written, but exercised via adaptation 2026-07-12
      (manip-trial, lerobot 0.6.0, M2 Pro/MPS): the same command shape with
      `--steps=200 --save_freq=200 --batch_size=8 --policy.device=mps` trained in
      31 s (loss 14.4→3.5) and at `--steps=10000` in 14.5 min (loss→0.33); the
      `checkpoints/last/pretrained_model` eval path below resolved exactly as
      written.
      
      **Source:** the command shape below is assembled from two directly-fetched
      upstream sources on 2026-07-10: huggingface/lerobot's README ("Training a
      policy is as simple as running a script configuration", `lerobot-train
      --policy.type=act --dataset.repo_id=...`) and its `docs/source/cheat-sheet.mdx`'s
      training section (`--output_dir`, `--job_name`, `--policy.device`,
      `--wandb.enable`, `--policy.repo_id`, `--steps`). The dataset (`lerobot/pusht`)
      was confirmed to exist and to already be on LeRobotDataset v3.0 via a direct
      fetch of its `meta/info.json` from huggingface.co on 2026-07-10 (206
      episodes, 25,650 frames). `--steps=3000` below is a deliberately small
      smoke-run value, not an upstream default; see the reasoning below and
      `references/policies-and-training.md`'s compute-sizing table before scaling
      it up for a real run.
      
      Pairs with `load-dataset-snippet.py` in this same directory (same dataset,
      `lerobot/pusht`). Use it only when PushT is a useful smoke path for the
      application; its purpose is to prove train → checkpoint → evaluation with a
      self-trained checkpoint before spending substantial compute.
      
      Requires the `training` and `pusht` extras (`uv add
      "lerobot[training,pusht]"`; verify the current extras in LeRobot's install
      documentation; `core_scripts` was only needed for hardware CLIs in the tested
      release).
      
      ```bash
      uv run lerobot-train \
        --dataset.repo_id=lerobot/pusht \
        --policy.type=act \
        --output_dir=outputs/train/act_pusht_smoke \
        --job_name=act_pusht_smoke \
        --policy.device=cuda \
        --steps=3000 \
        --save_freq=1000 \
        --wandb.enable=false \
        --policy.repo_id=${HF_USER}/act_pusht_smoke \
        --policy.push_to_hub=false
      ```
      
      **Why these values:**
      
      - `--policy.type=act`: the smallest policy family (`references/policies-and-training.md`'s
        compute table: ~2-6 GB VRAM, ~30-60 min for 5 epochs on an RTX 4090-class
        GPU), the right choice for a first smoke run, not a VRAM-bound VLA policy.
      - `--steps=3000`: deliberately short. `lerobot/pusht` has 25,650 frames;
        3,000 steps at the ACT default batch size is well under one full epoch,
        enough to confirm the loss curve moves and a checkpoint saves/evaluates
        cleanly, not a converged policy. ACT trains at a constant LR (its
        `get_scheduler_preset()` returns no scheduler, confirmed by fetching
        `configuration_act.py` directly on 2026-07-10), so there's no LR-decay
        schedule to rescale when changing `--steps`, unlike scheduler-based
        policies (diffusion, SmolVLA, the Pi0 family); see
        `references/policies-and-training.md` if switching `--policy.type` to one
        of those.
      - `--policy.device=cuda`: swap for `mps` (Apple Silicon) or `cpu` only after
        checking the policy's current accelerator support; see `FAILURES.md` for
        measured limits of those smoke paths.
      - `--wandb.enable=false` and `--policy.push_to_hub=false`: kept off for a
        throwaway smoke run; flip both on for a real tracked run (`wandb login`
        first; dropping `--policy.push_to_hub=false` pushes the trained policy to
        `--policy.repo_id` on the Hub, which is the `huggingface` skill's
        territory once you get there).
      
      **After this run**, evaluate the checkpoint using the current CLI shape in
      `references/eval-and-sim.md`, pointing `--policy.path` at
      `outputs/train/act_pusht_smoke/checkpoints/last/pretrained_model` (path
      verified 2026-07-12):
      
      ```bash
      uv run lerobot-eval \
        --policy.path=outputs/train/act_pusht_smoke/checkpoints/last/pretrained_model \
        --env.type=pusht \
        --eval.batch_size=10 \
        --eval.n_episodes=10 \
        --eval.use_async_envs=false \
        --policy.device=cuda
      ```
      
      A smoke run at this scale is not expected to solve PushT (success needs
      ≥95% T-coverage; even 10k steps measured `pc_success` 0 with
      `avg_max_reward` 0.28); the goal is confirming the train -> checkpoint ->
      eval loop works end to end before committing to a long run.
      
  • references
    • datasets.md 9.4 KB
      # LeRobotDataset: format, loading, recording
      
      The LeRobotDataset format (currently **v3.0**), how to load or stream one
      for training, how recording and dataset-editing CLIs use it, and how
      episode visualization hands off to Rerun. Hub mechanics beyond "this is the
      LeRobotDataset shape a hub repo has" (auth, upload/download, model/dataset
      cards) are the `huggingface` skill's territory, not this file's.
      
      Sources: `huggingface/lerobot`'s `docs/source/lerobot-dataset-v3.mdx`,
      `docs/source/il_robots.mdx`, `docs/source/using_dataset_tools.mdx`, and the
      README, all fetched directly via raw GitHub URLs on 2026-07-10 (`main`
      branch). The `lerobot/pusht` dataset's `meta/info.json` was fetched directly
      from `huggingface.co/datasets/lerobot/pusht` to confirm it is already on
      format v3.0 (206 episodes, 25,650 frames); this is the dataset this
      skill's examples use throughout.
      
      ## Format v3.0: what changed and why
      
      v3.0 (included in `lerobot >= 0.4.0`; the installed `lerobot` on 2026-07-10
      is 0.6.1) replaced v2.1's one-file-per-episode layout with **file-based
      storage**: many episodes are concatenated into fewer, larger Parquet and MP4
      files, with episode boundaries resolved through metadata rather than
      filenames. This is what makes `StreamingLeRobotDataset` (below) practical at
      scale: fewer, larger files mean less filesystem overhead when streaming
      directly from the Hub instead of downloading first.
      
      **Three storage pillars:**
      
      1. **Tabular data** (states, actions, timestamps): Apache Parquet, memory-
         mapped or streamed via the `datasets` stack.
      2. **Visual data** (camera frames): MP4, frames from the same episode
         grouped, videos sharded per camera.
      3. **Metadata**: JSON/Parquet describing schema, FPS, normalization stats,
         and episode segmentation (start/end offsets into the shared files).
      
      **Directory layout (simplified):**
      
      - `meta/info.json`: schema (feature names/dtypes/shapes), FPS, codebase
        version, path templates for locating data/video shards.
      - `meta/stats.json`: global feature statistics (mean/std/min/max) for
        normalization, exposed as `dataset.meta.stats`.
      - `meta/tasks.jsonl`: natural-language task descriptions mapped to integer
        IDs, for task-conditioned policies.
      - `meta/episodes/`: per-episode records (lengths, tasks, offsets) as
        chunked Parquet.
      - `data/`: frame-by-frame Parquet shards, each typically holding many
        episodes.
      - `videos/`: MP4 shards per camera, each typically holding many episodes.
      
      ## Loading a dataset for training
      
      Install the dataset dependencies required by the pinned LeRobot release before
      testing video-backed episodes. Robium observed that bare `lerobot==0.6.0`
      imported successfully but could not decode dataset video; the matching
      `lerobot[dataset]` extra supplied TorchCodec and restored the load path. Re-check
      the current extra and decoder requirements rather than treating import success
      as a dataset smoke.
      
      ```python
      from lerobot.datasets import LeRobotDataset
      
      repo_id = "lerobot/pusht"
      dataset = LeRobotDataset(repo_id)   # downloads + caches locally
      
      sample = dataset[100]
      # {'observation.state': tensor(...), 'action': tensor(...),
      #  'observation.image': tensor([C, H, W]), 'timestamp': tensor(...), ...}
      ```
      
      `delta_timestamps` requests a temporal window (seconds relative to the
      current frame) instead of a single frame per key, e.g.
      `{"observation.image": [-0.2, -0.1, 0.0]}` returns a `[T, C, H, W]` stack.
      `LeRobotDataset` returns plain dicts of PyTorch tensors and works directly
      with `torch.utils.data.DataLoader`. See `examples/load-dataset-snippet.py`
      for a runnable version against `lerobot/pusht`.
      
      Inspect units per feature channel before converting rows into simulator or
      robot commands. A single vector may combine degree-valued arm joints with a
      percentage-valued gripper. Prefer the environment's published row-to-action
      conversion helper; a vector-wide radians conversion can silently corrupt the
      gripper while leaving the shapes valid.
      
      The README's own quick example uses a slightly different import path,
      `from lerobot.datasets.lerobot_dataset import LeRobotDataset`; both resolve
      to the same class; the shorter `from lerobot.datasets import LeRobotDataset`
      (used in the v3 doc and above) is the one this skill uses consistently.
      
      ## Streaming without downloading
      
      ```python
      from lerobot.datasets import StreamingLeRobotDataset
      
      dataset = StreamingLeRobotDataset("lerobot/pusht")  # iterates from the Hub directly
      ```
      
      Useful for datasets too large to comfortably cache locally, or a quick look
      before committing to a full download.
      
      ## Image transforms (training-time augmentation)
      
      Transforms (`ColorJitter`-based brightness/contrast/saturation/hue,
      `SharpnessJitter`, or arbitrary `torchvision.transforms.v2`) are applied at
      **training time only**: recording/creation always stores raw images, so
      augmentation choices can change later without re-recording. Pass an
      `ImageTransforms` instance (built from an `ImageTransformsConfig`, disabled
      by default) as `LeRobotDataset(..., image_transforms=...)`. Preview the
      effect of a config before a real run with:
      
      ```bash
      lerobot-imgtransform-viz --repo-id=<id> --output-dir=./transform_examples --n-examples=5
      ```
      
      This is a `references/policies-and-training.md`-adjacent concern (it only
      matters once training starts) but lives in the dataset loading path, so it's
      documented here.
      
      ## Recording episodes
      
      `lerobot-record` drives a real robot through a teleoperation device, saves
      frames into a LeRobotDataset, and pushes it to the Hub on completion:
      
      ```bash
      lerobot-record \
        --robot.type=so101_follower \
        --robot.port=/dev/tty.usbmodem585A0076841 \
        --robot.id=my_awesome_follower_arm \
        --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}}" \
        --teleop.type=so101_leader \
        --teleop.port=/dev/tty.usbmodem58760431551 \
        --teleop.id=my_awesome_leader_arm \
        --display_data=true \
        --dataset.repo_id=${HF_USER}/record-test \
        --dataset.num_episodes=5 \
        --dataset.single_task="Grab the black cube"
      ```
      
      Status: unverified; fetched directly from `docs/source/lerobot-dataset-v3.mdx`
      and `docs/source/il_robots.mdx` on 2026-07-10; robot/teleop type strings and
      camera indices above are placeholders; real values are hardware-specific.
      
      Episode-boundary keyboard controls during recording (Right Arrow/`n` = next,
      Left Arrow/`r` = re-record, Esc/`q` = stop) work over X11, Wayland, and
      headless/SSH sessions as long as `lerobot-record` runs in an interactive
      terminal. **Keyboard teleoperation itself** (as opposed to the recording
      control flow) needs a global key backend and only works on X11, a Windows
        desktop, or macOS with Accessibility permission granted, not Wayland or
        headless. Authenticate through the secure workflow in the `huggingface` skill
        before a recording that pushes to the Hub; do not put an access token in the
        LeRobot command or this dataset workflow.
      
      **Always call `finalize()` before `push_to_hub()`** when hand-rolling a
      recording loop (as opposed to using `lerobot-record`, which does this for
      you); it flushes buffered episode metadata and closes Parquet writers.
      Skipping it leaves corrupt Parquet files that won't load:
      
      ```python
      dataset = LeRobotDataset.create(...)
      for episode in range(num_episodes):
          for frame in episode_data:
              dataset.add_frame(frame)
          dataset.save_episode()
      dataset.finalize()      # required before push_to_hub()
      dataset.push_to_hub()
      ```
      
      ## Replaying an episode
      
      `lerobot-replay --robot.type=<id> --dataset.repo_id=<id> --dataset.episode=0`
      drives a real robot through a previously recorded episode's actions, useful
      for testing repeatability or cross-robot transfer within the same model.
      
      ## Editing an existing dataset
      
      `lerobot-edit-dataset` covers delete-episodes, split, merge, add/remove
      feature, and image-to-video conversion operations without hand-writing
      Parquet/MP4 manipulation:
      
      ```bash
      lerobot-edit-dataset \
        --repo_id lerobot/pusht \
        --operation.type delete_episodes \
        --operation.episode_indices "[0, 2, 5]"
      ```
      
      `--new_repo_id` preserves the original dataset and writes the result to a
      new repo id instead of modifying in place. Run `lerobot-edit-dataset --help`
      for the full operation list.
      
      ## Visualizing episodes
      
      `lerobot-dataset-viz --repo-id=lerobot/pusht --episode-index=0` renders a
      full episode (all modalities) through Rerun; the Rerun mechanics themselves
      (what you're looking at, session/recording concepts) belong to the `rerun`
      skill; this skill only owns invoking the
      command. Two more modes (verified 2026-07-12, manip-trial):
      `--display-mode foxglove` serves the episode to the Foxglove app (bind
      port via `--web-port`, default 8765) instead of Rerun, and
      `--save 1 --output-dir <dir>` writes a `.rrd` file headlessly with no
      viewer at all, the CI-friendly artifact path. A hosted viewer also exists at
      [huggingface.co/spaces/lerobot/visualize_dataset](https://huggingface.co/spaces/lerobot/visualize_dataset)
      for browsing a dataset without running anything locally. For headless or remote
      viewer selection, use the `rerun` skill; verify the current LeRobot distant-mode
      flags before invoking them.
      
      ## Migrating v2.1 → v3.0
      
      A converter script aggregates per-episode Parquet/MP4 files into the larger
      v3.0 shards and rewrites `meta/episodes/*`:
      
      ```bash
      python -m lerobot.scripts.convert_dataset_v21_to_v30 --repo-id=<HF_USER/DATASET_ID>
      ```
      
      Most current-generation hub datasets (including `lerobot/pusht`, confirmed
      via its `meta/info.json` on 2026-07-10) are already on v3.0; this is
      relevant mainly for older datasets recorded before the format change.
      
    • eval-and-sim.md 7.3 KB
      # Evaluation and simulation environments
      
      The `lerobot-eval` CLI, which sim environments LeRobot ships (as opposed to
      loads from EnvHub), headless rendering, and `lerobot-rollout` for
      real-hardware deployment.
      
      Sources: `huggingface/lerobot`'s README, `docs/source/libero.mdx`,
      `docs/source/envhub.mdx`, `docs/source/il_robots.mdx`, and
      `src/lerobot/scripts/lerobot_eval.py`'s own module docstring, all fetched
      directly via raw GitHub URLs and the GitHub Contents API on 2026-07-10
      (`main` branch). The registered `EnvConfig` subclasses (`aloha`, `pusht`,
      `libero`, `libero_plus`, `metaworld`, `robocasa`, `robotwin`, `robomme`,
      `vlabench`, `isaaclab_arena`) were confirmed by fetching
      `src/lerobot/envs/configs.py` directly, not inferred from the docs alone;
      some of these (`aloha`, `pusht`) are defined inline in that file rather than
      having their own module, so they don't show up in a directory listing of
      `src/lerobot/envs/`.
      
      ## The `lerobot-eval` CLI
      
      Adapted from `lerobot_eval.py`'s own docstring example with two
      corrections learned in a real build (2026-07-12, manip-trial): the
      docstring's `--policy.path=lerobot/diffusion_pusht` no longer loads
      (pre-0.6 checkpoint, missing processor files), and the async-env default
      crashed in the tested release, so point at your own checkpoint and force sync
      envs. See `FAILURES.md` and re-check both behaviors against the installed
      version:
      
      ```bash
      lerobot-eval \
        --policy.path=outputs/train/act_pusht_smoke/checkpoints/last/pretrained_model \
        --env.type=pusht \
        --eval.batch_size=10 \
        --eval.n_episodes=10 \
        --eval.use_async_envs=false \
        --policy.use_amp=false \
        --policy.device=cuda
      ```
      
      `--policy.path` accepts a hub id or a local checkpoint directory; on 0.6+
      that directory must contain `config.json`, `model.safetensors`, **and**
      the processor-pipeline files (`policy_preprocessor.json` /
      `policy_postprocessor.json` + their `.safetensors` stats); training
      writes all of them (verified 2026-07-12: `checkpoints/<step>/
      pretrained_model/` plus a `checkpoints/last` pointer). `--env.type`
      selects the environment (see table below); `--eval.batch_size` controls how
      many environments run in parallel, `--eval.n_episodes` how many episodes
      per task.
      
      Results are written to `<output_dir>/eval_info.json` with top-level keys
      `per_task`, `per_group`, and `overall`; the aggregate metrics
      (`pc_success`, `avg_sum_reward`, `avg_max_reward`, `n_episodes`, `eval_s`,
      `video_paths`) live under `overall` (verified 2026-07-12 on 0.6.0;
      older versions used an `aggregated` key). Rollout MP4s land in
      `<output_dir>/videos/` with no display required.
      
      ## Sim environments shipped
      
      | `--env.type` | What it is | Notes |
      | --- | --- | --- |
      | `pusht` | 2D pushing task (`gym-pusht`) | `task="PushT-v0"`, 10 fps, single task; pairs with the `lerobot/pusht` dataset used throughout this skill. Install via `lerobot[pusht]`. |
      | `aloha` | Bimanual manipulation sim (`gym-aloha`) | `task="AlohaInsertion-v0"` by default, 50 fps, 14-dim action (two 7-DoF arms). Install via `lerobot[aloha]`. |
      | `libero` | LIBERO lifelong-learning benchmark | 5 task suites (`libero_spatial`, `libero_object`, `libero_goal`, `libero_90`, `libero_10`), 130 tasks total. Linux-only (MuJoCo); see Headless rendering below. Install via `lerobot[libero]`. |
      | `libero_plus` | Extended LIBERO variant | Subclasses `libero`'s config. |
      | `metaworld` | MetaWorld manipulation benchmark | |
      | `robocasa` | Kitchen-scale manipulation sim | |
      | `robotwin` | Bimanual manipulation benchmark | |
      | `robomme` | Multi-modal-evaluation env | |
      | `vlabench` | VLA-focused benchmark suite | |
      | `isaaclab_arena` | Isaac Lab Arena, loaded via EnvHub (`HubEnvConfig`) | This is the seam to the NVIDIA RL stack: deep Isaac Lab usage is `isaac-lab`'s territory; this skill only notes that LeRobot can evaluate against it. |
      
      `aloha` and `pusht` are LeRobot's original, longest-supported sim envs:
      the default choice for validating a new pipeline. Their many hub-hosted
      pretrained baselines mostly predate the 0.6
      processor-pipeline format and no longer load, so validate with a smoke
      train of your own rather than a hub checkpoint.
      `libero` is the standard published benchmark for comparing VLA policies.
      
      ### LIBERO example (multi-suite)
      
      ```bash
      lerobot-eval \
        --policy.path="your-policy-id" \
        --env.type=libero \
        --env.task=libero_spatial,libero_object,libero_goal,libero_10 \
        --eval.batch_size=1 \
        --eval.n_episodes=10 \
        --env.max_parallel_tasks=1
      ```
      
      `--env.task` accepts a comma-separated suite list; `--env.task_ids`
      restricts to specific task indices within a suite (`[0]`, `[1,2,3]`) and
      defaults to all tasks. `--env.control_mode` (`relative` default, or
      `absolute`) must match how the target policy was trained; different VLA
      checkpoints use different action parameterizations.
      
      ## EnvHub: loading a sim env from the Hub without installing it
      
      Beyond the built-in `--env.type` list above, `lerobot.envs.make_env` can
      load an arbitrary environment published on the Hub as a Git repo containing
      an `env.py` with a `make_env(n_envs, use_async_envs)` entry point, no
      package install required:
      
      ```python
      from lerobot.envs import make_env
      
      env = make_env("lerobot/cartpole-env", trust_remote_code=True)
      ```
      
      `trust_remote_code=True` is mandatory and executes third-party Python code;
      review the `env.py` first and pin to a specific commit
      (`"user/repo@<commit-sha>"`) for anything beyond local experimentation. This
      is how `isaaclab_arena` and community-contributed sim envs are consumed;
      building/publishing a new EnvHub env is out of this skill's depth (see the
      upstream `docs/source/envhub.mdx` if that's the actual task).
      
      ## Headless rendering on remote/server hosts
      
      `lerobot-eval` itself needs no display: sim envs render via
      `render_mode="rgb_array"` and videos are written to disk. But the
      **rendering backend** the sim uses to produce those frames still needs a
      headless-capable path on a server with no attached display:
      
      ```bash
      export MUJOCO_GL=egl   # required for LIBERO (MuJoCo-based) on headless servers
      ```
      
      This is orthogonal to `environments`' general headless/remote-display
      guidance (X11 forwarding vs. web-based viz): `MUJOCO_GL` controls how
      MuJoCo itself renders, not how a human views the result. See
      `references/datasets.md` for the equivalent concern on the visualization
      side (`lerobot-dataset-viz --mode distant`).
      
      ## `lerobot-rollout`: real-hardware deployment
      
      Evaluation *in simulation* uses `lerobot-eval` (above); running a trained
      policy *on a real robot* uses `lerobot-rollout` instead, a different
      script because it drives physical hardware rather than a gym vector env:
      
      ```bash
      lerobot-rollout \
        --strategy.type=base \
        --policy.path=${HF_USER}/my_policy \
        --robot.type=so100_follower \
        --robot.port=/dev/ttyACM1 \
        --task="Put lego brick into the transparent box" \
        --duration=60
      ```
      
      `--strategy.type` selects the execution mode: `base` (no recording, quick
      check), `sentry` (continuous recording with auto-upload, for large-scale
      evaluation), `highlight` (ring-buffer recording, save-on-keystroke),
      `dagger` (human-in-the-loop data collection), `episodic` (episode-oriented
      with reset phases). All strategies support `--inference.type=rtc` for
      smoother execution with slower VLA policies (Pi0, Pi0.5, SmolVLA). Real-
      hardware bring-up (ports, calibration, camera setup) is deliberately not
      covered in depth by this skill; see LeRobot's current hardware documentation
      for the specific robot.
      
    • policies-and-training.md 5 KB
      # Policies and training
      
      Use this guide after the dataset and embodiment contract are known. LeRobot's
      policy catalog, configuration fields, accelerator support, and hosted-job path
      move quickly; inspect the installed CLI and current official guide before
      copying a recipe.
      
      ## Choose a policy for the next unknown
      
      - Start with ACT when the goal is to prove a first imitation-learning pipeline.
        LeRobot's current documentation recommends it as the first policy because it
        is comparatively light and fast. This is a starting probe, not a claim that
        ACT is best for every task.
      - Choose a larger VLA only when language conditioning, pretrained visual
        knowledge, or cross-task generalization is part of the requirement. First
        confirm that its camera, state, action, precision, dependency, and accelerator
        contracts fit the dataset and target runtime.
      - Treat every policy not shown by the installed release and current
        [policy catalog](https://huggingface.co/docs/lerobot/main/en/api/policies) as
        unavailable until verified. Do not maintain negative lists of policies that
        LeRobot does not ship; those become obsolete as integrations land.
      - Use the policy's own current guide for its training recipe. Architecture
        names alone do not establish compatible processors, checkpoints, or compute.
      
      ## Prove training before sizing it
      
      The common CLI shape is:
      
      ```bash
      lerobot-train \
        --dataset.repo_id=${HF_USER}/so101_test \
        --policy.type=act \
        --output_dir=outputs/train/act_so101_test \
        --job_name=act_so101_test \
        --policy.device=cuda \
        --policy.repo_id=${HF_USER}/my_policy \
        --steps=<small-explicit-count> \
        --save_freq=<at-least-one-checkpoint>
      ```
      
      - Confirm this shape with `lerobot-train --help` and the current
        [cheat sheet](https://huggingface.co/docs/lerobot/main/cheat-sheet).
      - For fine-tuning, use `--policy.path=<hub-id-or-local-dir>` only after
        inspecting that checkpoint's config and processor files. Do not also assume a
        policy type from its repository name.
      - A smoke run proves data loading, forward/backward passes, logging, checkpoint
        creation, and reload. It does not establish convergence or policy quality.
      - Shorten any save and scheduler horizons that would otherwise fall beyond the
        smoke run. Read the resolved policy configuration first; scheduler fields are
        not uniform across families.
      - Measure peak memory and steps per second on a small batch, then choose batch
        size, steps, and hardware. Do not use a timeless policy-to-VRAM table as a
        procurement promise.
      
      Robium evidence, scoped to its measured conditions: on 2026-07-12, ACT on
      `lerobot/pusht` with 96×96 images and batch 8 sustained about 11.6 steps/s on an
      M2 Pro through MPS; a 200-step pipeline smoke train took about 31 seconds. That
      does not predict a 640×480 real-robot dataset or another policy family.
      
      ## Camera and processor contracts
      
      - Compare dataset feature keys with the selected policy's current input
        features before training. Camera count and names can differ even between a
        base checkpoint and its fine-tune.
      - Where the installed release supports them, use `rename_map` to align real
        feature keys and `empty_cameras` only for policy-supported masked slots. Check
        the current policy guide rather than assuming every VLA supports the same
        fields.
      - Inspect the saved preprocessor and postprocessor after training. At
        evaluation, do not apply a rename a second time when the saved processor
        already owns it.
      
      Robium's 2026-07-14 SmolVLA trial found that a base checkpoint and fine-tune
      could carry different camera layouts in their processor configuration. Preserve
      that as a failure signature, not a universal list of camera keys.
      
      ## Remote training
      
      Local training is the normal path. If the installed release exposes
      `--job.target`, treat it as paid external compute:
      
      - Use `huggingface` to verify identity, current hardware, pricing, approval,
        logs, output repositories, and cancellation.
      - Keep `--output_dir` inside the remote container and discover the actual model
        destination from the completed job rather than assuming a requested Hub repo
        was honored.
      - Do not preserve a flavor list, price, or timeout here. Read live CLI help and
        current Hugging Face Jobs documentation immediately before submission.
      
      Robium's 2026-07-14 VLA trial observed three version-specific signatures: an
      account without prepaid credit failed at submission with HTTP 402; the managed
      path ignored the requested policy repository and logged an auto-generated one;
      and a macOS host path passed as `output_dir` failed only when the remote run
      saved. Compare these signatures with current behavior before relying on them.
      
      ## Current sources
      
      Re-checked on 2026-09-07; use the live pages rather than treating this date as a
      version guarantee.
      
      - [LeRobot overview](https://huggingface.co/docs/lerobot/main/en/index)
      - [ACT guide](https://huggingface.co/docs/lerobot/act)
      - [Policy API and catalog](https://huggingface.co/docs/lerobot/main/en/api/policies)
      - [Training cheat sheet](https://huggingface.co/docs/lerobot/main/cheat-sheet)
      
  • evals.yaml 555 B
    triggers:
      positive:
        - phrase: inspect the processor files in this SmolVLA checkpoint before evaluating it
        - phrase: train an ACT policy and publish the LeRobot checkpoint to my Hub repo
      negative:
        - phrase: download this generic language model from the Hugging Face Hub
          expect: huggingface
    tasks:
      - name: pusht-dataset-loads
        example: examples/load-dataset-snippet.py
        command: "uv run --with 'lerobot[dataset]' python skills/lerobot/examples/load-dataset-snippet.py"
        pass_criteria: "observation\\.image"
        timeout: 900
    
  • FAILURES.md 3.3 KB
    # When a LeRobot pipeline fails
    
    Start at the first contract that has contrary evidence.
    
    ## A checkpoint will not load
    
    - Inspect the repository or directory for model configuration, weights,
      `policy_preprocessor.json`, `policy_postprocessor.json`, and their statistics.
    - Pre-processor-pipeline checkpoints can have valid weights but remain
      unloadable on current LeRobot. In Robium's 0.6.0 trial,
      `lerobot/diffusion_pusht` failed because the processor files were absent.
    - Do not substitute a similarly named base or fine-tuned checkpoint without
      comparing processor configuration and expected features.
    
    ## Training rejects dataset features
    
    - Compare dataset feature names and shapes with the policy's expected state,
      action, and camera inputs.
    - VLA policies may require fixed camera keys. Consult the current rename-map
      and empty-camera support before adapting them.
    - A rename saved into the fine-tuned checkpoint's processor should not be
      applied a second time during evaluation.
    
    ## Evaluation ends in BrokenPipe or missing environments
    
    - Find the first worker exception rather than the parent's final pipe error.
    - Robium observed shipped environments failing under the asynchronous vector
      environment default on LeRobot 0.6.0; synchronous evaluation worked. Verify
      whether that still applies to the installed release before carrying the
      workaround forward.
    - Match environment observation/action shapes and control mode to the trained
      policy before changing evaluation parallelism.
    
    ## Training runs but is impractically slow
    
    - Separate a pipeline smoke test from a viable training target. CPU and Apple
      MPS can prove that some loops start; they do not make large VLA fine-tuning
      practical.
    - In the Robium VLA trial, SmolVLA on MPS advanced only about 20 of 20,000
      steps in roughly two hours. Preserve that as a measured warning, not a
      universal benchmark.
    - Check data-loading time, image size, batch size, accelerator use, and policy
      memory before adding GPUs. Rescale a policy's learning-rate schedule when a
      smoke run drastically shortens its configured steps.
    
    ## A remote Job does not behave like the local command
    
    - A 402 response means paid-compute credit is unavailable, not that the job is
      waiting.
    - Robium observed the Jobs path ignoring `--policy.repo_id` and publishing to
      an auto-generated repository; read the logs for the actual destination.
    - Local absolute output paths are passed into the remote container unchanged.
      In the same trial, a `/Users/...` path trained successfully and failed only
      when saving. Use a container-local path after verifying current behavior.
    
    ## Visualization or recording fails
    
    - Resolve `lerobot` and `rerun-sdk` constraints together. Robium observed the
      LeRobot 0.6.0 `viz` extra conflicting with `gradio_rerun` 0.34.1; dropping
      that extra and pinning the required Rerun SDK resolved that application.
    - Dataset visualization can save an `.rrd` or serve a remote viewer without a
      local display; do not force `spawn()` on a headless host.
    - Recording controls can work in an interactive terminal while keyboard
      teleoperation still fails. Global keyboard capture needs a supported desktop
      session or macOS Accessibility permission.
    - Video decode failures belong first to the installed FFmpeg/TorchCodec path,
      not the dataset schema.
    
  • SKILL.md 3.1 KB
    ---
    name: lerobot
    description: Build and debug LeRobot datasets, training, and policy evaluation. For a first pretrained robot-arm demo, start with architect's reference-app selection.
    ---
    
    # LeRobot
    
    Follow one contract chain through LeRobot: embodiment, dataset, processor,
    checkpoint, runtime observations and actions, then evaluation. Find the first
    contract that does not match.
    
    ## Establish the contract
    
    - For a new manipulation app or first pretrained-policy demo, read
      [architect](../architect/SKILL.md) before creating an environment or training
      pipeline. It finds the saved apps checkout and selects a compatible baseline.
      If selection already happened, continue here. Existing dataset, training,
      or evaluation work does not need onboarding; an inference-only request does
      not authorize training.
    - Inspect the installed LeRobot version and current CLI help before writing
      flags. Dataset formats, policy families, scripts, and extras change quickly.
    - Match the robot's state, action space, cameras, rates, and task to the
      dataset. A policy adapts to those features; it cannot repair a mismatched
      embodiment.
    - Inspect every checkpoint's configuration and processor files, not only its
      weights. Base and fine-tuned checkpoints from one family can expect different
      camera layouts.
    - Keep Hub identity, transfer, publication, and Jobs lifecycle at the Hugging
      Face boundary. Keep source-selection strategy in data.
    
    ## Prove the loop cheaply
    
    - Start with a small shipped policy and a known dataset/environment pair.
    - Run a short train that writes a checkpoint, then load that exact checkpoint
      through evaluation. Completion and numeric metrics are the smoke-test result;
      policy quality is not.
    - Confirm loss, saved processors, input/output shapes, rollout metrics, and
      video or real-robot behavior before increasing steps or hardware cost.
    - Treat real-hardware rollout as a new safety boundary even when simulation
      evaluation passed.
    
    ## Go deeper only when needed
    
    - For loading, recording, editing, migration, and episode visualization, read
      [references/datasets.md](references/datasets.md).
    - For policy choice, camera remapping, training, compute sizing, and remote
      Jobs behavior, read
      [references/policies-and-training.md](references/policies-and-training.md).
    - For simulation evaluation, headless rendering, EnvHub, or real-hardware
      rollout, read [references/eval-and-sim.md](references/eval-and-sim.md).
    - When a checkpoint, feature contract, evaluation worker, dependency, or remote
      run fails, start with [FAILURES.md](FAILURES.md).
    - The concrete PushT examples are useful only when that smoke path matches the
      application: [load dataset](examples/load-dataset-snippet.py) and
      [train ACT](examples/train-act-command.md).
    - Use the current [LeRobot documentation](https://huggingface.co/docs/lerobot)
      and [source](https://github.com/huggingface/lerobot) for version-sensitive
      APIs and the shipped policy/environment list.
    
    ## Done
    
    - The target dataset loads, the checkpoint carries its processors, evaluation
      exercises matching observations and actions, and measured results justify
      any longer run or hardware deployment.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related