Claude
Cursor
Skill
lerobot
Build and debug LeRobot datasets, training, and policy evaluation. For a first pretrained robot-arm demo, start with architect's reference-app selection.
Virus-scanned
Reviewed automatically before listing.
Download
robium-ai-robium-skills_lerobot-498ea4e.zip · 17 KB
Install
skills CLI
npx skills add https://github.com/robium-ai/robium/tree/main/skills/lerobot
Claude Code
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install robium-ai-robium@llmmart
Git
git clone https://github.com/robium-ai/robium.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole robium-ai/robium collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
LeRobot
Follow one contract chain through LeRobot: embodiment, dataset, processor, checkpoint, runtime observations and actions, then evaluation. Find the first contract that does not match.
Establish the contract
- For a new manipulation app or first pretrained-policy demo, read architect before creating an environment or training pipeline. It finds the saved apps checkout and selects a compatible baseline. If selection already happened, continue here. Existing dataset, training, or evaluation work does not need onboarding; an inference-only request does not authorize training.
- Inspect the installed LeRobot version and current CLI help before writing flags. Dataset formats, policy families, scripts, and extras change quickly.
- Match the robot's state, action space, cameras, rates, and task to the dataset. A policy adapts to those features; it cannot repair a mismatched embodiment.
- Inspect every checkpoint's configuration and processor files, not only its weights. Base and fine-tuned checkpoints from one family can expect different camera layouts.
- Keep Hub identity, transfer, publication, and Jobs lifecycle at the Hugging Face boundary. Keep source-selection strategy in data.
Prove the loop cheaply
- Start with a small shipped policy and a known dataset/environment pair.
- Run a short train that writes a checkpoint, then load that exact checkpoint through evaluation. Completion and numeric metrics are the smoke-test result; policy quality is not.
- Confirm loss, saved processors, input/output shapes, rollout metrics, and video or real-robot behavior before increasing steps or hardware cost.
- Treat real-hardware rollout as a new safety boundary even when simulation evaluation passed.
Go deeper only when needed
- For loading, recording, editing, migration, and episode visualization, read references/datasets.md.
- For policy choice, camera remapping, training, compute sizing, and remote Jobs behavior, read references/policies-and-training.md.
- For simulation evaluation, headless rendering, EnvHub, or real-hardware rollout, read references/eval-and-sim.md.
- When a checkpoint, feature contract, evaluation worker, dependency, or remote run fails, start with FAILURES.md.
- The concrete PushT examples are useful only when that smoke path matches the application: load dataset and train ACT.
- Use the current LeRobot documentation and source for version-sensitive APIs and the shipped policy/environment list.
Done
- The target dataset loads, the checkpoint carries its processors, evaluation exercises matching observations and actions, and measured results justify any longer run or hardware deployment.
Files (robium)
-
examples
-
load-dataset-snippet.py 2.4 KB
# status: verified 2026-08-02 (deep-verify: pusht-dataset-loads) # source: assembled from huggingface/lerobot's docs/source/lerobot-dataset-v3.mdx # ("Load a dataset for training" section) and its README's own LeRobotDataset # example, both fetched directly via raw GitHub URLs on 2026-07-10 (main # branch). The `lerobot/pusht` repo id and its shape (206 episodes, 25,650 # frames, "observation.image" + "observation.state" + "action" features) # were confirmed by fetching that dataset's meta/info.json directly from # huggingface.co on 2026-07-10, not assumed from memory. # # Pairs with examples/train-act-command.md, which trains an ACT policy # against this same dataset (lerobot/pusht); the feature names printed # below (observation.image, observation.state, action) are exactly what # that training command's policy adapts to automatically. # # Requires: uv add "lerobot[dataset]" (verify the current extra in LeRobot's # install docs; references/datasets.md explains the dataset contract.) from lerobot.datasets import LeRobotDataset REPO_ID = "lerobot/pusht" def main() -> None: # Downloads and caches the dataset under ~/.cache/huggingface/lerobot/ # the first time it's loaded; subsequent loads reuse the local cache. dataset = LeRobotDataset(REPO_ID) print(f"repo_id: {REPO_ID}") print(f"num_episodes: {dataset.num_episodes}") print(f"num_frames: {dataset.num_frames}") print(f"fps: {dataset.fps}") print(f"features: {sorted(dataset.features.keys())}") # Random access by frame index: returns a dict of PyTorch tensors. sample = dataset[0] print(f"observation.state shape: {tuple(sample['observation.state'].shape)}") print(f"observation.image shape: {tuple(sample['observation.image'].shape)}") print(f"action shape: {tuple(sample['action'].shape)}") # A temporal window instead of a single frame: request the current frame # plus the two preceding it (seconds relative to t, so this depends on # `dataset.fps`). Useful for policies that condition on recent history. delta_timestamps = {"observation.image": [-0.2, -0.1, 0.0]} windowed = LeRobotDataset(REPO_ID, delta_timestamps=delta_timestamps) windowed_sample = windowed[0] # Shape becomes [T, C, H, W] instead of [C, H, W] for the windowed key. print( "windowed observation.image shape: " f"{tuple(windowed_sample['observation.image'].shape)}" ) if __name__ == "__main__": main() -
train-act-command.md 4.2 KB
# Small-scale ACT training smoke run on `lerobot/pusht` **Status:** unverified as written, but exercised via adaptation 2026-07-12 (manip-trial, lerobot 0.6.0, M2 Pro/MPS): the same command shape with `--steps=200 --save_freq=200 --batch_size=8 --policy.device=mps` trained in 31 s (loss 14.4→3.5) and at `--steps=10000` in 14.5 min (loss→0.33); the `checkpoints/last/pretrained_model` eval path below resolved exactly as written. **Source:** the command shape below is assembled from two directly-fetched upstream sources on 2026-07-10: huggingface/lerobot's README ("Training a policy is as simple as running a script configuration", `lerobot-train --policy.type=act --dataset.repo_id=...`) and its `docs/source/cheat-sheet.mdx`'s training section (`--output_dir`, `--job_name`, `--policy.device`, `--wandb.enable`, `--policy.repo_id`, `--steps`). The dataset (`lerobot/pusht`) was confirmed to exist and to already be on LeRobotDataset v3.0 via a direct fetch of its `meta/info.json` from huggingface.co on 2026-07-10 (206 episodes, 25,650 frames). `--steps=3000` below is a deliberately small smoke-run value, not an upstream default; see the reasoning below and `references/policies-and-training.md`'s compute-sizing table before scaling it up for a real run. Pairs with `load-dataset-snippet.py` in this same directory (same dataset, `lerobot/pusht`). Use it only when PushT is a useful smoke path for the application; its purpose is to prove train → checkpoint → evaluation with a self-trained checkpoint before spending substantial compute. Requires the `training` and `pusht` extras (`uv add "lerobot[training,pusht]"`; verify the current extras in LeRobot's install documentation; `core_scripts` was only needed for hardware CLIs in the tested release). ```bash uv run lerobot-train \ --dataset.repo_id=lerobot/pusht \ --policy.type=act \ --output_dir=outputs/train/act_pusht_smoke \ --job_name=act_pusht_smoke \ --policy.device=cuda \ --steps=3000 \ --save_freq=1000 \ --wandb.enable=false \ --policy.repo_id=${HF_USER}/act_pusht_smoke \ --policy.push_to_hub=false ``` **Why these values:** - `--policy.type=act`: the smallest policy family (`references/policies-and-training.md`'s compute table: ~2-6 GB VRAM, ~30-60 min for 5 epochs on an RTX 4090-class GPU), the right choice for a first smoke run, not a VRAM-bound VLA policy. - `--steps=3000`: deliberately short. `lerobot/pusht` has 25,650 frames; 3,000 steps at the ACT default batch size is well under one full epoch, enough to confirm the loss curve moves and a checkpoint saves/evaluates cleanly, not a converged policy. ACT trains at a constant LR (its `get_scheduler_preset()` returns no scheduler, confirmed by fetching `configuration_act.py` directly on 2026-07-10), so there's no LR-decay schedule to rescale when changing `--steps`, unlike scheduler-based policies (diffusion, SmolVLA, the Pi0 family); see `references/policies-and-training.md` if switching `--policy.type` to one of those. - `--policy.device=cuda`: swap for `mps` (Apple Silicon) or `cpu` only after checking the policy's current accelerator support; see `FAILURES.md` for measured limits of those smoke paths. - `--wandb.enable=false` and `--policy.push_to_hub=false`: kept off for a throwaway smoke run; flip both on for a real tracked run (`wandb login` first; dropping `--policy.push_to_hub=false` pushes the trained policy to `--policy.repo_id` on the Hub, which is the `huggingface` skill's territory once you get there). **After this run**, evaluate the checkpoint using the current CLI shape in `references/eval-and-sim.md`, pointing `--policy.path` at `outputs/train/act_pusht_smoke/checkpoints/last/pretrained_model` (path verified 2026-07-12): ```bash uv run lerobot-eval \ --policy.path=outputs/train/act_pusht_smoke/checkpoints/last/pretrained_model \ --env.type=pusht \ --eval.batch_size=10 \ --eval.n_episodes=10 \ --eval.use_async_envs=false \ --policy.device=cuda ``` A smoke run at this scale is not expected to solve PushT (success needs ≥95% T-coverage; even 10k steps measured `pc_success` 0 with `avg_max_reward` 0.28); the goal is confirming the train -> checkpoint -> eval loop works end to end before committing to a long run.
-
-
references
-
datasets.md 9.4 KB
# LeRobotDataset: format, loading, recording The LeRobotDataset format (currently **v3.0**), how to load or stream one for training, how recording and dataset-editing CLIs use it, and how episode visualization hands off to Rerun. Hub mechanics beyond "this is the LeRobotDataset shape a hub repo has" (auth, upload/download, model/dataset cards) are the `huggingface` skill's territory, not this file's. Sources: `huggingface/lerobot`'s `docs/source/lerobot-dataset-v3.mdx`, `docs/source/il_robots.mdx`, `docs/source/using_dataset_tools.mdx`, and the README, all fetched directly via raw GitHub URLs on 2026-07-10 (`main` branch). The `lerobot/pusht` dataset's `meta/info.json` was fetched directly from `huggingface.co/datasets/lerobot/pusht` to confirm it is already on format v3.0 (206 episodes, 25,650 frames); this is the dataset this skill's examples use throughout. ## Format v3.0: what changed and why v3.0 (included in `lerobot >= 0.4.0`; the installed `lerobot` on 2026-07-10 is 0.6.1) replaced v2.1's one-file-per-episode layout with **file-based storage**: many episodes are concatenated into fewer, larger Parquet and MP4 files, with episode boundaries resolved through metadata rather than filenames. This is what makes `StreamingLeRobotDataset` (below) practical at scale: fewer, larger files mean less filesystem overhead when streaming directly from the Hub instead of downloading first. **Three storage pillars:** 1. **Tabular data** (states, actions, timestamps): Apache Parquet, memory- mapped or streamed via the `datasets` stack. 2. **Visual data** (camera frames): MP4, frames from the same episode grouped, videos sharded per camera. 3. **Metadata**: JSON/Parquet describing schema, FPS, normalization stats, and episode segmentation (start/end offsets into the shared files). **Directory layout (simplified):** - `meta/info.json`: schema (feature names/dtypes/shapes), FPS, codebase version, path templates for locating data/video shards. - `meta/stats.json`: global feature statistics (mean/std/min/max) for normalization, exposed as `dataset.meta.stats`. - `meta/tasks.jsonl`: natural-language task descriptions mapped to integer IDs, for task-conditioned policies. - `meta/episodes/`: per-episode records (lengths, tasks, offsets) as chunked Parquet. - `data/`: frame-by-frame Parquet shards, each typically holding many episodes. - `videos/`: MP4 shards per camera, each typically holding many episodes. ## Loading a dataset for training Install the dataset dependencies required by the pinned LeRobot release before testing video-backed episodes. Robium observed that bare `lerobot==0.6.0` imported successfully but could not decode dataset video; the matching `lerobot[dataset]` extra supplied TorchCodec and restored the load path. Re-check the current extra and decoder requirements rather than treating import success as a dataset smoke. ```python from lerobot.datasets import LeRobotDataset repo_id = "lerobot/pusht" dataset = LeRobotDataset(repo_id) # downloads + caches locally sample = dataset[100] # {'observation.state': tensor(...), 'action': tensor(...), # 'observation.image': tensor([C, H, W]), 'timestamp': tensor(...), ...} ``` `delta_timestamps` requests a temporal window (seconds relative to the current frame) instead of a single frame per key, e.g. `{"observation.image": [-0.2, -0.1, 0.0]}` returns a `[T, C, H, W]` stack. `LeRobotDataset` returns plain dicts of PyTorch tensors and works directly with `torch.utils.data.DataLoader`. See `examples/load-dataset-snippet.py` for a runnable version against `lerobot/pusht`. Inspect units per feature channel before converting rows into simulator or robot commands. A single vector may combine degree-valued arm joints with a percentage-valued gripper. Prefer the environment's published row-to-action conversion helper; a vector-wide radians conversion can silently corrupt the gripper while leaving the shapes valid. The README's own quick example uses a slightly different import path, `from lerobot.datasets.lerobot_dataset import LeRobotDataset`; both resolve to the same class; the shorter `from lerobot.datasets import LeRobotDataset` (used in the v3 doc and above) is the one this skill uses consistently. ## Streaming without downloading ```python from lerobot.datasets import StreamingLeRobotDataset dataset = StreamingLeRobotDataset("lerobot/pusht") # iterates from the Hub directly ``` Useful for datasets too large to comfortably cache locally, or a quick look before committing to a full download. ## Image transforms (training-time augmentation) Transforms (`ColorJitter`-based brightness/contrast/saturation/hue, `SharpnessJitter`, or arbitrary `torchvision.transforms.v2`) are applied at **training time only**: recording/creation always stores raw images, so augmentation choices can change later without re-recording. Pass an `ImageTransforms` instance (built from an `ImageTransformsConfig`, disabled by default) as `LeRobotDataset(..., image_transforms=...)`. Preview the effect of a config before a real run with: ```bash lerobot-imgtransform-viz --repo-id=<id> --output-dir=./transform_examples --n-examples=5 ``` This is a `references/policies-and-training.md`-adjacent concern (it only matters once training starts) but lives in the dataset loading path, so it's documented here. ## Recording episodes `lerobot-record` drives a real robot through a teleoperation device, saves frames into a LeRobotDataset, and pushes it to the Hub on completion: ```bash lerobot-record \ --robot.type=so101_follower \ --robot.port=/dev/tty.usbmodem585A0076841 \ --robot.id=my_awesome_follower_arm \ --robot.cameras="{ front: {type: opencv, index_or_path: 0, width: 1920, height: 1080, fps: 30}}" \ --teleop.type=so101_leader \ --teleop.port=/dev/tty.usbmodem58760431551 \ --teleop.id=my_awesome_leader_arm \ --display_data=true \ --dataset.repo_id=${HF_USER}/record-test \ --dataset.num_episodes=5 \ --dataset.single_task="Grab the black cube" ``` Status: unverified; fetched directly from `docs/source/lerobot-dataset-v3.mdx` and `docs/source/il_robots.mdx` on 2026-07-10; robot/teleop type strings and camera indices above are placeholders; real values are hardware-specific. Episode-boundary keyboard controls during recording (Right Arrow/`n` = next, Left Arrow/`r` = re-record, Esc/`q` = stop) work over X11, Wayland, and headless/SSH sessions as long as `lerobot-record` runs in an interactive terminal. **Keyboard teleoperation itself** (as opposed to the recording control flow) needs a global key backend and only works on X11, a Windows desktop, or macOS with Accessibility permission granted, not Wayland or headless. Authenticate through the secure workflow in the `huggingface` skill before a recording that pushes to the Hub; do not put an access token in the LeRobot command or this dataset workflow. **Always call `finalize()` before `push_to_hub()`** when hand-rolling a recording loop (as opposed to using `lerobot-record`, which does this for you); it flushes buffered episode metadata and closes Parquet writers. Skipping it leaves corrupt Parquet files that won't load: ```python dataset = LeRobotDataset.create(...) for episode in range(num_episodes): for frame in episode_data: dataset.add_frame(frame) dataset.save_episode() dataset.finalize() # required before push_to_hub() dataset.push_to_hub() ``` ## Replaying an episode `lerobot-replay --robot.type=<id> --dataset.repo_id=<id> --dataset.episode=0` drives a real robot through a previously recorded episode's actions, useful for testing repeatability or cross-robot transfer within the same model. ## Editing an existing dataset `lerobot-edit-dataset` covers delete-episodes, split, merge, add/remove feature, and image-to-video conversion operations without hand-writing Parquet/MP4 manipulation: ```bash lerobot-edit-dataset \ --repo_id lerobot/pusht \ --operation.type delete_episodes \ --operation.episode_indices "[0, 2, 5]" ``` `--new_repo_id` preserves the original dataset and writes the result to a new repo id instead of modifying in place. Run `lerobot-edit-dataset --help` for the full operation list. ## Visualizing episodes `lerobot-dataset-viz --repo-id=lerobot/pusht --episode-index=0` renders a full episode (all modalities) through Rerun; the Rerun mechanics themselves (what you're looking at, session/recording concepts) belong to the `rerun` skill; this skill only owns invoking the command. Two more modes (verified 2026-07-12, manip-trial): `--display-mode foxglove` serves the episode to the Foxglove app (bind port via `--web-port`, default 8765) instead of Rerun, and `--save 1 --output-dir <dir>` writes a `.rrd` file headlessly with no viewer at all, the CI-friendly artifact path. A hosted viewer also exists at [huggingface.co/spaces/lerobot/visualize_dataset](https://huggingface.co/spaces/lerobot/visualize_dataset) for browsing a dataset without running anything locally. For headless or remote viewer selection, use the `rerun` skill; verify the current LeRobot distant-mode flags before invoking them. ## Migrating v2.1 → v3.0 A converter script aggregates per-episode Parquet/MP4 files into the larger v3.0 shards and rewrites `meta/episodes/*`: ```bash python -m lerobot.scripts.convert_dataset_v21_to_v30 --repo-id=<HF_USER/DATASET_ID> ``` Most current-generation hub datasets (including `lerobot/pusht`, confirmed via its `meta/info.json` on 2026-07-10) are already on v3.0; this is relevant mainly for older datasets recorded before the format change. -
eval-and-sim.md 7.3 KB
# Evaluation and simulation environments The `lerobot-eval` CLI, which sim environments LeRobot ships (as opposed to loads from EnvHub), headless rendering, and `lerobot-rollout` for real-hardware deployment. Sources: `huggingface/lerobot`'s README, `docs/source/libero.mdx`, `docs/source/envhub.mdx`, `docs/source/il_robots.mdx`, and `src/lerobot/scripts/lerobot_eval.py`'s own module docstring, all fetched directly via raw GitHub URLs and the GitHub Contents API on 2026-07-10 (`main` branch). The registered `EnvConfig` subclasses (`aloha`, `pusht`, `libero`, `libero_plus`, `metaworld`, `robocasa`, `robotwin`, `robomme`, `vlabench`, `isaaclab_arena`) were confirmed by fetching `src/lerobot/envs/configs.py` directly, not inferred from the docs alone; some of these (`aloha`, `pusht`) are defined inline in that file rather than having their own module, so they don't show up in a directory listing of `src/lerobot/envs/`. ## The `lerobot-eval` CLI Adapted from `lerobot_eval.py`'s own docstring example with two corrections learned in a real build (2026-07-12, manip-trial): the docstring's `--policy.path=lerobot/diffusion_pusht` no longer loads (pre-0.6 checkpoint, missing processor files), and the async-env default crashed in the tested release, so point at your own checkpoint and force sync envs. See `FAILURES.md` and re-check both behaviors against the installed version: ```bash lerobot-eval \ --policy.path=outputs/train/act_pusht_smoke/checkpoints/last/pretrained_model \ --env.type=pusht \ --eval.batch_size=10 \ --eval.n_episodes=10 \ --eval.use_async_envs=false \ --policy.use_amp=false \ --policy.device=cuda ``` `--policy.path` accepts a hub id or a local checkpoint directory; on 0.6+ that directory must contain `config.json`, `model.safetensors`, **and** the processor-pipeline files (`policy_preprocessor.json` / `policy_postprocessor.json` + their `.safetensors` stats); training writes all of them (verified 2026-07-12: `checkpoints/<step>/ pretrained_model/` plus a `checkpoints/last` pointer). `--env.type` selects the environment (see table below); `--eval.batch_size` controls how many environments run in parallel, `--eval.n_episodes` how many episodes per task. Results are written to `<output_dir>/eval_info.json` with top-level keys `per_task`, `per_group`, and `overall`; the aggregate metrics (`pc_success`, `avg_sum_reward`, `avg_max_reward`, `n_episodes`, `eval_s`, `video_paths`) live under `overall` (verified 2026-07-12 on 0.6.0; older versions used an `aggregated` key). Rollout MP4s land in `<output_dir>/videos/` with no display required. ## Sim environments shipped | `--env.type` | What it is | Notes | | --- | --- | --- | | `pusht` | 2D pushing task (`gym-pusht`) | `task="PushT-v0"`, 10 fps, single task; pairs with the `lerobot/pusht` dataset used throughout this skill. Install via `lerobot[pusht]`. | | `aloha` | Bimanual manipulation sim (`gym-aloha`) | `task="AlohaInsertion-v0"` by default, 50 fps, 14-dim action (two 7-DoF arms). Install via `lerobot[aloha]`. | | `libero` | LIBERO lifelong-learning benchmark | 5 task suites (`libero_spatial`, `libero_object`, `libero_goal`, `libero_90`, `libero_10`), 130 tasks total. Linux-only (MuJoCo); see Headless rendering below. Install via `lerobot[libero]`. | | `libero_plus` | Extended LIBERO variant | Subclasses `libero`'s config. | | `metaworld` | MetaWorld manipulation benchmark | | | `robocasa` | Kitchen-scale manipulation sim | | | `robotwin` | Bimanual manipulation benchmark | | | `robomme` | Multi-modal-evaluation env | | | `vlabench` | VLA-focused benchmark suite | | | `isaaclab_arena` | Isaac Lab Arena, loaded via EnvHub (`HubEnvConfig`) | This is the seam to the NVIDIA RL stack: deep Isaac Lab usage is `isaac-lab`'s territory; this skill only notes that LeRobot can evaluate against it. | `aloha` and `pusht` are LeRobot's original, longest-supported sim envs: the default choice for validating a new pipeline. Their many hub-hosted pretrained baselines mostly predate the 0.6 processor-pipeline format and no longer load, so validate with a smoke train of your own rather than a hub checkpoint. `libero` is the standard published benchmark for comparing VLA policies. ### LIBERO example (multi-suite) ```bash lerobot-eval \ --policy.path="your-policy-id" \ --env.type=libero \ --env.task=libero_spatial,libero_object,libero_goal,libero_10 \ --eval.batch_size=1 \ --eval.n_episodes=10 \ --env.max_parallel_tasks=1 ``` `--env.task` accepts a comma-separated suite list; `--env.task_ids` restricts to specific task indices within a suite (`[0]`, `[1,2,3]`) and defaults to all tasks. `--env.control_mode` (`relative` default, or `absolute`) must match how the target policy was trained; different VLA checkpoints use different action parameterizations. ## EnvHub: loading a sim env from the Hub without installing it Beyond the built-in `--env.type` list above, `lerobot.envs.make_env` can load an arbitrary environment published on the Hub as a Git repo containing an `env.py` with a `make_env(n_envs, use_async_envs)` entry point, no package install required: ```python from lerobot.envs import make_env env = make_env("lerobot/cartpole-env", trust_remote_code=True) ``` `trust_remote_code=True` is mandatory and executes third-party Python code; review the `env.py` first and pin to a specific commit (`"user/repo@<commit-sha>"`) for anything beyond local experimentation. This is how `isaaclab_arena` and community-contributed sim envs are consumed; building/publishing a new EnvHub env is out of this skill's depth (see the upstream `docs/source/envhub.mdx` if that's the actual task). ## Headless rendering on remote/server hosts `lerobot-eval` itself needs no display: sim envs render via `render_mode="rgb_array"` and videos are written to disk. But the **rendering backend** the sim uses to produce those frames still needs a headless-capable path on a server with no attached display: ```bash export MUJOCO_GL=egl # required for LIBERO (MuJoCo-based) on headless servers ``` This is orthogonal to `environments`' general headless/remote-display guidance (X11 forwarding vs. web-based viz): `MUJOCO_GL` controls how MuJoCo itself renders, not how a human views the result. See `references/datasets.md` for the equivalent concern on the visualization side (`lerobot-dataset-viz --mode distant`). ## `lerobot-rollout`: real-hardware deployment Evaluation *in simulation* uses `lerobot-eval` (above); running a trained policy *on a real robot* uses `lerobot-rollout` instead, a different script because it drives physical hardware rather than a gym vector env: ```bash lerobot-rollout \ --strategy.type=base \ --policy.path=${HF_USER}/my_policy \ --robot.type=so100_follower \ --robot.port=/dev/ttyACM1 \ --task="Put lego brick into the transparent box" \ --duration=60 ``` `--strategy.type` selects the execution mode: `base` (no recording, quick check), `sentry` (continuous recording with auto-upload, for large-scale evaluation), `highlight` (ring-buffer recording, save-on-keystroke), `dagger` (human-in-the-loop data collection), `episodic` (episode-oriented with reset phases). All strategies support `--inference.type=rtc` for smoother execution with slower VLA policies (Pi0, Pi0.5, SmolVLA). Real- hardware bring-up (ports, calibration, camera setup) is deliberately not covered in depth by this skill; see LeRobot's current hardware documentation for the specific robot. -
policies-and-training.md 5 KB
# Policies and training Use this guide after the dataset and embodiment contract are known. LeRobot's policy catalog, configuration fields, accelerator support, and hosted-job path move quickly; inspect the installed CLI and current official guide before copying a recipe. ## Choose a policy for the next unknown - Start with ACT when the goal is to prove a first imitation-learning pipeline. LeRobot's current documentation recommends it as the first policy because it is comparatively light and fast. This is a starting probe, not a claim that ACT is best for every task. - Choose a larger VLA only when language conditioning, pretrained visual knowledge, or cross-task generalization is part of the requirement. First confirm that its camera, state, action, precision, dependency, and accelerator contracts fit the dataset and target runtime. - Treat every policy not shown by the installed release and current [policy catalog](https://huggingface.co/docs/lerobot/main/en/api/policies) as unavailable until verified. Do not maintain negative lists of policies that LeRobot does not ship; those become obsolete as integrations land. - Use the policy's own current guide for its training recipe. Architecture names alone do not establish compatible processors, checkpoints, or compute. ## Prove training before sizing it The common CLI shape is: ```bash lerobot-train \ --dataset.repo_id=${HF_USER}/so101_test \ --policy.type=act \ --output_dir=outputs/train/act_so101_test \ --job_name=act_so101_test \ --policy.device=cuda \ --policy.repo_id=${HF_USER}/my_policy \ --steps=<small-explicit-count> \ --save_freq=<at-least-one-checkpoint> ``` - Confirm this shape with `lerobot-train --help` and the current [cheat sheet](https://huggingface.co/docs/lerobot/main/cheat-sheet). - For fine-tuning, use `--policy.path=<hub-id-or-local-dir>` only after inspecting that checkpoint's config and processor files. Do not also assume a policy type from its repository name. - A smoke run proves data loading, forward/backward passes, logging, checkpoint creation, and reload. It does not establish convergence or policy quality. - Shorten any save and scheduler horizons that would otherwise fall beyond the smoke run. Read the resolved policy configuration first; scheduler fields are not uniform across families. - Measure peak memory and steps per second on a small batch, then choose batch size, steps, and hardware. Do not use a timeless policy-to-VRAM table as a procurement promise. Robium evidence, scoped to its measured conditions: on 2026-07-12, ACT on `lerobot/pusht` with 96×96 images and batch 8 sustained about 11.6 steps/s on an M2 Pro through MPS; a 200-step pipeline smoke train took about 31 seconds. That does not predict a 640×480 real-robot dataset or another policy family. ## Camera and processor contracts - Compare dataset feature keys with the selected policy's current input features before training. Camera count and names can differ even between a base checkpoint and its fine-tune. - Where the installed release supports them, use `rename_map` to align real feature keys and `empty_cameras` only for policy-supported masked slots. Check the current policy guide rather than assuming every VLA supports the same fields. - Inspect the saved preprocessor and postprocessor after training. At evaluation, do not apply a rename a second time when the saved processor already owns it. Robium's 2026-07-14 SmolVLA trial found that a base checkpoint and fine-tune could carry different camera layouts in their processor configuration. Preserve that as a failure signature, not a universal list of camera keys. ## Remote training Local training is the normal path. If the installed release exposes `--job.target`, treat it as paid external compute: - Use `huggingface` to verify identity, current hardware, pricing, approval, logs, output repositories, and cancellation. - Keep `--output_dir` inside the remote container and discover the actual model destination from the completed job rather than assuming a requested Hub repo was honored. - Do not preserve a flavor list, price, or timeout here. Read live CLI help and current Hugging Face Jobs documentation immediately before submission. Robium's 2026-07-14 VLA trial observed three version-specific signatures: an account without prepaid credit failed at submission with HTTP 402; the managed path ignored the requested policy repository and logged an auto-generated one; and a macOS host path passed as `output_dir` failed only when the remote run saved. Compare these signatures with current behavior before relying on them. ## Current sources Re-checked on 2026-09-07; use the live pages rather than treating this date as a version guarantee. - [LeRobot overview](https://huggingface.co/docs/lerobot/main/en/index) - [ACT guide](https://huggingface.co/docs/lerobot/act) - [Policy API and catalog](https://huggingface.co/docs/lerobot/main/en/api/policies) - [Training cheat sheet](https://huggingface.co/docs/lerobot/main/cheat-sheet)
-
-
evals.yaml 555 B
triggers: positive: - phrase: inspect the processor files in this SmolVLA checkpoint before evaluating it - phrase: train an ACT policy and publish the LeRobot checkpoint to my Hub repo negative: - phrase: download this generic language model from the Hugging Face Hub expect: huggingface tasks: - name: pusht-dataset-loads example: examples/load-dataset-snippet.py command: "uv run --with 'lerobot[dataset]' python skills/lerobot/examples/load-dataset-snippet.py" pass_criteria: "observation\\.image" timeout: 900 -
FAILURES.md 3.3 KB
# When a LeRobot pipeline fails Start at the first contract that has contrary evidence. ## A checkpoint will not load - Inspect the repository or directory for model configuration, weights, `policy_preprocessor.json`, `policy_postprocessor.json`, and their statistics. - Pre-processor-pipeline checkpoints can have valid weights but remain unloadable on current LeRobot. In Robium's 0.6.0 trial, `lerobot/diffusion_pusht` failed because the processor files were absent. - Do not substitute a similarly named base or fine-tuned checkpoint without comparing processor configuration and expected features. ## Training rejects dataset features - Compare dataset feature names and shapes with the policy's expected state, action, and camera inputs. - VLA policies may require fixed camera keys. Consult the current rename-map and empty-camera support before adapting them. - A rename saved into the fine-tuned checkpoint's processor should not be applied a second time during evaluation. ## Evaluation ends in BrokenPipe or missing environments - Find the first worker exception rather than the parent's final pipe error. - Robium observed shipped environments failing under the asynchronous vector environment default on LeRobot 0.6.0; synchronous evaluation worked. Verify whether that still applies to the installed release before carrying the workaround forward. - Match environment observation/action shapes and control mode to the trained policy before changing evaluation parallelism. ## Training runs but is impractically slow - Separate a pipeline smoke test from a viable training target. CPU and Apple MPS can prove that some loops start; they do not make large VLA fine-tuning practical. - In the Robium VLA trial, SmolVLA on MPS advanced only about 20 of 20,000 steps in roughly two hours. Preserve that as a measured warning, not a universal benchmark. - Check data-loading time, image size, batch size, accelerator use, and policy memory before adding GPUs. Rescale a policy's learning-rate schedule when a smoke run drastically shortens its configured steps. ## A remote Job does not behave like the local command - A 402 response means paid-compute credit is unavailable, not that the job is waiting. - Robium observed the Jobs path ignoring `--policy.repo_id` and publishing to an auto-generated repository; read the logs for the actual destination. - Local absolute output paths are passed into the remote container unchanged. In the same trial, a `/Users/...` path trained successfully and failed only when saving. Use a container-local path after verifying current behavior. ## Visualization or recording fails - Resolve `lerobot` and `rerun-sdk` constraints together. Robium observed the LeRobot 0.6.0 `viz` extra conflicting with `gradio_rerun` 0.34.1; dropping that extra and pinning the required Rerun SDK resolved that application. - Dataset visualization can save an `.rrd` or serve a remote viewer without a local display; do not force `spawn()` on a headless host. - Recording controls can work in an interactive terminal while keyboard teleoperation still fails. Global keyboard capture needs a supported desktop session or macOS Accessibility permission. - Video decode failures belong first to the installed FFmpeg/TorchCodec path, not the dataset schema. -
SKILL.md 3.1 KB
--- name: lerobot description: Build and debug LeRobot datasets, training, and policy evaluation. For a first pretrained robot-arm demo, start with architect's reference-app selection. --- # LeRobot Follow one contract chain through LeRobot: embodiment, dataset, processor, checkpoint, runtime observations and actions, then evaluation. Find the first contract that does not match. ## Establish the contract - For a new manipulation app or first pretrained-policy demo, read [architect](../architect/SKILL.md) before creating an environment or training pipeline. It finds the saved apps checkout and selects a compatible baseline. If selection already happened, continue here. Existing dataset, training, or evaluation work does not need onboarding; an inference-only request does not authorize training. - Inspect the installed LeRobot version and current CLI help before writing flags. Dataset formats, policy families, scripts, and extras change quickly. - Match the robot's state, action space, cameras, rates, and task to the dataset. A policy adapts to those features; it cannot repair a mismatched embodiment. - Inspect every checkpoint's configuration and processor files, not only its weights. Base and fine-tuned checkpoints from one family can expect different camera layouts. - Keep Hub identity, transfer, publication, and Jobs lifecycle at the Hugging Face boundary. Keep source-selection strategy in data. ## Prove the loop cheaply - Start with a small shipped policy and a known dataset/environment pair. - Run a short train that writes a checkpoint, then load that exact checkpoint through evaluation. Completion and numeric metrics are the smoke-test result; policy quality is not. - Confirm loss, saved processors, input/output shapes, rollout metrics, and video or real-robot behavior before increasing steps or hardware cost. - Treat real-hardware rollout as a new safety boundary even when simulation evaluation passed. ## Go deeper only when needed - For loading, recording, editing, migration, and episode visualization, read [references/datasets.md](references/datasets.md). - For policy choice, camera remapping, training, compute sizing, and remote Jobs behavior, read [references/policies-and-training.md](references/policies-and-training.md). - For simulation evaluation, headless rendering, EnvHub, or real-hardware rollout, read [references/eval-and-sim.md](references/eval-and-sim.md). - When a checkpoint, feature contract, evaluation worker, dependency, or remote run fails, start with [FAILURES.md](FAILURES.md). - The concrete PushT examples are useful only when that smoke path matches the application: [load dataset](examples/load-dataset-snippet.py) and [train ACT](examples/train-act-command.md). - Use the current [LeRobot documentation](https://huggingface.co/docs/lerobot) and [source](https://github.com/huggingface/lerobot) for version-sensitive APIs and the shipped policy/environment list. ## Done - The target dataset loads, the checkpoint carries its processors, evaluation exercises matching observations and actions, and measured results justify any longer run or hardware deployment.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.