# Robotics test failure router

- **Build passes, robot behavior is still unproven**
  - If claiming the behavior works, run the smallest existing check that drives
    it and observes the outcome; add a probe only if needed. Otherwise label it
    unverified and give a concrete manual check, not an automatic new harness.
  - Process presence or a healthy port is readiness evidence, not task success.

- **Test passes after measuring nothing**
  - Count devices, episodes, samples, assertions, and expected artifacts.
  - Fail explicitly when the measured count is zero or the fixture did not
    load.

- **Simulation test flakes**
  - Compare seed, initial state, simulation time, readiness wait, real-time
    factor, and timeout separately.
  - Use event-driven readiness and tolerance bands before increasing retries.

- **Test breaks only in CI**
  - Check display/headless mode, system packages, ROS environment sourcing,
    GPU availability, clocks, filesystem paths, and fixture provenance.
  - Reproduce in the same immutable environment before weakening the assertion.

- **Suite is too slow for normal iteration**
  - Move logic down to unit tests, shrink scenarios and episode sets, and mark
    benchmarks or qualification runs as slow/scheduled.
  - Keep default checks affordable and relevant. Run expensive behavior smokes
    at affected-behavior boundaries or manually/scheduled, not after every edit.

- **A configuration change did not break its test**
  - Look for re-typed literals, assertions on rendered strings, or a test that
    never imports the source constant.

- **Map or sensor-quality assertion is suspiciously easy**
  - Inspect the data encoding and unknown/missing-data representation. A
    numeric midpoint is not automatically a valid free/occupied threshold.

- **Paid remote run fails before application work begins**
  - Separate provider allocation, image pull, hardware compatibility, data
    access, and application startup using provider and workload evidence.
  - Do not spend on another attempt until the failed boundary is known.

# Learned-policy test patterns

## Pipeline smoke

- For training-pipeline changes, train at tiny scale, then run a few deterministic evaluation
  episodes.
- For an inference-only demo, reuse the pinned pretrained checkpoint and one
  bounded existing inference path. Do not train, add a harness, or run a
  multi-seed evaluation merely to demonstrate bring-up. One rollout is not a
  policy-reliability claim.
- Assert completion, expected artifacts, schemas, and numeric metrics.
- Do not require a success threshold from an intentionally undertrained policy.
  Robium's 2026-07-12 manipulation trial legitimately scored zero while still
  proving the pipeline worked.

## Regression gate

- Use a pinned known-good checkpoint, fixed environment and dataset revisions,
  deterministic seeds, a bounded episode set, and a stated threshold.
- Record per-episode results and aggregate the metric the product claim uses.
- Keep broad multi-task or multi-seed qualification separate from the quick
  default gate.

## Benchmarks are different

- A benchmark measures latency, memory, throughput, or quality across a broader
  matrix. It does not become a regression test merely because pytest launches
  it.
- Mark expensive benchmark work slow or otherwise exclude it from the default
  suite. One Robium suite grew from roughly 3.5 to 8 minutes when a large-model
  benchmark ran on every push.
- When first-use compilation dominates, report it separately from steady-state
  measurements.

## Harness guards

- Fail when zero devices, episodes, or samples were measured. A per-device
  exception loop can otherwise exit zero with no evidence; Robium observed this
  during a transient Hugging Face 401.
- Import configuration constants when they are the subject of the assertion.
  Re-typing their rendered values can leave a stale test green after the source
  changes.
- Verify artifacts independently after the run; do not treat a final log line
  as the sole success signal.

## Before paid or remote evaluation

- Run the same path locally with tiny data and CPU where possible.
- Pin the immutable image, checkpoint, dataset, framework, device, seed, and
  stop condition.
- Bound cost and lifetime, then preserve result artifacts before cleanup.
- Provider allocation and cleanup belong to its owning provider skill.

# ROS 2 and simulation test patterns

Use these patterns when automated evidence is warranted, not as a checklist
requiring a new test layer for every demo or edit.

## Choose the layer

- Use ordinary pytest for logic that does not require ROS.
- Use `launch_testing` when assertions require launched nodes, lifecycle, or
  ROS interfaces. Follow the current
  [ROS 2 testing guide](https://docs.ros.org/en/rolling/Tutorials/Intermediate/Testing/Testing-Main.html)
  and [launch_testing README](https://github.com/ros2/launch/blob/rolling/launch_testing/README.md)
  for API details.
- Use a simulator scenario only when the claim depends on physics, sensors,
  time, or closed-loop robot behavior.

## A useful ROS smoke

- Launch the smallest application composition that owns the behavior.
- Wait on the actual readiness boundary rather than a fixed sleep.
- Check the expected node or lifecycle state and the interfaces needed by the
  behavior.
- Drive one bounded action and assert its observable result.
- Capture process exits and perform shutdown assertions so a passing behavior
  does not hide a crashing component.
- Select relevant packages through the project's normal `colcon test` path,
  rather than testing the whole workspace after each edit, and inspect
  `colcon test-result` when it fails.

## Simulation evidence

- Pin the world, robot or model revision, seed, initial state, and simulated
  time behavior.
- Assert task outcomes such as reaching a goal, avoiding an obstacle, or
  completing a manipulation stage. Avoid pixel-perfect or timing-exact goldens
  unless the system actually promises them.
- Separate startup allowance from behavior timeout, and record real-time factor
  when it changes how the result should be interpreted.
- Keep the default scenario small. Longer stress or multi-seed runs can be
  scheduled or manual.

## Headless and platform constraints

- CI simulators should use their supported headless mode instead of assuming a
  display.
- Native ROS 2 and Gazebo CI is Linux-oriented. On macOS, use the project's
  established container or remote environment rather than pretending the same
  native path exists.
- Isaac Sim tests inherit its current NVIDIA GPU and driver requirements. Keep
  them off runners that do not satisfy the verified floor.

## Occupancy-map assertion observed by Robium

In the 2026-07-10 navigation trial, a free-space assertion using
`pixel >= 0.75 * maxval` falsely passed an all-unknown trinary occupancy map.
The observed encoding was free 254, unknown 205, occupied 0. A threshold above
`0.80 * maxval` separated unknown from free in that fixture; 0.9 worked.

Treat those values as evidence for that ROS map encoding and file, not a
universal image rule. Inspect the map format and metadata before reusing the
assertion.

---
name: testing
description: Choose proportional evidence for robotics software before claiming it works.
---

# Testing

Test the claim at the cheapest layer that can falsify it.

## Decide whether a test is needed

- Batch coherent changes, then run the smallest relevant existing check. Do
  not test after every edit or repeat passing checks without a relevant change,
  failure, or unresolved uncertainty. A full suite is not the default.
- Inspection can suffice for low-risk prose or straightforward edits. For
  uncertain API usage, inspect installed source, help, or current official docs
  first; use one isolated probe if uncertainty remains, not a new test suite.
- Write a new automated test when it protects a reproduced bug, a risky durable
  contract, or behavior worth repeatedly checking, or when the user requests it.
  A one-off demo does not automatically need a test harness.
- Stop when evidence supports the requested claim. Do not add platform matrices,
  benchmarks, repeated trials, or qualification work to a simple tryout.

## Match evidence to the claim

- When automation is warranted, test pure logic without ROS or a simulator.
- A node or launch claim needs the expected processes and interfaces to appear,
  not merely a successful build.
- Behavior that depends on physics, sensors, timing, or closed-loop control
  needs a deterministic scenario in simulation or on the target robot.
- A learned-policy pipeline smoke proves that data, training, evaluation, and
  metrics connect. A regression claim needs a known checkpoint and a stated
  performance bar.
- Add costlier layers only when the risk or claim requires them. Claim a demo
  worked only after observing its actual behavior; a launch or manual handoff
  alone must be labeled as behavior not yet verified.

Read [ROS2-AND-SIM.md](ROS2-AND-SIM.md) for ROS launch, simulator, map, and
headless-CI concerns. Read [POLICY-EVAL.md](POLICY-EVAL.md) for learned-policy
smokes and regression gates. Tool-specific syntax remains in official docs and
the owning `ros2`, simulator, or `lerobot` skill.

## Keep interaction and handoff lightweight

- Use browser/computer interaction only when the relevant visual or interaction
  behavior cannot be established more cheaply, a suspected UI bug needs it,
  or the user asks you to look. Prefer existing CLI, API, logs, and focused
  checks for nonvisual claims; do not tour every screen after each change.
- For low-risk visual acceptance or checks requiring the user's environment
  or access, hand off one short manual check: exact command or URL, expected
  result, and what remains unverified. Do not claim that the user has run it.
- Do not defer safety-critical evidence needed before physical motion,
  destructive actions, security-sensitive changes, or paid resource use merely
  to save testing time. Verify the affected boundary or stop before that action;
  testing never substitutes for authorization.

## Make the result trustworthy

- Seed controllable randomness, use simulation time consistently, bound the
  run, and record the fixture and environment that produced the result.
- Assert observable behavior and interfaces, not duplicated configuration
  literals or log wording.
- When checking camera/render correctness or a suspected blank viewer, inspect
  a frame for meaningful content. Non-black mean and non-flat variance are
  cheap blank-frame guards, not proof of the correct scene. HTTP success and a
  “camera ready” status can both pass while the viewer is blank.
- Guard measurement tools against an empty run. Exiting successfully after
  measuring zero devices or episodes is a test-harness failure.
- Keep benchmarks and long evaluations outside the default suite unless their
  cost is justified on every change.
- Use `test-assets` when fixtures, worlds, datasets, recordings, or goldens need
  provenance and maintenance rules.

If a suite is flaky, slow, green without useful evidence, or fails only on a
particular runner, read [FAILURES.md](FAILURES.md).

## Keep remote cost proportional

- Run the same pipeline locally at tiny scale before any paid remote test.
- Before deploying or spending on changed allocation, isolation, cancellation,
  or cleanup behavior, verify the affected lifecycle boundaries in the local
  fake deployment first. Rebuild its image when its inputs change; a stale
  digest does not test the change. Do not rerun this for unrelated edits.
- Keep GPU- or hardware-dependent tests explicit and schedulable; do not make a
  default CI job depend on unavailable hardware.
- A remote success should preserve the exact image, model, data, seed,
  hardware, result, and cost window needed to understand the claim.

## Done

- Report the evidence obtained and any explicitly deferred manual checks.
- A robotics app's smoke exercises the behavior it exists to demonstrate, not
  only process health.
- CI runs the stable, affordable evidence by default and clearly separates
  slow, paid, hardware, and manual checks.