triggers:
  positive:
    - phrase: connect Gemini Robotics ER 2 Streaming to our TurtleBot camera and guarded Nav2 tools
      source: learnings/2026-09-07-silly-turtlebot.md#lrn-0907-01
    - phrase: my Gemini robotics Live API session stalls after the navigate function call
    - phrase: attach an on-demand camera frame to the Gemini Robotics look tool response on our voice robot
      source: learnings/2026-09-17-stackchan-er2.md#lrn-0917-01
  negative:
    - phrase: Nav2 plans a path but the robot never receives velocity commands
      expect: navigation
    - phrase: decide whether this new robot application should use a VLM at all
      expect: architect
tasks: []

# Gemini Robotics streaming failures

Start with the first missing event in the session rather than rewriting the
prompt.

## The session stops after a tool call

- The Live API does not execute or answer functions automatically. Build a
  `FunctionResponse` for every requested call and send the complete response
  list with `send_tool_response`.
- Preserve each call's ID and name. A robot action completing locally does not
  unblock the model until its response reaches the session.
- Keep the receive loop alive after the response; one tool result may lead to
  another tool call before the turn completes.

## Physical actions overlap or the model plans from an unfinished action

- Confirm every physical function declaration uses `behavior: BLOCKING`.
- Do not report success when an action was only accepted. Wait for its terminal
  result or return an explicit timeout/failure.
- Keep asynchronous/non-blocking tool behavior out of the physical-action path
  even when another Live model supports it.

## The model sees frames but does not react

- Video input alone does not initiate a reasoning turn. Send user text/audio or
  a deliberate heartbeat after the latest frame.
- Respect the current JPEG rate limit. Dropped or delayed frames can otherwise
  look like a reasoning failure.
- Heartbeats can interrupt an in-progress response. Serialize them with turns
  when interruption is not the intended behavior.

## The next decision describes the old scene

- Capture after the robot action reaches a terminal state and send that fresh
  frame before asking for the next decision.
- Attach timestamps or freshness metadata at the camera/adapter boundary. Do
  not infer freshness from a non-empty JPEG.
- Keep camera acquisition outside the cloud receive loop so a stalled camera
  cannot silently stall tool-response delivery.

## Audio input never completes

- Confirm raw signed 16-bit mono PCM, little-endian, at 16 kHz and the matching
  MIME type.
- Preserve sample alignment when chunking; every chunk must contain an even
  number of bytes.
- Send `audio_stream_end=True` for a finite utterance. For continuous listening,
  configure turn detection deliberately instead of sending arbitrary file
  boundaries.

## No speech comes from the robot

- Gemini Robotics ER 2 Streaming produces text output. Feed completed text to
  a TTS adapter or let the model call a bounded `speak` tool.
- Keep TTS failure separate from motion completion so audio-device trouble
  cannot make a successful navigation action appear failed.

## A model-generated action bypasses robot constraints

- Treat this as a missing application guard, not a prompt-tuning problem.
- Reject unknown functions, extra arguments, unconfigured waypoints, out-of-
  range values, and object identifiers not issued by current perception.
- Make stop/cancel callable by the host on deadline or disconnect. The model
  must not be the only component able to stop motion.

Re-check the current [Live API tool guide](https://ai.google.dev/gemini-api/docs/live-api/tools)
when message fields or function-calling behavior differ from the application.

# Silly TurtleBot evidence

This card records one Robium application developed from 2026-09-07 through
2026-09-14. It is evidence for the narrow Gemini Robotics integration path, not
a universal robot design.

## Observed stack

- Python 3.10 with `google-genai>=2.22.0,<3`.
- `gemini-robotics-er-2-streaming-preview` through
  `client.aio.live.connect`, with text output and manual function responses.
- A cloud-side agent calling a loopback HTTP adapter; a ROS 2 process owned
  Nav2 actions, camera subscriptions, cancellation, and TTS.
- Semantic tools for named navigation, bounded quarter-turn inspection,
  approaching a perception-issued object ID, facing a person without
  approaching, speaking, and stopping.

## What passed

- A live model turn called a guarded fake-robot tool successfully.
- Thirteen hardware-free tests covered the mission guard, audio framing, manual
  Live API tool mediation, persistent-session reuse, heartbeat filtering, SSE
  replay, cancellation, terminal response ordering, camera flow, and the HTTP
  adapter.
- An unrecognized `publish_cmd_vel` call was returned to the model as rejected.
- The Gazebo Harmonic TurtleBot 4 runtime exposed Nav2 actions and a fresh
  simulated OAK-D JPEG; forward and quarter-turn actions completed.
- The physical TurtleBot 4 smoke verified ROS/Nav2 state, fresh full-resolution
  OAK-D frames, a separate low-bandwidth operator preview, and bounded neural
  speech without commanding an autonomous mission.

## What the app changed after implementation evidence

- Tool schemas remained semantic and blocking; the independent guard enforced
  exact arguments, named waypoints, ranges, and latest-scan object IDs.
- A successful look action caused the robot adapter to fetch and queue a fresh
  JPEG, which the receive loop streamed before the next model decision.
- A model-turn deadline invoked the same stop path available to the operator.
- Text-to-speech stayed behind a replaceable adapter because the endpoint's
  output modality is text.
- The long-lived session kept local tool progress and camera frames flowing
  during blocking motion, but withheld model-facing heartbeats until the tool
  response and turn were resolved.

## Not yet proven

- A complete Gemini-driven navigation mission has not yet been repeated three
  times on the physical TurtleBot 4 under supervision.
- RGB-depth grounding of floor objects and person detection remain application
  work, so this skill does not prescribe either pipeline.

---
name: gemini-robotics
description: Integrate and debug Gemini Robotics ER perception, function calls, and guarded execution. For a first natural-language robot assistant demo, start with architect's reference-app selection.
---

# Gemini Robotics

Keep Gemini on the perception and planning side of an actuator boundary. The
model may choose a capability; deterministic robot software validates and
executes it.

## Choose the endpoint from the interaction

- For a new robot assistant or first simulation demo, read
  [architect](../architect/SKILL.md) before building a model/robot integration.
  It discovers the saved apps checkout and checks compatible examples. If a
  baseline was already selected, continue here; existing integration fixes
  and API explanations do not need onboarding.
- Use `gemini-robotics-er-2-streaming-preview` for a stateful Live API session
  that receives text, JPEG frames, or audio and orchestrates robot tools with
  low latency.
- Use `gemini-robotics-er-2-preview` for discrete embodied-reasoning requests
  such as spatial analysis or offline video work. The standard endpoint does
  not provide the Live API.
- Treat both model IDs and feature support as preview surfaces. Re-check the
  current [model overview](https://ai.google.dev/gemini-api/docs/robotics-overview)
  before changing dependencies or deployment assumptions.

## Make the stream an observe-act-observe loop

- Keep one `client.aio.live.connect` session open for the task and run a receive
  loop that handles both model content and tool calls.
- Serialize user turns and model-facing heartbeats around unresolved turns and
  blocking tools. A text heartbeat is a new reasoning input, not a transport
  keepalive, and can interrupt an action as barge-in.
- Declare physical actions with `behavior: BLOCKING`. Execute each call through
  the robot adapter, then manually return a `FunctionResponse` with the call ID,
  name, and structured result using `send_tool_response`.
- Stream raw 16-bit, 16 kHz, little-endian PCM for speech input and explicitly
  end finite audio with `audio_stream_end=True`. Send JPEG camera frames at no
  more than the endpoint's current one-frame-per-second limit.
- A camera frame alone updates context but does not trigger reasoning. Pair it
  with user audio/text, or use an intentional heartbeat prompt. Heartbeats are
  turns and can interrupt generation.
- When a tool exists specifically to observe the world, attach its fresh image
  to that call's `FunctionResponse` when the SDK supports inline media. This
  binds the evidence to the requesting call more deterministically than placing
  an unrelated realtime frame immediately before the response.
- The streaming endpoint returns text, not synthesized audio. Route speech
  through an independently replaceable TTS adapter or expose speaking as a
  bounded tool.

Use Google's current [robotics streaming guide](https://ai.google.dev/gemini-api/docs/robotics-streaming)
for the volatile SDK syntax. Read [FAILURES.md](FAILURES.md) when a session
stalls, ignores images, overlaps actions, or never finishes an audio turn.

## Guard the robot outside the model

- Expose semantic capabilities such as named-waypoint navigation, bounded
  inspection, or grasping a currently grounded object. Do not expose raw motor
  commands, arbitrary poses, or unrestricted coordinates merely because the
  function schema can describe them.
- Validate the tool allowlist, exact arguments, ranges, named resources, and
  current perception-issued object IDs in ordinary code. A system instruction
  and JSON schema improve model behavior but are not the safety boundary.
- Build the advertised tool list from capabilities that passed preflight. Do
  not leave a disconnected robot, camera, or accessory visible to the model as
  a callable tool.
- Return completion, rejection, and failure states to the model. After motion,
  send a fresh observation so the next decision is based on the resulting
  scene rather than the pre-action frame.
- Give every long-running action cancellation and a deadline. On session or
  tool timeout, invoke the robot's stop/cancel path independently of the model.
- On half-duplex hardware, pause microphone ingestion before speech or another
  device action and resume it explicitly afterward. Keep this device handoff
  outside the model's control.
- Prove the same semantic contract against a fake adapter, representative
  simulation, and finally supervised hardware. Keep simulator- and robot-
  specific motion details behind the adapter.

For the evidence behind these choices and their current validation limits, read
[SILLY-TURTLEBOT.md](SILLY-TURTLEBOT.md) for a ROS/Nav2 mobile robot and
[STACKCHAN-ER2.md](STACKCHAN-ER2.md) for a USB, audio, camera, and BLE companion.
Use `integration` for process or transport boundaries, `ros2` and `navigation`
for deterministic mobile-robot execution, and `testing` for the
fake-to-simulation-to-hardware acceptance ladder.

## Done

- A complete user turn can stream input, execute a blocking semantic action,
  return its result, and reason from a fresh observation.
- An undeclared or invalid action is rejected before reaching the robot SDK.
- Timeout and cancellation behavior is proven without depending on a model
  response.

# STACK-CHAN ER 2 evidence

This card records the STACK-CHAN ER 2 Companion application validated on real
hardware from 2026-09-14 through 2026-09-17. It supports the reusable Gemini
session and tool-boundary guidance; its device limits are not general defaults.

## Observed stack

- `gemini-robotics-er-2-streaming-preview` in one persistent Live API session,
  with automatic voice activity detection over continuous 250 ms microphone
  chunks and text output.
- A Python host process owning the model session, guarded actions, macOS TTS,
  one USB serial request at a time, and an optional dedicated BLE worker.
- M5Stack STACK-CHAN K151 firmware owning the display, servos, microphones,
  speaker, and on-demand 320 by 240 camera capture.
- Six semantic tools: bounded head movement, short display text, bounded speech,
  one fresh `look`, one of five named animations, and enum-only LEGO motion.

## What passed

- Thirty-five hardware-free tests covered the tool allowlist and guards, PCM
  framing, continuous and push-to-talk audio boundaries, inline camera tool
  responses, silent action-only turns, serial framing, BLE selection, and
  confirmed motion-to-stop transmission.
- Real hardware passed display, bounded head motion, microphone capture,
  acknowledged speaker playback, camera capture, and automatic microphone
  resume after half-duplex actions.
- A physical voice turn caused ER 2 to request one image, describe the current
  view, speak through the robot, and resume listening. Ordinary conversation
  did not capture an image.
- A separate end-to-end voice turn selected one bounded LEGO direction; the
  host confirmed the transmitted zero-power revision before returning success,
  while the hub retained an independent 400 ms watchdog.

## What changed after physical evidence

- A `look` image moved from realtime video input into the matching
  `FunctionResponse.parts`. Two opposite views in one session then produced
  answers about the new view rather than stale context.
- Continuous audio and speaker playback became an explicit half-duplex handoff:
  pause input, run the guarded action, wait for device readiness, then resume.
- The LEGO tool is removed from the model declarations when no controller is
  connected; prompt text does not stand in for capability availability.
- Named firmware animations and fixed LEGO directions replaced raw servo modes,
  wheel powers, and durations at the model boundary.

## Limits

- The application continuously sends room audio while running and uses visible
  listening LEDs plus explicit process start/stop rather than a local wake-word
  engine.
- Camera capture is model-selected and single-frame, not a continuous video
  stream. The device temporarily hands shared I2C pins to the camera and then
  restores the audio/display peripherals.
- STACK-CHAN and its LEGO accessory are supervised desktop hardware. These
  results do not establish safe autonomous navigation or manipulation.