Esp32 Iphone Agent

Let an AI model use a real iPhone with a $7 ESP32. Just the phone and the board: no computer, no jailbreak.

English · 简体中文

Demo video and updates: X @iamhenryyuan · YouTube demo

New: the computer is gone. The board is now the AirPlay receiver and the mouse, and the agent runs in an iPhone app on the same phone. The original version (a Mac/Linux computer running UxPlay and a Python server) is kept as tag v0.1.0 and branch legacy. See docs/legacy.md.

"Set an alarm for 8:00 AM that repeats every Monday and Tuesday, then save it": the agent leaves its own app, opens Clock, types the time into the wheel, ticks the days and saves. (Photo from v0.1.0; the task is the same.)

Left: the real phone. Right: what the agent sees through the board's AirPlay mirror. The second shot is the pointer calibration.

How it works

iPhone ──Screen Mirroring (Wi-Fi)──▶ ESP32-S3: AirPlay receiver ──BLE──▶ ESP32 Agent app (same iPhone)
   ▲                                    own hotspot, H.264 passthrough        decode → vision model → action
   └──────── USB HID mouse + keyboard ◀──────────── BLE commands ◀────────────────────┘

• Eyes — AirPlay to the board. The board opens its own Wi-Fi hotspot and appears in the phone's Screen Mirroring list as ESP32 Agent. It keeps the H.264 stream (no decoding on the board) and forwards it to the app over BLE. The phone keeps using cellular data for the model while it is on the board's hotspot.
• Hands — USB HID. The board is a standard USB mouse and keyboard. With AssistiveTouch on, the mouse moves an on-screen pointer, so a click is a tap and a drag is a swipe.
• Brain — the app. The app decodes the stream, sends screenshots to any OpenAI-compatible vision model, and runs a planner and an actor: tap, swipe, draw, type, press keys, go home, finish. It runs in the background while it works; the Dynamic Island shows what it is doing.
• Closing the loop. The pointer is relative, so the app tracks it and re-homes it in a corner; before precise taps it wiggles the pointer, finds it on screen and corrects the miss. After each action it waits for the screen to settle and tells the model whether anything changed. A lost board, hotspot or mirroring link pauses the run and resumes it from the same step.
• Review. Every run is kept on the phone: replay it step by step with the taps and swipes drawn on top. Export the screenshots, actions and saved annotations from Runs → open a run → ⋯ → Export trace for analysis by another agent. See Export runs for agent review for batch export, USB copying and file details.

Hardware