ui-control
Infrastructure skill for application UI observation, scoped action, waits, and CUA trajectory recording when a typed DCC tool cannot expose the needed interface. Use this skill when the user asks for CUA, UI Control, dcc-cua, or application UI automation in a DCC. Prefer typed DC
Install
npx skills add https://github.com/dcc-mcp/dcc-mcp-core/tree/main/python/dcc_mcp_core/skills/ui-control
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install dcc-mcp-dcc-mcp-core@llmmart
git clone https://github.com/dcc-mcp/dcc-mcp-core.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole dcc-mcp/dcc-mcp-core collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
DCC UI Control
The Agent Host owns user authorization. Core never enables the standalone Host's legacy native confirmation popup, including when menu invocation is allowed. Missing authorization remains a typed refusal; do not auto-approve or retry it through another input route. For bounded task authorization, use the project-owned DCC-CUA task bridge and its exact target/scope checks.
Use application UI automation only when a typed DCC skill, host API, or adapter
script reports unsupported or capability_missing. Policy denial, user
interruption, authentication, and desktop unavailability are stop conditions.
Backend order
- Use a typed DCC-MCP tool whenever one exists.
- For Chrome, Edge, launchers, and embedded webviews with DevTools access, set
DCC_MCP_UI_CONTROL_BACKEND=chrome(oredge) and use CDP. - For native application UI, set
DCC_MCP_UI_CONTROL_BACKEND=cuaand use the standalonedcc-cuaCLI/Host. mockis an explicit deterministic test backend only; production defaults todcc-cuaand never silently falls back to mock automation.
Core does not package an automation Host in its own release assets. The official
CLI installer reconciles the independently released dcc-cua companion; verify
it with dcc-mcp-cli components status dcc-cua or repair it with
dcc-mcp-cli components ensure dcc-cua --yes. For custom layouts, install
dcc-cua 0.4.0 or newer separately. Core probes the configured
DCC_MCP_INSTALL_DIR, the platform's standard dcc-mcp bin directory,
versioned standalone installs, and then PATH; set DCC_MCP_CUA_BINARY to an
absolute executable path only for a custom location. Core validates
dcc-cua manifest, ensures the shared Host, and keeps one
persistent JSONL bridge per active UI session. It prefers shared-memory image
transport when the native Core extension is present and otherwise uses bounded
binary attachments.
The CUA Host owns native accessibility, capture, banner/border/cursor markers, input scheduling, Escape interruption, browser CDP routing, and platform adapters. Core owns only DCC policy, normalized tool results, audit events, and artifact publication.
An adapter may opt into an integrity-pinned, owned public-MCP pixels_only
runtime through server-owned UiControlRuntimeOptions. This preserves the
default shared JSONL path for other servers. The owner must bind both PID and
HWND; tool arguments cannot select a binary, transport, or action grant. See
owned pixels MCP before using that mode.
Owned pixels mode can separately permit explicit secondary popup observations
through the owner-only allow_native_capture_popup_keyboard ceiling. Select
one through eight exact HWNDs from the root's fresh native capture inventory
with snapshot(native_capture_popup_window_handles=[...]). The main image keeps
its honest redactions; each popup has a separate labeled PNG and full native
provenance. This permits one unheld keyboard action to the original main root
within five seconds, subject to native coverage, identity and foreground checks.
It grants no pointer input or popup rebinding. Every attempt consumes that root
observation; re-observe instead of retrying. See the reference for the complete
boundary and cleanup requirements.
Exact application scope
DccServerBase injects its trusted DccServerOptions process/window context
into every in-process ui-control call. Dedicated or custom servers may
override that automatic binding with:
DCC_MCP_UI_CONTROL_PROCESS_IDDCC_MCP_UI_CONTROL_WINDOW_HANDLE- optional
DCC_MCP_UI_CONTROL_PROCESS_NAME - optional
DCC_MCP_UI_CONTROL_WINDOW_TITLE
Request arguments may narrow that scope but cannot widen it. Native mouse and
keyboard input are enabled inside that exact scope by default; operators can
disable them with DCC_MCP_CUA_ALLOW_RAW_INPUT=false. Semantic accessibility
actions remain preferred. For a multi-window process, pass a window_title
that narrows the trusted PID to one window. The Host resolves that title once,
then returns and enforces an exact window capability; use an explicit handle if
the title still matches more than one window.
A redundant process_name may accompany a trusted PID or HWND as metadata or
an additional constraint. It never replaces the exact native binding: a
process name or title by itself cannot mint a capability, and a process name
that conflicts with an operator-trusted name fails with invalid_target.
Multiple agents may control different applications concurrently. Each logical session has its own grant, window capability, observation fences, and bridge. The shared Host isolates session state, serializes raw input, and broadcasts Escape as the operator stop signal.
Observe, act, verify
Passive temporary capture is an independent owner-granted route:
ui_control__act(action="get_window_state"), then
ui_control__capture_preparation(operation="begin", window_state_id=..., lifetime_ms=...)
on the same explicit session/PID/HWND. Read preparation state, optionally request
operation="snapshot", then operation="stop" and verify cleanup_verified.
These images carry no input token. Preserve restore_pending and cleanup_unknown;
never reinterpret temporary visibility as actual foreground or input readiness.
Follow this loop:
- For authorized foreground interaction, call
ui_control__prepare_foregroundwith the currentsession_id, exactprocess_idandwindow_handle, and explicitoperation="restore_activate"(oractivatefor an already visible window). It activates, reads back actual foreground/visibility, and returns a fresh same-session snapshot. For observation alone, callui_control__snapshot; it never performs foreground preparation. - Use
ui_control__findfor semantic controls when needed. - Call
ui_control__actonce with the currentsnapshot_id. - Call
ui_control__wait_forfor an explicit condition or take a fresh snapshot. - Call
ui_control__stop_computer_usewhen the application task is complete.
Never reuse an observation after an action, target change, display change, or
resume. CUA actions are fenced by both observation_id and
accessibility_state_id; stale actions fail closed.
For a native application menu bar, use
ui_control__act(action="invoke_menu", menu_path=[...]) when a semantic menu
click or Alt mnemonic cannot prove that a Qt popup opened. This exact-window
native path does not require a prior snapshot, fails closed on missing or
ambiguous levels, and invalidates any current observation. Check
verification_required, then take a fresh snapshot and verify the popup or
resulting application state before another mutation.
Prefer semantic control_id actions. Coordinate clicks, drag paths, raw typing,
shortcuts, and game navigation are native input and require the raw-input gate.
Do not use UI Control to enter credentials, solve authentication challenges, or
cross a secure-desktop boundary.
Recording
Trajectory recording is session-scoped and intentionally asynchronous:
- Call
ui_control__recording_startwith an absoluteoutput_dirand optionalrecord_video=true. - Perform the UI actions that should become evidence.
- Optionally call
ui_control__recording_state. - Call
ui_control__recording_stopto finalize the trajectory. - Preserve the finalized artifacts and structured state returned by CUA.
This replaces the removed synchronous JPEG-sequence record_clip contract.
Core does not duplicate CUA's recording format.
In explicit owned pixels mode, recording needs
the separate operator-owned UiControlRecordingOptions ceiling. It grants no
input and never starts automatically. Use the same session and call
recording_start with output_dir/record_video omitted: the runtime supplies its
immutable task directory and video-only request. trajectory_available=false.
Retain paused/degraded/failed state and partial artifacts. A failed stop is not
finalization; read recording_state explicitly and always finish with
stop_computer_use, preserving the actual cleanup ACK or failed/unknown outcome.
Semantic profiles and trusted handoff
Run dcc-cua profiles, then inspect the chosen application with
dcc-cua profile --id <id>. Profile target IDs stay stable across languages;
localized aliases are matching inputs, not translated identifiers. Dispatch a
browser_dom surface through the exact-bound dcc-cua browser route, never the
in-app Browser skill. A fallback such as ue/fab/download to
fab/launcher_download requires a new Epic Games Launcher binding and fresh
observation. Cloudflare challenges, authentication, purchases, and operating
system security confirmations require trusted human action; full agent access
does not bypass them.
Browser control
CDP is the first choice for browser content because it exposes DOM semantics, stable selectors, network state, and background operation. Use native CUA only for browser chrome, file pickers, permission prompts, canvas-only content, or when CDP is unavailable. Never attach a second automation stack to the same page concurrently.
Error handling
Treat these as recoverable only when the stated precondition can be restored:
stale_observation: take a fresh snapshot.missing_windoworinvalid_target: rediscover and rebind the exact app.desktop_unavailable: wait for the interactive desktop to return.backend_unavailable: install/configuredcc-cuaor restore CDP.user_interrupted: stop until the operator explicitly resumes.policy_disabledorpermission_denied: do not bypass policy.
Successful snapshots include capture_provenance. Preserve it with any visual
evidence. Audit logs redact typed text and other sensitive action payloads.
Files (dcc-mcp-core)
-
references
-
OWNED_PIXELS_MCP.md 8.9 KB
Loading…
-
-
scripts
-
act.py 390 B
Loading… -
capture_preparation.py 500 B
Loading… -
find.py 395 B
Loading… -
prepare_foreground.py 488 B
Loading… -
recording_start.py 441 B
Loading… -
recording_state.py 446 B
Loading… -
recording_stop.py 435 B
Loading… -
snapshot.py 415 B
Loading… -
stop_computer_use.py 460 B
Loading… -
wait_for.py 415 B
Loading… -
_backend.py 25.1 KB
Loading… -
_cdp_runtime.py 19.7 KB
Loading… -
_chrome_backend.py 29 KB
Loading… -
_cua_backend.py 39.8 KB
Loading… -
_cua_cli_host_client.py 13.4 KB
Loading… -
_cua_lifecycle.py 2.3 KB
Loading… -
_cua_passive_preparation.py 5.9 KB
Loading… -
_cua_pixels.py 3.8 KB
Loading… -
_cua_preparation.py 16.9 KB
Loading… -
_cua_snapshot_result.py 3.2 KB
Loading… -
_cua_support.py 25.4 KB
Loading… -
_cua_window_frame.py 6.6 KB
Loading… -
_entrypoint.py 21.8 KB
Loading… -
_scenario.py 31.3 KB
Loading…
-
-
SKILL.md 10.6 KB
--- name: ui-control description: >- Infrastructure skill for application UI observation, scoped action, waits, and CUA trajectory recording when a typed DCC tool cannot expose the needed interface. Use this skill when the user asks for CUA, UI Control, dcc-cua, or application UI automation in a DCC. Prefer typed DCC tools first and CDP for browsers; use the standalone dcc-cua backend for native applications. license: MIT compatibility: "dcc-cua 0.4.0+, Python 3.7+" metadata: dcc-mcp: dcc: python recall-context: app_type: python domain: ui-automation workflow_stage: automation task_category: mutate version: "0.5.0" layer: infrastructure search-hint: "dcc ui control, ui control, UI Control, cua, CUA, dcc-cua, dcc cua, computer use, ui automation, chrome cdp, edge cdp, screenshot, click, type, keypress, scroll, wait, trajectory recording" tags: "ui-control, cua, dcc-cua, computer-use, ui-automation, chrome-cdp, diagnostics, infrastructure" tools: tools.yaml --- # DCC UI Control The Agent Host owns user authorization. Core never enables the standalone Host's legacy native confirmation popup, including when menu invocation is allowed. Missing authorization remains a typed refusal; do not auto-approve or retry it through another input route. For bounded task authorization, use the project-owned DCC-CUA task bridge and its exact target/scope checks. Use application UI automation only when a typed DCC skill, host API, or adapter script reports `unsupported` or `capability_missing`. Policy denial, user interruption, authentication, and desktop unavailability are stop conditions. ## Backend order 1. Use a typed DCC-MCP tool whenever one exists. 2. For Chrome, Edge, launchers, and embedded webviews with DevTools access, set `DCC_MCP_UI_CONTROL_BACKEND=chrome` (or `edge`) and use CDP. 3. For native application UI, set `DCC_MCP_UI_CONTROL_BACKEND=cua` and use the standalone `dcc-cua` CLI/Host. 4. `mock` is an explicit deterministic test backend only; production defaults to `dcc-cua` and never silently falls back to mock automation. Core does not package an automation Host in its own release assets. The official CLI installer reconciles the independently released `dcc-cua` companion; verify it with `dcc-mcp-cli components status dcc-cua` or repair it with `dcc-mcp-cli components ensure dcc-cua --yes`. For custom layouts, install `dcc-cua` 0.4.0 or newer separately. Core probes the configured `DCC_MCP_INSTALL_DIR`, the platform's standard dcc-mcp bin directory, versioned standalone installs, and then `PATH`; set `DCC_MCP_CUA_BINARY` to an absolute executable path only for a custom location. Core validates `dcc-cua manifest`, ensures the shared Host, and keeps one persistent JSONL bridge per active UI session. It prefers shared-memory image transport when the native Core extension is present and otherwise uses bounded binary attachments. The CUA Host owns native accessibility, capture, banner/border/cursor markers, input scheduling, Escape interruption, browser CDP routing, and platform adapters. Core owns only DCC policy, normalized tool results, audit events, and artifact publication. An adapter may opt into an integrity-pinned, owned public-MCP `pixels_only` runtime through server-owned `UiControlRuntimeOptions`. This preserves the default shared JSONL path for other servers. The owner must bind both PID and HWND; tool arguments cannot select a binary, transport, or action grant. See [owned pixels MCP](references/OWNED_PIXELS_MCP.md) before using that mode. Owned pixels mode can separately permit explicit secondary popup observations through the owner-only `allow_native_capture_popup_keyboard` ceiling. Select one through eight exact HWNDs from the root's fresh native capture inventory with `snapshot(native_capture_popup_window_handles=[...])`. The main image keeps its honest redactions; each popup has a separate labeled PNG and full native provenance. This permits one unheld keyboard action to the original main root within five seconds, subject to native coverage, identity and foreground checks. It grants no pointer input or popup rebinding. Every attempt consumes that root observation; re-observe instead of retrying. See the reference for the complete boundary and cleanup requirements. ## Exact application scope `DccServerBase` injects its trusted `DccServerOptions` process/window context into every in-process `ui-control` call. Dedicated or custom servers may override that automatic binding with: - `DCC_MCP_UI_CONTROL_PROCESS_ID` - `DCC_MCP_UI_CONTROL_WINDOW_HANDLE` - optional `DCC_MCP_UI_CONTROL_PROCESS_NAME` - optional `DCC_MCP_UI_CONTROL_WINDOW_TITLE` Request arguments may narrow that scope but cannot widen it. Native mouse and keyboard input are enabled inside that exact scope by default; operators can disable them with `DCC_MCP_CUA_ALLOW_RAW_INPUT=false`. Semantic accessibility actions remain preferred. For a multi-window process, pass a `window_title` that narrows the trusted PID to one window. The Host resolves that title once, then returns and enforces an exact window capability; use an explicit handle if the title still matches more than one window. A redundant `process_name` may accompany a trusted PID or HWND as metadata or an additional constraint. It never replaces the exact native binding: a process name or title by itself cannot mint a capability, and a process name that conflicts with an operator-trusted name fails with `invalid_target`. Multiple agents may control different applications concurrently. Each logical session has its own grant, window capability, observation fences, and bridge. The shared Host isolates session state, serializes raw input, and broadcasts Escape as the operator stop signal. ## Observe, act, verify Passive temporary capture is an independent owner-granted route: `ui_control__act(action="get_window_state")`, then `ui_control__capture_preparation(operation="begin", window_state_id=..., lifetime_ms=...)` on the same explicit session/PID/HWND. Read preparation state, optionally request `operation="snapshot"`, then `operation="stop"` and verify `cleanup_verified`. These images carry no input token. Preserve `restore_pending` and `cleanup_unknown`; never reinterpret temporary visibility as actual foreground or input readiness. Follow this loop: 1. For authorized foreground interaction, call `ui_control__prepare_foreground` with the current `session_id`, exact `process_id` and `window_handle`, and explicit `operation="restore_activate"` (or `activate` for an already visible window). It activates, reads back actual foreground/visibility, and returns a fresh same-session snapshot. For observation alone, call `ui_control__snapshot`; it never performs foreground preparation. 2. Use `ui_control__find` for semantic controls when needed. 3. Call `ui_control__act` once with the current `snapshot_id`. 4. Call `ui_control__wait_for` for an explicit condition or take a fresh snapshot. 5. Call `ui_control__stop_computer_use` when the application task is complete. Never reuse an observation after an action, target change, display change, or resume. CUA actions are fenced by both `observation_id` and `accessibility_state_id`; stale actions fail closed. For a native application menu bar, use `ui_control__act(action="invoke_menu", menu_path=[...])` when a semantic menu click or Alt mnemonic cannot prove that a Qt popup opened. This exact-window native path does not require a prior snapshot, fails closed on missing or ambiguous levels, and invalidates any current observation. Check `verification_required`, then take a fresh snapshot and verify the popup or resulting application state before another mutation. Prefer semantic `control_id` actions. Coordinate clicks, drag paths, raw typing, shortcuts, and game navigation are native input and require the raw-input gate. Do not use UI Control to enter credentials, solve authentication challenges, or cross a secure-desktop boundary. ## Recording Trajectory recording is session-scoped and intentionally asynchronous: 1. Call `ui_control__recording_start` with an absolute `output_dir` and optional `record_video=true`. 2. Perform the UI actions that should become evidence. 3. Optionally call `ui_control__recording_state`. 4. Call `ui_control__recording_stop` to finalize the trajectory. 5. Preserve the finalized artifacts and structured state returned by CUA. This replaces the removed synchronous JPEG-sequence `record_clip` contract. Core does not duplicate CUA's recording format. In explicit [owned pixels mode](references/OWNED_PIXELS_MCP.md), recording needs the separate operator-owned `UiControlRecordingOptions` ceiling. It grants no input and never starts automatically. Use the same session and call `recording_start` with output_dir/record_video omitted: the runtime supplies its immutable task directory and video-only request. `trajectory_available=false`. Retain paused/degraded/failed state and partial artifacts. A failed stop is not finalization; read `recording_state` explicitly and always finish with `stop_computer_use`, preserving the actual cleanup ACK or failed/unknown outcome. ## Semantic profiles and trusted handoff Run `dcc-cua profiles`, then inspect the chosen application with `dcc-cua profile --id <id>`. Profile target IDs stay stable across languages; localized aliases are matching inputs, not translated identifiers. Dispatch a `browser_dom` surface through the exact-bound `dcc-cua` browser route, never the in-app Browser skill. A fallback such as `ue/fab/download` to `fab/launcher_download` requires a new Epic Games Launcher binding and fresh observation. Cloudflare challenges, authentication, purchases, and operating system security confirmations require trusted human action; full agent access does not bypass them. ## Browser control CDP is the first choice for browser content because it exposes DOM semantics, stable selectors, network state, and background operation. Use native CUA only for browser chrome, file pickers, permission prompts, canvas-only content, or when CDP is unavailable. Never attach a second automation stack to the same page concurrently. ## Error handling Treat these as recoverable only when the stated precondition can be restored: - `stale_observation`: take a fresh snapshot. - `missing_window` or `invalid_target`: rediscover and rebind the exact app. - `desktop_unavailable`: wait for the interactive desktop to return. - `backend_unavailable`: install/configure `dcc-cua` or restore CDP. - `user_interrupted`: stop until the operator explicitly resumes. - `policy_disabled` or `permission_denied`: do not bypass policy. Successful snapshots include `capture_provenance`. Preserve it with any visual evidence. Audit logs redact typed text and other sensitive action payloads. -
tools.yaml 28.7 KB
Loading…
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.