# Evaluation Report

Evaluation of the `doca-bare-metal-deployment` skill before publication through Skill Evaluator.

This benchmark summarizes 3-Tier Evaluation from Skill Evaluator results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.

## Evaluation Summary

- Skill: `doca-bare-metal-deployment`
- Evaluation date: 2026-07-25
- Environment: `k8s-sandbox`
- Dataset: 4 evaluation tasks
- Attempts per task: 1
- Pass threshold: 50%
- Overall verdict: PASS

## Agents Used

- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`)
- Codex (`openai/openai/gpt-5.5`)

## Metrics Used

Reported benchmark dimensions:

- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
- Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
- Effectiveness: checks whether the agent performs measurably better with the skill than without it.
- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.

Underlying evaluation signals used in this run:

- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access.
- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow.
- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.
- `accuracy` (Accuracy): grades final-answer correctness against the reference answer.
- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully.
- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations.

## Test Tasks

The benchmark dataset contained 4 evaluation tasks:

- Positive tasks: 3 tasks where the skill was expected to activate.
- Negative tasks: 1 tasks where no skill was expected.
- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.

Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.

## Results

| Dimension | Num | Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`) | Codex (`openai/openai/gpt-5.5`) |
|---|---:|---:|---:|
| Security | 4 | 100% (+0%) | 100% (+0%) |
| Correctness | 4 | 100% (+35%) | 75% (+10%) |
| Discoverability | 4 | 100% (+50%) | 95% (+45%) |
| Effectiveness | 4 | 92% (+58%) | 100% (+57%) |
| Efficiency | 4 | 94% (+56%) | 100% (+75%) |

Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.

## Tier 1: Static Validation Summary

Tier 1 validation passed with observations. Skill Evaluator ran 1 checks and found 5 total findings.

Top findings:

- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/doca-bare-metal-deployment/SKILL.md`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/doca-bare-metal-deployment/SKILL.md`)
- MEDIUM SCHEMA/author_missing: Author not specified in metadata (`skills/doca-bare-metal-deployment/SKILL.md`)
- LOW SCHEMA/unexpected_file: Unexpected 'CAPABILITIES.md' in skill root (`skills/doca-bare-metal-deployment/CAPABILITIES.md`)
- LOW SCHEMA/unexpected_file: Unexpected 'TASKS.md' in skill root (`skills/doca-bare-metal-deployment/TASKS.md`)

## Tier 2: Deduplication Summary

This tier was not run or did not produce findings in this report.

## Publication Recommendation

The skill is suitable to proceed toward Skill Evaluator publication based on this benchmark. Skill owners should keep this file with the skill and refresh it when the evaluation dataset, skill behavior, or target agents materially change.

# DOCA bare-metal deployment — Capabilities

**Where to start:** The pattern overview below names the recurring
bare-metal deployment patterns the agent walks for any DOCA-linked
binary. Pick the pattern first, then drill into the H2 that owns the
substance. For the *how* of executing each pattern, jump to
[TASKS.md](TASKS.md). For the container-path counterpart of the same
patterns (kubelet standalone on the BlueField Arm watching a
documented static-pod manifests directory), see the sibling skill
[`doca-container-deployment`](../doca-container-deployment/SKILL.md).
For the cross-cutting safety meta-policy this skill overlays, see
[`doca-hardware-safety`](../doca-hardware-safety/SKILL.md).

This file enumerates the cross-cutting bare-metal deployment runtime
contract as described in the public **DOCA Programming Guide**, the
public **DOCA Installation Guide**, and the public
**BlueField / DPU User Manual** (all reachable through
[`doca-public-knowledge-map`](../doca-public-knowledge-map/SKILL.md)),
plus the standard Linux primitives the bare-metal path inherits
(`numactl(8)`, `taskset(1)`, `systemd.service(5)`,
`systemd.unit(5)`, cgroup-v2, network namespaces). Treat this file
as a *map of what is documented*, not a substitute for reading the
live guides against the operator's target hardware.

## Pattern overview

Every bare-metal-deployment question this skill teaches resolves into
one of SIX patterns. The patterns are CLASSES — they apply across
both supported host modes (host x86, BlueField Arm bare-metal) and
across any DOCA library the binary may have linked against (DOCA-Flow,
DOCA-RDMA, DOCA-DMA, DOCA-Comch, DOCA-GPUNetIO, …), not just one
library.

| Bare-metal deployment pattern | Class shape | Where the substance lives |
| --- | --- | --- |
| 1. Recognise the host mode — host x86 with a remote BlueField NIC over PCIe, OR BlueField Arm bare-metal with the binary on the DPU itself | Where DOCA is installed (host side vs Arm side); which `devlink dev` the operator sees; which representor naming convention is in play; which CPUs the binary may bind to without crossing a NUMA boundary the NIC owns | [`## Capabilities and modes`](#capabilities-and-modes) host-mode table |
| 2. Pick the launch mode — direct foreground, tmux / screen, or systemd-supervised | Direct = interactive debug, foreground stdout/stderr; tmux = long-running with manual reattach; systemd = restart-after-reboot, journald-integrated logs, documented `Restart=` policy | [`## Capabilities and modes`](#capabilities-and-modes) launch-mode table |
| 3. Bind the process to hardware — PF / VF / representor, NUMA node, CPU set, IRQ affinity | The DOCA-linked binary touches a specific PCIe function and a specific NUMA-local memory pool; pinning it correctly is the difference between baseline performance and the symptom *"my app works but throughput is a third of the documented number"* | [`## Capabilities and modes`](#capabilities-and-modes) hardware-binding rules |
| 4. Isolate per tenant — cgroup-v2 cpu / memory / io, network namespaces, `numactl` / `taskset` | Multiple DOCA processes co-tenant on the same BlueField is a supported shape; the isolation primitives are standard Linux, the gotchas (NUMA-locality, hugepage accounting, representor-vs-namespace) are DOCA-specific | [`## Capabilities and modes`](#capabilities-and-modes) isolation rules |
| 5. Map a failure back to its layer | Won't start → exits immediately → can't find device → library error → OOM / signal → restart loop → co-tenant noise; seven layers, each with its own owner | [`## Error taxonomy`](#error-taxonomy) layered split |
| 6. Smoke before bulk — trivial-arg invocation, then liveness equivalent, then real workload | The binary's stdout / journald lines BEFORE the workload moves; same discipline as the container path, applied to a process instead of a pod | [`## Safety policy`](#safety-policy) smoke-before-bulk rule |

Two cross-cutting rules apply to *every* pattern above:

- **Operate the documented path; do not invent one.** PCI addresses,
  representor names, NUMA node numbers, devlink paths, hugepage
  allocation amounts, and systemd `Restart=` mode names all come
  from the public DOCA Programming Guide, the public BlueField /
  DPU User Manual, the Linux man pages, or `--help` on the
  installed tool. Inferring them from generic Linux intuition or
  from a previous host's PCI BDF is the most common hallucination
  failure for this skill.
- **The deployment shape branches early.** Container vs bare metal
  is a fork the operator picks once, in the recognition step
  routed by [`doca-setup`](../doca-setup/SKILL.md); once the
  operator is on the bare-metal path this skill owns the runtime
  contract end-to-end, and the per-library skill layers on top.
  Re-stating the bare-metal runtime inside a per-library skill is
  the failure mode this skill exists to prevent.

## Capabilities and modes

### Host modes — host x86 vs BlueField Arm bare-metal

A DOCA-linked binary on the bare-metal path runs in one of two host
modes. The runtime contract is uniform; the install location,
device-enumeration surface, and representor naming convention
differ.

| Property | Host x86 mode | BlueField Arm bare-metal mode |
| --- | --- | --- |
| Where DOCA is installed | On the host x86 OS, per the public DOCA Installation Guide's host-side install path (reached through [`doca-public-knowledge-map`](../doca-public-knowledge-map/SKILL.md)) | On the BlueField Arm side, per the BlueField OS image's documented DOCA install path |
| Where the binary runs | On the host x86 CPU; calls into DOCA libraries that talk to the BlueField NIC over PCIe | On the BlueField Arm cores; calls into DOCA libraries that talk to the local NIC silicon directly |
| Device enumeration | `devlink dev show` on the host lists the BlueField as a remote NIC; `lspci -d 15b3:` enumerates the BlueField PFs / VFs | `devlink dev show` on the BlueField Arm side lists the local NIC; representor naming follows the documented BlueField OS convention |
| NUMA topology to bind against | The host's NUMA topology; the NUMA node owning the PCIe root complex that the BlueField is plugged into is the load-bearing one | The BlueField Arm's NUMA topology per the BlueField OS image's documentation |
| Hugepage backing | Reserved on the host per [`doca-setup ## configure`](../doca-setup/TASKS.md#configure) step 7 | Reserved on the BlueField Arm side per the BlueField OS image's documented procedure (routed via the BlueField / DPU User Manual reached through [`doca-public-knowledge-map`](../doca-public-knowledge-map/SKILL.md)) |
| Sibling deployment shape | The container path for this mode is *also* host-side (a container running on the host x86 that talks to the BlueField NIC over PCIe); the kubelet-standalone shape covered by [`doca-container-deployment`](../doca-container-deployment/SKILL.md) is the BlueField-Arm analog | The container path for this mode is the kubelet-standalone shape covered by [`doca-container-deployment`](../doca-container-deployment/SKILL.md) — the operator chooses container or bare metal at the recognition step in [`doca-setup`](../doca-setup/SKILL.md), not after the fact |

Operators frequently have BOTH host modes in play (a host x86 with a
DOCA host install AND a BlueField with DOCA on its Arm side). When
that is the case, the agent's rule is to confirm WHICH side the
binary in question is built for (`pkg-config doca-common
--variable=arch` against the install that produced it) and route
the workflow at the matching install — not to assume the host x86
install is the *only* install in scope.

### Launch modes — direct, tmux / screen, systemd-supervised

A DOCA-linked binary on the bare-metal path can be launched in one
of three modes. The choice is made up front in
[`TASKS.md ## configure`](TASKS.md#configure) and feeds the
observability surface and the restart / recovery semantics below.

| Launch mode | When it fits | Observability surface | Restart / recovery posture |
| --- | --- | --- | --- |
| Direct (foreground CLI) | Interactive debug; first-launch smoke; one-shot evaluation against a trivial workload. The shell session owns the process | stdout / stderr arrive on the terminal the operator launched from; closing the shell terminates the process unless the operator backgrounded it | None: a crash exits the shell session; restart is a manual re-invocation. The operator gets full visibility but no resilience |
| tmux / screen (long-running, manual reattach) | Long-running workload where the operator wants the process to outlive the SSH session and be able to reattach to read live stdout / stderr | stdout / stderr arrive on the tmux / screen pane; `tmux attach` / `screen -r` reattaches; output may scroll out of the pane buffer | None beyond what direct gives: a crash leaves the pane with the failed process; restart is still a manual re-invocation. Useful when the operator wants the live terminal surface without a supervisor |
| systemd-supervised | Long-running production workload; restart-after-reboot required; journald-integrated logs and a documented `Restart=` policy are valuable | stdout / stderr captured into journald per the unit's `StandardOutput=` / `StandardError=` settings; `journalctl -u <unit> -f` is the live tail | systemd's `Restart=` policy applies — see the restart / recovery semantics below. The HIGH-STAKES rule in [`## Safety policy`](#safety-policy) explicitly covers the case where the documented `Restart=` mode auto-restarts a process whose underlying failure has not yet been cleared |

The agent does NOT pre-bake sample systemd units, sample tmux
invocations, or sample direct command lines. The launch-mode
decision is the operator's; the agent's job is to surface the trade-
offs and quote the documented mode names (per `systemd.service(5)`
and `systemd.unit(5)`) when the operator picks systemd.

### Hardware-resource binding

A DOCA-linked process is bound to a specific PCIe function (the PF
or VF the DOCA library is documented to attach to), and that
binding implies a specific NUMA-local memory pool and a specific
CPU set that should drive it. Getting these bindings right is the
difference between baseline performance and the canonical *"my
DOCA-Flow app builds and runs but throughput is a third of what
the docs claim"* trap.

The binding surface, named at class level:

- **PF / VF / representor enumeration.** `lspci -d 15b3:` lists
  Mellanox PCIe functions on the local PCI tree;
  `devlink dev show` lists the BlueField devices the kernel sees;
  the documented representor naming convention for the operator's
  BlueField OS image lives in the public BlueField / DPU User
  Manual. The agent does NOT quote a specific BDF (e.g.
  `0000:01:00.0`) or a specific representor name (e.g.
  `pf0vf0`) from memory; both vary per host and per image.
- **NUMA topology discovery.** `numactl --hardware` and
  `lscpu` describe the host's NUMA layout. The load-bearing fact
  for a DOCA process is *"which NUMA node owns the PCIe root
  complex the BlueField is plugged into"*; that is the NUMA node
  the process should pin to, and crossing it costs measurable
  latency and bandwidth. The exact NUMA number is per-host —
  derive it from the live output, do not infer it.
- **CPU pinning rationale.** A DOCA dataplane binary typically
  wants a fixed CPU set for its workers (the *"polling threads"*
  in DPDK-based libraries) and isolated from the kernel
  scheduler's general balancing. `taskset(1)` and `numactl(8)`
  are the standard primitives. The agent quotes the public DOCA
  Programming Guide's documented CPU-pinning guidance for the
  library in use rather than picking a CPU set from generic
  Linux intuition.
- **IRQ affinity rules.** Interrupts from the BlueField PCIe
  function should be steered to the same NUMA node the polling
  threads run on, per the public BlueField / DPU User Manual's
  documented IRQ-affinity guidance. The Linux-side mechanism is
  `/proc/irq/<n>/smp_affinity`; the rule is to mirror the
  CPU-pinning choice on the IRQ side, not to invent a new IRQ
  mask.

### Per-tenant isolation

Multiple DOCA processes co-tenant on the same BlueField is a
supported shape (e.g. one DOCA-Flow process per representor plus
one DOCA-RDMA process for storage). The isolation primitives are
standard Linux; the DOCA-specific gotchas are what this skill owns.

- **cgroup-v2 cpu / memory / io controllers.** Per-tenant cpu
  shares (or hard caps), memory limits, and io weights enforce
  the resource split. The Linux-side mechanics live in the
  kernel cgroup-v2 documentation; the DOCA-specific gotcha is
  that a DOCA process's hugepage backing is accounted against the
  cgroup's memory budget — a `memory.max` value that fits the
  binary's resident set but starves the hugepage pool surfaces
  as a startup failure that LOOKS like a config error.
- **Network namespaces.** A per-tenant netns is the standard
  Linux mechanism for traffic isolation. The DOCA-specific
  gotcha is that a representor moved into a tenant netns has
  documented enumeration semantics — the representor is only
  visible to DOCA inside the netns that owns it, and a DOCA
  library called from a process in a different netns will not
  see it. The agent walks the operator through which netns owns
  which representor; it does not guess.
- **`numactl` / `taskset` per-tenant binding.** Each tenant's
  process gets its own CPU set bound via `numactl
  --cpunodebind` + `--membind` (NUMA-local memory) or
  `taskset -c <cpu-list>` (CPU-list only). The DOCA-specific
  gotcha is the rule named under [hardware-resource
  binding](#hardware-resource-binding): the chosen CPU set must
  not cross the NUMA boundary the NIC owns, or the per-tenant
  isolation gain is canceled by a NUMA-crossing latency hit.

The agent's discipline: confirm the tenant-isolation primitives
are in place BEFORE the workload starts, not after a co-tenant
complains. The corresponding workflow step lives at
[`TASKS.md ### isolation`](TASKS.md#isolation).

### Restart / recovery semantics

How a failed bare-metal process recovers depends on the launch mode
picked at [`TASKS.md ## configure`](TASKS.md#configure):

- **Direct / tmux launch.** No supervisor. A crash terminates the
  process; restart is a manual re-invocation. This is fine for
  interactive debug; it is the WRONG mode for a production
  workload where the operator expects the binary to come back
  after a host reboot.
- **systemd-supervised launch.** systemd's `Restart=` modes
  (documented in `systemd.service(5)`) govern auto-restart on
  exit. The mode names the operator may choose are the ones
  listed in `systemd.service(5)`; the agent does NOT invent
  modes (e.g. `Restart=on-failure-with-burst-cap` is not a real
  mode, but is the canonical class-of-thing the agent
  hallucinates when working from memory).
- **Crash-and-investigate vs supervisor-driven restart.** The
  cross-cutting rule (from
  [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md)
  via [`## Safety policy`](#safety-policy) below) is that a
  failed bare-metal DOCA process that touched the device is
  HIGH-STAKES — the root cause MUST be cleared before the
  operator allows the supervisor to restart-loop the process.
  The auto-restart is the right default for a transient failure
  and the WRONG default for a recurring one.

## Version compatibility

For the canonical DOCA version-detection chain, the four-way match
rule (host package ↔ binary build ↔ BlueField firmware ↔
DOCA-version policy), and the headers-win-over-docs rule, see
[`doca-version`](../doca-version/SKILL.md). The body lives there;
this skill does not duplicate it.

**The bare-metal-deployment-specific overlay** is:

- **The binary's link-time `pkg-config doca-*` version MUST match
  the runtime `LD_LIBRARY_PATH`'d install.** A DOCA-linked binary
  resolves DOCA symbols against whatever `libdoca_*.so` is on the
  runtime `LD_LIBRARY_PATH` (or the default loader search path).
  If the binary was built against (for example) `doca-common 3.3.x`
  headers but the runtime install is `doca-common 3.2.x` (or vice
  versa), the binary fails at the first symbol that moved
  between releases — sometimes immediately at `dlopen` time,
  sometimes at the first call site, sometimes silently with
  wrong-behavior. Diagnose with
  [`doca-version TASKS.md ## debug`](../doca-version/TASKS.md#debug)
  layer 2.
- **`LD_LIBRARY_PATH` vs runtime-loader configuration.** When the
  operator picks the trace flavor at runtime
  (`/opt/mellanox/doca/lib/<arch>-linux-gnu/trace/`) per
  [`doca-setup ## configure`](../doca-setup/TASKS.md#configure)
  step 3, the runtime `LD_LIBRARY_PATH` is the contract. A
  systemd unit that does not propagate `LD_LIBRARY_PATH` into
  the unit's `Environment=` will silently use the release
  flavor and the operator's *"I'm running the trace build"*
  assumption is false.
- **Host x86 install version and BlueField device anchors are
  BOTH version anchors — and the device side splits into TWO
  distinct legs.** Same four-way overlay every per-library skill
  carries. The host-side `pkg-config --modversion doca-common`
  is the binary's link / runtime anchor; the device side splits
  into (i) NIC *firmware version* read via `flint -d <bdf> q`
  (look for the `FW Version:` line — this is the firmware image
  the silicon runs) and (ii) BFB-image DOCA version read via
  `bfver` + `cat /etc/mlnx-release` on the BlueField Arm
  console (this is the DOCA userland inside the BFB). Do NOT
  substitute `mlxconfig -d <bdf> q` for the FW-version leg —
  `mlxconfig` returns the firmware *configuration* dump
  (NV-config toggles like `LINK_TYPE_P1`, `INTERNAL_CPU_MODEL`),
  NOT the firmware version. Do NOT substitute `mlxprivhost`
  (configures privileged-host mode, not BFB version) or
  `bfb-info` (not a real NVIDIA-documented tool) for the
  BFB-image leg — both are common hallucinations explicitly
  banned in [`doca-version CAPABILITIES.md ## Capabilities and
  modes`](../doca-version/CAPABILITIES.md#capabilities-and-modes).
  Mismatched anchors are the canonical *"the docs say this
  should work but it does not"* failure mode. Capture all three
  (host pkg-config, per-device `flint q`, per-BlueField
  `bfver`) before debugging — and capture per-device, not
  globally, because BF2 and BF3 (or two BF3s from different
  procurement waves) on the same host are independent silicon
  with independent FW levels.

## Host-side DOCA upgrade workflow

When the operator is moving an already-running host from one DOCA
release to another (the canonical case: 3.1 → 3.3), the agent walks
THIS ordered ladder before quoting any `apt install` line. The
host-side portion is fully in-scope for this skill; the **BlueField
BSP / BFB / RShim / TMFIFO portion is OUT OF SCOPE** for this
bundle (per [`AGENTS.md ## Non-goals`](../../AGENTS.md#non-goals-questions-the-agent-should-recognize-and-refuse-politely)
item 7) and the agent routes the operator to NVIDIA's BlueField
BSP / DOCA Platform Framework documentation for those steps.

1. **Audit the host's current four-source state.** Per
   [`doca-version CAPABILITIES.md ## Version compatibility`](../doca-version/CAPABILITIES.md#version-compatibility),
   capture all four legs. The audit is read-only and is the rollback
   anchor — without a captured BEFORE state, "rollback" is a phrase
   the operator says, not a thing they can do.
2. **Verify the target release line.** Pin to a specific DOCA `X.Y.Z`
   from NVIDIA's public release notes (route via
   [`doca-public-knowledge-map`](../doca-public-knowledge-map/SKILL.md)
   — do NOT invent version strings). Read off the target's required
   OFED / kernel module version, the target BFB-image DOCA, and the
   target NIC firmware level from the release-notes *Supported NICs
   and Firmware* table.
3. **Walk the apt-repo + OS-matrix preconditions BEFORE any install
   line.** See
   [`doca-version CAPABILITIES.md ## Apt-repo and OS-matrix preconditions`](../doca-version/CAPABILITIES.md#apt-repo-and-os-matrix-preconditions).
   Pin the apt repo to the explicit `X.Y.Z` URL form, NOT to `latest`.
   Run `apt-cache policy <pkg>` for every package about to be
   installed. Confirm the host OS family AND point release is inside
   the target's supported sub-range — if it's outside, STOP and
   surface to the operator, do not silently proceed.
4. **Host-side upgrade is the first mutating step.** Quote the
   release-pinned `apt install` / `apt upgrade` line. Do NOT issue
   `apt install doca-all` reflexively; prefer the granular per-
   package list per the four-source partial-install table in
   `doca-version` so absent-source recovery is a single targeted
   install. Do NOT auto-reboot — let the operator schedule the
   reboot inside the documented maintenance window per
   [`doca-hardware-safety CAPABILITIES.md ## Safety policy`](../doca-hardware-safety/CAPABILITIES.md#safety-policy).
5. **Reboot the host, then re-walk the four-source audit AFTER.**
   The host-side upgrade is not declared "done" until the AFTER
   audit shows (i) all four sources coherent at the target version,
   (ii) `mlx5_core` / OFED at the target level (`modinfo mlx5_core
   | grep ^version`), (iii) all host-side host-PF / representor
   PCIe devices still enumerated (`lspci -d 15b3:`), (iv) host-
   visible NIC firmware version matches the per-device target from
   the release notes (`flint -d <bdf> q`).
6. **Handoff to BFB install — external tooling, in-scope sequencing.**
   Once the host is green, the BFB-side install (push BFB to BlueField via
   RShim, eMMC image install, BFB-side firmware update, BFB-side OS bring-
   up to `Linux up` / `DPU is ready`, TMFIFO recovery, BFB-side apt
   vocabulary, `bf.cfg`) is **externally-productized** (BlueField BSP layer
   plus, for fleet-scale deployments, DOCA Platform Framework). The
   *operational sequencing ladder* for the single-host BFB lifecycle
   itself — what evidence to capture, in which order, and which recovery
   action each evidence pattern lattices to — is in scope and lives in
   [`TASKS.md ## bluefield-lifecycle`](TASKS.md#bluefield-lifecycle); the
   *productized framework / BSP / BFB image / firmware tooling* is out
   of scope for this bundle's strict-1:1 monorepo alignment, but the
   bundle's `AGENTS.md ## Non-goals #7` contract still applies: the agent
   MUST produce the three-part response shape (recognize + name boundary
   + **route with substance**). For step (c), consult the per-product
   rows in the `doca-public-knowledge-map` routing table:
   [BlueField BSP / BFB / `bfb-install` / RShim / TMFIFO / `bf.cfg`](../doca-public-knowledge-map/SKILL.md#externally-productized-doca-software--not-in-this-bundle-but-here-is-where-to-route)
   for the single-host case, and [DOCA Platform Framework (DPF)](../doca-public-knowledge-map/SKILL.md#externally-productized-doca-software--not-in-this-bundle-but-here-is-where-to-route)
   for fleet-scale K8s-driven DPU provisioning. If the BFB-install symptom
   touches firmware state (`mlxconfig` / `flint` / `mlxfwmanager`), also
   load the [NVIDIA Firmware Tools (MFT)](../doca-public-knowledge-map/SKILL.md#externally-productized-doca-software--not-in-this-bundle-but-here-is-where-to-route)
   row. If the BFB-install left the BlueField unreachable, the recovery
   path is the [BlueField BMC](../doca-public-knowledge-map/SKILL.md#externally-productized-doca-software--not-in-this-bundle-but-here-is-where-to-route)
   row. The agent **must** name the symptom-matching gotcha class from the
   relevant row (e.g. "`bfb-install` exit 0 with `NIC firmware update failed`
   is the BSP row's #1 partial-failure signature — bisect with `flint -d
   <bdf> q`") rather than handing the user a bare URL. The agent does NOT
   synthesize BFB install / RShim / TMFIFO / `bf.cfg` mechanics from
   training memory.
7. **End-to-end success contract.** A DOCA host-side upgrade is
   "done" only when, AFTER the BFB-side portion completes
   independently, all of the following hold:
   - Host four-source audit at target version, coherent.
   - BlueField-side four-source audit at target version, coherent.
   - Per-device `flint -d <bdf> q` FW Version on each BF / NIC matches
     the release-notes target.
   - `doca_caps --list-devs` enumerates every expected device with
     expected capabilities.
   - The operator's own canonical smoke (a doca-flow / doca-rdma /
     doca-comch sample) runs end-to-end against the target install.

The discipline mirrors the universal verification contract end-to-
end: every step's preconditions established by the prior steps, no
"upgrade complete" claimed on exit-code alone, the BFB / BSP
boundary explicitly named and routed out rather than silently
synthesized. See [`AGENTS.md ## The universal verification contract`](../../AGENTS.md#the-universal-verification-contract).

### RShim instance ↔ BlueField disambiguation (canonical one-liner)

Multi-DPU hosts expose multiple RShim character devices under
`/dev/rshim<N>/`. To map each RShim instance to a specific
BlueField (so `bfb-install --rshim /dev/rshim<N>` targets the
right DPU, and `cat /dev/rshim<N>/misc` reads the right
console), read the `DEV_NAME` field of each `misc` file:

```bash
for r in /dev/rshim*/misc; do
  echo "== $r =="
  grep -E '^DEV_NAME' "$r"
done
```

`DEV_NAME` returns the PCIe BDF that RShim instance is attached
to (e.g. `DEV_NAME pcie-0000:03:00.2`); cross-match against
`lspci -d 15b3: -nn` to identify which physical BlueField is on
which `/dev/rshim<N>`. This one-liner is the canonical
disambiguation step before any per-DPU `bfb-install`, `rshim`
config edit, or console capture on a multi-DPU host; skipping it
is the #1 cause of *"I flashed the wrong DPU"* incidents. The
mapping is consumed downstream by every step in
[`TASKS.md ## bluefield-lifecycle`](TASKS.md#bluefield-lifecycle)
that takes a `--rshim` or `/dev/rshim<N>` argument.

## Error taxonomy

Bare-metal-deployment errors fall into SEVEN layers, each with its
own owner. The agent walks the layers in this order; conflating
them wastes debug time and blames the wrong layer.

1. **Process won't start at all.** Symptoms: the shell reports
   *"command not found"*, *"permission denied"*, *"cannot
   execute binary file: Exec format error"*, or a missing
   shared object error from the dynamic loader (`error while
   loading shared libraries: libdoca_*.so: cannot open shared
   object file`). Causes: binary not on `PATH` or not executable
   (`chmod +x`); binary built for a different arch than the
   host (x86 binary on the BlueField Arm or vice versa);
   `LD_LIBRARY_PATH` not set so the loader cannot find
   `libdoca_*.so`. Resolution: confirm the binary's arch
   (`file <binary>`) matches the host's arch (`uname -m`);
   confirm the binary's required shared objects are reachable
   (`ldd <binary>`); set `LD_LIBRARY_PATH` per
   [`doca-setup ## configure`](../doca-setup/TASKS.md#configure)
   step 3. Owner: this skill + [`doca-setup`](../doca-setup/SKILL.md).
2. **Process starts but exits immediately.** Symptoms: the
   process runs for milliseconds, exits with a non-zero status,
   prints a short message about a missing config file, a
   missing environment variable, or `EAL: Cannot get hugepage
   information` from DPDK. Causes: a required env var (e.g.
   `DOCA_LOG_LEVEL`, a per-library env knob documented in the
   public DOCA Programming Guide) is unset; the config file the
   binary expects is missing or at the wrong path; hugepages
   are not reserved per [`doca-setup ## configure`](../doca-setup/TASKS.md#configure)
   step 4; the binary's argp surface rejected the operator's CLI
   args. Resolution: read the binary's stdout / stderr from
   the launch mode in use (direct = terminal; tmux = pane;
   systemd = `journalctl -u <unit>`); cross-check the
   documented env vars in the public DOCA Programming Guide;
   re-run hugepage reservation. Owner: this skill + the
   per-library skill (for library-specific env vars).
3. **Process runs but cannot find the device.** Symptoms: the
   binary stays up but the per-library bring-up reports *"no
   matching device"*, *"representor not found"*, or
   `DOCA_ERROR_NOT_FOUND` from the documented device-open call.
   Causes: PCI address wrong (the operator passed a BDF that
   does not exist or does not match the BlueField); representor
   not enumerated by the kernel (the eswitch is not in
   `switchdev` mode); devlink mode wrong; the binary is in a
   netns that does not own the representor. Resolution: confirm
   the device shape from `devlink dev show` and `lspci -d
   15b3:` per [`doca-setup ## configure`](../doca-setup/TASKS.md#configure)
   step 5; re-read the public DOCA Programming Guide's
   device-open contract for the library in use; do NOT
   substitute a BDF from memory. Owner: this skill +
   [`doca-setup`](../doca-setup/SKILL.md).
4. **Process attaches to the device but the workload errors.**
   Symptoms: the binary stays up, the device is open, the
   per-library bring-up looks clean, but a runtime call returns
   `DOCA_ERROR_*` per the cross-library taxonomy in
   [`doca-programming-guide CAPABILITIES.md ## Error taxonomy`](../doca-programming-guide/CAPABILITIES.md#error-taxonomy).
   Causes: per-library configuration error (a pipe / queue /
   buffer the per-library skill is the authority on); a
   capability the operator's BlueField does not have at the
   running firmware version. Resolution: defer to
   [`doca-debug`](../doca-debug/SKILL.md) for the cross-cutting
   ladder, then to the matching `libs/<library>` skill for the
   library-specific debug overlay. Owner: the per-library skill
   + [`doca-debug`](../doca-debug/SKILL.md).
5. **Process runs correctly but OOMs / gets killed /
   signal-mishandled.** Symptoms: the binary disappears with
   no log line; `dmesg` shows an OOM-killer entry naming the
   binary; the supervisor reports the process exited with
   `SIGKILL` / `SIGTERM`; counters reset because the process
   restarted. Causes: the cgroup-v2 `memory.max` is too small
   for the binary's working set plus hugepage backing; the
   host is over-committed; the binary does not handle `SIGTERM`
   from the supervisor cleanly; NUMA imbalance forces a
   memory blowup on one node. Resolution: read the supervisor's
   exit-status / signal record; check `dmesg` for OOM
   evidence; re-check cgroup limits per
   [`### isolation`](TASKS.md#isolation); confirm the binary's
   documented signal-handling contract from the public DOCA
   Programming Guide. Owner: this skill + the operator's host
   OS team.
6. **Process is in a restart loop.** Symptoms: the supervisor
   keeps re-launching the binary; each launch exits with the
   same exit signature; the device the binary touches is
   reporting odd errors that may be caused by the loop itself.
   Causes: a real underlying failure (any of layers 1-5) that
   the supervisor's `Restart=` policy is hiding behind
   automatic re-launches. Resolution: STOP the supervisor (per
   the HIGH-STAKES rule in [`## Safety policy`](#safety-policy));
   read the binary's LAST full log; walk the taxonomy from
   layer 1 against the captured evidence; only re-enable the
   supervisor once the root cause is identified. Owner: this
   skill (the cross-cutting rule) + whichever layer the root
   cause turns out to live in.
7. **Co-tenant noise — another bare-metal process on the same
   NUMA / same NIC is interfering.** Symptoms: the binary
   behaves correctly in isolation; introducing a second DOCA
   process (or any other process) on the same BlueField makes
   the first one's per-library counters degrade, latency
   climb, or throughput collapse; the symptom does NOT
   reproduce on a quiet host. Causes: cgroup-v2 limits not in
   place; CPU pinning not in place; the second process is
   cross-NUMA against the BlueField's PCIe root complex; the
   second process is competing for the same hugepage pool.
   Resolution: confirm tenant-isolation primitives per
   [`### isolation`](TASKS.md#isolation) BEFORE introducing
   the second tenant, not after; if the symptom only appears
   under co-tenancy, the diagnosis is multi-tenant, not
   per-library. Owner: this skill.

DOCA library calls inside the binary return `DOCA_ERROR_*`
according to the cross-library taxonomy in
[`doca-programming-guide CAPABILITIES.md ## Error taxonomy`](../doca-programming-guide/CAPABILITIES.md#error-taxonomy);
that taxonomy becomes relevant at layer 4 above and is owned by
[`doca-debug`](../doca-debug/SKILL.md) plus the matching per-library
skill, not by this one.

## Observability

Documented observability surfaces, in the order the agent reaches
for them. Three layers, each tied to a launch mode and a documented
host-side primitive. Healthy means all three agree.

- **Process-output layer (FIRST).** The binary's own stdout /
  stderr is the first place to look. Where it arrives depends on
  the launch mode: direct = the terminal the operator launched
  from; tmux / screen = the pane buffer (reattach with `tmux
  attach` / `screen -r`; output may have scrolled off if the
  pane buffer is small); systemd-supervised = captured into
  journald per the unit's `StandardOutput=` / `StandardError=`
  settings, tailed with `journalctl -u <unit> -f` (the unit name
  is the operator's choice and lives in their systemd policy).
  The agent does NOT invent a journald query against an
  unnamed unit; the unit name comes from the operator's
  deployment, not from memory.
- **Device-state introspection layer (SECOND).** The host-side
  picture of the device the binary is touching: `devlink dev
  show` lists the BlueField PCIe device; `ip link show` lists
  the device's netdevs and any representors; `mlxconfig -d
  <pci> query` is the QUERY-ONLY surface for the device's
  firmware configuration. The agent uses `mlxconfig query`
  freely as an observability primitive but treats `mlxconfig
  set` as a hardware-state change — it MUST defer to
  [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md)
  before recommending any `mlxconfig set` invocation. The
  documented query commands the agent may quote live in the
  public BlueField / DPU User Manual reached through
  [`doca-public-knowledge-map`](../doca-public-knowledge-map/SKILL.md).
- **Per-tenant resource visibility layer (THIRD, load-bearing
  for multi-tenant).** When the binary is one of several DOCA
  processes on the same BlueField, the per-tenant resource
  picture is what tells the operator whether the tenant
  isolation primitives are still in effect. cgroup-v2 reports
  `cpu.stat` / `memory.stat` / `io.stat` per cgroup;
  `numactl --hardware` reports the NUMA topology and per-node
  memory pressure; `perf stat -p <pid>` reports per-process
  CPU / cache behavior. The agent quotes these tools at class
  level; the exact metric names live in the Linux man pages
  and in the kernel cgroup-v2 documentation.

Cross-cutting host-side debug (kernel version, driver loaded /
not loaded, hugepage allocation health beyond reserved-vs-used,
PCIe link state, `dmesg`) lives in
[`doca-debug TASKS.md ## debug`](../doca-debug/TASKS.md#debug)
and [`doca-setup TASKS.md ## Command appendix`](../doca-setup/TASKS.md#command-appendix);
this skill names only the bare-metal-deployment-specific
surfaces.

## Safety policy

> **Overlay on the bundle-wide hardware-safety meta-policy.** The
> rules below are this skill's per-artifact overlay on the
> cross-cutting rules in
> [`doca-hardware-safety` CAPABILITIES.md ## Safety policy](../doca-hardware-safety/CAPABILITIES.md#safety-policy)
> (specifically
> [### Per-artifact overlay pattern](../doca-hardware-safety/CAPABILITIES.md#per-artifact-overlay-pattern)).
> When the two layers disagree, the stricter wins; when either
> layer says STOP, the agent stops.

The cross-cutting safety surface for any DOCA-linked binary
deployed bare-metal. Per-library skills add their own overlays
(e.g. doca-flow's pipe-validate-before-commit rule); the
cross-cutting rules below apply across every library.

- **Smoke before bulk (load-bearing).** Before any real workload
  touches the device, the agent walks the smoke sequence in
  [`TASKS.md ## test`](TASKS.md#test): (a) trivial-arg
  invocation (the binary's `--help` or `--version` equivalent,
  proving the binary executes and the loader resolves DOCA
  symbols); (b) liveness-equivalent invocation (the binary
  starts, opens the device, prints its documented bring-up
  lines, exits cleanly on `SIGTERM`); (c) trivial-workload
  invocation (one packet / one operation through the binary,
  per the library's documented liveness contract); only then is
  the binary ready for production workload. Skipping this and
  going straight to bulk is the most common reason *"the binary
  works on the bench but blows up under load"*.
- **Failed bare-metal process is HIGH-STAKES — clear the root
  cause BEFORE restarting.** A bare-metal DOCA process that
  touched the device and then failed has potentially left the
  device in a documented-but-not-guaranteed-clean state. The
  operator's rule: do NOT let a supervisor's `Restart=` policy
  re-launch the binary while the underlying failure is
  uncleared. A binary in an auto-restart loop that keeps
  touching the device burns BlueField cycles, fills the log
  surface, and obscures the actual root cause. This rule is
  the bare-metal analog of the failed-pod-restart-is-HIGH-STAKES
  rule in [`doca-container-deployment`](../doca-container-deployment/SKILL.md);
  it applies just as strongly to systemd-supervised binaries.
- **Do not invent PCI addresses, NUMA node numbers, representor
  names, devlink paths, hugepage allocation amounts, or
  `Restart=` mode names.** All of these are per-host or
  per-image specifics. The PCI BDF the binary should attach to
  lives in `lspci -d 15b3:` output on the target host; the NUMA
  topology lives in `numactl --hardware`; the representor
  naming convention lives in the public BlueField / DPU User
  Manual for the operator's BlueField OS image; the `Restart=`
  mode names live in `systemd.service(5)`. Quote from these
  live sources; do NOT supply a BDF, a representor name, or a
  systemd mode name from memory. This is the load-bearing
  first-run failure mode for this skill — the operator pastes
  the agent's BDF into a launch line, the binary attaches to
  the wrong function (or to no function), and the resulting
  debug spiral is hours long.
- **Confirm tenant-isolation primitives BEFORE the workload
  starts, not after a co-tenant complains.** Per
  [`### isolation`](TASKS.md#isolation), the cgroup-v2 / netns /
  `numactl` configuration must be in place when the first
  tenant launches. Adding isolation after a co-tenant is
  already misbehaving means the first symptom the operator
  sees is co-tenant noise from layer 7 of the error taxonomy —
  which looks like *"the binary regressed"* and burns time on
  the wrong diagnosis.
- **Hardware-state changes leave this skill.** Any change that
  touches device firmware (`mlxconfig set`, BFB reflash,
  firmware burn, BlueField mode flip, kernel-boot-parameter
  changes for IOMMU or hugepage reservation) is OUT OF SCOPE
  for this skill. The agent MUST hand off to
  [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md) for
  the change-application discipline and only return here once
  the hardware-state change is complete.
- **The per-library skill owns the per-library safety overlay.**
  Doca-flow's pipe-validate-before-commit rule, doca-rdma's
  queue-pair-state-machine rule, doca-comch's
  channel-handshake-discipline rule, and similar per-library
  safety rules are owned by the matching per-library skill,
  not by this one. The agent reads both layers in parallel and
  applies them; this skill names the cross-cutting baseline.

## Public-source pointer

The canonical public sources for the bare-metal deployment runtime
are:

- The **DOCA Programming Guide** on `docs.nvidia.com`, reachable
  through
  [`doca-public-knowledge-map ## Library- and module-specific guides`](../doca-public-knowledge-map/SKILL.md#library--and-module-specific-guides),
  for the per-library device-open / capability / runtime
  contracts.
- The **DOCA Installation Guide** on `docs.nvidia.com`, reachable
  through
  [`doca-public-knowledge-map ## Public documentation entry points`](../doca-public-knowledge-map/SKILL.md#public-documentation-entry-points),
  for the host-side and BlueField-Arm-side install layouts plus
  the documented env-var surface.
- The **BlueField / DPU User Manual** on `docs.nvidia.com`,
  reachable through the same routing table, for the documented
  device-enumeration surface, representor-naming convention,
  IRQ-affinity guidance, and firmware-query surface.
- The Linux man pages for `numactl(8)`, `taskset(1)`,
  `systemd.service(5)`, `systemd.unit(5)`, and the kernel
  cgroup-v2 documentation, for the standard primitives this
  skill inherits.

Verify that the version of each guide matches the host's DOCA
install version, the BlueField's firmware version, and the
binary's link-time version per
[`## Version compatibility`](#version-compatibility) — flag names,
documented device-enumeration commands, and `Restart=` modes
evolve, so anything quoted from memory is suspect.

## Description: <br>
Use this skill for launching, supervising, debugging, or platform lifecycle on a BlueField — BFB install, RShim/TMFIFO, host PF rebind, post-BFB recovery — taking a DOCA-linked binary to a healthy run directly on hardware (host x86 + BlueField NIC over PCIe, or BlueField Arm bare-metal). <br>

This skill is ready for commercial/non-commercial use. <br>

## Owner
NVIDIA <br>

### License/Terms of Use: <br>
Apache 2.0 AND CC-BY-4.0 <br>
## Use Case: <br>
Developers and operators who have a DOCA-linked application binary and want to run it directly on hardware — launching, binding to PCI/NUMA/CPU/IRQ resources, configuring co-tenant isolation, diagnosing bare-metal failures, and managing BlueField lifecycle operations. <br>

### Deployment Geography for Use: <br>
Global <br>

## Requirements / Dependencies: <br>
**Requires API Key or External Credential:** [No] <br>
**Credential Type(s):** [None] <br>

Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. <br>

## Known Risks and Mitigations: <br>
Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br>
Mitigation: Review and scan skill before deployment. <br>

## Reference(s): <br>
- [details.md](references/details.md) <br>
- [NVIDIA DOCA SDK Documentation](https://docs.nvidia.com/doca/sdk/index.html) <br>
- [DOCA Samples](https://github.com/NVIDIA-DOCA/doca-samples) <br>
- [DOCA Platform Framework](https://github.com/NVIDIA/doca-platform) <br>


## Skill Output: <br>
**Output Type(s):** [Shell commands, Configuration instructions, Analysis] <br>
**Output Format:** [Markdown with inline bash code blocks] <br>
**Output Parameters:** [1D] <br>
**Other Properties Related to Output:** [None] <br>

## Evaluation Agents Used: <br>
- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`) <br>
- Codex (`openai/openai/gpt-5.5`) <br>



## Evaluation Tasks: <br>
Evaluated against 4 evaluation tasks (3 positive skill-activation, 1 negative). <br>

## Evaluation Metrics Used: <br>
Reported benchmark dimensions: <br>
- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
- Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>

Underlying evaluation signals used in this run: <br>
- `security`: Checks for unsafe operations, secret leakage, and unauthorized access. <br>
- `skill_execution`: Verifies that the agent loaded the expected skill and workflow. <br>
- `skill_efficiency`: Checks routing quality, decoy avoidance, and redundant tool usage. <br>
- `accuracy`: Grades final-answer correctness against the reference answer. <br>
- `goal_accuracy`: Checks whether the overall user task completed successfully. <br>
- `behavior_check`: Verifies expected behavior steps, including safety expectations. <br>



## Evaluation Results: <br>
| Dimension | Num | Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`) | Codex (`openai/openai/gpt-5.5`) |
|---|---:|---:|---:|
| Security | 4 | 100% (+0%) | 100% (+0%) |
| Correctness | 4 | 100% (+35%) | 75% (+10%) |
| Discoverability | 4 | 100% (+50%) | 95% (+45%) |
| Effectiveness | 4 | 92% (+58%) | 100% (+57%) |
| Efficiency | 4 | 94% (+56%) | 100% (+75%) |

## Skill Version(s): <br>
ffe362c (source: git SHA, committed 2026-07-25) <br>

## Ethical Considerations: <br>
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

(For Release on NVIDIA Platforms Only) <br>
Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). <br>

---
license: Apache-2.0
name: doca-bare-metal-deployment
description: >
  Use this skill for launching, supervising, debugging, OR
  platform lifecycle on a BlueField — BFB install, RShim/TMFIFO,
  host PF rebind, post-BFB recovery — taking a DOCA-linked binary
  to a healthy run directly on hardware (host x86 + BlueField NIC
  over PCIe, or BlueField Arm bare-metal). No container, no
  kubelet. Covers launch mode (direct, tmux, systemd), PCI/NUMA/
  CPU/IRQ binding, co-tenant isolation (cgroup-v2/netns/numactl),
  a seven-layer error taxonomy, and a six-state BlueField
  lifecycle classifier. Trigger even when user does not say
  "bare-metal" — implicit phrasings include "binary exits 1 right
  after launch", "systemd keeps restarting it", "no matching
  device on the BF", "bfb-install exited 0 but DPU is dead",
  "ping 192.168.100.2 works but ssh fails", "host PFs aren't
  showing netdevs". Destructive firmware burn / mlxconfig set
  requires explicit confirmation via doca-hardware-safety;
  containers, library APIs, env prep, and build use other skills.
metadata:
  kind: library
compatibility: >
  No DOCA install required to read this skill (it is an overlay
  loaded against any DOCA artifact skill); the validation steps
  within this skill require a live DOCA install at /opt/mellanox/doca on
  a host or BlueField with a built DOCA-linked binary.
---

# DOCA bare-metal deployment

**Where to start:** This skill is the bundle's home for *operating*
a DOCA-linked application binary **directly on hardware** — no
container, no kubelet, no static-pod manifest. It is the parallel
of [`doca-container-deployment`](../doca-container-deployment/SKILL.md)
for the non-container path. If the user has a DOCA-linked binary
they built (per the canonical workflow in
[`doca-programming-guide`](../doca-programming-guide/SKILL.md))
and they want to know *how to actually run it on the host or on
the BlueField Arm cores correctly*, open
[`TASKS.md`](TASKS.md) and start at
[`## configure`](TASKS.md#configure). If the question is *what
shape does the bare-metal runtime even have and what is the
deployment contract*, start at [`CAPABILITIES.md`](CAPABILITIES.md).
If the user is not yet sure whether their target system shape is
the container path or the bare-metal path, route the recognition
step to [`doca-setup`](../doca-setup/SKILL.md) first; only return
here once *bare-metal* is the confirmed shape.

## Audience

This skill serves **external DOCA developers and operators who
have a DOCA-linked application binary they built and want to run
it directly on hardware** — i.e., people who already have:

- a DOCA-linked application binary they built per
  [`doca-programming-guide ## build`](../doca-programming-guide/TASKS.md#build),
- a real BlueField NIC and a host that talks to it (the **host
  x86** path — DOCA host install on the host talks to the
  BlueField NIC over PCIe), OR a BlueField with a console or SSH
  to the Arm side (the **BlueField Arm bare-metal** path — DOCA
  installed on the DPU Arm cores; the binary runs there
  directly), and
- a desire to RUN that binary directly on the hardware, not
  inside a kubelet-standalone-managed container.

It is **not** for:

- kernel-driver developers contributing to `mlx5_*` or the
  BlueField OS,
- DOCA library contributors (those changes go to the internal
  DOCA tree, not to a bare-metal deployment),
- full-Kubernetes-cluster operators managing a fleet of
  BlueFields (the bundle covers
  [`doca-container-deployment`](../doca-container-deployment/SKILL.md)
  for the single-host kubelet-standalone shape; **fleet/production-scale
  deployment is fleet-orchestration scope** — route to the orchestration
  entry-point in
  [`doca-public-knowledge-map ## Deploying DOCA services at scale`](../doca-public-knowledge-map/references/map.md#deploying-doca-services-at-scale--orchestration-entry-point-personascale-routing)
  (DPF / Network Operator / Launch Kit), not hand-rolled static-pod loops),
- fresh-laptop-no-hardware users with no DOCA install yet — those
  belong on
  [`doca-setup ## no-install`](../doca-setup/TASKS.md#no-install).

The skill teaches the agent the bare-metal-deployment *procedure*
and the rules for quoting documented commands from the public DOCA
Programming Guide and the public BlueField / DPU User Manual via
[`doca-public-knowledge-map`](../doca-public-knowledge-map/SKILL.md);
it does not invent flag names, PCI BDFs, NUMA numbers, devlink
paths, representor strings, or systemd `Restart=` mode names from
memory.

## When to load this skill

Load this skill when the user is doing **hands-on bare-metal
deployment of a DOCA-linked application binary** on either of the
two supported host modes (host x86 or BlueField Arm), or asking a
cross-cutting bare-metal question that is not specific to one
library's API. Concretely:

- Launching a DOCA-linked binary for the first time on a host
  with a BlueField NIC in a PCIe slot, with DOCA installed on the
  host.
- Launching a DOCA-linked binary on the BlueField Arm cores
  directly (BlueField Arm bare-metal mode), with DOCA installed
  on the Arm side per the BlueField OS image.
- Deciding which launch mode to use (direct foreground for
  interactive debug; tmux/screen for long-running with manual
  reattach; systemd-supervised for restart-after-reboot,
  journald-integrated logs, and Restart= policy).
- Binding the DOCA process to the right PCIe function, the right
  representor, the right NUMA node, and the right CPU set — and
  pinning IRQs to match — without inventing the addresses or the
  flag names.
- Setting up per-tenant isolation (cgroup-v2 cpu / memory / io
  controllers, network namespaces for multi-tenant deployments,
  `numactl` / `taskset` for CPU + NUMA binding) so multiple DOCA
  processes co-tenant on the same BlueField without crushing each
  other.
- Diagnosing a bare-metal launch that is misbehaving — won't
  start, starts and exits immediately, runs but can't find the
  device, attaches to the device but the workload errors, OOMs or
  is signal-killed, is in a restart loop under a supervisor, or
  is being interfered with by a co-tenant.
- Cross-cutting questions: *"should I run this in tmux or as a
  systemd unit"*, *"what is the smoke-before-bulk loop for a
  binary on bare metal"*, *"my binary works in a container on the
  BlueField but not when I run it directly on the Arm — what
  changed"*.

Do **not** load this skill for the container-path equivalent
(those questions go to
[`doca-container-deployment`](../doca-container-deployment/SKILL.md));
for full-Kubernetes-cluster operations (out of scope per the
bundle's non-goals); for library-API questions (route to the
matching `libs/<library>` skill); for env-preparation questions
including hugepages, IOMMU, pkg-config, and devlink mode flips
(use [`doca-setup`](../doca-setup/SKILL.md)); for any
hardware-state-changing operation including `mlxconfig` writes
and BFB reflashes (route to
[`doca-hardware-safety`](../doca-hardware-safety/SKILL.md) for the
cross-cutting meta-policy); or for cross-library programming
questions (use
[`doca-programming-guide`](../doca-programming-guide/SKILL.md)).

## What this skill provides

This is a **thin loader**. Substantive material lives in two
companion files:

- `CAPABILITIES.md` — the bare-metal deployment runtime contract
  for a DOCA-linked binary: the two host modes (host x86 vs
  BlueField Arm bare-metal), the three launch modes (direct,
  tmux/screen, systemd-supervised), the hardware-resource-binding
  surface (PF / VF / representor enumeration; NUMA topology
  discovery; CPU pinning rationale; IRQ affinity rules), the
  per-tenant isolation surface (cgroup-v2 cpu / memory / io,
  network namespaces, `numactl` / `taskset`), the restart and
  recovery semantics (documented `systemd` `Restart=` modes vs
  crash-and-investigate vs supervisor-driven restart), the
  bare-metal-specific version overlay on the four-way version
  match owned by
  [`doca-version`](../doca-version/SKILL.md), the cross-cutting
  error taxonomy (seven layers, walked in order), the observability
  surface (stdout/stderr discipline by launch mode; device-state
  introspection via `devlink` / `sysfs` / `mlxconfig` *query*;
  per-tenant resource visibility), and the safety policy (overlay
  on
  [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md):
  smoke-before-bulk for binaries; failed bare-metal process is
  HIGH-STAKES; do not invent PCI addresses, NUMA numbers,
  representor names, devlink paths, or systemd `Restart=` mode
  names; confirm tenant-isolation primitives BEFORE the workload
  starts).
- `TASKS.md` — step-by-step workflows for the in-scope bare-metal
  verbs: `configure`, `build`, `modify`, `run` (with an explicit
  `### isolation` sub-anchor covering cgroup-v2 / namespaces /
  numactl per-tenant primitives), `test`, `debug`,
  `bluefield-lifecycle` (the BFB-install → RShim/TMFIFO →
  post-BFB-recovery operational sequencing ladder, with the
  six-state `bluefield-state-classifier` sub-anchor), the
  `Command appendix` (documented commands the agent may quote,
  each cross-linked to its public-doc source — no invented
  commands), and the `Deferred task verbs` block routing
  container-path / cluster / library-API / env-prep /
  hardware-state-change / cross-library questions out to their
  owning skills. (The change-application discipline for any
  mutating burn invoked from `## bluefield-lifecycle` is still
  meta-policy owned by
  [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md),
  loaded alongside.)

The skill assumes a host or BlueField target where:

- DOCA is already installed and healthy (per
  [`doca-setup ## test`](../doca-setup/TASKS.md#test)),
- the user has a DOCA-linked application binary they built (per
  [`doca-programming-guide ## build`](../doca-programming-guide/TASKS.md#build)),
- the user has the host-OS permissions to enumerate devices,
  reserve hugepages, write systemd units (if they choose that
  launch mode), and bind processes to NUMA nodes.

It does not cover installing DOCA — that path goes through
[`doca-setup`](../doca-setup/SKILL.md) — and it does not cover
building the binary — that path goes through
[`doca-programming-guide`](../doca-programming-guide/SKILL.md).

## Loading order

1. Read this `SKILL.md` first to confirm the user's question is
   in scope (bare-metal launch of a DOCA-linked binary on host
   x86 or BlueField Arm; NOT the container path, NOT a full
   cluster, NOT a library-API question).
2. **For the runtime contract (two host modes, three launch
   modes, hardware-binding surface, per-tenant isolation, version
   overlay, seven-layer error taxonomy, observability surface,
   bare-metal safety overlay), see
   [CAPABILITIES.md](CAPABILITIES.md).**
3. **For step-by-step workflows — `configure`, `build` (routing
   stub), `modify` (routing stub), `run` (with `### isolation`
   sub-anchor), `test`, `debug`, `bluefield-lifecycle` (BFB
   install + RShim/TMFIFO + post-BFB recovery + the six-state
   `bluefield-state-classifier`), plus the `Command appendix` and
   the `Deferred task verbs` block — see [TASKS.md](TASKS.md).**

## Example questions this skill answers well

See [`references/details.md`](references/details.md#example-questions-this-skill-answers-well).
## What this skill deliberately does not ship

See [`references/details.md`](references/details.md#what-this-skill-deliberately-does-not-ship).
## Related skills

See [`references/details.md`](references/details.md#related-skills).

# DOCA bare-metal deployment — Tasks

**Where to start:** The verb order is `configure → build → modify →
run → test → debug`. For bare-metal deployment, `build` and `modify`
are *routing stubs* — building the binary and modifying its source
both live in
[`doca-programming-guide`](../doca-programming-guide/SKILL.md); this
skill owns deployment of an already-built binary on the operator's
hardware. The `## test` verb is an iterative smoke-before-bulk loop,
not a one-shot pass.

> **⚠️ Destructive / irreversible operations require explicit
> confirmation.** Before any firmware burn, `mlxconfig set`, BFB
> reflash, or reboot/power-cycle action, load
> [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md), show the
> exact impact and rollback plan beside the command, and obtain the
> operator's explicit confirmation. Do not issue the command from
> this skill alone.

These verbs cover the in-scope cross-cutting bare-metal-deployment
workflows for an external operator launching any DOCA-linked
application binary on either supported host mode — host x86 with a
remote BlueField NIC over PCIe, or BlueField Arm bare-metal with the
binary on the DPU directly. Every step assumes the operator has
consulted the live public DOCA Programming Guide, the public
BlueField / DPU User Manual, and the Linux man pages for
`numactl(8)` / `taskset(1)` / `systemd.service(5)` /
`systemd.unit(5)` (all reachable through
[doca-public-knowledge-map ## Public documentation entry points](../doca-public-knowledge-map/SKILL.md#public-documentation-entry-points))
and is using them as the authoritative reference; this file
prescribes the *order* and *what to look up where*, not a copy-paste
runbook.

## configure

Preparing the host (or BlueField Arm) target, confirming every
precondition the bare-metal launch will rely on, and picking the
launch mode BEFORE any binary is invoked. This is also the verb
where the smoke-before-bulk posture is established up front — every
later verb assumes the operator has read it here.

1. **Confirm the env is healthy first.** This skill expects DOCA
   installed and healthy on whichever side the binary is built for
   (host x86 OR BlueField Arm). If install health is unverified,
   run
   [`doca-setup ## test`](../doca-setup/TASKS.md#test) on the
   target first. If the operator has no install at all, route to
   [`doca-setup ## no-install`](../doca-setup/TASKS.md#no-install)
   for the public NGC DOCA container path; the bare-metal launch
   itself cannot run inside the NGC container, but the operator
   may use the container to build the binary they will deploy.
2. **Recognise the host mode.** Confirm with the operator which
   side the binary is going to run on — host x86 against a remote
   BlueField NIC over PCIe, OR BlueField Arm cores directly. The
   distinction governs every later step (which install layout,
   which `devlink`, which NUMA topology). If the operator is
   ambiguous, derive the answer from `file <binary>` (x86_64 vs
   aarch64) and the install location (`/opt/mellanox/doca` on
   the host vs on the BlueField Arm). Do NOT guess.
3. **Confirm the bare-metal preconditions are closed.** Walk the
   precondition surface in
   [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
   against the operator's target:
    - DOCA installed and healthy per
      [`doca-setup ## test`](../doca-setup/TASKS.md#test).
    - Hugepages mounted and reserved per
      [`doca-setup ## configure`](../doca-setup/TASKS.md#configure)
      step 7 (or its BlueField-Arm equivalent on the BlueField OS
      image).
    - Devices and representors visible per
      [`doca-setup ## configure`](../doca-setup/TASKS.md#configure)
      step 8.
    - `LD_LIBRARY_PATH` set correctly per
      [`doca-setup ## configure`](../doca-setup/TASKS.md#configure)
      step 6 — the runtime install version MUST match the binary's
      link-time `pkg-config doca-*` version per
      [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility).
    - The four-way version match is closed per
      [`doca-version TASKS.md ## configure`](../doca-version/TASKS.md#configure).
4. **Pick the launch mode.** Per the launch-mode table in
   [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes),
   the operator picks ONE of: direct foreground (best for first
   launch, interactive debug, smoke); tmux / screen (best for
   long-running with manual reattach, no supervisor); systemd-
   supervised (best for restart-after-reboot, journald-integrated
   logs, documented `Restart=` policy). The choice feeds the
   observability surface in
   [`## run`](#run) step 4 and the restart / recovery surface in
   [`## debug`](#debug) layer 6. The agent does NOT pre-bake a
   sample systemd unit, a sample tmux invocation, or a sample
   direct command line — the operator authors the launch invocation
   against their environment.
5. **Plan the hardware-resource binding.** Per the
   hardware-binding rules in
   [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes),
   capture (do NOT invent):
    - The PCI BDF the binary should attach to, derived from
      `lspci -d 15b3:` on the target host.
    - The NUMA node owning that BDF's PCIe root complex, derived
      from `numactl --hardware` / `lscpu`.
    - The CPU set the binary's polling threads should pin to,
      chosen to be NUMA-local to the BDF.
    - The IRQ affinity mask that mirrors the CPU set on the
      `/proc/irq/<n>/smp_affinity` side, per the public
      BlueField / DPU User Manual reached through
      [`doca-public-knowledge-map`](../doca-public-knowledge-map/SKILL.md).
   Record each choice; the same record drives the launch
   invocation in [`## run`](#run) and the isolation primitives in
   [`### isolation`](#isolation).
6. **Plan the rollback path.** For any deployment that will
   exercise device state under load (production traffic on
   doca-flow, production RDMA queue pairs on doca-rdma, production
   I/O on a doca-comch channel), every launch on a live target
   must have: (a) the pre-launch device state captured (host
   networking, representor list, firmware-query output); (b) the
   previous-known-good launch invocation (or a no-binary baseline)
   ready to re-apply; (c) an out-of-band way to reach the target
   if the launch disrupts host connectivity (BlueField console,
   redundant management path, IPMI to the host); (d) a maintenance
   window agreed with whoever uses the host. For interactive
   first-launch on a non-production target, the rollback bar is
   lower but the *"be able to revert"* rule still applies. When
   the planned launch touches hardware state itself (not just runs
   a DOCA workload), route to
   [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md)
   FIRST.
7. **Confirm the four-way version match.** Per
   [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility),
   record BOTH the host (or BlueField Arm) DOCA install version
   via
   [`doca-version TASKS.md ## configure`](../doca-version/TASKS.md#configure),
   the binary's link-time `pkg-config doca-*` version (captured
   when the binary was built per
   [`doca-programming-guide ## build`](../doca-programming-guide/TASKS.md#build)),
   the BlueField firmware version (per the documented firmware-
   query surface in the BlueField / DPU User Manual), and the
   DOCA-version-policy row for the operator's release. A mismatch
   among any two is the silent *"the docs say this should work"*
   trap; align them explicitly.

## build

Bare-metal deployment is the *deploy* verb for an already-built DOCA
binary. There is no *application* artifact for the operator to build
inside this skill — the binary the operator deploys here is the one
they built per
[`doca-programming-guide ## build`](../doca-programming-guide/TASKS.md#build),
quoting the canonical `pkg-config doca-<library>` + meson pattern
that lives there.

If the user is asking how to build the binary they want to deploy,
hand off to
[`doca-programming-guide ## build`](../doca-programming-guide/TASKS.md#build)
(the canonical pattern, with C/C++ + non-C language tracks). Once
that returns a binary, control returns here at
[`## configure`](#configure) step 2 (host-mode recognition) for the
deploy.

If the user is asking how to build a systemd unit, a `numactl`
invocation, or any other launch-shell artifact, the answer is *the
launch artifact is composed against the live env*, not built ahead
of time. See [`## run`](#run) for how the launch-mode choice from
[`## configure`](#configure) step 4 composes the launch invocation
against the operator's PCI BDF / NUMA node / CPU set / `Restart=`
policy choice.

If the user is asking how to build a *DOCA library* itself, that is
a DOCA contributor workflow — out of scope for this skill per the
audience boundary in [`SKILL.md`](SKILL.md).

## modify

Bare-metal deployment does not have a *modify a sample program*
workflow analogous to DOCA libraries; the deployment-side analog of
"modify" is **re-walk the deploy after the binary changes, after
the launch invocation changes, or after the underlying env changes**:

1. **A binary change is a deploy event.** Any rebuild of the
   binary (a fresh `meson compile` per
   [`doca-programming-guide ## modify`](../doca-programming-guide/TASKS.md#modify))
   re-opens the smoke loop in [`## test`](#test) — the new binary
   may link against a different DOCA version, may have a different
   env-var contract, may exit on different signals. Re-walk
   [`## configure`](#configure) step 7 (version anchors), then
   [`## run`](#run), then [`## test`](#test).
2. **A launch-invocation change is a deploy event.** Editing the
   CPU pin set, the NUMA node, the PCI BDF, the env vars, or the
   `Restart=` policy on a systemd unit changes the deployment
   contract; treat each edit as a fresh deploy. Re-walk
   [`## run`](#run) step 3 (launch) and
   [`## test`](#test) step 1 (smoke).
3. **An env change is a deploy event.** A hugepage reservation
   change, an `LD_LIBRARY_PATH` change, a kernel-module load /
   unload, a devlink mode flip, or a representor moved into a
   different netns all change the deploy preconditions; re-walk
   [`## configure`](#configure) step 3 (preconditions).
4. **A hardware-state change leaves this verb entirely.** Any
   change touching device firmware (`mlxconfig set`, BFB
   reflash, firmware burn, BlueField mode flip) is owned by
   [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md);
   this skill does NOT walk that workflow. Control returns here
   at [`## configure`](#configure) step 3 once the hardware-state
   change is complete and verified per the meta-policy.
5. **Modify the binary's source only at the programming-guide
   layer.** If the operator is modifying the binary's source
   (adding a new pipe, changing a queue depth, adapting to a
   different DOCA library version), that change is owned by
   [`doca-programming-guide ## modify`](../doca-programming-guide/TASKS.md#modify)
   + the matching `libs/<library>` skill. Once the modified
   binary is rebuilt, control returns here at step 1 above.

The agent's anti-pattern alert: editing a launch invocation in
place without re-walking the smoke is the canonical *"my deploy
silently degraded after a small change"* failure. Treat every
launch-side change as a fresh deploy.

## run

Bringing up the DOCA-linked binary, confirming the device-attach
layer reaches a healthy state, and confirming the trivial-workload
liveness signal before layering any real workload on top. Every step
here assumes the prerequisites in [`## configure`](#configure) are
done.

1. **Confirm the binary is executable and the loader resolves
   DOCA.** From the launch host, confirm `file <binary>` reports
   the expected arch (matches `uname -m`), `chmod +x` is set, and
   `ldd <binary>` resolves every `libdoca_*.so` against the
   install on the runtime `LD_LIBRARY_PATH` per
   [`doca-setup ## configure`](../doca-setup/TASKS.md#configure)
   step 3. A failure here is a layer-1 symptom in
   [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy)
   and must be resolved before the binary is launched.
2. **Compose the launch invocation against the captured
   bindings.** Per
   [`## configure`](#configure) step 5, the launch invocation
   names: the PCI BDF (or representor) the binary should attach
   to, derived from live `lspci -d 15b3:` / `devlink dev show`
   output; the CPU set and NUMA node (via the documented
   `numactl --cpunodebind` / `--membind` form or `taskset -c
   <cpu-list>`); the env-var surface the binary's documented
   contract requires (per the public DOCA Programming Guide for
   the library in use). The launch invocation is composed from
   live values; do NOT substitute a BDF, a NUMA number, or a
   representor name from memory.
3. **Launch in the chosen launch mode.** Per
   [`## configure`](#configure) step 4:
    - **Direct.** Invoke the binary in the foreground from the
      shell session; stdout / stderr arrive on the terminal.
    - **tmux / screen.** Open a named tmux / screen session,
      invoke the binary inside it, detach; reattach with the
      documented commands when reading live output.
    - **systemd-supervised.** Drop the operator's authored
      `.service` unit into the documented systemd unit path
      (per `systemd.unit(5)`), `systemctl daemon-reload`,
      `systemctl start <unit>`. The `Restart=` policy in the
      unit must be one of the modes named in
      `systemd.service(5)`; do NOT invent a mode.
4. **Verify the process is up and the device attached.** Per the
   observability surface in
   [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability):
    - Direct: the binary's own stdout / stderr show the
      documented bring-up lines from the public DOCA Programming
      Guide for the library in use; no documented error lines
      repeat.
    - tmux: reattach and read the live pane buffer.
    - systemd: `journalctl -u <unit> -f` shows the same
      documented bring-up lines.
   A process that is up but printing layer-3 errors *"no
   matching device"* / *"representor not found"* / *"PCI BDF not
   found"* per [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy)
   is NOT ready; drop to [`## debug`](#debug) layer 3 BEFORE
   proceeding.
5. **Verify the trivial-workload liveness signal.** The
   documented liveness signal for the library in use (a single
   matched packet for doca-flow, a single RDMA write-with-imm for
   doca-rdma, a single comch send for doca-comch, etc.) lives in
   the matching `libs/<library>` skill — read it now from there
   per [`doca-programming-guide CAPABILITIES.md ## Error taxonomy`](../doca-programming-guide/CAPABILITIES.md#error-taxonomy)
   for the cross-library escalation rules. If the per-library
   liveness signal is NOT healthy, drop to
   [`## debug`](#debug) layer 4 (library error) or to the
   per-library skill's debug ladder — not to *"restart the
   binary and hope"*.
6. **Smoke before bulk (next: [`## test`](#test) step 1).**
   Before driving any real workload, walk [`## test`](#test)
   step 1 once to confirm end-to-end readiness; only then layer
   the workload on top.

### isolation

This sub-anchor covers per-tenant isolation for multiple DOCA
processes co-tenant on the same BlueField. It is reached from
[`## run`](#run) step 2 when the launch invocation crosses a
tenant boundary, and from [`## debug`](#debug) layer 7 when the
symptom is co-tenant noise. The isolation rules layered here are
the bare-metal-specific overlay on the standard Linux primitives
named at class level in
[`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes).

1. **Compose the cgroup-v2 cpu / memory / io budget per
   tenant.** Per the kernel cgroup-v2 documentation, each
   tenant's process is placed in its own cgroup with
   `cpu.weight` (or `cpu.max` for hard caps), `memory.max`
   (sized to cover the binary's resident set PLUS the hugepage
   accounting noted in
   [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)),
   and `io.weight` if the binary touches block I/O. Set the
   budget BEFORE the launch; verifying after the launch is too
   late.
2. **Compose the network-namespace per tenant when traffic
   isolation is required.** The standard Linux primitive is
   `ip netns add <name>` + moving the per-tenant representor
   into the netns per the documented `ip link set <repr> netns
   <name>` form. The DOCA-specific gotcha: the DOCA library is
   only able to see the representor inside the netns that owns
   it; launching the binary in a different netns (or in the
   root netns when the representor was moved out) reproduces
   as a layer-3 *"no matching device"* error.
3. **Compose the CPU / NUMA pinning per tenant via `numactl` /
   `taskset`.** Per `numactl(8)` and `taskset(1)`, the
   per-tenant CPU set is NUMA-local to the PCI BDF the binary
   attaches to (per
   [`## configure`](#configure) step 5). The DOCA-specific
   gotcha: a per-tenant CPU set that crosses the NUMA boundary
   owning the NIC cancels the isolation gain — both tenants
   then compete on memory bandwidth across the NUMA
   interconnect.
4. **Confirm hugepage accounting per tenant.** Hugepages
   reserved per
   [`doca-setup ## configure`](../doca-setup/TASKS.md#configure)
   step 4 are a global pool; each tenant's allocation is
   accounted against its cgroup's `memory.max` per the kernel
   cgroup-v2 hugepage accounting rules. A `memory.max` value
   that fits the binary's resident set but starves the
   hugepage pool surfaces as layer-2 *"process starts but
   exits immediately"* with an `EAL: Cannot get hugepage
   information` line.
5. **Verify the isolation primitives BEFORE the workload
   starts.** Per the safety rule in
   [`CAPABILITIES.md ## Safety policy`](CAPABILITIES.md#safety-policy):
   `systemd-cgls` (or `cat /sys/fs/cgroup/<path>/cpu.max`
   etc.) confirms the cgroup limits are in effect;
   `ip netns exec <name> ip link show` confirms the
   per-tenant representor is in the right netns; `taskset -p
   <pid>` / `numactl --show -p <pid>` confirms the running
   binary is pinned to the planned CPU set / NUMA node. Adding
   the verification after a co-tenant complains is the wrong
   order.

## test

Bare-metal deployment has no *compile and unit-test* workflow —
testing is operational and end-to-end against real hardware.

**`## test` is an iterative loop, not a one-shot pass.** Every
mutation (binary rebuild, launch-invocation edit, env change,
co-tenant addition) re-opens the smoke sweep. Skipping the re-run
after a mutation is the failure mode this loop replaces.

The smoke-before-bulk loop (rows apply to every DOCA-linked binary
on the bare-metal path, not just one library):

| Step | Why this is a loop, not a step | Where the substance lives |
| --- | --- | --- |
| 1 → 3 → 1 | Step 3 (per-library liveness probe) often reveals an as-launched gap in the binding (wrong representor, wrong NUMA node) that masquerades as a binary problem; loop back to step 1 | [`## test`](#test) step 3 |
| 1 → ## debug | When trivial-arg invocation fails, the binary cannot reach DOCA at all — escalate to [`## debug`](#debug) immediately, do not run later steps | [`## debug`](#debug) |
| 2 → ## configure → 2 | When the liveness-equivalent invocation shows a precondition was not closed (hugepages, devlink mode, representor visibility), loop back to [`## configure`](#configure) step 3 and re-walk the preconditions | [`## configure`](#configure) |
| 1..5 → ## run | Each loop iteration ends with a documented smoke; if all five pass, hand off to live [`## run`](#run) traffic | [`## run`](#run) |

The five steps of the smoke:

1. **Trivial-arg smoke.** Invoke the binary with its `--help` /
   `--version` equivalent (whichever the binary actually
   supports — the agent does NOT invent a flag). Confirm the
   binary executes, the dynamic loader resolves every
   `libdoca_*.so`, and the documented version / usage string
   appears. A failure here is layer 1 of
   [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy).
2. **Liveness-equivalent smoke.** Invoke the binary against the
   captured bindings from
   [`## configure`](#configure) step 5 but with NO real
   workload offered. Confirm the binary reaches the documented
   bring-up lines, opens the device, and exits cleanly on
   `SIGTERM`. A failure here is layer 2 or layer 3 of
   [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy).
3. **Trivial-workload smoke.** Drive ONE operation through the
   binary — one packet for doca-flow, one queue-pair operation
   for doca-rdma, one channel send for doca-comch, etc. (per
   the matching `libs/<library>` skill's documented liveness
   contract). Confirm the per-library counter advances by
   exactly one. A failure here is layer 4 of
   [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy).
4. **Capability + launch snapshot.** Save the *as-launched*
   answer to: which binary version was running, which DOCA
   install version it linked against (per
   [`doca-version TASKS.md ## run`](../doca-version/TASKS.md#run)),
   which BlueField firmware version the device was on, which
   PCI BDF / NUMA node / CPU set the binary attached to, the
   chosen launch mode, the `Restart=` policy if systemd-
   supervised, the captured stdout / journald lines from the
   bring-up. This snapshot is the artifact that lets future
   debug sessions skip rediscovery — and on a HIGH-STAKES
   deploy it is the rollback baseline.
5. **Multi-tenant smoke (only when co-tenants are planned).**
   Bring up each additional tenant AFTER this binary, one at a
   time, and re-run step 3 (trivial-workload smoke) against
   THIS binary between additions. A regression in this
   binary's per-library counter that only appears once a
   co-tenant is added is layer 7 of
   [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy);
   walk [`### isolation`](#isolation) before queueing more
   tenants.

Loop termination: all applicable smoke steps passing ends the loop
successfully and hands off to live traffic in [`## run`](#run).
Otherwise stop and escalate when either (a) the **same smoke step**
has failed twice with the same pass/fail outcome and unchanged saved
capability + launch evidence from step 4, or (b) ten total smoke-loop
iterations have completed without a green result. Neither stop
condition proves which lower layer is at fault. Escalate to the
per-library skill's debug ladder plus
[`doca-debug TASKS.md ## debug`](../doca-debug/TASKS.md#debug)
with the captured layer evidence.

## debug

Layered diagnosis. Walk the seven deployment layers in this order;
do not skip down without clearing the layer above. After those
seven, run the cross-cutting host check. The seven layers match
[`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy).

1. **Process-won't-start layer (layer 1).** Is the binary even
   executing? Symptoms: *"command not found"*, *"permission
   denied"*, *"cannot execute binary file: Exec format
   error"*, or `error while loading shared libraries:
   libdoca_*.so` from the dynamic loader. Resolution: confirm
   the binary's arch via `file <binary>` matches `uname -m`;
   confirm executability (`ls -l`); resolve every shared
   object with `ldd <binary>`; set `LD_LIBRARY_PATH` per
   [`doca-setup ## configure`](../doca-setup/TASKS.md#configure)
   step 3. Do NOT add a randomly-chosen `libdoca_*.so` to
   `LD_LIBRARY_PATH` from memory — quote from the live install
   layout per
   [`doca-public-knowledge-map ## Layout of an installed DOCA package`](../doca-public-knowledge-map/SKILL.md#layout-of-an-installed-doca-package).
2. **Process-starts-and-exits-immediately layer (layer 2).**
   Binary launches, runs for milliseconds, exits non-zero.
   Symptoms: short stderr message about a missing env var, a
   missing config file, or `EAL: Cannot get hugepage
   information`; supervisor reports the process restarted.
   Resolution: read the binary's stdout / stderr (terminal /
   tmux pane / `journalctl -u <unit>` per the launch mode);
   re-walk the documented env-var surface in the public DOCA
   Programming Guide for the library in use; re-run hugepage
   reservation per
   [`doca-setup ## configure`](../doca-setup/TASKS.md#configure)
   step 4. NEVER paste an env-var name from memory; quote from
   the per-library guide.
3. **Process-runs-but-cannot-find-the-device layer (layer 3).**
   Binary stays up but per-library bring-up reports *"no
   matching device"*, *"representor not found"*, or
   `DOCA_ERROR_NOT_FOUND` from the device-open call.
   Resolution: re-walk the device-visibility surface per
   [`doca-setup ## configure`](../doca-setup/TASKS.md#configure)
   step 5 — confirm the PCI BDF the binary was launched with
   matches a live `lspci -d 15b3:` entry; confirm the
   representor is enumerated in `ip link show` and is in the
   netns the binary launched in; confirm the eswitch is in
   the documented mode per the public BlueField / DPU User
   Manual. Do NOT substitute a BDF from a previous deploy.
4. **Library-error layer (layer 4).** Binary attaches to the
   device but the workload errors with a `DOCA_ERROR_*` per
   [`doca-programming-guide CAPABILITIES.md ## Error taxonomy`](../doca-programming-guide/CAPABILITIES.md#error-taxonomy).
   Resolution: defer to the cross-cutting debug ladder at
   [`doca-debug TASKS.md ## debug`](../doca-debug/TASKS.md#debug)
   FIRST, then to the matching `libs/<library>` skill for the
   library-specific overlay. Per-library error codes are owned
   by the per-library skill, not by this one.
5. **OOM / signal / resource-limit layer (layer 5).** Binary
   disappears with no log line, or supervisor reports an
   unexpected exit signal. Symptoms: `dmesg` shows an
   OOM-killer entry naming the binary; supervisor reports
   `SIGKILL` / `SIGTERM`; restart count climbs without a
   matching binary-side error log. Resolution: read the
   supervisor's exit-status record; check `dmesg` for OOM;
   re-check the cgroup-v2 budget per
   [`### isolation`](#isolation) step 1; confirm the binary's
   documented signal-handling contract from the public DOCA
   Programming Guide. A `memory.max` that fits the resident
   set but starves the hugepage pool is the canonical trap.
6. **Restart-loop layer (layer 6, HIGH-STAKES).** Supervisor
   keeps re-launching the binary; each launch exits with the
   same exit signature; the device the binary touches may be
   reporting odd errors caused by the loop itself. Per the
   HIGH-STAKES rule in
   [`CAPABILITIES.md ## Safety policy`](CAPABILITIES.md#safety-policy):
   STOP the supervisor (`systemctl stop <unit>` for systemd-
   supervised launches; manual termination for tmux); read
   the binary's LAST full log surface; walk layers 1-5 above
   against the captured evidence; only re-enable the supervisor
   once the root cause is identified. Letting the supervisor
   loop a known-broken binary is delayed diagnosis, not
   resilience.
7. **Co-tenant-noise layer (layer 7).** This binary behaves
   correctly in isolation; introducing a second DOCA process
   (or any other process) on the same BlueField makes this
   binary's counters degrade. Resolution: re-walk
   [`### isolation`](#isolation) for the second tenant's
   cgroup-v2 / netns / `numactl` configuration BEFORE drawing
   a per-library conclusion; if the symptom only reproduces
   under co-tenancy, the diagnosis is multi-tenant isolation,
   not a per-library bug.
**Cross-cutting host check (after the seven layers; not an eighth
taxonomy layer).** Deployment looks
   healthy at every layer above but a cross-cutting host issue
   (kernel version, driver loaded / not loaded, PCIe link
   state, hugepage allocation health beyond reserved-vs-used,
   BFB log surface) breaks the binary's downstream behavior.
   Resolution: drop to
   [`doca-debug TASKS.md ## debug`](../doca-debug/TASKS.md#debug)
   for the cross-cutting debug ladder; if the symptom is a
   hardware-state change the operator is contemplating, route
   to
   [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md)
   instead of touching device state from this skill.

## bluefield-lifecycle

The BlueField **platform lifecycle** anchor — the workflow for taking
a BlueField from "powered card in the slot" to "Arm OS healthy,
TMFIFO up, host PFs bound, four-way version match closed, DOCA-linked
binary safely launchable per [`## run`](#run)". This anchor exists
because the bare-metal-deployment skill's downstream verbs (`## run`,
`## test`, `## debug`) all assume a working BlueField; when that
assumption breaks, the bundle previously routed the operator out to
[`doca-hardware-safety`](../doca-hardware-safety/SKILL.md) for the
mutating-change meta-policy AND out to
[`doca-public-knowledge-map ## Externally-productized DOCA software`](../doca-public-knowledge-map/SKILL.md#externally-productized-doca-software--not-in-this-bundle-but-here-is-where-to-route)
for the BSP/BFB documentation entry, but the **operational sequencing
ladder itself** (which evidence to collect in which order, which
failure mode each evidence pattern points to, which recovery action
lattices to which evidence) had no home. This section is that home.

This is a **reasoning ladder, not a script.** The agent does not ship
a `bfb-install-wrapper.sh` or a `classify-bluefield-state.sh`; it
prescribes the *order in which an operator should collect evidence
and the recovery action each evidence pattern lattices to*, quoting
documented commands from the public BlueField Platform Software
Manual (reached through
[`doca-public-knowledge-map`](../doca-public-knowledge-map/SKILL.md)),
the MFT manual (`flint`, `mlxconfig`, `mlxfwmanager`), and the Linux
man pages for `modprobe(8)`, `lspci(8)`, `ip-route(8)`. Every
mutating step (firmware burn, BFB reflash, `mlxconfig set`, kernel
boot parameter change) STILL routes through
[`doca-hardware-safety`](../doca-hardware-safety/SKILL.md) for the
meta-policy (preflight, OOB console, maintenance window, rollback)
— this section adds only the *bare-metal-deployment-specific
sequencing* on top.

### bfb-install lifecycle

The canonical "push a BFB image from the host to the BlueField" flow.
The agent walks the operator through this verb-by-verb; it does NOT
fabricate the host-side `bfb-install` flag set, the BFB image
filename, the RShim character-device path, or the `bf.cfg` schema —
all four come from live `--help` on the installed tool and the
public BlueField Platform Software Manual.

1. **Pre-flight inventory.** Capture, BEFORE any push:
    - The current BFB image / BSP version on the BlueField, from the
      BSP version-query path documented in the BlueField Platform
      Software Manual (do NOT guess a command name).
    - The current ConnectX firmware version on the BlueField's NIC
      side, from `flint -d <bdf> q` (per the MFT manual); on a
      BlueField this is the NIC PSID + firmware revision the new
      BFB will or will not match.
    - The host-side RShim userspace daemon state (`dpkg -s
      rshim` / `rpm -q rshim` for package install, `systemctl
      status rshim` for `active (running)`, `pgrep -a rshim` for
      a live `/usr/sbin/rshim` process, `ls /dev/rshim*` for the
      character-device tree — all per the BSP manual). On DOCA
      3.3+ there is NO `rshim` kernel module (the in-tree
      module was removed); `lsmod | grep rshim` is expected to
      be empty and is NOT failure evidence. If the daemon /
      `/dev/rshim*` tree is missing, the host has no path to
      push.
    - The OOB management path the operator will use if the push
      breaks the Arm OS: BMC console-over-Redfish, BMC IPMI
      serial-over-LAN, or the physical UART (per the
      [`BlueField BMC Software`](../doca-public-knowledge-map/SKILL.md#externally-productized-doca-software--not-in-this-bundle-but-here-is-where-to-route)
      row in the public-knowledge-map). Without one of these, the
      agent MUST stop and escalate to the operator responsible for
      the target with the captured pre-flight state and rollback
      plan; it must not proceed without an OOB path, per
      [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md).
    - The BFB image's SHA matches the SHA the operator
      downloaded from the documented DOCA Downloads page (per
      [`doca-public-knowledge-map`](../doca-public-knowledge-map/SKILL.md#public-documentation-entry-points))
      — pushing a corrupted BFB is the load-bearing first-run
      failure for the entire flow.
2. **Author `bf.cfg` from the documented schema.** The BFB-install
   path takes an installer configuration file (`bf.cfg`) that
   controls post-install state on the Arm side — root/ubuntu
   password, hostname, and `bfb_modify_os()` shell-script hooks
   that run during the install to seed any state the documented
   `bf.cfg` parameters do NOT cover directly. **Two
   operator-relevant rules**:
   (a) If the operator wants passwordless SSH to survive the
   install, the `bf.cfg` MUST set the documented `ubuntu_PASSWORD`
   parameter (or equivalent per the schema, e.g. `ROOT_PASSWORD`)
   AND seed the SSH public key via a `bfb_modify_os()` hook that
   writes `/mnt/home/ubuntu/.ssh/authorized_keys` on the
   to-be-installed rootfs (the rootfs is mounted under `/mnt`
   during install per the BSP manual). The default BFB install
   rewrites `/home/ubuntu/.ssh/`, so any pre-existing key on the
   previous image is GONE unless reseeded by this hook. Do NOT
   invent an `authorized_keys` top-level `bf.cfg` parameter — the
   schema does not have one; the mechanism is the
   `bfb_modify_os()` hook.
   (b) For separated-host / bump-in-the-wire / scalable-function
   deployments where the BlueField must boot in a specific
   internal-CPU / port-owner / SF mode (e.g. `SEPARATED_HOST(0)`,
   `EMBEDDED_CPU(1)`), the required `mlxconfig set` invocations
   are run from a `bfb_modify_os()` hook in `bf.cfg` at
   BFB-install time; reconfiguring the same modes after the
   install typically requires another BFB push (per the BSP
   manual). The agent quotes the `bf.cfg` parameter keys from the
   public schema (and references the `bfb_modify_os()` script
   pattern) and does NOT invent key names from memory.
3. **Push the BFB.** Run the host-side `bfb-install` invocation
   per its `--help` and the BSP manual. The push streams the BFB
   to the BlueField over RShim/PCIe; the Arm side reboots through
   UEFI → Linux up → first-boot init.
4. **Do not trust `bfb-install` exit code 0 alone.** This is the
   single most expensive failure mode in the operator's loop.
   `bfb-install` has been observed in the field to exit 0 while
   the Arm-side flow only partially completed — the canonical
   field-reported signature is *"Ubuntu installation completed"*
   (or *"Ubuntu installation finished"*, both phrasings have
   been seen in different BFB releases) followed by an
   `INFO[MISC]: NIC firmware update failed` line in the RShim
   console / log, meaning the OS image landed but the
   firmware-update sub-step silently failed. The agent ALWAYS
   parses the actual console / log output the installer wrote,
   in addition to the exit code, and looks for: (a) any `[MISC]`
   or `[ERR]` line whose text contains a failure verb
   ("failed", "error", "abort"), (b) the documented `Linux up`
   marker, (c) the documented `DPU is ready` marker. If any of
   `(a)` is present or `(b)` / `(c)` are absent, the install is
   treated as **partial**, not complete, and the agent advances
   to
   [`### bluefield-state-classifier`](#bluefield-state-classifier)
   instead of declaring success.
5. **Distinguish "BFB install completed" from "readiness wait
   failed".** "Completed" means the installer's I/O is done; it
   does NOT mean the Arm OS is up, TMFIFO is reachable, SSH is
   live, or host PFs are bound. The agent always runs the
   readiness sequence in
   [`### post-bfb-recovery`](#post-bfb-recovery) before re-declaring
   the BlueField healthy.
6. **Routing for the firmware burn itself.** A BFB push is a
   mutating change against live device state — the meta-policy in
   [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md)
   governs the preflight / OOB-console / rollback discipline that
   wraps this step. This section adds only the *sequencing
   ladder*; the agent loads `doca-hardware-safety` ALONGSIDE this
   skill whenever the operator's question reaches step 3 above.

### rshim and tmfifo

RShim is the host-side surface that exposes the BlueField's
Arm-side console (`/dev/rshim<N>/console`) AND the host-side
network endpoint of the TMFIFO recovery interface (factory
defaults per the BlueField Platform Software Manual: host-side
address `192.168.100.1/30`, BlueField-side address
`192.168.100.2/30`; the agent does NOT fabricate the subnet from
memory). On DOCA 3.3+ hosts the RShim surface ships as a
**userspace daemon** (`/usr/sbin/rshim` started by
`rshim.service`); the legacy in-tree kernel module is no longer
shipped. The TMFIFO interface is the *recovery* path when the
BlueField's normal management network is broken; it is NOT a
primary data path.

1. **Verify RShim is attached on the host.** The agent runs all
   three of: `dpkg -s rshim` / `rpm -q rshim` (the userspace
   package is installed), `systemctl status rshim` (the daemon is
   `active (running)`), and `ls /dev/rshim*` (character-device
   tree is present per the BSP manual). On DOCA 3.3+
   `lsmod | grep rshim` is EXPECTED to be empty and is NOT
   evidence of failure — the in-tree kernel module is gone; the
   surface is delivered entirely by the userspace daemon. If
   `systemctl status rshim` is not `active (running)` OR the
   `/dev/rshim*` tree is missing, the host has no path to the
   BlueField's recovery surface and downstream TMFIFO checks are
   meaningless.
2. **Verify the TMFIFO network endpoint on the host.** `ip
   addr show tmfifo_net0` (per the BSP manual's documented
   interface name; verify the name on the operator's host —
   different driver versions have shipped slightly different
   names). The address should be the host-side documented address.
3. **Critical TMFIFO gotcha — ALWAYS `ip route get` before
   `ping`.** A real failure mode the bundle has hit in the wild
   is: the BlueField-side TMFIFO address (e.g. `192.168.100.2`)
   gets accidentally added to the **host's** loopback or to
   `tmfifo_net0` itself, so `ping 192.168.100.2` from the host
   succeeds — but it is pinging *the host*, not the BlueField.
   The diagnostic that catches this in one command is:
    - `ip route get <bf-tmfifo-address>` on the host. **Healthy
      outputs** the agent should accept as "the route is going to
      the BlueField": `<bf-addr> dev tmfifo_net0 src <host-addr>`
      (driver versions that expose the TMFIFO interface directly)
      OR `<bf-addr> dev tm-br src <host-addr>` (BSP / DOCA-host
      installs that bridge `tmfifo_net0` into a `tm-br` bridge —
      observed in the wild on DOCA 3.3 hosts where the BFB-side
      RShim driver is configured to bridge the TMFIFO endpoint
      with the host-side management bridge). **Broken output** is
      always: `<bf-addr> dev lo src 127.0.0.1` or
      `local <bf-addr> dev lo` — the address has been bound
      LOCALLY and *every* ping / ssh / curl to it is hitting the
      host, not the BlueField. The agent ALWAYS runs
      `ip route get` before trusting `ping` for TMFIFO recovery,
      and accepts either `tmfifo_net0` or `tm-br` as the egress
      interface on the host side.
4. **Soft-reset is a recovery action, not a routine action.** The
   `rshim` soft-reset path documented in the BSP manual is
   appropriate when the Arm side is stuck in a known-recoverable
   state (UEFI hang post-BFB-install, console responsive but
   userspace dead); it is NOT routine. Routing: the operator
   captures the BlueField state per
   [`### bluefield-state-classifier`](#bluefield-state-classifier)
   FIRST, then decides whether soft-reset, cold power cycle, or
   re-push BFB is the appropriate recovery — and loads
   [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md)
   ALONGSIDE for the meta-policy on any of those.

### post-bfb-recovery

After a BFB push lands, the BlueField's downstream surface (Arm
OS health, TMFIFO reachability, SSH liveness, host PFs bound,
firmware version match) must be re-verified before the
bare-metal-deployment skill returns to [`## run`](#run). This is
the "did the install actually take?" gate.

1. **Wait for the documented readiness markers, not for a
   timer.** A wall-clock sleep is not equivalent to a readiness
   probe. The agent polls for the documented `Linux up` / `DPU is
   ready` markers in the RShim console buffer (per the BSP
   manual), AND for the documented Arm-side SSH endpoint
   responding, AND for the documented BMC health endpoint
   reporting `OK`. If any of these never report ready within the
   manual's documented bound, the BlueField is in a partial state
   and the agent advances to
   [`### bluefield-state-classifier`](#bluefield-state-classifier)
   rather than declaring success.
2. **Host PF rebind sequence.** A BFB push can leave the host
   side's mlx5 driver in a stale state: the PCI devices for the
   BlueField PFs are present (`lspci -d 15b3:` lists them) but
   `ip link show` does not enumerate the netdevs, RDMA enumeration
   is empty, and any DOCA program that attaches by representor
   name fails. The documented recovery is:
    - `modprobe mlx5_core` (no-op if already loaded; loads if not).
    - For each BlueField PF BDF captured at pre-flight (e.g.
      `0000:b3:00.0`, `0000:b3:00.1`): `echo <bdf> >
      /sys/bus/pci/drivers/mlx5_core/bind` per the kernel sysfs
      driver-binding documentation.
    - Re-verify: `ip link show` enumerates the BlueField netdevs;
      `ibv_devinfo` enumerates the BlueField RDMA devices;
      `devlink dev show` lists the BlueField devlink instance.
   The agent does NOT invent the BDF strings from memory; they
   come from the pre-flight `lspci -d 15b3:` capture.
3. **`/home/ubuntu` operational gotcha (Arm-side BlueField OS).**
   On certain BlueField OS images, `/home/ubuntu` ships owned by
   `root` rather than `ubuntu`, which breaks the normal pattern of
   the `ubuntu` user writing logs / scratch files under their own
   home directory. The agent: (a) checks `stat -c '%U:%G' /home/ubuntu`
   after first SSH; (b) if it is `root:root`, flags it to the
   operator and proposes the documented `chown -R ubuntu:ubuntu
   /home/ubuntu` fix (per the BSP manual) BEFORE the operator
   pastes any script that writes there. Because this recursively
   changes ownership, capture the current ownership and obtain the
   operator's explicit confirmation before applying it. The fix
   itself is trivially documented Linux; the value is the recognition
   *during* lifecycle recovery instead of after a script fails.
4. **Log copy-back to host workspace.** All install / readiness /
   recovery evidence collected on the Arm side (the RShim console
   buffer, the cloud-init log, the documented BSP install log,
   the readiness probe output) should be copied BACK to the host
   workspace before the operator re-attempts the workload. Two
   rules: (a) use `scp ubuntu@<bf-mgmt-addr>:<path> .` (or the
   documented BSP log-export path) from the host side — pulling
   is safer than pushing host credentials onto the BF; (b) do NOT
   wrap the copy step in `sudo` on the host unless the operator
   explicitly entered `sudo` mode for this lifecycle session, per
   the smoke-before-bulk rule in
   [`## run`](#run) step 4.
5. **Four-way version-match re-close.** Once Arm OS is healthy
   and host PFs are bound, the BlueField's new BFB / firmware
   stack must satisfy the four-way version match owned by
   [`doca-version TASKS.md`](../doca-version/TASKS.md). The
   agent walks the four-way match against the new BlueField state
   BEFORE returning to [`## configure`](#configure) step 7. A
   skipped re-close after a BFB push is the most common cause of
   "ran fine yesterday, breaks today" symptoms.

### bluefield-state-classifier

When a BFB push, a soft-reset, or a host PF rebind has been done
and the BlueField is *not yet* confirmed healthy, the agent walks
this six-state classifier IN ORDER and records every matching state
in that order. It never stops at the first match. Each state names
the evidence that
identifies it, the most-likely root cause class, and the
*sequencing* of the recovery action (mutating steps still load
[`doca-hardware-safety`](../doca-hardware-safety/SKILL.md)
alongside; this ladder names the order, not the burns).

This is a **reasoning ladder, not a binary**: an Arm OS can be
"Linux up" AND "host PFs unbound" simultaneously; the agent
walks the ladder top-to-bottom and reports every state that
matches, not just the first. The point is the *sequencing of
evidence collection*, so two ops engineers reading the same
console output reach the same triage step.

1. **`installer-still-running`.** Evidence: `bfb-install` is still
   resident on the host (`ps -ef | grep bfb-install`), AND the
   RShim console buffer is still emitting documented installer
   progress lines per the BSP manual. Root cause class: install
   in flight, not failure. Recovery: WAIT, do not abort. The
   `bfb-install` push can take many minutes on first-flash;
   aborting it mid-write is what *creates* the next state down.
2. **`uefi-only`.** Evidence: the RShim console buffer reports
   `exit Boot Service` (or the BSP manual's equivalent UEFI-exit
   marker) but never reaches the documented `Linux up` marker.
   Root cause class: kernel did not hand off to userspace —
   common after a partial BFB-install with a firmware-update
   error (see [`### bfb-install lifecycle`](#bfb-install-lifecycle)
   step 4). Recovery sequencing: capture full RShim console
   buffer to host, capture host-side `dmesg`, do NOT immediately
   re-push BFB; first perform a    documented cold power cycle of
   the BlueField (via BMC per the
   [`BlueField BMC Software`](../doca-public-knowledge-map/SKILL.md#externally-productized-doca-software--not-in-this-bundle-but-here-is-where-to-route)
   row, not just `reboot` from the unresponsive Arm side); only
   if cold power cycle does NOT recover, re-push BFB under
   [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md)
   meta-policy.
3. **`linux-up-tmfifo-down`.** Evidence: documented `Linux up`
   marker present in RShim console, BUT TMFIFO probe per
   [`### rshim and tmfifo`](#rshim-and-tmfifo) step 2 returns
   no host-side address. Root cause class: TMFIFO interface
   never came up (driver / udev / link-state). Recovery
   sequencing: re-check host-side RShim daemon, run the
   documented TMFIFO bring-up procedure from the BSP manual,
   THEN re-run [`### rshim and tmfifo`](#rshim-and-tmfifo)
   step 3 (the `ip route get` gotcha) — a freshly-bound TMFIFO
   on the host can land in the loopback failure mode and look
   like it is working.
4. **`tmfifo-up-ssh-down`.** Evidence: documented TMFIFO probe
   passes (and `ip route get` confirms the route is going to the
   BlueField, not local loopback), BUT SSH to the documented
   Arm-side management endpoint refuses or hangs. Root cause
   class: Arm-side sshd not yet listening (still in init), OR
   the operator's authorized_keys / `ubuntu_PASSWORD` was NOT in
   the `bf.cfg` per [`### bfb-install lifecycle`](#bfb-install-lifecycle)
   step 2. Recovery sequencing: wait the documented sshd-ready
   bound from the BSP manual; if still down, fall through to the
   RShim console (`/dev/rshim<N>/console`) for a userspace
   prompt and re-seed credentials there; on the next BFB push,
   put the `authorized_keys` IN the `bf.cfg`.
5. **`arm-ok-host-pfs-unbound`.** Evidence: Arm-side SSH alive,
   Arm OS reports healthy (uptime > a few seconds, `dmesg` clean),
   BUT host-side enumeration is broken — `lspci -d 15b3:` shows
   the BlueField PFs, `ip link show` does NOT show the BlueField
   netdevs, `ibv_devinfo` is empty, DOCA programs cannot attach
   by representor name. Root cause class: stale host mlx5
   driver-binding state post-BFB push. Recovery sequencing: run
   [`### post-bfb-recovery`](#post-bfb-recovery) step 2 (the
   documented PF-rebind sequence). Do NOT proceed to launch any
   DOCA-linked binary in this state — every device-open will
   fail with a misleading error.
6. **`host-bf-version-mismatch`.** Evidence: everything above
   looks healthy, but the four-way version match owned by
   [`doca-version`](../doca-version/SKILL.md) does not close —
   host DOCA install version, BlueField BFB / BSP version,
   ConnectX firmware version, and binary's link-time
   `pkg-config doca-*` version do not satisfy the documented
   compatibility matrix in the DOCA Compatibility Policy
   (linked from
   [`doca-public-knowledge-map`](../doca-public-knowledge-map/SKILL.md#public-documentation-entry-points)).
   Root cause class: the operator pushed a BFB but did not
   simultaneously align the host DOCA-Host install, OR the
   `/etc/apt/sources.list.d/doca.list` is pointed at a
      different release channel than what is now installed (see
      [`doca-version TASKS.md ## apt-source consistency`](../doca-version/TASKS.md#apt-source-consistency)).
   Recovery sequencing: walk
   [`doca-version TASKS.md`](../doca-version/TASKS.md) in full;
   resolve any apt-source / repo-pin drift BEFORE installing
   anything new; route any host-side DOCA reinstall through
   [`doca-setup`](../doca-setup/SKILL.md).

Two cross-cutting rules for this classifier:

- **Match multiple states; do not stop at the first.** A
  BlueField that just came back from a partial BFB-install can
  match `uefi-only` initially, then `arm-ok-host-pfs-unbound`
  after a cold power cycle. The agent reports the *current*
  state set and the most-recently-observed transition, not just
  the first match.
- **Sequence recovery in classifier order.** When multiple states
  match, apply the earliest matching state's recovery first, then
  re-run the classifier from state 1 before applying another
  recovery. Every mutating recovery still requires the explicit
  confirmation gate from `doca-hardware-safety`.
- **Never declare healthy from absence of evidence.** "TMFIFO
  ping succeeded" without `ip route get` is NOT evidence the
  BlueField is reachable (see [`### rshim and tmfifo`](#rshim-and-tmfifo)
  step 3). "Arm SSH connected" without `uptime` / `dmesg` / a
  documented health probe is NOT evidence the Arm OS finished
  initialising. "host-side `lspci` shows the PFs" is NOT
  evidence the PFs are usable (see
  [`### post-bfb-recovery`](#post-bfb-recovery) step 2). The
  classifier states are walked top-to-bottom precisely so the
  agent does not skip an unverified gate.

## Command appendix

Bare-metal-deployment commands the verbs above reach for, grouped
by purpose so the agent picks the right family without searching
prose. Every row is a CLASS — the agent must not invent flags or
specific values beyond what the row names; flag and value discovery
is `--help` on the installed tool, the public DOCA / BlueField docs,
or the Linux man pages, not prose recall.

**Infra-aware preamble (every row below).** Per the bundle's
detect → prefer → fall back → report contract documented in
[`doca-structured-tools-contract ## The agent behavior contract`](../doca-structured-tools-contract/SKILL.md#the-agent-behavior-contract),
the agent should:

1. Probe for the matching structured helper FIRST (`doca-env
   --json` for version + devices + libraries + drivers +
   hugepages in one shot; `collect-host-state` /
   `collect-dpu-state` for the host-side device topology;
   `version-matrix.json` for *"available since"* lookups).
2. If the probe succeeds, the structured tool's output is the
   authoritative answer and the agent SHOULD NOT also run the
   manual command in the row below. Report *"using structured
   `<tool>`"*.
3. If the probe fails, fall back to the manual command in the
   row. Report *"falling back to manual chain"*.
4. The schemas the structured tools emit are defined in
   [`doca-structured-tools-contract ## Schemas`](../doca-structured-tools-contract/SKILL.md#schemas);
   the version-handling semantics (four-way match, NGC,
   headers-win) are owned by
   [`doca-version`](../doca-version/SKILL.md).

| Purpose | Command (class shape) | Owning step | Reads as healthy when … |
| --- | --- | --- | --- |
| Binary arch + loader resolution | `file <binary>` ; `ldd <binary>` | [`## run`](#run) step 1; [`## debug`](#debug) layer 1 | Arch matches `uname -m`; every `libdoca_*.so` resolves to a path under the active DOCA install. |
| Binary executability | `ls -l <binary>` ; `chmod +x <binary>` if needed | [`## run`](#run) step 1; [`## debug`](#debug) layer 1 | The execute bit is set for the user that will launch the binary. |
| PCI function enumeration | `lspci -d 15b3:` (Mellanox vendor ID); the documented form lives in the public BlueField / DPU User Manual | [`## configure`](#configure) step 5; [`## debug`](#debug) layer 3 | Lists the BlueField PF (and any VFs / SFs the operator expects); BDFs match what the binary's launch invocation will quote. |
| Device + representor enumeration | `devlink dev show` ; `ip link show` ; `cat /sys/class/net/*/phys_port_name` | [`## configure`](#configure) step 3 + step 5; [`## debug`](#debug) layer 3 | Lists the expected BlueField devices, netdevs, and representors; representor naming matches the BlueField OS image's documented convention. |
| NUMA topology | `numactl --hardware` ; `lscpu` | [`## configure`](#configure) step 5; [`### isolation`](#isolation) step 3 | Identifies the NUMA node owning the BlueField's PCIe root complex; CPU IDs per NUMA node match what the launch invocation will pin to. |
| CPU / NUMA pinning at launch | `numactl --cpunodebind=<node> --membind=<node> -- <binary> [args]` ; OR `taskset -c <cpu-list> <binary> [args]` (per `numactl(8)` / `taskset(1)`) | [`## run`](#run) step 2 + step 3; [`### isolation`](#isolation) step 3 | The launched process's `numactl --show -p <pid>` / `taskset -p <pid>` matches the planned set. |
| IRQ affinity inspection | `cat /proc/irq/<n>/smp_affinity` for the IRQ(s) the BlueField NIC owns (per the public BlueField / DPU User Manual) | [`## configure`](#configure) step 5; [`## debug`](#debug) layer 7 | IRQ mask matches the planned CPU set on the same NUMA node. |
| Hugepage state | `mount \| grep huge` ; `cat /proc/meminfo \| grep -i huge` | [`## configure`](#configure) step 3; [`## debug`](#debug) layer 2 + layer 5 | hugetlbfs is mounted; `HugePages_Total` > 0; `HugePages_Free` > 0 even after the binary attaches. |
| Process output (direct launch) | The terminal the operator launched from | [`## run`](#run) step 4; [`## debug`](#debug) layers 2-5 | Documented bring-up lines from the public DOCA Programming Guide appear; no documented error lines repeat. |
| Process output (tmux / screen) | `tmux attach -t <session>` ; `screen -r <session>` (per the operator's session name) | [`## run`](#run) step 4; [`## debug`](#debug) layers 2-5 | Same as direct, read from the reattached pane buffer. |
| Process output (systemd-supervised) | `systemctl status <unit>` ; `journalctl -u <unit> -f` (per the operator's unit name) | [`## run`](#run) step 4; [`## debug`](#debug) layers 2-6 | Unit is `active (running)`; journald tail shows the documented bring-up lines; restart count is stable. |
| Firmware configuration query (READ-ONLY) | `mlxconfig -d <bdf> q` per the MFT / BlueField documentation — configuration only; `mlxconfig set` is a hardware-state change owned by [`doca-hardware-safety`](../doca-hardware-safety/SKILL.md) | [`## configure`](#configure) step 7; cross-cutting host check | Reports current / next-boot NV-config values; it does **not** report the running firmware version. |
| Firmware version query (READ-ONLY) | `flint -d <bdf> q`; read the `FW Version:` field per the MFT manual | [`## configure`](#configure) step 7; cross-cutting host check | Reports the running NIC firmware-version anchor used by [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility). |
| cgroup-v2 budget inspection | `systemd-cgls` ; `cat /sys/fs/cgroup/<path>/cpu.max` ; `cat /sys/fs/cgroup/<path>/memory.stat` (per the kernel cgroup-v2 documentation) | [`### isolation`](#isolation) step 5; [`## debug`](#debug) layer 5 + layer 7 | Per-tenant cgroup limits match the planned budget; `memory.stat` does not show OOM evidence. |
| Network-namespace inspection | `ip netns list` ; `ip netns exec <name> ip link show` (per `ip-netns(8)`) | [`### isolation`](#isolation) step 2 + step 5; [`## debug`](#debug) layer 3 | The per-tenant representor is in the netns the binary launches into; the root netns does not own a representor a netns-scoped binary is trying to reach. |
| Version anchor — host DOCA install | `pkg-config --modversion doca-common` ; `doca_caps --version` on the active install | [`## configure`](#configure) step 7; [`## debug`](#debug) layer 4 | The two strings match each other and match the binary's link-time `pkg-config doca-*` capture from [`doca-programming-guide ## build`](../doca-programming-guide/TASKS.md#build). |
| Version anchor — BlueField firmware | `flint -d <bdf> q`; read `FW Version:` (per the MFT manual) | [`## configure`](#configure) step 7; [`## debug`](#debug) layer 4 | Reports a firmware version the public DOCA Programming Guide certifies for the operator's DOCA release. |

Three cross-cutting rules for this appendix:

- **Never invent a PCI BDF, a representor name, a NUMA node
  number, a hugepage allocation amount, or a `Restart=` mode
  name.** The public DOCA Programming Guide, the public
  BlueField / DPU User Manual, the Linux man pages, and live
  `lspci` / `numactl --hardware` / `devlink dev show` output
  on the target host are the contract; prose-derived strings
  are the most common hallucination failure for this skill.
- **Process output before device state before per-tenant
  resource picture.** When triaging, read the binary's own
  stdout / stderr / journald surface first (did the binary
  parse its args and reach DOCA?); only then read the
  device-state surface (`devlink` / `ip link` / `mlxconfig
  query`); only then read the per-tenant resource surface
  (cgroup-v2 / `numactl --show`). Reading the per-tenant
  resource picture against a binary that never reached its
  device-open call is meaningless.
- **Cross-link instead of duplicate.** Cross-cutting env
  commands (`pkg-config --modversion`, `doca_caps --version`,
  `mount | grep huge`, `lsmod | grep mlx5`) live in
  [`doca-setup TASKS.md ## Command appendix`](../doca-setup/TASKS.md#command-appendix);
  cross-cutting debug commands (`gdb`, `valgrind`, `strace`,
  `--sdk-log-level`, the `doca-<lib>-trace` build flavor) live
  in
  [`doca-debug TASKS.md ## debug`](../doca-debug/TASKS.md#debug);
  this appendix names only the bare-metal-deployment-specific
  ones.

## Deferred task verbs

- **Container-path deployment of any DOCA service container** on
  the BlueField (kubelet-standalone mode, static-pod manifests
  directory, pod-spec YAML, image-pull from NGC) — out of scope
  here. Route to
  [`doca-container-deployment`](../doca-container-deployment/SKILL.md)
  for the sibling deployment path.
- **Full-Kubernetes-cluster operations** (cluster API,
  `kubectl`, `Deployment` / `Service` / `Ingress` objects,
  cluster-wide observab