# Evaluation Report

Evaluation of the `doca-bench` skill before publication through Skill Evaluator.

This benchmark summarizes 3-Tier Evaluation from Skill Evaluator results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.

## Evaluation Summary

- Skill: `doca-bench`
- Evaluation date: 2026-07-26
- Environment: `k8s-sandbox`
- Dataset: 4 evaluation tasks
- Attempts per task: 1
- Pass threshold: 50%
- Overall verdict: PASS

## Agents Used

- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`)
- Codex (`openai/openai/gpt-5.5`)

## Metrics Used

Reported benchmark dimensions:

- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
- Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
- Effectiveness: checks whether the agent performs measurably better with the skill than without it.
- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.

Underlying evaluation signals used in this run:

- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access.
- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow.
- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.
- `accuracy` (Accuracy): grades final-answer correctness against the reference answer.
- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully.
- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations.

## Test Tasks

The benchmark dataset contained 4 evaluation tasks:

- Positive tasks: 3 tasks where the skill was expected to activate.
- Negative tasks: 1 tasks where no skill was expected.
- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.

Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.

## Results

| Dimension | Num | Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`) | Codex (`openai/openai/gpt-5.5`) |
|---|---:|---:|---:|
| Security | 4 | 100% (+0%) | 100% (+0%) |
| Correctness | 4 | 100% (+25%) | 100% (+20%) |
| Discoverability | 4 | 97% (+34%) | 95% (+45%) |
| Effectiveness | 4 | 100% (+81%) | 86% (+44%) |
| Efficiency | 4 | 91% (+37%) | 78% (+53%) |

Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.

## Tier 1: Static Validation Summary

Tier 1 validation passed with observations. Skill Evaluator ran 1 checks and found 7 total findings.

Top findings:

- MEDIUM SCHEMA/folder_hierarchy: Unexpected nesting depth for general skill (`skills/tools/doca-bench`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/tools/doca-bench/SKILL.md`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/tools/doca-bench/SKILL.md`)
- MEDIUM SCHEMA/author_missing: Author not specified in metadata (`skills/tools/doca-bench/SKILL.md`)
- LOW SCHEMA/unexpected_file: Unexpected 'SKILLCARD.yaml' in skill root (`skills/tools/doca-bench/SKILLCARD.yaml`)

## Tier 2: Deduplication Summary

This tier was not run or did not produce findings in this report.

## Publication Recommendation

The skill is suitable to proceed toward Skill Evaluator publication based on this benchmark. Skill owners should keep this file with the skill and refresh it when the evaluation dataset, skill behavior, or target agents materially change.

# DOCA Bench — Capabilities

**Where to start:** `doca_bench` is a single CLI binary (plus a
companion app for the remote half of remote-memory / RDMA / Eth
scenarios); the pattern overview below names the recurring
`doca_bench`-class questions. Pick the pattern first, then drill
into the H2 that owns the substance. For the *how* of executing
each pattern, jump to [TASKS.md](TASKS.md).

This file is loaded by [`SKILL.md`](SKILL.md). It documents *what
`doca_bench` is*, *what it can measure across DOCA libraries*,
*what versions it ships in*, *what its layered error and
observability surfaces look like*, and *the safety posture* the
public guide stakes out (notably the *"not for production
deployments"* warning on the companion-app channel). For
step-by-step invocations and the smoke-before-bulk workflow, see
[`TASKS.md`](TASKS.md).

## Pattern overview

Every `doca_bench`-class question this skill teaches resolves into
one of SIX patterns. The patterns are CLASSES — they apply across
every DOCA library bench can drive, not just one.

| `doca_bench` pattern | Class shape | Where the substance lives |
| --- | --- | --- |
| 1. Pick the target library | Cross-library — bench can drive AES-GCM, Comch, Compress, DMA, EC, Eth, RDMA, SHA, GPUNetIO. Picking the target library is axis 1 of the three-axis configuration. | [`## Capabilities and modes`](#capabilities-and-modes) cross-library table + [TASKS.md ## configure](TASKS.md#configure) step 1 |
| 2. Pick the workload shape | The pipeline-of-steps model is axis 2 — what *operation* is the device being asked to do on each iteration (e.g. compress vs decompress, send vs receive, encrypt vs decrypt), with the data provider, batching, and remote-memory choices that go with it. | [`## Capabilities and modes`](#capabilities-and-modes) workload-shape rules + [TASKS.md ## configure](TASKS.md#configure) step 3 |
| 3. Pick the measurement axis | Throughput vs bulk-latency vs precision-latency vs max-bandwidth is axis 3 — the shipped `doca_bench` binary defines FOUR benchmark modes in `tools/bench/doca_bench/configuration.hpp`. The public guide documents the modes as **not interchangeable**; a single number reported without naming the mode is ambiguous and the agent must surface that. | [`## Capabilities and modes`](#capabilities-and-modes) measurement-axis table + [TASKS.md ## configure](TASKS.md#configure) step 4 |
| 4. Smoke-before-bulk | Confirm the bench can talk to the target device with a trivial workload before a long run; predeclare the workload's acceptable run-to-run tolerance, then require two consecutive runs within it before calling the result stable | [TASKS.md ## configure](TASKS.md#configure) tolerance step + [TASKS.md ## run](TASKS.md#run) smoke flow + [TASKS.md ## test](TASKS.md#test) eval-loop overlay |
| 5. Diagnose a bench failure | Walk the layered error taxonomy in [`## Error taxonomy`](#error-taxonomy) — config-syntax / device-binding / library-precondition / workload-precondition / measurement-soundness / version / cross-cutting — instead of guessing at causes from a stack trace. | [`## Error taxonomy`](#error-taxonomy) + [TASKS.md ## debug](TASKS.md#debug) |
| 6. Interpret a bench number | A bench number is only meaningful with the (command line + DOCA version + device + as-deployed environment) four-tuple. Quoting one without the other three is the cross-version regression-hunt failure mode. | [`## Observability`](#observability) + [TASKS.md ## test](TASKS.md#test) baseline-capture rule |

Two cross-cutting rules that apply to *every* pattern above:

- **`doca_bench` is cross-library, not single-library.** Recommendations
  that frame it as *"the doca_rdma benchmarker"* or *"the
  doca_compress benchmarker"* are categorically wrong; bench is
  the harness, the library is the unit under test. An agent
  picking one DOCA library and never naming the others is
  silently narrowing the answer.
- **Warm-up is part of the measurement, not part of the bug.**
  The public guide documents a warm-up period (a non-zero number
  of jobs run through the pipeline before measurement starts).
  Treating a first-iteration number as steady-state, or stripping
  the warm-up to *"get a faster answer"*, is the canonical
  measurement-soundness failure mode and the reason the bench
  ships the concept in the first place.

## Capabilities and modes

`doca_bench` is shipped as a single CLI binary at
`/opt/mellanox/doca/tools/doca_bench` on every DOCA install (host
or BlueField Arm) since DOCA 2.7.0, plus a separate **companion
app** that is part of the same install and provides the remote
half of remote-memory / RDMA / Eth scenarios. The two halves
communicate over an out-of-band channel (TCP/IP sockets or a DOCA
Comch channel per the public guide). There is no daemon, no
library to link against, and no programmatic API for the harness
itself — the entire interaction model is *configure the
invocation, run the binary, read the printed and/or CSV output*.

**Three-axis configuration model — the load-bearing concept.**
Every `doca_bench` invocation commits to a point in this space;
omitting any axis produces an ambiguous result.

| Axis | What it picks | Why the agent must name it |
| --- | --- | --- |
| 1. Target library | Which DOCA library is the unit under test (AES-GCM / Comch / Compress / DMA / EC / Eth / RDMA / SHA / GPUNetIO per the public guide's documented surface). | The cross-library scope is the bench's defining feature; an answer that picks one library and never names the others has silently narrowed the bench to a single-library tool. |
| 2. Workload shape | The pipeline-of-steps the device runs each iteration — direction (e.g. send vs receive for RDMA / Eth, encrypt vs decrypt for AES-GCM, compress vs decompress for COMPRESS), the data provider (file / file-set / random-data per the public guide), batching, and whether remote-memory input / output is in play. | Two runs against the *same* library can report wildly different numbers if the workload shape differs; quoting "DOCA Compress throughput" without naming compress-vs-decompress, the data provider, and the buffer sizing is ambiguous. |
| 3. Measurement axis | Throughput vs bulk-latency vs precision-latency vs max-bandwidth per the four documented operating modes (the shipped binary's `benchmark_mode` enum has four values; the public guide enumerates them). | The modes are explicitly **not interchangeable** — precision-latency disables batching, bulk-latency uses bucketing, throughput maximizes pipeline occupancy, max-bandwidth holds the pipeline at the saturation point. Comparing a precision-latency number to a throughput-mode number (or a max-bandwidth number to either) is the cross-mode apples-to-oranges failure. |

**Cross-library scope — what bench can drive.** The public DOCA
Bench guide enumerates the supported DOCA libraries explicitly;
the bench's own built-in query system reports which subset is
*actually installed and exposed on the running host* (the
"granular build support" model). The library set on any given
install is the intersection of "documented by bench" and
"installed by this DOCA package selection" — the agent must
probe, not assume.

**Pipeline of steps — the workload primitive.** Bench expresses
a workload as a pipeline of steps, where each step is a
documented operation on a DOCA library (e.g. an Ethernet
receive, a SHA hash, a compress). The public guide reports that
DOCA currently supports running a single pipeline at a time per
invocation. Multi-step pipelines (e.g. receive → decompress →
hash) are explicitly in scope; multi-pipeline-per-invocation is
out of scope and the agent must not propose it.

**Multi-core / multi-thread scaling.** Bench creates execution
threads with CPU affinities the operator picks (the public guide
documents `--core-mask`, `--core-list`, `--core-count`,
`--threads-per-core` as the family — exact flag names should be
re-confirmed against `--help` on the installed version). Each
thread independently runs the configured pipeline against its
own jobs pool. The number of cores / threads and their NUMA
placement is part of axis 2 (workload shape) for any number a
later run is going to be compared against.

**Operating modes.** Per the public guide:

| Mode | What it optimizes for | What it sacrifices |
| --- | --- | --- |
| `throughput` (default) | Maximum pipeline occupancy and aggregate bandwidth / op-rate | Per-job latency precision; results are aggregate, not per-job |
| `bulk-latency` | A balance — submits batches, measures the time from first-submit to last-complete per batch, uses bucketing to report latency distribution | Per-job precision; latency is per-batch + bucketed, not per-job |
| `precision-latency` | Per-job latency precision (min / max / median / percentiles) by submitting one job at a time | Throughput; the pipeline is deliberately serialized and capacity is wasted by design |
| `max-bandwidth` | Pushes the workload at maximum sustainable bandwidth — separate operating mode from `throughput` in the parser (`tools/bench/doca_bench/configuration.hpp` defines four `benchmark_mode` enum values: `throughput`, `bulk_latency`, `precision_latency`, **`max_bandwidth`**; the CLI parser at `tools/bench/doca_bench/impl/configuration_parser.cpp` accepts the kebab-case spelling `max-bandwidth`). Use when the question is about the achievable bandwidth ceiling rather than aggregate op-rate. | Per-job latency precision; bandwidth-pinned runs deliberately hold the pipeline at the saturation point. |

**Built-in query system.** Bench ships a documented query
surface that reports which DOCA libraries are *installed* on
the running host and the supported sweep attributes. The agent
treats the query output as the install-side ground truth for
*"is library X benchable on this box"*; absence in the query
output is the canonical *"granular build does not include X"*
answer.

**Companion app.** Remote-memory / RDMA / Eth scenarios require
the companion app on the far side, communicating with the bench
host over the documented out-of-band channel (TCP/IP sockets or
DOCA Comch). The agent's rule for remote scenarios: a companion
app is not optional — bench will not synthesize the remote half
on its own.

## Version compatibility

For the canonical DOCA version-detection chain, the four-way
match rule, NGC container semantics, and the headers-win-over-docs
rule, see [`doca-version`](../../doca-version/SKILL.md). The body
lives there; this skill does not duplicate it.

**The `doca_bench`-specific overlay** is:

- **`doca_bench` is available since DOCA 2.7.0** per the public
  DOCA Bench guide's prerequisites section. On older installs
  the binary is not present; the right answer for *"I can't
  find doca_bench"* is to confirm the installed version per
  [`doca-version TASKS.md ## configure`](../../doca-version/TASKS.md#configure)
  and, if `< 2.7.0`, route to
  [`doca-setup`](../../doca-setup/SKILL.md) for an upgrade
  rather than recommending alternative tools.
- **Granular build means the *available library* set is
  install-specific.** The public guide documents granular build
  support — bench probes the install and only exposes libraries
  that are actually present. A flag / scenario reachable on one
  install can be silently absent on another even at the same
  DOCA version. The agent's rule: re-run the built-in query
  per [`TASKS.md ## configure`](TASKS.md#configure) step 2
  before quoting which libraries are benchable.
- **Companion app version must match the bench version.** Bench
  and the companion app are shipped together; mixing a bench
  binary from one DOCA version with a companion app from
  another is unsupported and falls into the partial-install
  layer of [`doca-version CAPABILITIES.md ## Version compatibility`](../../doca-version/CAPABILITIES.md#version-compatibility).
- **Output format stability is not contractually frozen.** The
  documented operating modes are stable; the exact textual
  layout of stdout, the CSV column order, and the screen-mode
  histogram rendering can shift across releases. Agents that
  need to consume bench output programmatically should prefer
  the structured helper per
  [`doca-structured-tools-contract`](../../doca-structured-tools-contract/SKILL.md#schemas)
  when present and re-verify the textual layout against the
  user's installed version when absent.
- **Per-platform support matrix per the public guide.** The
  documented BlueField generation support varies per library
  (e.g. some library operations are documented for BlueField-3
  but not BlueField-2). Do not copy a library-availability
  claim from one BlueField generation to another; re-read the
  public matrix per
  [`doca-public-knowledge-map ## DOCA tools`](../../doca-public-knowledge-map/SKILL.md#doca-tools).

## Error taxonomy

`doca_bench`'s error surface is broader than `doca_caps` because
the tool *does* configure devices, allocate buffers, drive
hardware, and produce measured numbers — each of which has its
own failure mode. The error layers the agent should distinguish,
in escalating order:

1. **Config-syntax.** The invocation itself does not parse:
   unknown flag, malformed value (e.g. a unit suffix bench does
   not accept), missing required argument for the chosen mode,
   conflicting flags. Cause: the operator wrote a flag string
   that does not exist in `--help` on the installed version
   (often a flag taken from prose / blog / older release).
   Routing: re-read `--help` on the installed binary, and the
   public DOCA Bench guide via
   [`doca-public-knowledge-map ## DOCA tools`](../../doca-public-knowledge-map/SKILL.md#doca-tools);
   do not guess.
2. **Device-binding.** Invocation parses; bench cannot bind the
   target device. Cause: device PCIe address / IB name /
   interface name does not exist on this host, NUMA placement
   of the chosen cores is wrong, or the underlying driver stack
   (`mlx5_core`, IB stack, etc.) is not loaded. Routing: first
   verify the device is visible to DOCA at all via
   [`doca-caps ## run`](../doca-caps/TASKS.md#run) and
   [`doca-setup ## test`](../../doca-setup/TASKS.md#test); only
   re-attempt the bench invocation once the device is on the
   capability snapshot.
3. **Library-precondition.** Device bound; bench refuses to
   exercise the requested DOCA library on this install. Cause:
   the granular-build setup did not include the library (its
   `pkg-config` module is missing, its samples are missing);
   the library is present but the device does not support the
   requested operation per its capability surface; the library
   is present but the BlueField generation is outside the
   public guide's per-platform support matrix. Routing: bench's
   own query system per [`TASKS.md ## configure`](TASKS.md#configure)
   step 2 + the per-library skill (e.g.
   [`doca-comch`](../../libs/doca-comch/SKILL.md),
   [`doca-compress`](../../libs/doca-compress/SKILL.md)) for
   the library-internal capability rules.
4. **Workload-precondition.** Library exercisable; the workload
   shape is invalid for the library. Cause: a data provider
   the library does not accept (e.g. random-data fed to a
   decompression scenario that needs structured input), a
   buffer / job sizing the device does not support, a remote-
   memory choice without the companion app on the far side, a
   pipeline-of-steps the library cannot chain. Routing: re-walk
   axis 2 (workload shape) of the three-axis model in
   [`## Capabilities and modes`](#capabilities-and-modes);
   bench will not silently substitute a workload, and the
   agent must not either.
5. **Measurement-soundness.** The run completes and reports
   numbers, but the numbers are unsound and must not be quoted
   as-is. Three sub-layers, all documented by the public guide:
    - *Warm-up not applied / too short.* Reported numbers
      include cold-cache / cold-pipeline iterations and are
      lower than steady-state. Fix: confirm the warm-up
      configuration matches the public guide's documented
      default and is appropriate for the chosen library /
      mode.
    - *Steady-state not reached.* Run duration / job count is
      too small for the pipeline to settle, and the reported
      number is in the transient region. Fix: lengthen the
      run via the documented duration / job-count limits and
      re-iterate per [`TASKS.md ## test`](TASKS.md#test).
    - *Outliers / distribution unreported.* A single
      throughput average hides a heavy tail; a precision-
      latency mean hides a 99.99-percentile spike that the
      consumer workload will actually feel. Fix: report the
      distribution (the bulk-latency histogram or the
      precision-latency percentile breakdown) alongside any
      single number.
6. **Version.** Cross-cutting partial-install / mixed-version
   layer per [`doca-version CAPABILITIES.md ## Version compatibility`](../../doca-version/CAPABILITIES.md#version-compatibility).
   Symptoms: bench binary version disagrees with `doca_caps
   --version` or `pkg-config --modversion doca-common`,
   companion-app version disagrees with the bench-host version,
   public guide version the operator is reading disagrees with
   the install. Routing: walk
   [`doca-version TASKS.md ## debug`](../../doca-version/TASKS.md#debug)
   before any further investigation.
7. **Cross-cutting.** The cause is below DOCA — driver /
   firmware / NUMA / hugepages / OS. Symptoms that do not fit
   layers 1-6 (e.g. throughput numbers that fall sharply only
   on one NUMA node, latency spikes correlated with kernel-
   thread scheduling, throughput tied to firmware version
   independent of DOCA version). Routing: hand off to
   [`doca-debug ## debug`](../../doca-debug/SKILL.md) and
   [`doca-setup ## debug`](../../doca-setup/TASKS.md#debug);
   the bench surface has reached its limit.

`doca_bench` does not *itself* participate in the cross-library
`DOCA_ERROR_*` taxonomy that DOCA libraries return through their
C API; bench is a CLI driving libraries, not a library call. For
the cross-library `DOCA_ERROR_*` taxonomy and the program-side
debug order, see
[`doca-programming-guide CAPABILITIES.md ## Error taxonomy`](../../doca-programming-guide/CAPABILITIES.md#error-taxonomy).

## Observability

`doca_bench`'s observability surface is **the measurement output
itself**, plus the built-in query system. Specifically:

- **Stdout summary.** Every run prints an aggregate summary on
  stdout per the public guide — the documented examples include
  duration, enqueued / dequeued job counts, throughput / ingress
  / egress rates for throughput mode, and a histogram + bucket
  breakdown for bulk-latency mode, and min / max / median / mean
  / percentile rows for precision-latency mode. The exact
  textual layout is install-specific; re-verify against the
  user's run.
- **CSV output.** The public guide documents a CSV output path
  with options to append, separate dynamic values, and emit
  environment information alongside the numbers. CSV is the
  documented machine-readable surface — the agent's rule for
  any baseline that will be re-read by a later run: capture
  CSV, not just stdout.
- **Real-time stats interval.** The public guide documents a
  real-time stats interval so the operator can see the run
  progress without waiting for completion. Useful when
  diagnosing whether a long run has reached steady state.
- **Query subsystem.** The bench's `Queries` section in the
  public guide documents a built-in mechanism for reporting
  device capabilities and supported sweep attributes. This is
  the install-side ground truth for *"can this run even
  attempt library X"* before committing to a full run.
- **Reported invocation echo.** Per the public guide, bench
  echoes the full set of configured values at the start of the
  run so a captured log self-documents the (command line +
  effective defaults) the numbers belong to. The agent must
  preserve this echo in any captured baseline; it is the
  *"what command produced this number"* leg of the four-tuple
  in [`## Pattern overview`](#pattern-overview) pattern 6.

For the cross-cutting env-side observability primitives
(representor enumeration, `devlink dev show`, `mlxconfig`) see
[`doca-setup CAPABILITIES.md ## Observability`](../../doca-setup/CAPABILITIES.md#observability).
For the program-side observability surface (DOCA log levels,
`DOCA_LOG_LEVEL`, `--sdk-log-level`) see
[`doca-programming-guide CAPABILITIES.md ## Observability`](../../doca-programming-guide/CAPABILITIES.md#observability).

## Safety policy

> **Overlay on the bundle-wide hardware-safety meta-policy.** The rules below are this skill's per-artifact overlay on the cross-cutting rules in [`doca-hardware-safety` CAPABILITIES.md ## Safety policy](../../doca-hardware-safety/CAPABILITIES.md#safety-policy) (specifically [### Per-artifact overlay pattern](../../doca-hardware-safety/CAPABILITIES.md#per-artifact-overlay-pattern)). When the two layers disagree, the stricter wins; when either layer says STOP, the agent stops.

`doca_bench` is a measurement tool and a more powerful surface
than `doca_caps`; it *does* allocate buffers, bind devices, and
drive hardware, including remote-memory operations over an
out-of-band channel. The safety rules:

- **Not for production deployments.** The public guide carries
  an explicit warning that the bench is not intended for
  production deployment, and that the companion-app out-of-band
  channel can carry sensitive information. The agent must
  surface this warning whenever the user proposes running bench
  on a host that also serves production traffic, and must not
  recommend running the companion app over an untrusted network
  segment without the public guide's secure-channel guidance.
- **Smoke-before-bulk; never run a long sweep first.** A swept
  run on the wrong device, wrong workload, or wrong mode
  consumes hours and produces unusable data. The agent's rule
  is the [`TASKS.md ## run`](TASKS.md#run) smoke step (trivial
  workload, short duration) before any sweep or long run.
- **Quote the (command line + version + device + environment)
  four-tuple, not just the number.** A bench number quoted
  without the four-tuple is unreplicable and unfalsifiable.
  This rule applies to every output of this skill — the *most
  common* downstream misuse of bench is quoting a screenshot
  from one platform as if it described another.
- **Predeclare tolerance; require two consecutive stable runs.**
  The workload owner supplies an acceptable absolute or percentage
  delta before measurement. A stable result requires two
  consecutive runs within that tolerance. If the bounded re-run
  cannot meet it, report the variance and escalate; do not invent
  or relax a tolerance after seeing the numbers.
- **Do not invent flags, scenario names, attribute names, or
  metric names.** The documented invocations and the installed
  `--help` are the authoritative surface. Prose-derived flags
  are the most common hallucination failure for this skill;
  see the cross-cutting rule in
  [`TASKS.md ## Command appendix`](TASKS.md#command-appendix).
- **Host vs BlueField Arm execution rule.** Per the public
  guide the binary is the same on both sides; the *measured*
  numbers differ because the environment differs. An agent
  comparing a host-side number to a BlueField-Arm-side number
  without naming where each ran is making the cross-platform
  apples-to-oranges mistake.

## Public-source pointer

The single canonical public source for `doca_bench` is the
**DOCA Bench** page on `docs.nvidia.com`, reachable through
[`doca-public-knowledge-map ## DOCA tools`](../../doca-public-knowledge-map/SKILL.md#doca-tools).
Do not invent flags, scenario names, attribute names, metric
names, or supported library entries beyond what that page
documents — and re-verify against `--help` on the user's
installed binary, since granular-build support means the
*available* surface is install-specific within the *documented*
surface.

## Description: <br>
Run doca_bench (DOCA 2.7.0 or newer) to measure throughput, bulk latency, precision latency, or maximum bandwidth for RDMA, Compress, AES-GCM, SHA, DMA, EC, Ethernet, Comch, or GPUNetIO on a host or BlueField Arm. <br>

This skill is ready for commercial/non-commercial use. <br>

## Owner
NVIDIA <br>

### License/Terms of Use: <br>
Apache 2.0 AND CC-BY-4.0 <br>
## Use Case: <br>
Developers and engineers who need a reproducible, vendor-supported way to measure DOCA library performance on their actual install and device, including baseline capture, regression testing, and cross-library throughput comparison. <br>

### Deployment Geography for Use: <br>
Global <br>

## Requirements / Dependencies: <br>
**Requires API Key or External Credential:** [No] <br>
**Credential Type(s):** [None] <br>

Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. <br>

## Known Risks and Mitigations: <br>
Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br>
Mitigation: Review and scan skill before deployment. <br>

## Reference(s): <br>
- [DOCA SDK Documentation](https://docs.nvidia.com/doca/sdk/index.html) <br>
- [DOCA Samples (GitHub)](https://github.com/NVIDIA-DOCA/doca-samples) <br>
- [DOCA Platform Framework (GitHub)](https://github.com/NVIDIA/doca-platform) <br>
- [DOCA Developer Forum](https://forums.developer.nvidia.com/c/infrastructure/doca/370) <br>


## Skill Output: <br>
**Output Type(s):** [Shell commands, Analysis, Configuration instructions] <br>
**Output Format:** [Markdown with inline bash code blocks] <br>
**Output Parameters:** [1D] <br>
**Other Properties Related to Output:** [None] <br>

## Evaluation Agents Used: <br>
- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`) <br>
- Codex (`openai/openai/gpt-5.5`) <br>



## Evaluation Tasks: <br>
Evaluated against 4 evaluation tasks (3 positive skill-activation, 1 negative). <br>

## Evaluation Metrics Used: <br>
Reported benchmark dimensions: <br>
- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
- Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>

Underlying evaluation signals used in this run: <br>
- `security`: Checks for unsafe operations, secret leakage, and unauthorized access. <br>
- `skill_execution`: Verifies that the agent loaded the expected skill and workflow. <br>
- `skill_efficiency`: Checks routing quality, decoy avoidance, and redundant tool usage. <br>
- `accuracy`: Grades final-answer correctness against the reference answer. <br>
- `goal_accuracy`: Checks whether the overall user task completed successfully. <br>
- `behavior_check`: Verifies expected behavior steps, including safety expectations. <br>



## Evaluation Results: <br>
| Dimension | Num | Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`) | Codex (`openai/openai/gpt-5.5`) |
|---|---:|---:|---:|
| Security | 4 | 100% (+0%) | 100% (+0%) |
| Correctness | 4 | 100% (+25%) | 100% (+20%) |
| Discoverability | 4 | 97% (+34%) | 95% (+45%) |
| Effectiveness | 4 | 100% (+81%) | 86% (+44%) |
| Efficiency | 4 | 91% (+37%) | 78% (+53%) |

## Skill Version(s): <br>
56cf891 (source: git SHA, committed 2026-07-26) <br>

## Ethical Considerations: <br>
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

(For Release on NVIDIA Platforms Only) <br>
Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). <br>

---
license: Apache-2.0
name: doca-bench
description: >
  Run `doca_bench` (DOCA 2.7.0 or newer) to measure throughput,
  bulk latency, precision latency, or maximum bandwidth for RDMA,
  Compress, AES-GCM, SHA, DMA, EC, Ethernet, Comch, or GPUNetIO on
  a host or BlueField Arm. Use it to discover enabled benchmark
  libraries, capture a reproducible command/version/device/environment
  baseline, compare stable runs against a declared tolerance, or
  diagnose configuration, device-binding, workload-precondition, and
  measurement failures. Trigger for requests such as measuring
  BlueField compression speed, NIC RDMA throughput, crypto latency,
  or a pre-upgrade baseline. Do not use for application end-to-end
  timing, custom benchmark code, DOCA installation, or binary patches.
metadata:
  kind: tool
compatibility: >
  Requires DOCA SDK ≥ 2.7.0 installed at /opt/mellanox/doca on
  Linux (Ubuntu 22.04/24.04 or RHEL/SLES) with a BlueField DPU or
  ConnectX NIC attached and the `doca_bench` binary present at
  /opt/mellanox/doca/tools/doca_bench. Companion app must run on
  the far side for remote-memory / RDMA / Eth scenarios; host and
  BlueField-Arm execution both supported.
---

# DOCA Bench (`doca_bench`)

**Where to start:** This is a tool skill for invoking `doca_bench`,
the cross-library micro-benchmark harness. Open
[`TASKS.md`](TASKS.md) and start at
[`## configure`](TASKS.md#configure) for the three-axis decision
(target library × workload shape × measurement axis), then
[`## run`](TASKS.md#run) for the smoke-before-bulk flow. Open
[`CAPABILITIES.md`](CAPABILITIES.md) when the question is *what
`doca_bench` can measure*, *which DOCA libraries it can drive*, or
*how to interpret throughput / latency / op-rate output without
fooling yourself on warm-up or steady-state*. If DOCA is not
installed yet, route to
[`doca-setup`](../../doca-setup/SKILL.md) first; if the install
version is < 2.7.0, `doca_bench` is not shipped on this host.

## Example questions this skill answers well

The CLASSES of `doca_bench` questions this skill is built to answer,
each with one worked example. The class is the load-bearing piece;
the worked example is one instance.

- **"What does this DOCA library actually deliver on this device?"** —
  worked example: *"throughput of DOCA Compress on my BlueField-3"*.
  Answered by the three-axis configuration in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  + the smoke-before-bulk flow in
  [`TASKS.md ## run`](TASKS.md#run). The *same* shape answers
  *"send-side throughput of DOCA RDMA"* — `doca_bench` is
  cross-library, not single-library.
- **"Which DOCA libraries can `doca_bench` actually drive on this
  install?"** — worked example: *"is doca_sha enumerable on a
  granular-build install"*. Answered by the built-in query system
  surfaced in
  [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
  + [`TASKS.md ## configure`](TASKS.md#configure) step 2
  (probe-before-bench). Empty enumeration = library not installed,
  not bench failure.
- **"Is this number reliable, or did I miss the warm-up?"** —
  worked example: *"why does my first-second number differ from my
  steady-state number"*. Answered by the measurement-soundness
  overlay in
  [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy)
  layer 5 + [`TASKS.md ## test`](TASKS.md#test) (the eval-loop
  overlay treats warm-up / steady-state / outliers as
  re-iteration triggers, not one-shot facts).
- **"Bench reports zero throughput / hangs at start / disagrees
  with the public docs."** — worked example: *"`doca_bench` shows
  zero ops for AES-GCM but `doca_caps` says the device supports
  it"*. Answered by the layered error taxonomy in
  [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy)
  (config-syntax → device-binding → library-precondition →
  workload-precondition → measurement-soundness → version →
  cross-cutting) + [`TASKS.md ## debug`](TASKS.md#debug).
- **"How do I capture a baseline I can later regression-test
  against?"** — worked example: *"snapshot decompress throughput
  on this BlueField + DOCA version before a firmware update"*.
  Answered by the CSV output + version-overlay rule in
  [`TASKS.md ## test`](TASKS.md#test) (capture command line +
  version + device + as-deployed environment alongside the
  numbers; quoting numbers without the four-tuple is the
  cross-version regression-hunt failure mode).
- **"`doca_bench` returns nothing for library X — what does that
  mean?"** — worked example: *"empty output for DOCA SHA"*.
  Answered by the empty-output interpretation rules in
  [`TASKS.md ## debug`](TASKS.md#debug) +
  [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy).
  Re-route through
  [`doca-caps`](../doca-caps/SKILL.md) for the coarse
  per-device per-library capability ground truth, then back
  into bench once the capability is confirmed present.

## Audience

This skill serves **external operators, developers, and AI agents
who need a reproducible, vendor-supported way to measure DOCA
library performance on the user's actual install and device**.
Concretely:

- An external developer choosing between DOCA libraries (e.g.
  COMPRESS vs SHA vs DMA throughput) before committing an
  application design.
- A platform operator validating a tuning change (NUMA pinning,
  driver upgrade, firmware burn) by re-running a captured
  `doca_bench` baseline against the new state.
- An SRE / performance engineer producing a *"this is what the
  device delivers today"* artifact that downstream consumers
  (capacity planning, regression bisection) can cite.
- An AI agent answering *"what throughput / latency should I
  expect from DOCA library X on device Y?"* honestly — with a
  measured number, the command line that produced it, and the
  version + device + environment that scopes it — instead of
  guessing from datasheet headlines.

It is **not** for users debugging the `doca_bench` source code,
and **not** a substitute for the live public DOCA Bench guide on
`docs.nvidia.com`.

`doca_bench` is shipped as a **tool** (a single CLI binary plus a
companion app for the remote half of remote-memory / RDMA / Eth
scenarios), not a library you link against. The skill uses the
same `kind: tool` three-file shape as the rest of the bundle so
the agent's task-verb contract
(`configure / build / modify / run / test / debug`) is uniform
across libraries, services, and tools — even when individual
verbs collapse to a routing stub for a shipped binary.

## When to load this skill

Load this skill when the user is — or the agent needs to — invoke
`doca_bench` on a real host with DOCA ≥ 2.7.0 installed (or
inside the public NGC DOCA container with the equivalent version)
to measure performance of a DOCA library. Concretely:

- Picking *which* DOCA library to benchmark for a candidate
  workload (RDMA vs COMPRESS vs DMA, etc.).
- Picking *which* measurement axis to ask for (throughput vs bulk
  latency vs precision latency vs max-bandwidth) — the four modes
  defined in `tools/bench/doca_bench/configuration.hpp` are not
  interchangeable.
- Probing the install's granular-build state so the agent can
  honestly report *"this library is not exposed on this install"*
  instead of inventing a workload.
- Capturing a documented baseline (command line + version + device
  + as-deployed environment + numbers) for later regression hunts.
- Requiring the workload owner to predeclare acceptable variance
  and obtaining two consecutive runs within that tolerance before
  reporting a stable result; otherwise escalating the variance.
- Diagnosing why a bench run reported zero / unstable / unexpected
  results (the error-taxonomy walk in
  [`TASKS.md ## debug`](TASKS.md#debug)).

Do **not** load this skill for general DOCA orientation, library
API work, or installation. For those, use
[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md),
the matching `libs/<library>` skill, or
[`doca-setup`](../../doca-setup/SKILL.md). Do not load it for
*application-level* end-to-end benchmarking either — `doca_bench`
measures the DOCA library surface, not the user's application
above it.

## What this skill provides

This is a **thin loader**. Substantive material lives in two
companion files:

- `CAPABILITIES.md` — what `doca_bench` can measure (the
  cross-library scope, the three-axis configuration model, the
  documented operating modes, the warm-up / pipeline / multi-core
  concepts that constrain measurement soundness), the version
  overlay (`doca-bench`-specific facts on top of the canonical
  `doca-version` rules), the layered error taxonomy
  (config-syntax / device-binding / library-precondition /
  workload-precondition / measurement-soundness / version /
  cross-cutting), the observability surface (screen + CSV
  output, real-time stats, query system), and the safety
  posture (the public guide's *"not for production"* warning,
  the host vs BlueField execution rule, the companion-app
  attack surface).
- `TASKS.md` — step-by-step workflows for the in-scope task
  verbs: `configure` (the three-axis decision + the
  probe-before-bench step), `build` (route to install — the
  binary is shipped, the companion app is shipped), `modify`
  (refuse — do not patch the bench binary; modify the bench
  *invocation* instead), `run` (the smoke-before-bulk flow),
  `test` (the eval loop — warm-up, steady-state, outliers,
  cross-version), `debug` (walk the error taxonomy layer by
  layer), plus a `Deferred task verbs` block routing
  out-of-scope questions and a `Command appendix` of
  `doca_bench`-specific invocation classes.

The skill assumes a host where DOCA ≥ 2.7.0 is already installed
(or the public NGC DOCA container is running at an equivalent
version) and the operator has whatever permissions the public
guide requires for `doca_bench` to bind devices and allocate
resources on their platform.

## What this skill deliberately does not ship

This skill is **agent guidance**, not a samples or scripts
bundle. To keep the boundary clean, it deliberately does not
contain — and pull requests should not add:

- **Specific flag strings or scenario / metric / attribute names
  beyond what the public DOCA Bench guide documents.** The flag
  surface evolves and is install-specific; the documented
  invocations + `--help` on the installed version are the
  authoritative answer. Inventing a flag is the most common
  hallucination failure for this skill.
- **Pre-baked example output or expected throughput numbers.**
  Bench output is device-, version-, firmware-, NUMA-, and
  tuning-specific. A captured number pinned to one platform and
  one DOCA version misleads operators on a different
  platform / version.
- **Wrappers, parsers, or scripts** in any language that consume
  `doca_bench` CSV or stdout. The output formats are documented;
  if a user wants to script against them, the right answer is
  "read the live guide, write the parser against your installed
  version".
- **A `samples/` or `reference/` subtree.** This is a thin
  loader for a documented CLI; substantive material lives on
  the public page and in `--help`.

## Loading order

1. Read this `SKILL.md` first to confirm the user's question is
   in scope (the user actually wants to invoke `doca_bench` for
   measurement, not learn about a DOCA library in general).
2. **For what `doca_bench` measures, the three-axis model, the
   version overlay, the error taxonomy, observability surface,
   and safety posture, see [CAPABILITIES.md](CAPABILITIES.md).**
3. **For the documented invocations and the smoke-before-bulk
   workflow — `configure`, `build`, `modify`, `run`, `test`,
   `debug` — see [TASKS.md](TASKS.md).**

## Related skills

- [`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md)
  — routing to the public DOCA Bench page on `docs.nvidia.com`
  and the rest of the public DOCA documentation set.
- [`doca-version`](../../doca-version/SKILL.md) — the canonical
  version-detection chain, four-way match rule, NGC container
  semantics, and headers-win-over-docs rule. The
  `## Version compatibility` section in this skill is a thin
  overlay on top of `doca-version`; the body lives there.
- [`doca-structured-tools-contract`](../../doca-structured-tools-contract/SKILL.md)
  — the bundle-wide contract for structured-output helper tools.
  Bench-runner / bench-snapshot executables that satisfy the
  detect-prefer-fallback-report loop are deferred to PR2; the
  contract is consumed here in advance so the
  `## Command appendix` in [`TASKS.md`](TASKS.md) is infra-aware
  from PR1.
- [`doca-setup`](../../doca-setup/SKILL.md) — env preparation,
  install verification, hugepages, NUMA awareness, and the *I
  have no install yet* path with the public NGC DOCA container.
- [`doca-debug`](../../doca-debug/SKILL.md) — the cross-cutting
  debug ladder. Bench surfaces *its own* error taxonomy in
  [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy);
  when the cause turns out to be below DOCA (driver, firmware,
  NUMA), the bench taxonomy hands off to `doca-debug`.
- [`doca-caps`](../doca-caps/SKILL.md) — the sibling DOCA tool
  for the coarse per-device per-library capability snapshot.
  Bench probes capability at finer grain via its own query
  system; `doca_caps` is the cheaper first step to confirm the
  device is even visible to DOCA.
- The matching `libs/<library>` skill — e.g.
  [`doca-comch`](../../libs/doca-comch/SKILL.md),
  [`doca-compress`](../../libs/doca-compress/SKILL.md) — for
  the workload-side preconditions, capability-query rules, and
  error-taxonomy overlays of the library under test. Bench
  drives the library; the library skill explains *what
  "healthy" means for it*.

# SKILLCARD.yaml — provisional, signed at NVCARPS scan time
# Schema: https://github.com/NVIDIA/skills/blob/main/docs/skill-cards.mdx
# This file is generated by `runner/build_skillcards.py`; do not hand-edit.
# Re-run after any change to SKILL.md frontmatter or the deep-E2E baseline.

schema_version: "1"

identity:
  name: "doca-bench"
  artifact_id: "doca-bench"
  kind: "tool"
  version: "TBD-at-sign-time"          # NVCARPS fills from the source ref
  signature: "TBD-at-sign-time"        # NVCARPS-generated SHA + ed25519 sig
  signed_by: "TBD-at-sign-time"        # NVCARPS identity (NVIDIA-PSIRT signing key)

description: "Use this skill when the user is running `doca_bench` (DOCA ≥\n2.7.0) — the cross-library micro-benchmark harness — to measure\nthroughput, bulk-latency, precision-latency, or max-bandwidth\n(the four `benchmark_mode` enum values the binary ships) of a\nDOCA library (RDMA, COMPRESS, AES-GCM, SHA, DMA, EC, ETH,\nComch, GPUNetIO) on host or BlueField Arm, probe the granular-\nbuild query for which libraries the install exposes, capture a\nbaseline four-tuple (command, version, device, environment), or\ndiagnose a bench failure in the config-syntax, device-binding,\nlibrary/workload-precondition, or measurement-soundness layer.\nTrigger even without the word \"doca_bench\" — typical implicit\nforms include \"how fast is compress on my BlueField\", \"what\nRDMA throughput can this NIC do\", \"measure AES-GCM latency\",\nor \"baseline before a firmware update\". Refuse and route\nelsewhere for application-level end-to-end timing, custom\nbenchmark programs, DOCA install/upgrade, or patching the\nbinary."

source:
  repo: "NVIDIA-DOCA/doca-skills"
  path: "skills/tools/doca-bench/SKILL.md"
  branch: "main"
  license: "Apache-2.0 AND CC-BY-4.0"

provenance:
  scan_run_id: "TBD-at-sign-time"      # NVCARPS scan run UUID
  signed_at:   "TBD-at-sign-time"      # ISO-8601 timestamp from NVCARPS
  upstream_catalog: "https://github.com/NVIDIA/skills"
  upstream_component: "doca"           # see components.d/doca.yml

quality:
  baseline:
    verdict: "PASS"
    blocker_findings: 0
    suite:    "runner/reports/2026-05-26_r15_full_e2e"
    aggregate: "runner/reports/2026-05-26_r15_full_e2e/aggregate.json"
    summary:   "runner/reports/2026-05-26_r15_full_e2e/summary.md"
  regression_contract:
    file:    "runner/baseline_grades.json"
    rule:    "no PASS->FAIL on variant C since last green run"
    gate:    "ci/check-no-regression.py --variant C"

behavior_boundaries:
  documentation_only: true             # bundle ships no runnable code
  hardware_touching_changes_routed_to: "skills/doca-hardware-safety/"
  version_policy_routed_to: "skills/doca-version/"
  external_productized_software_routed_to: "skills/doca-public-knowledge-map/"
  see_also:
    - "AGENTS.md"                       # universal rules
    - "AUTHORING.md"                    # per-skill class-shape contract
    - "BENCHMARK.md"                    # how this skill is graded
    - "SECURITY.md"                     # safety-bug intake

contact:
  bug_intake: "https://github.com/NVIDIA-DOCA/doca-skills/issues"
  safety_bug_intake: "https://github.com/NVIDIA-DOCA/doca-skills/issues/new?labels=safety-bug"
  security:   "https://www.nvidia.com/en-us/security/"

# DOCA Bench — Tasks

**Where to start:** The verbs that carry real workflow content are
`## configure`, `## run`, `## test`, and `## debug`. The other two
substantive verbs (`build`, `modify`) carry routing stubs because
`doca_bench` is a shipped binary, not a source artifact the user
compiles or patches. The `## test` verb is an iterative loop, not
a one-shot pass — see the eval-loop overlay in `## test` below.

This file is loaded by [`SKILL.md`](SKILL.md) after
[`CAPABILITIES.md`](CAPABILITIES.md). It walks the agent through
the six task verbs every artifact in this bundle exposes
(`configure / build / modify / run / test / debug`), then
explicitly defers task verbs that do not belong here.

For `doca_bench`, the verbs that carry real workflow content are
`configure`, `run`, `test`, and `debug`. The other two verbs
*exist as anchors* because the agent's task-verb contract is
uniform across libraries, services, and tools — and each one
carries a meaningful **routing stub** that names where the user's
question really belongs.

## configure

The bench's *configuration* is the invocation: there is no
separate config file, no daemon, no env knob the public guide
documents as required (DOCA-wide env vars like `DOCA_LOG_LEVEL`
still apply, but they are owned by
[`doca-programming-guide CAPABILITIES.md ## Observability`](../../doca-programming-guide/CAPABILITIES.md#observability),
not by bench). What the agent has to *configure* is the three-axis
decision documented in
[`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes).

Steps the agent should walk the user through, in order:

1. **Confirm DOCA ≥ 2.7.0 is installed and the binary is
   present at `/opt/mellanox/doca/tools/doca_bench`.** If
   not, route to
   [`doca-version TASKS.md ## configure`](../../doca-version/TASKS.md#configure)
   for the detection chain and
   [`doca-setup ## install`](../../doca-setup/TASKS.md#configure)
   for the upgrade path. Do not propose alternative tools; on
   `< 2.7.0` installs the bench is genuinely not there.
2. **Probe the install's granular build first.** Use the
   bench's built-in query system (per
   [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability))
   to enumerate which DOCA libraries are *actually exposed* on
   this install before committing to a target. An empty
   enumeration for the user's target library means the
   granular build did not include it — route to
   [`doca-setup ## install`](../../doca-setup/TASKS.md#configure)
   for the install profile, not to a bench invocation.
3. **Axis 1 — pick the target library.** Per the cross-library
   table in
   [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes),
   commit explicitly to which DOCA library is the unit under
   test (e.g. COMPRESS, RDMA, AES-GCM, SHA, DMA, EC, Eth,
   Comch, GPUNetIO). An answer that picks one without naming
   the alternatives has silently narrowed the bench to a
   single-library tool; surface that to the user.
4. **Axis 2 — pick the workload shape.** Direction (e.g.
   send vs receive, encrypt vs decrypt, compress vs
   decompress), data provider (file / file-set / random-data
   per the public guide), buffer / job sizing, batching, NUMA
   placement of the chosen cores, and whether remote-memory
   input / output is in play (which pulls in the companion
   app — see step 6). Re-cross-check against the per-platform
   support matrix in
   [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility);
   not every workload shape is documented for every BlueField
   generation.
5. **Axis 3 — pick the measurement mode.** Throughput vs
   bulk-latency vs precision-latency vs max-bandwidth per
   the shipped binary's four `benchmark_mode` values and the
   public guide. Quote back to the user *why* you
   picked the mode — *"throughput because the user asked
   about aggregate op-rate"*, *"max-bandwidth because the
   user asked for the saturation ceiling"*,
   *"precision-latency because the user
   asked about per-job tail latency"* — so the user can
   challenge the framing if it does not match intent. The
   modes are not interchangeable; see
   [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
   for the trade-off table.
6. **If the workload is remote (RDMA, Eth, remote-memory),
   plan the companion app deployment.** The companion app is
   shipped with bench but runs on the far side, with the
   out-of-band channel between the two halves. Surface the
   public guide's *"not for production"* warning per
   [`CAPABILITIES.md ## Safety policy`](CAPABILITIES.md#safety-policy)
   and confirm a non-production segment is available.
7. **Predeclare workload tolerance.** Before the first
   measurement, record the metric being judged and the
   acceptable run-to-run absolute or percentage delta for this
   workload. The operator or workload owner supplies this
   tolerance; the agent must not invent `X%` after seeing the
   results. No declared tolerance means the run can be captured
   as exploratory evidence but cannot be called stable.

For the canonical DOCA universal lifecycle that underlies
program-side configuration (which `doca_bench` itself runs
internally per library), see
[`doca-programming-guide TASKS.md ## configure`](../../doca-programming-guide/TASKS.md#configure).
This skill is concerned with the *operator*-side configuration
of the bench invocation, not the program-side lifecycle of the
library under test.

## build

`doca_bench` and its companion app are **shipped pre-built** as
part of every DOCA install since 2.7.0
(`/opt/mellanox/doca/tools/doca_bench`). There is no source tree
the external user is expected to compile, no build flags, no
`meson` or `make` workflow for the bench itself.

Routing for nearby "build" questions:

- *"The binary isn't there — do I need to build it?"* → no.
  Route to
  [`doca-setup ## install`](../../doca-setup/TASKS.md#configure).
  The fix is to install (or re-install) DOCA at ≥ 2.7.0, or
  use the public NGC DOCA container per
  [`doca-setup ## no-install`](../../doca-setup/TASKS.md#no-install)
  at an equivalent version.
- *"I want to build my own benchmark program against DOCA
  library X."* → not a `doca_bench` question. Route to
  [`doca-programming-guide ## build`](../../doca-programming-guide/TASKS.md#build)
  for the cross-library build pattern and the matching
  `libs/<library>` skill (e.g.
  [`doca-comch ## build`](../../libs/doca-comch/TASKS.md#build),
  [`doca-compress`](../../libs/doca-compress/SKILL.md)) for
  the library-specific build overlay. The bench is the
  shipped harness; the user's bespoke harness is a
  different artifact.
- *"I want to extend the bench with a new scenario."* →
  out of scope here; this skill is for external operators
  consuming the shipped bench, not for contributors
  extending it.

The `## What this skill deliberately does not ship` block in
[`SKILL.md`](SKILL.md) explicitly forbids adding a build recipe
or wrappers for `doca_bench`; revisit that policy before
changing this section.

## modify

**Do not modify the shipped `doca_bench` binary.** It is an
NVIDIA-shipped CLI; there is no documented public way to change
its behavior, output format, scenario set, or attribute surface,
and none should be invented.

What the agent *does* modify, every time, is the **bench
invocation** — the flags, the chosen library, the workload
shape, the mode, the core / thread layout. That is the
configuration loop in [`## configure`](#configure) above and the
iteration loop in [`## test`](#test) below; treat *modify the
invocation, not the binary* as the operating mode.

Routing for nearby "modify" questions:

- *"The output format is inconvenient — can I change it?"* →
  the documented surfaces are stdout, CSV, and the real-time
  stats interval per
  [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability).
  If those are insufficient, the right answer is *"write a
  parser against the documented CSV format on your installed
  version"* — not a binary patch — and even that scripting is
  out of scope per
  [`SKILL.md ## What this skill deliberately does not ship`](SKILL.md#what-this-skill-deliberately-does-not-ship).
- *"Can I patch `doca_bench` to add scenario X?"* → out of
  scope for external users; this skill is for consumers of
  the shipped tool, not contributors to it.
- *"I need a *different measurement* than `doca_bench`
  reports."* → re-examine axis 3 (measurement mode) in
  [`## configure`](#configure) first; the four documented
  modes cover aggregate throughput, bulk-latency
  distribution, per-job precision latency, and the
  max-bandwidth saturation ceiling. If the question is genuinely outside
  bench's surface (e.g. application-level end-to-end timing),
  route to
  [`doca-programming-guide`](../../doca-programming-guide/SKILL.md)
  and the matching `libs/<library>` skill — the user's own
  program is the right place to measure end-to-end.

## run

The smoke-before-bulk flow — every bench session goes through
it, no exceptions. The full invocation surface lives in the
public DOCA Bench guide; this section names the *shape* of the
flow, not the verbatim command lines (per
[`CAPABILITIES.md ## Safety policy`](CAPABILITIES.md#safety-policy)
*"do not invent flags"*).

> **Do-not-invent guard (specific flag names).** Real downstream
> agents have hallucinated the following plausibly-named but
> non-existent flags for `doca_bench`: `--pipeline`, `--mode`,
> `--device`, `--csv-output`, `--target-library`. None of these
> appear in the bundle or in `doca_bench --help` on the public
> DOCA release. The flag inventory the bundle DOES name verbatim
> is `--core-mask`, `--core-list`, `--core-count`,
> `--threads-per-core`, the `--run-limit-*` family, `--sweep`,
> and the `--csv-*` family — and those are named as *classes*,
> not as ready-to-paste literals; the agent MUST read
> `doca_bench --help` on the installed bench and use only the
> names that appear there. Any example invocation written in
> this skill MUST keep tokens like `<flag-from-help>` or
> `<chosen-mode-from-help>` as placeholders for the
> per-install-variable inventory.

1. **Confirm the binary, version, and granular-build
   inventory.** Per [`## configure`](#configure) steps 1-2;
   without this the next four steps will burn the operator's
   time on a configuration that the install does not support.
2. **Smoke run — trivial workload, short duration.** Pick the
   smallest defensible workload for the chosen target library
   and mode (e.g. a tiny job count, a short time limit, the
   minimum data provider that the library will accept). The
   goal is *"bench can bind the device, start the pipeline,
   and emit numbers"*, not a usable measurement. The public
   guide documents the duration / job-count limit families
   (`--run-limit-*` per the user's installed `--help`); use
   the smallest defensible value of one of them.
3. **Read the echoed invocation.** Per
   [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability)
   the bench prints the full set of configured values at the
   start; this is the user's chance to catch a defaulted
   value that does not match intent (e.g. wrong NUMA node,
   wrong data provider, warm-up turned off) before the run
   completes.
4. **Inspect the summary and reject silently-bad runs.** Exit
   0 + a number is not enough — verify the number is in a
   defensible order of magnitude for the library / device,
   verify warm-up actually happened, verify the mode in the
   summary matches the requested mode. If anything looks off,
   loop back to [`## debug`](#debug) before sinking time
   into a longer run.
5. **Plan the bulk / swept run** only after the smoke is
   green and the workload tolerance from `## configure`
   step 7 is recorded. The public guide documents the `--sweep` family for
   parameter sweeps; the agent's rule for sweep planning is
   *enumerate the swept dimension explicitly, estimate the
   total run time, and confirm the operator is OK with the
   wall-clock cost* before committing.
6. **For remote scenarios, start the companion app first.**
   The companion app must be listening before the bench host
   tries to drive it; per
   [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
   the companion app is part of the install, not optional.

When recording the run for downstream consumers (the *baseline*
pattern), write down: the DOCA version, the host platform (host
vs BlueField Arm, OS, kernel, firmware), the exact command line
used (the bench's own echo line covers this), the device target
(PCIe address / IB name / interface name), and the full
unredacted summary + CSV. The downstream `## test` and `## debug`
workflows depend on those five fields.

## test

`doca_bench` is **a measurement tool**, so its `## test` verb is
about *testing the measurement* — i.e. confirming the numbers
are sound and reproducible — not unit-testing the bench itself.

**`## test` is an iterative loop, not a one-shot pass.** A bench
run that completes is not the same as a bench run that produced
a defensible number; each iteration tightens one axis of
measurement soundness (warm-up, steady-state, outliers, NUMA
placement, cross-run reproducibility, cross-version delta) and
loops back to [`## run`](#run).

The eval-loop overlay (rows apply to every bench run, not just
one library × mode):

| Iteration trigger | What it looks like | What changes next iteration |
| --- | --- | --- |
| Smoke completed; number is far below datasheet headline | Could be cold pipeline, wrong workload shape, wrong NUMA, or actually-right for this install. Do not assume datasheet first. | Confirm warm-up applied per [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy) layer 5; re-check axis 2 (workload shape) in [`## configure`](#configure) step 4; only then question hardware. |
| Throughput- or max-bandwidth-mode result exceeds the workload's predeclared tolerance across re-runs | Steady-state not reached; outlier-dominated run | Lengthen the run via the documented duration / job-count limit and re-run. Do not substitute an after-the-fact tolerance. |
| Precision-latency mean looks good; 99.99th percentile is huge | Tail-latency story is the actual answer; the mean is misleading | Quote the percentile breakdown, not the mean, per [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability) the precision-latency mode's reported distribution. |
| Same invocation produces different numbers on two hosts at the same DOCA version | NUMA / firmware / driver delta below DOCA | Walk axis 2 environment (cores / threads / NUMA) and the version layer per [`doca-version TASKS.md ## test`](../../doca-version/TASKS.md#test) before blaming bench. |
| Same invocation produces different numbers on the same host across DOCA versions | This *is* a regression signal — provided both four-tuples are captured | Cross-link the two baselines, name the changed fields, route to [`doca-version TASKS.md ## debug`](../../doca-version/TASKS.md#debug) for the version-delta diagnosis. |
| `--sweep` of a parameter shows a discontinuity | Could be a real performance cliff (capacity / cache / queue depth) or a measurement artefact at the swept value | Re-run the boundary points without the sweep to confirm; if real, that is the answer the user came for. |
| Bench reports zero / hung after extended wait | Device-binding or library-precondition layer; not a measurement-soundness issue | Stop iterating on the workload shape; jump to [`## debug`](#debug) layers 2-3. |

The agent's rule: every change to the invocation re-opens the
loop. Re-running with a tweaked flag and quoting the new
number without re-checking warm-up / steady-state / outliers /
distribution is exactly the failure mode this loop replaces.

**Baseline-capture rule.** When the goal of the bench session
is a baseline (vs an ad-hoc question), the captured artifact
must include the *four-tuple* per
[`## Pattern overview`](CAPABILITIES.md#pattern-overview)
pattern 6 — command line + DOCA version + device target +
as-deployed environment (firmware, kernel, NUMA, hugepages) —
alongside the summary and CSV. Without all four, the baseline
cannot be regression-tested later; quoting a number without the
four-tuple is the cross-version regression-hunt failure mode.

Loop termination requires **two consecutive runs within the
predeclared workload tolerance** for the selected metric. Record
both values and the tolerance. If two consecutive stable runs
cannot be obtained after the bounded re-run above, do not report
a stable benchmark; escalate the captured four-tuples and
variance to [`doca-debug ## debug`](../../doca-debug/SKILL.md)
or the workload owner. Escalate cross-version or cross-host comparisons to
[`doca-version TASKS.md ## test`](../../doca-version/TASKS.md#test)
or [`doca-debug ## debug`](../../doca-debug/SKILL.md) with the
captured four-tuples as evidence.

This skill does **not** ship a "test fixture" or pre-recorded
expected output. The expected output is install-, device-,
firmware-, and tuning-specific; pinning one would mislead
operators on a different platform / version. See
[`SKILL.md ## What this skill deliberately does not ship`](SKILL.md#what-this-skill-deliberately-does-not-ship).

## debug

When `doca_bench` fails to start, fails to produce numbers, or
produces numbers that do not look defensible, walk the
[`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy)
layers in order. The shape of the diagnosis:

1. **Config-syntax.** Invocation does not parse. Confirm the
   flag exists in `--help` on the *installed* binary (not a
   blog or older release). Confirm units / values are in the
   documented form. Confirm the flag is not mutually
   exclusive with another in the same invocation. Route to
   the public DOCA Bench guide via
   [`doca-public-knowledge-map ## DOCA tools`](../../doca-public-knowledge-map/SKILL.md#doca-tools);
   do not infer from generic CLI knowledge.
2. **Device-binding.** Invocation parses; bench cannot bind
   the target device. Confirm the device is visible to DOCA
   at all via
   [`doca-caps ## run`](../doca-caps/TASKS.md#run); confirm
   the driver stack is loaded
   ([`doca-setup ## debug`](../../doca-setup/TASKS.md#debug)
   layer Driver); confirm the chosen NUMA / core layout
   matches the device's actual NUMA node. Re-attempt the
   bench invocation only after the device is on the
   capability snapshot.
3. **Library-precondition.** Device bound; bench refuses to
   exercise the requested DOCA library. Re-run the bench
   query system per [`## configure`](#configure) step 2 to
   confirm the library is actually exposed; re-cross-check
   the per-platform support matrix per
   [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility);
   walk the per-library skill for library-internal
   capability rules (e.g.
   [`doca-comch CAPABILITIES.md ## Capabilities and modes`](../../libs/doca-comch/CAPABILITIES.md#capabilities-and-modes),
   [`doca-compress`](../../libs/doca-compress/SKILL.md) for
   compress-specific capability boundaries).
4. **Workload-precondition.** Library exercisable; the
   workload shape is invalid. Re-walk axis 2 of the
   three-axis model per [`## configure`](#configure) step 4;
   the most common failure here is feeding random-data to a
   structured workload (e.g. decompression / decryption)
   that needs valid input, or sizing buffers below / above
   what the library accepts.
5. **Measurement-soundness.** The run completes and reports
   numbers, but the numbers are unsound. Walk the three
   sub-layers per
   [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy)
   layer 5 — warm-up applied? steady-state reached?
   distribution reported alongside the single number? —
   before quoting any number.
6. **Version.** Cross-cutting partial-install / mixed-version.
   Walk [`doca-version TASKS.md ## debug`](../../doca-version/TASKS.md#debug)
   end-to-end; common bench-specific symptom is a
   companion-app version that does not match the bench host
   version.
7. **Cross-cutting.** Cause is below DOCA. Hand off to
   [`doca-debug ## debug`](../../doca-debug/SKILL.md) for
   the cross-cutting debug ladder and
   [`doca-setup ## debug`](../../doca-setup/TASKS.md#debug)
   for the env-side layers (driver / firmware / hugepages /
   NUMA).

In every case: **quote what the bench reported.** Do not
paraphrase the summary, do not reorder fields, do not
"summarize" a histogram into a single number. The bench is in
the loop precisely to break the agent out of the
inference-from-datasheet trap.

## Deferred task verbs

The four verbs below are not `doca_bench` work and should be
routed out before the agent does any of them under this skill's
name.

- **install** ⇒ [`doca-setup ## install`](../../doca-setup/TASKS.md#configure)
  (and [`## no-install`](../../doca-setup/TASKS.md#no-install)
  for the public NGC DOCA container path). The bench is
  shipped by the install at ≥ 2.7.0; this skill does not own
  the install workflow.
- **build a custom DOCA benchmark or application** ⇒
  [`doca-programming-guide ## build`](../../doca-programming-guide/TASKS.md#build)
  for the cross-library pattern, plus the matching
  `libs/<library>` skill for the library-specific build
  details. `doca_bench` is the shipped harness; a bespoke
  harness is a different artifact.
- **library-internal benchmarking** (e.g. an application-level
  end-to-end measurement, or a library-internal performance
  counter the bench does not expose) ⇒ the matching
  `libs/<library>` skill plus
  [`doca-programming-guide`](../../doca-programming-guide/SKILL.md).
  Bench is a uniform harness; library-internal performance
  questions go to the library.
- **streaming telemetry / live metrics from a production
  workload** ⇒ not a bench feature, and bench is explicitly
  *not for production* per
  [`CAPABILITIES.md ## Safety policy`](CAPABILITIES.md#safety-policy).
  The DOCA Telemetry Service (DTS) is the documented telemetry
  surface; routing belongs in
  [`doca-public-knowledge-map ## DOCA services`](../../doca-public-knowledge-map/SKILL.md#doca-services).

## Command appendix

`doca_bench`-specific invocation classes the verbs above reach
for. Every row is a CLASS — the agent must not invent flags
beyond `--help` on the installed binary and the public DOCA
Bench guide. The five-class symmetry below is the load-bearing
piece; one worked example per class is shown.

**Infra-aware preamble (every row below).** Per the bundle's
detect → prefer → fall back → report contract documented in
[`doca-structured-tools-contract ## The agent behavior contract`](../../doca-structured-tools-contract/SKILL.md#the-agent-behavior-contract),
the agent should:

1. Probe for the matching structured helper FIRST (`doca-env --json`
   for version + devices + libraries + drivers + hugepages in one
   shot; `doca-capability-snapshot` for per-device capability
   flags; `version-matrix.json` for *"available since"* lookups;
   a future bench-runner / bench-snapshot helper for the
   four-tuple-capturing baseline pattern when it lands per
   [`doca-structured-tools-contract ## Relationship to PR2 executables`](../../doca-structured-tools-contract/SKILL.md#relationship-to-pr2-executables)).
2. If the probe succeeds, the structured tool's output is the
   authoritative answer and the agent SHOULD NOT also run the
   manual command in the row below. Report *"using structured
   `<tool>`"*.
3. If the probe fails, fall back to the manual command in the
   row. Report *"falling back to manual chain"*.
4. The schemas the structured tools emit are defined in
   [`doca-structured-tools-contract ## Schemas`](../../doca-structured-tools-contract/SKILL.md#schemas);
   the version-handling semantics (four-way match, NGC,
   headers-win) are owned by
   [`doca-version`](../../doca-version/SKILL.md).

| Purpose (class) | Invocation (shape) | Owning step | Reads as healthy when … |
| --- | --- | --- | --- |
| Discover the documented flag surface | `doca_bench --help` (and the public DOCA Bench guide for the long-form documentation) | [`## configure`](#configure) step 1; [`## debug`](#debug) layer 1 | Prints the documented flag inventory the agent uses as the only source of truth for flag names; the public guide is the secondary source. |
| Probe the granular-build inventory | The bench's built-in query family for installed libraries + supported sweep attributes per [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability) | [`## configure`](#configure) step 2 | Reports the libraries this install actually exposes; absence of the target library is the canonical *"granular build does not include X"* answer. |
| Drive a single-library micro-benchmark | The documented bench invocation for the chosen (target library × workload shape × measurement mode) — flag names re-confirmed against `--help` on the installed binary | [`## run`](#run) steps 2-4; [`## test`](#test) eval loop | The bench echoes the invocation, applies the documented warm-up, completes in the requested duration / job count, and prints a summary in the chosen mode's documented format. |
| Capture a CSV baseline alongside stdout | The documented CSV-output family per [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability) — `--csv-*` family on the installed `--help` | [`## test`](#test) baseline-capture rule | A CSV file is written, the stdout summary matches the CSV aggregate, and the captured four-tuple (command line + version + device + environment) accompanies the CSV. |
| Sweep a parameter across a planned range | The documented sweep family (`--sweep` per the public guide); the agent estimates total wall-clock cost before committing | [`## run`](#run) step 5 | The smoke run for the boundary values passed first; the sweep completes; the resulting series has no implausible discontinuities that disappear when re-running the boundary point without the sweep. |
| Drive a remote scenario with the companion app | The bench invocation paired with the companion app on the far side, communicating over the documented out-of-band channel | [`## configure`](#configure) step 6; [`## run`](#run) step 6 | The companion app is up and reachable, the channel comes up, the bench drives it without the *"not for production"* warning being triggered on a production segment. |

Three cross-cutting rules for this appendix:

- **Never invent a `doca_bench` flag, scenario name, attribute
  name, or metric name.** `--help` on the installed binary and
  the public DOCA Bench guide are the joint contract;
  prose-derived flag strings are the most common hallucination
  failure for this skill.
- **Smoke before bulk.** Every row above presumes the smoke
  row succeeded first; running a sweep or a long single-library
  run without the smoke is the canonical operator-time-waste
  failure mode.
- **Cross-link instead of duplicate.** Cross-cutting commands
  (`pkg-config --modversion`, `doca_caps --list-devs`,
  `dmesg`, `mlxconfig -d <bdf> q`, `numactl --hardware`) live
  in [`doca-debug ## debug`](../../doca-debug/SKILL.md) and
  [`doca-setup TASKS.md ## debug`](../../doca-setup/TASKS.md#debug);
  this appendix names only `doca_bench`-specific invocation
  classes.

## Cross-cutting

A few rules that apply across every verb in this file, restated
here so they are visible at the point of action and not buried
in [`SKILL.md`](SKILL.md):

- The **public DOCA Bench guide** plus the installed `--help`
  are the joint source of truth. When they disagree (e.g. a
  flag landed in a release this skill was not written
  against), the *installed* `--help` wins for the user's
  actual run.
- `doca_bench` *does* drive hardware and *does* allocate
  resources; smoke-before-bulk is mandatory, and re-running
  a long sweep "to confirm" without the smoke step is
  exactly the failure mode this skill is here to prevent.
- **Quote the four-tuple, not just the number.** Command line
  + DOCA version + device target + as-deployed environment
  is the minimum unit a bench number is meaningful in. The
  agent must surface all four whenever reporting a number to
  the user.
- This skill **assumes a healthy DOCA install at ≥ 2.7.0**
  (or the public NGC DOCA container at an equivalent
  version). If the install is in doubt, route to
  [`doca-version TASKS.md ## configure`](../../doca-version/TASKS.md#configure)
  and
  [`doca-setup`](../../doca-setup/SKILL.md) before running
  anything else here.