# Applications and testing

## CLI Tools (clap)

- Use the derive API: `#[derive(Parser)]` + `#[derive(Subcommand)]`. Less boilerplate, types drive the help text.
- One `enum Commands` variant per subcommand; flatten shared flags into a `#[command(flatten)] struct CommonArgs`.
- `--json` flag on query commands for agent/pipe consumption. Emit via `serde_json::to_string(&value)?`.
- Exit codes: 0 success, 1 for errors `main` returned, 2 for argparse (clap handles this), reserve 3+ for domain meanings documented in `--help`.
- Provide `--version` automatically via `#[command(version)]`.

See [cli-tools.md](./cli-tools.md) for config layering, logging setup, progress reporting, and shell completions.


## HTTP Services (axum)

- Framework default: **axum** (tokio-native, tower middleware, extractor-based handlers). Pick `actix-web` only if an existing codebase uses it.
- Handlers return `Result<impl IntoResponse, AppError>`. Implement `IntoResponse` for `AppError` to centralize error → status mapping.
- Validate input at the boundary: `axum::extract::Json<T>` where `T: Deserialize + Validate` (use `validator` crate). Internal services trust input was validated.
- Share state via `State<Arc<AppState>>` — not globals, not `lazy_static`.
- Middleware via `tower::ServiceBuilder`: tracing → timeout → auth → CORS → handler. Order matters.
- **Resilience layers** (outbound clients, shared services): combine `LoadShed` + `ConcurrencyLimit` for backpressure, not unbounded queueing; full tower stack in [production-resilience.md](./production-resilience.md).

See [axum-service.md](./axum-service.md) for project layout, extractors, error types, graceful shutdown, and OpenAPI generation.


## Testing

- Built-in `#[test]`. Prefer `cargo nextest run --workspace` over `cargo test` — it runs tests in parallel processes with proper isolation.
- Unit tests live in `mod tests { ... }` at the bottom of the file (access to private items).
- Integration tests in `tests/` directory. One file per public surface area.
- `#[tokio::test]` for async tests. Add `flavor = "multi_thread"` when the code under test spawns tasks.
- `rstest` for parametrized tests and fixtures. `proptest` / `quickcheck` for property-based tests on pure logic.
- `insta` for snapshot testing CLI output, serialization, large structs. Review diffs with `cargo insta review`.
- `assert_cmd` + `predicates` for CLI integration tests (invokes the binary, asserts on stdout/stderr/exit code).
- **Assert on error variants with `matches!`**: `assert!(matches!(result.unwrap_err(), MyError::Validation(_)))` — no `match` arms to update when unrelated variants are added.
- Coverage: `cargo llvm-cov --workspace --html`. Target 70%+ on application code, higher on library crates.
- **Fuzzing for parsers**: `cargo fuzz` + `libfuzzer-sys` on any code parsing untrusted input; nightly runs surface panics and UB unit tests miss.
- **Never mutate process-global state in a test.** `set_var("TMPDIR", …)` in one test makes every *concurrent* `tempfile::tempdir()` create its scratch dir inside that test's `TempDir` — recursively deleted when it drops. The smoking gun is nested temp paths (`/tmp/.tmpXXXX/.tmpYYYY/…`) and `ENOENT` on files a victim just created, with the failing test rotating between runs. A mutex around the env-mutating tests does not fix it: the victims never take the mutex. Refactor the function under test into a thin env-reading wrapper over an env-free core that takes the values as parameters, and test the core.
- **A concurrent `Command::spawn` briefly extends the lifetime of every open file descriptor.** `fork` duplicates the whole parent fd table and only `exec`'s `CLOEXEC` closes the copies, so in that window a sibling test holds the caller's lock fd or its just-written script's write fd. Two symptoms, one cause: an `flock` that outlives its guard's `drop` (a test asserting "drop released it, re-acquire succeeds immediately" fails roughly 1 run in 12 next to spawn-heavy tests, 0 in N alone — fix with a bounded poll-acquire to a deadline, not a one-shot assert) and `ErrorKind::ExecutableFileBusy` on exec'ing a file just `chmod +x`'d (fix with a bounded retry around the spawn). The already-held assertion needs no change in either case.

For generic test discipline (anti-patterns, mock rules, rationalization resistance), see the `ia-writing-tests` skill.


## Production Resilience

When productionizing a service (config validation, `/health` + `/ready` endpoints, graceful shutdown, retries/timeouts/jitter, deny-by-default fallback when the call is the security decision, connection pools, diagnostic secret redaction), load [production-resilience.md](./production-resilience.md).


## Observability

For logging (`tracing` + `tracing-subscriber` with init recipe), `#[instrument]` spans, correlation IDs, metrics, and distributed tracing patterns, load [observability.md](./observability.md). Never use `println!` or `log::` in new code.

# Axum HTTP Services

Patterns for building production HTTP services with `axum` + `tokio` + `tower`.

## Project Layout

```
src/
  main.rs           # Entrypoint: config load, tracing init, server bind, graceful shutdown
  app.rs            # Router assembly: `pub fn router(state: AppState) -> Router`
  state.rs          # AppState struct (pools, clients, config)
  error.rs          # AppError enum + IntoResponse impl
  routes/
    mod.rs
    users.rs        # One module per resource
    health.rs
  services/         # Business logic, no HTTP types
  repo/             # Data access (sqlx), no HTTP types
  config.rs
  telemetry.rs      # tracing + metrics setup
tests/
  api.rs           # Integration tests hitting the router directly
```

Rules mirror the layered architecture from `ia-nodejs-backend`:
- Routes parse + call services + format response. No business logic.
- Services never import from `axum` or `http`. No HTTP status codes leak in.
- Repos never construct `AppError` variants that map to HTTP — they return typed storage errors that services convert.

## AppState

```rust
#[derive(Clone)]
pub struct AppState {
    pub db: sqlx::PgPool,
    pub http: reqwest::Client,
    pub config: Arc<Config>,
}
```

`Clone` is cheap because the expensive members are `Arc` inside. Inject with `State<AppState>` extractor — don't use globals or `OnceCell`.

## Error Type

```rust
use axum::{http::StatusCode, response::{IntoResponse, Response}, Json};
use serde_json::json;
use thiserror::Error;

#[derive(Debug, Error)]
pub enum AppError {
    #[error("not found")]
    NotFound,
    #[error("invalid input: {0}")]
    Validation(String),
    #[error("unauthorized")]
    Unauthorized,
    #[error(transparent)]
    Sqlx(#[from] sqlx::Error),
    #[error(transparent)]
    Other(#[from] anyhow::Error),
}

impl IntoResponse for AppError {
    fn into_response(self) -> Response {
        let (status, code) = match &self {
            AppError::NotFound => (StatusCode::NOT_FOUND, "not_found"),
            AppError::Validation(_) => (StatusCode::BAD_REQUEST, "validation"),
            AppError::Unauthorized => (StatusCode::UNAUTHORIZED, "unauthorized"),
            AppError::Sqlx(_) | AppError::Other(_) => {
                tracing::error!(error = ?self, "internal error");
                (StatusCode::INTERNAL_SERVER_ERROR, "internal")
            }
        };
        let message = if status == StatusCode::INTERNAL_SERVER_ERROR {
            "internal server error".to_owned()
        } else {
            self.to_string()
        };
        let body = Json(json!({
            "error": { "code": code, "message": message }
        }));
        (status, body).into_response()
    }
}
```

- One error envelope shape across every handler. Callers parse `.error.code` once.
- Log the full error with `?self` for `INTERNAL_SERVER_ERROR` paths; never leak internal messages to the client.
- Use `?` in handlers freely — `From` impls convert `sqlx::Error`, `anyhow::Error` into `AppError`.

## Handlers

```rust
#[tracing::instrument(skip(state), fields(user_id))]
pub async fn get_user(
    State(state): State<AppState>,
    Path(id): Path<Uuid>,
) -> Result<Json<UserResponse>, AppError> {
    tracing::Span::current().record("user_id", %id);
    let user = state.users.find(id).await?.ok_or(AppError::NotFound)?;
    Ok(Json(user.into()))
}
```

- Handlers take extractors first, body last. Extractors run in order; the body extractor must be last (it consumes the request).
- Return `Result<Json<T>, AppError>` for JSON endpoints, `Result<impl IntoResponse, AppError>` for flexible responses.
- Use `#[tracing::instrument(skip(state))]` on every handler — skip the state so secrets don't land in logs.

## Validation

```rust
use validator::Validate;

#[derive(Deserialize, Validate)]
pub struct CreateUser {
    #[validate(length(min = 1, max = 100))]
    pub name: String,
    #[validate(email)]
    pub email: String,
}

pub async fn create_user(
    State(state): State<AppState>,
    Json(body): Json<CreateUser>,
) -> Result<(StatusCode, Json<UserResponse>), AppError> {
    body.validate().map_err(|e| AppError::Validation(e.to_string()))?;
    let user = state.users.create(body.into()).await?;
    Ok((StatusCode::CREATED, Json(user.into())))
}
```

Or centralize via a `ValidatedJson<T>` extractor so every handler calls `.validate()` without repetition.

## Middleware

```rust
use tower::ServiceBuilder;
use tower_http::{trace::TraceLayer, timeout::TimeoutLayer, cors::CorsLayer};

pub fn router(state: AppState) -> Router {
    Router::new()
        .route("/health", get(health::shallow))
        .route("/ready",  get(health::deep))
        .nest("/users", users::routes())
        .layer(
            ServiceBuilder::new()
                .layer(TraceLayer::new_for_http())
                .layer(TimeoutLayer::new(Duration::from_secs(30)))
                .layer(CorsLayer::permissive())
                .into_inner(),
        )
        .with_state(state)
}
```

Middleware order: tracing → timeout → rate limit → auth → CORS → handler. Tracing *outside* the timeout so you see timed-out requests.

## Graceful Shutdown

```rust
async fn shutdown_signal() {
    let ctrl_c = async {
        tokio::signal::ctrl_c().await.expect("install ctrl+c handler");
    };
    #[cfg(unix)]
    let terminate = async {
        tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
            .expect("install sigterm handler")
            .recv().await;
    };
    tokio::select! { _ = ctrl_c => {}, _ = terminate => {} }
    tracing::info!("shutdown signal received");
}

// main.rs
let listener = tokio::net::TcpListener::bind(&config.addr).await?;
axum::serve(listener, app::router(state))
    .with_graceful_shutdown(shutdown_signal())
    .await?;
```

- Drain in-flight requests up to a budget (30s typical), then hard-exit.
- Close DB pools and flush telemetry *after* the server returns, before `main` exits.

## Testing

```rust
use axum::body::Body;
use axum::http::{Request, StatusCode};
use tower::ServiceExt; // for `.oneshot()`

#[tokio::test]
async fn get_user_returns_404_for_unknown_id() {
    let state = test_state().await;
    let app = app::router(state);
    let response = app
        .oneshot(
            Request::builder()
                .uri(format!("/users/{}", Uuid::new_v4()))
                .body(Body::empty())
                .unwrap(),
        )
        .await
        .unwrap();
    assert_eq!(response.status(), StatusCode::NOT_FOUND);
}
```

- Exercise the router directly with `tower::ServiceExt::oneshot` — no TCP bind needed, fast and deterministic.
- `sqlx::test` macro gives each test an isolated database via transactions or template DBs.
- For full end-to-end, use `reqwest` against a real bound port with `TcpListener::bind("127.0.0.1:0")` to pick a free port.

## Database (sqlx)

Prefer the compile-time-checked macros over the runtime builder:

```rust
// Schema drift caught at `cargo build`, not at 3am in prod.
let user = sqlx::query_as!(
    User,
    "SELECT id, email, created_at FROM users WHERE id = $1",
    id,
)
.fetch_optional(&state.db)
.await?;
```

- `sqlx::query_as!` / `sqlx::query!` verify SQL against the real database schema at compile time. Requires `DATABASE_URL` in the environment or a checked-in `.sqlx/` offline query cache (`cargo sqlx prepare`).
- Use `.fetch_optional()` for "by id" lookups — returns `Option<T>`, maps cleanly to `AppError::NotFound`.
- `.fetch_all()` only when you've bounded the result set with `LIMIT`. No unbounded selects from request handlers.
- Transactions: `let mut tx = state.db.begin().await?;` → do work → `tx.commit().await?;`. Dropping without commit rolls back.
- Prefer `query_as!` + explicit struct over `FromRow` derive when the struct and SQL columns are 1:1; derive when you want reuse across multiple queries.

Ship `.sqlx/` in the repo so CI builds don't need a live database. Regenerate with `cargo sqlx prepare --workspace` when queries change.

## OpenAPI

`utoipa` generates OpenAPI from derive macros on handlers + types. `aide` is an alternative with better runtime integration. Either way: the schema is derived from code, not maintained separately.

```rust
#[derive(OpenApi)]
#[openapi(paths(get_user, create_user), components(schemas(UserResponse, CreateUser)))]
struct ApiDoc;
```

Serve the spec at `/openapi.json` and Swagger UI at `/docs` in non-prod environments.

## Common Traps

- Don't put an `Arc<Mutex<T>>` in `AppState` for things that should be behind a DB. Shared mutable state across requests is almost always a design smell.
- Don't use `tokio::sync::RwLock` where `arc-swap::ArcSwap` fits — config reloads, rarely-changing snapshots.
- Don't forget `with_state` at the end of router assembly. The compiler error is confusing (`method not found on Router<AppState>`).
- Don't bind to `0.0.0.0` in local dev unless you mean it. Use `127.0.0.1` by default; make the bind address configurable.

# Build Profiles

Load this reference when setting up or tuning a Rust project's Cargo build profiles. Tune profiles for the shape of the binary — defaults ship fast debug builds and modest-optimization release builds, but application Rust benefits from more aggressive profiles.

## Profile definitions (Cargo.toml)

```toml
# Production release: maximum optimization, minimum binary
[profile.release]
lto = "fat"           # Link-time optimization across all crates
codegen-units = 1     # Single codegen unit trades compile time for runtime perf
strip = true          # Strip symbols from the final binary
panic = "abort"       # No unwinding tables — smaller binary, faster panics

# Release with symbols kept for profiling (perf, flamegraph, pprof)
[profile.release-dbg]
inherits = "release"
strip = false
debug = true

# Size-minimized release for distributable CLIs
[profile.release-min]
inherits = "release"
opt-level = "z"       # Optimize for size over speed
```

`panic = "abort"` breaks `catch_unwind`-based recovery — skip it for libraries others will link against, or for binaries that rely on panic hooks (some web frameworks do). For most CLIs and backend services, it's pure win.

## Dev-machine compile speedups (.cargo/config.toml)

Cut PR compile time on Linux with mold:

```toml
[build]
rustflags = ["-C", "link-arg=-fuse-ld=mold"]

[target.x86_64-unknown-linux-gnu]
rustflags = [
    "-C", "link-arg=-fuse-ld=mold",
    "-C", "target-cpu=native",   # dev machines only — bakes in CPU features
    "-Z", "share-generics=y",    # share monomorphizations across crates (nightly)
]

[alias]
t = "nextest run"
```

Mold is Linux-only (`lld` on other platforms). `target-cpu=native` is a developer-machine convenience — remove for reproducible CI and distributable binaries.

# Rust CI Pipeline — language-specific callouts

Load this reference when setting up or reviewing a CI pipeline for a Rust project. General CI design (matrix strategy, caching, deployment gating) lives with the `ia-infrastructure-engineer` agent — this file covers only the Rust-specific pieces.

- **`rustsec/audit-check`** — runs `cargo audit` against the RustSec Advisory DB. Catches published CVEs in your dependency graph. Wire it as a required PR check, not a nightly job; advisory hits on `main` are already too late.
- **Coverage via `cargo-llvm-cov`** — `cargo llvm-cov --workspace --lcov --output-path lcov.info` produces codecov-compatible output without the instrumentation overhead of `tarpaulin`. Prefer it for any new Rust project.
- **`Swatinem/rust-cache`** — caches `~/.cargo/registry`, `~/.cargo/git`, and `target/` keyed on `Cargo.lock`. Cuts cold-CI time from ~5min to ~90s on typical workspaces. Install before cargo commands.
- **`taiki-e/install-action`** — fast binary installer for cargo tools (nextest, llvm-cov, audit). Faster than `cargo install` on CI by orders of magnitude.
- **Matrix coverage**: at minimum `stable` on Linux. Add `beta` + `nightly` on Linux and `stable` on Windows + macOS if the crate is a library others consume; skip the full OS matrix for internal services.
- **Doc tests**: `cargo test --doc --all-features` as a separate step. Doc tests are easy to break with a refactor and easy to miss with `nextest run` alone.

# Rust CLI Tools

Patterns for building agent-friendly, scriptable CLIs with `clap`.

## Project Layout

```
src/
  main.rs       # CLI parse + dispatch only, no business logic
  cli.rs        # clap structs and enums
  commands/
    mod.rs
    index.rs    # one handler per subcommand
    search.rs
  config.rs     # TOML/env loading, validation
  output.rs    # JSON / human formatters
Cargo.toml
```

Keep `main.rs` under 20 lines. Every subcommand is a free function that takes parsed args + shared state and returns `Result<()>`. This makes each command independently testable.

## Clap Derive Patterns

```rust
use clap::{Parser, Subcommand, Args};

#[derive(Parser)]
#[command(name = "myapp", version, about)]
struct Cli {
    #[command(flatten)]
    global: GlobalOpts,

    #[command(subcommand)]
    command: Commands,
}

#[derive(Args)]
struct GlobalOpts {
    /// Increase logging verbosity (-v, -vv, -vvv)
    #[arg(short, long, global = true, action = clap::ArgAction::Count)]
    verbose: u8,

    /// Emit JSON instead of human output
    #[arg(long, global = true)]
    json: bool,
}

#[derive(Subcommand)]
enum Commands {
    /// Index the current project
    Index(IndexArgs),
    /// Search the index
    Search(SearchArgs),
}
```

- `global = true` flags apply to every subcommand without repeating.
- Use `value_enum` for typed string flags:
  ```rust
  #[arg(long, value_enum, default_value_t = Format::Md)]
  format: Format,
  ```
- `#[arg(value_parser = validate_path)]` to reject bad values before the handler runs.

## Layered Parsing for Non-Trivial CLIs

Once flag count crosses ~10 or commands start sharing complex validation, split parsing into two stages:

1. **Low stage** — `LowArgs` mirrors the raw CLI surface 1:1. Clap populates it. No cross-field validation, no domain types.
2. **High stage** — `HiArgs` (or `Config`) is what the rest of the program consumes. Constructing `HiArgs::from(low)` runs *all* semantic validation: mutual exclusions, path existence, glob compilation, range constraints, regex validity. Fails with one clear error message.

```rust
let low = Cli::parse();
let hi = HiArgs::try_from(low).context("invalid arguments")?;
run(hi).await
```

Downstream code accepts `&HiArgs` (or specific typed fields from it) and never re-checks. This keeps validation in one place and makes it impossible for a handler to receive an invalid combination. Pattern comes from ripgrep; worth it as soon as flag interactions become non-trivial.

For simple CLIs (one subcommand, a handful of flags), skip this — one clap-derive struct is enough.

## Config Layering

Priority (lowest to highest): built-in defaults → `~/.config/myapp/config.toml` → project `.myapp/config.toml` → env vars (`MYAPP_*`) → CLI flags.

Use `figment` or hand-roll with `serde` + `toml`:

```rust
#[derive(Deserialize, Debug)]
pub struct Config {
    #[serde(default = "default_limit")]
    pub limit: usize,
    pub api_key: Option<String>,
}

impl Config {
    pub fn load(project_root: &Path) -> Result<Self> {
        let path = project_root.join(".myapp/config.toml");
        let text = match std::fs::read_to_string(&path) {
            Ok(text) => text,
            Err(e) if e.kind() == std::io::ErrorKind::NotFound => String::new(),
            Err(e) => {
                return Err(e).with_context(|| format!("reading {}", path.display()));
            }
        };
        let mut cfg: Config = toml::from_str(&text)
            .with_context(|| format!("parsing {}", path.display()))?;
        if let Ok(key) = std::env::var("MYAPP_API_KEY") {
            cfg.api_key = Some(key);
        }
        cfg.validate()?;
        Ok(cfg)
    }
}
```

Validation runs in `load()` — never defer it to the first call site.

Only `NotFound` collapses to defaults. `read_to_string(p).unwrap_or_default()` also swallows permission-denied, invalid UTF-8, and transient I/O, so a config that exists but cannot be read becomes an empty one — and the next write replaces the comments and unrelated entries the read never surfaced.

## Logging

```rust
use tracing_subscriber::{EnvFilter, fmt};

fn init_logging(verbose: u8) {
    let level = match verbose {
        0 => "warn",
        1 => "info",
        2 => "debug",
        _ => "trace",
    };
    let filter = EnvFilter::try_from_default_env()
        .unwrap_or_else(|_| EnvFilter::new(format!("myapp={level}")));
    fmt().with_env_filter(filter).with_writer(std::io::stderr).init();
}
```

- Logs go to **stderr**. Only command results go to stdout. This keeps pipes clean.
- Respect `RUST_LOG` / `MYAPP_LOG` env var if set — it overrides `-v`.
- Never log at `info!` inside hot loops; budget is roughly one log line per user-visible action.

## Output

Every query command supports two modes:

```rust
if args.json {
    println!("{}", serde_json::to_string(&results)?);
} else {
    render_human(&results);
}
```

- JSON output must be a single line or a valid JSON document — no mixed human + JSON in the same stream.
- Exit with non-zero on failure even when `--json` is set; don't emit `{"error": "..."}` with exit 0.

## Progress

- Interactive terminals (check `std::io::IsTerminal::is_terminal(&stderr)`): `indicatif` progress bars on stderr.
- Non-interactive (pipes, CI, agents): plain periodic log lines. Never emit control codes to a non-tty.
- `--quiet` flag suppresses both.

## Shell Completions

```rust
use clap_complete::{generate, Shell};

Commands::Completions { shell } => {
    let mut cmd = Cli::command();
    generate(shell, &mut cmd, "myapp", &mut std::io::stdout());
}
```

Ship completions via the `completions` subcommand rather than pre-generated files — keeps them in sync with the actual flag set.

## Testing CLIs

```rust
use assert_cmd::Command;
use predicates::prelude::*;

#[test]
fn search_returns_json() {
    Command::cargo_bin("myapp").unwrap()
        .args(["search", "foo", "--json"])
        .assert()
        .success()
        .stdout(predicate::str::starts_with("["));
}
```

- `tempfile::TempDir` for isolated project roots.
- Snapshot stdout with `insta::assert_snapshot!` for human-readable output that changes rarely.
- Test exit codes explicitly — they're part of the CLI contract for scripts.

## Common Traps

- Don't print to stdout from library crates. Return structured data, let the binary format it.
- Don't swallow `SIGPIPE`. On Unix, when the reader closes a pipe early, the default is to die — let it. If you install a `tokio::signal` handler, re-raise or exit cleanly on pipe errors.
- Don't ship a CLI that panics on bad input. Map every user-facing error to a clean `anyhow` chain with `.context()`.

# Macro Hygiene and OS Boundaries

Two trap classes that type-check cleanly and fail at runtime or at a caller's crate. Load when authoring a `macro_rules!` or proc macro, or when touching filesystem paths, process output, or on-disk state.

## Declarative macros (`macro_rules!`)

These apply when *defining* a macro. An ordinary invocation of someone else's macro is not a site for any of them.

- **Interpolate an `$x:expr` fragment exactly once.** Each substitution re-evaluates the caller's expression, so `my_macro!(v.pop().unwrap())` runs the side effect once per mention. Bind it first inside the expansion (`let v = $x;`) and use the binding.
- **Reference items through `$crate`.** An exported macro expands in the *caller's* namespace, so a bare `helper()` or `Error` resolves against whatever the caller happens to have in scope. `$crate::helper()` binds to the defining crate regardless.
- **Parenthesize re-emitted `$t:tt` fragments.** A token-tree fragment is spliced verbatim, so `$a * $b` with `$a = 1 + 2` expands to `1 + 2 * b` and silently changes precedence. This does *not* apply to `:expr`, which is already parsed as one complete expression — wrapping those adds noise without preventing anything.
- **Do not assume identifier hygiene covers items.** Local variables introduced by an expansion are hygienic, but items (types, functions, consts) are not: two invocations in one module collide on any item the expansion names. Derive item names from a macro parameter, or emit them inside a generated module.

## Procedural macros

- **Emit `syn::Error` / `compile_error!`, never `panic!` or `unwrap()`.** A panic in a proc macro surfaces to the user as a compiler-internal failure with no source location. A `syn::Error` carries a span, so the error points at the offending token in their code.
- Compile the failure cases. `trybuild` fixtures for rejected input are the only way a macro's error contract stays honest through refactors.

## Paths and OS strings

- **`Path` is not UTF-8.** On Unix a filename is arbitrary bytes; on Windows it is potentially ill-formed UTF-16. `path.to_str().unwrap()` panics on filenames that are perfectly legal on the user's disk. Use `to_string_lossy()` where the value is only ever displayed, `OsStr`/`OsString` where it is passed through, and reserve `to_str()` for cases where non-UTF-8 is a genuine error the caller should see — with a real error, not an unwrap.
- The same applies to arguments and environment variables (`args_os()`, `var_os()`) when a value may originate outside the program.
- **An argv ban-list that converts to `String` first is bypassable.** The reasoning "a non-UTF-8 argument is prose, never a flag" is false: `--slug=\xff` is a non-UTF-8 argument that is very much a flag, and lossy conversion mangles it into something the ban-list no longer recognises while the downstream tool, receiving the original `OsString`, still parses it as the flag. Match on `OsStr::as_bytes()`, splitting on `b'='` before comparing.
- Subprocess output is bytes too. `String::from_utf8(output.stdout)` fails on any tool that emits non-UTF-8; decide deliberately between propagating that error and `from_utf8_lossy`.

## Crash-safe file updates

- **Write-then-rename.** Writing in place leaves a truncated file if the process dies mid-write, and the next run cannot distinguish it from a valid short file. Write to a temporary file in the *same directory* (rename is only atomic within a filesystem), then rename over the target.
- Persist the temporary explicitly rather than letting a `TempDir`/`NamedTempFile` guard delete it on the success path.
- Durability beyond the rename needs `sync_all()` on the file before renaming, and on the containing directory after, if the data must survive power loss rather than just process death. Skip both where the file is a rebuildable cache; state the choice either way.
- A retry after an interrupted update must be safe to run: the rename target either has the old content or the new one, never a blend.

# Observability for Rust Services

Load this reference when adding logging, tracing, metrics, or distributed tracing to a Rust service. `println!` and `log::` are forbidden in new code — use `tracing` + `tracing-subscriber`.

## Logging

- `tracing` + `tracing-subscriber` with `json()` formatter in production, `fmt().pretty()` in dev.
- **Init recipe**: build subscriber layers and register once at `main` entry. Respect `RUST_LOG` for runtime filter override, include thread IDs for concurrent contexts, gate OpenTelemetry behind a feature flag so dev builds don't pull the whole OTEL SDK:

  ```rust
  pub fn init_tracing() {
      let fmt_layer = tracing_subscriber::fmt::layer()
          .with_target(false)
          .with_thread_ids(true);
      let filter_layer = tracing_subscriber::EnvFilter::try_from_default_env()
          .unwrap_or_else(|_| "info".into());
      tracing_subscriber::registry()
          .with(filter_layer)
          .with(fmt_layer)
          .init();
  }
  ```

## Structured Spans

- `#[tracing::instrument(skip(large_arg), fields(user_id = %user.id))]` on service methods — automatic span creation, structured fields.
- Skip large args to keep spans lightweight; prefer named fields over stringified args.

## Correlation IDs

Extract or generate at ingress middleware, attach to the root span, propagate via `traceparent` header to downstream calls. Required for any multi-service system.

## Metrics

`metrics` crate with `metrics-exporter-prometheus`. Counter for traffic/errors, Histogram for latency, Gauge for saturation. Label cardinality bounded — no user IDs, no unbounded dimensions.

## Distributed Tracing

`tracing-opentelemetry` exports spans to Jaeger/Tempo/Honeycomb/Datadog. Gate the OpenTelemetry subscriber behind a feature flag to keep dev/test builds fast.

## Live Task Introspection (tokio-console)

Distinct from log/metric/trace export: `tokio-console` attaches to a running process and shows every Tokio task's state, poll count, busy/scheduled/idle durations with a poll-time histogram, and wakeup counts, and warns on self-wakes, lost wakers, and tasks that never yield -- the tool for a stuck or spinning task that emits no log line. Add the `console-subscriber` crate as a `tracing-subscriber` layer (`console_subscriber::init()` or `ConsoleLayer::builder()` alongside the fmt layer) and build with `RUSTFLAGS="--cfg tokio_unstable"` (or `rustflags = ["--cfg", "tokio_unstable"]` in `.cargo/config.toml`); without that cfg Tokio emits no task instrumentation. Keep it behind a feature flag like the OTel layer: it is a debugging aid, not production telemetry.

# Ownership and execution

## Error Handling

Split by crate role:

- **Libraries / lower crates**: define typed errors with `thiserror`. Consumers can pattern-match.
- **Binaries / top-level crates**: use `anyhow::Result` with `.context("what was being attempted")`. Human-readable error chains.
- Never return `Box<dyn Error>` from library APIs — it erases variant information.
- Use `?` liberally. Never `.unwrap()` or `.expect()` outside tests and `main`. An `expect("...")` is acceptable only when the invariant is provably upheld and the message explains why.
- Convert at boundaries: `#[from]` on thiserror variants for auto-conversion; `.map_err(MyError::from)` when explicit.
- `bail!("...")` / `ensure!(cond, "...")` in application code for early exits.
- Prefer `Result<T, E>` over panics for any recoverable error. Panics are for programmer bugs (broken invariants), not runtime failures.
- **`#[must_use]` on fallible APIs**: `Result` already warns on implicit unused results; annotate custom result wrappers or functions to add a specific diagnostic. Deny `unused_must_use` when that warning must fail the build. Explicit discard (`let _ = validate(x);`) bypasses the lint even when denied, so review intentional discards separately.
- **Make illegal call-sequences unrepresentable** — the type-state pattern: encode a mandatory call order as distinct types (`Client<Uninitialized>` → `Client<Connected>`) so an out-of-order call fails to compile instead of erroring at runtime.
- **`fs::read_to_string(p).unwrap_or_default()` to mean "an absent file is an empty config" swallows every read error, not just `NotFound`.** A file that exists but cannot be read — permission denied, invalid UTF-8, transient I/O — collapses to empty, and the next step writes a fresh file over the comments and unrelated entries the read never surfaced. Match the kind: `Err(e) if e.kind() == ErrorKind::NotFound => Ok(default)`, everything else propagates with context. Test it by writing invalid UTF-8 bytes to the path and asserting the operation returns `Err` *and* leaves the bytes untouched.


## Ownership Discipline

- Take `&str` over `&String`, `&[T]` over `&Vec<T>` in function signatures — accepts more call sites for free.
- Return owned (`String`, `Vec<T>`) from constructors and public APIs. Borrow in hot paths where lifetimes are obvious.
- Reach for `Arc<T>` only when sharing across threads. Single-threaded sharing uses `Rc<T>` or references.
- `Cow<'_, str>` when a function sometimes allocates and sometimes borrows (e.g. normalization).
- Rely on lifetime elision. More than one signature needing an explicit `'a` is a signal the type should own its data — convert the borrow to owned before adding lifetimes.
- Reducing hot-path allocations (SmallVec, ArrayVec, string interning, `Bytes`, vectored writes): profile first, then load [performance.md](./performance.md).
- **`str::lines()` splits on `\n` only.** A line-oriented scanner ported from a language with universal newlines (Python, Ruby) silently merges a bare-`\r` file into one line — in a redaction or filtering tool that is a security divergence, not a formatting one: the whole body rides through on whatever classification the merged first line matched. Write the splitter explicitly over CR, LF and CRLF, and emit the **original bytes** for every line the rules did not change rather than re-encoding a decoded copy — a round-trip through lossy decoding transcodes lines the tool was supposed to pass through untouched.
- **The `regex` crate has no look-around.** If a rule is defined by a lookbehind or lookahead, reach for `fancy-regex` rather than hand-rolling boundary checks, which drift from the reference on the one input nobody tried. Three neighbours that type-check and still diverge: `regex::bytes` still applies Unicode `\b` (a byte-oriented token scan wants `(?-u:\b)`, or `cafémb-x1z` passes a boundary check ASCII `\b` would have failed); `str::to_lowercase()` is not case folding (`ß` maps to `ss` only under casefold, so a hash key derived from case-folded text differs between implementations — use `caseless`); and `char::is_whitespace()` excludes U+001C–U+001F, which Python's `\s` and `str.strip()` include.


## Async with Tokio

- Default runtime: `#[tokio::main]` with `features = ["full"]` for apps; `features = ["rt", "macros", "sync"]` for libraries that need to stay slim.
- `tokio::spawn` for independent tasks. `JoinSet` for a dynamic group awaited together with cancellation.
- `tokio::select!` for racing futures (timeouts, cancellation, first-wins).
- Never block the runtime: `tokio::task::spawn_blocking` for sync CPU work or blocking I/O libs.
- `tokio::sync::Mutex` only when the guard must be held across `.await`. Otherwise `std::sync::Mutex` is faster.
- **`tokio::sync::RwLock` when reads dominate writes** (config snapshots, route tables, hot caches). Many readers proceed in parallel; `Mutex` serializes them. For snapshot-swap semantics (rarely-updated config), `arc-swap::ArcSwap` is faster still — no lock on the read path.
- Cancellation: `CancellationToken` (from `tokio-util`) propagates shutdown. Long-running tasks must check it.
- Backpressure via bounded `mpsc` channels — unbounded channels hide memory growth until OOM.
- **`Semaphore` for hard concurrency limits** on spawn paths that don't fit a channel model (e.g. "at most 50 concurrent outbound HTTP calls"). `let _permit = sem.acquire().await?;` inside the task; dropping the permit releases the slot. Pair with `Arc<Semaphore>` shared across spawners.
- Don't mix async runtimes. Pick `tokio` and stick with it; `async-std` and `smol` don't interop cleanly.
- **A manually-constructed `Runtime`'s `Drop` joins already-running `spawn_blocking` tasks.** A daemon whose shutdown must not wait on wedged blocking work (long inference, stuck I/O) has to finish its cleanup and `std::process::exit(0)` rather than let the runtime drop, or use `shutdown_timeout`. `JoinSet::abort_all` does not help — abort takes effect at an await point, and a blocking closure that has already started has none.
- **`std::process::exit` skips Rust `Drop` glue, not C++ static destructors.** It is an ordinary `exit(3)`, so every handler a native library registered through `__cxa_atexit` still runs — which is where a heap diagnostic from an FFI runtime's teardown fires, non-deterministically and under load. `libc::_exit(code)` walks no handler table at all. Reach for it only after proving nothing is left to run: flush stdio, drop owning values on the normal path, and confirm no atexit hook or tempfile destructor is being relied on.
- **A panic inside a spawned per-request task is worse than an error.** Without a `catch_unwind` the panic unwinds that one task: the connection survives, no response is ever sent for that request id, and the caller waits until its own timeout. So every reachable `unwrap`/`expect`/slice index in a handler — a DB row with an unexpected enum string, a model output of unexpected shape, an index derived from untrusted input — is a client hang rather than a crash anyone would notice. Running the handler body under `spawn_blocking` gives the boundary for free: a panic arrives as a `JoinError` you convert into an error response, and the same call offloads the blocking work.


## Concurrency

| Workload | Approach |
|----------|----------|
| Independent async I/O | `tokio::spawn` + `JoinSet` or `futures::join!` |
| Data-parallel CPU work | `rayon` with `par_iter` |
| Shared mutable state across threads | `Arc<Mutex<T>>` or `Arc<RwLock<T>>`, smallest scope possible |
| Single-producer pipelines | `tokio::sync::mpsc` (async) or `std::sync::mpsc` (sync) |
| Broadcast / fan-out | `tokio::sync::broadcast` |

`rayon` and `tokio` coexist — use `tokio::task::spawn_blocking` to call a rayon pool from async code. Never call `.block_on()` from inside a tokio task; it deadlocks the runtime.

# Hot-Path Performance

Load this reference when profiling shows allocation, copy, or syscall overhead on a hot path. These are optimizations — profile first. `Vec`/`String` on a cold path isn't the bottleneck.

## Reducing hot-path heap allocations

Use stack-or-inline collections when the typical size is small and known:

- `smallvec::SmallVec<[T; N]>` — inline for ≤N items, spills to heap beyond. Good for "usually 1-8 items" cases like parsed tag lists, lookup keys, small event batches.
- `arrayvec::ArrayVec<T, CAP>` — fixed capacity, never heap-allocates. Returns an error when full. Good for bounded message buffers or per-request scratch space.
- String interning for repeatedly-seen strings (enum-like values parsed from config, tenant IDs, route keys): `dashmap::DashMap<String, &'static str>` with `Box::leak` on miss gives `&'static str` comparisons without per-call allocations.

## Zero-copy buffer slicing

`bytes::Bytes` for zero-copy slicing of shared immutable buffers — network parsers, frame decoders, protocol handlers. `BytesMut` for building buffers that `split_to` / `split_off` into `Bytes` without reallocation. Prefer `Bytes` over `Arc<Vec<u8>>` when slicing is the dominant access pattern.

## Vectored writes

`write_vectored` + `std::io::IoSlice` coalesce many buffers — interleaved headers and payloads — into a single syscall when flushing a batch of messages to a socket; the kernel does the gather. Only for measured syscall-bound flush paths; a single `write_all` is fine elsewhere.

# Production Resilience

Load this reference when productionizing a Rust service — adding config validation, health endpoints, graceful shutdown, retry/timeout discipline, or connection pools. Not needed for CLI tools or dev-only code.

- **Fail-fast config**: parse and validate all config at startup with `serde` + a `Config::load() -> Result<Self>` that returns errors for missing/invalid values. Crash before binding the listen port, not on the first request.
- **Health endpoints**: `/health` (shallow liveness, returns 200 if the process responds) and `/ready` (deep readiness, verifies DB, cache, and downstream services). Load balancers route on `/ready`; orchestrators restart on `/health`. Diagnostic endpoints must redact secrets (tokens, passwords, API keys, PII) before returning.
- **Graceful shutdown**: install a `tokio::signal` handler, trigger a `CancellationToken`, drain in-flight requests with a timeout, then exit. Axum: `.with_graceful_shutdown(shutdown_signal)`.
- **Retries**: use `backon` or `tokio-retry` with exponential backoff + jitter. Retry only transient errors (connection reset, 429, 502/503/504). Never retry 4xx.
- **When the outbound call *is* the security decision, the fallback is deny.** For an authz check, trust score, entitlement or license gate, a shed request or an exhausted retry budget must resolve to "denied", never to "allowed because the check was unavailable". Any fail-open allowance scopes to transport failure alone — connection refused, DNS failure, timeout. A response that arrived but cannot be trusted (4xx/5xx, malformed body, a variant that does not deserialize, an unrecognized verdict) stays denied: the endpoint was reached and did not answer. Model this in the type system rather than in a branch: give the verdict enum exactly the variants the protocol defines and no `Default` impl, so an unparseable response has to be handled at the parse site. A `#[derive(Default)]` on that enum is the failure — it lets `unwrap_or_default()` compile and resolve an unreachable service to whichever variant carries `#[default]`. Same for a subject with no verdict on record yet: reject by default, and allow only through an explicit onboarding opt-in.
- **Timeouts on every network call** — no defaults. `tokio::time::timeout(dur, fut)` or `reqwest::Client::builder().timeout(dur)`.
- **Connection pools**: `sqlx::PgPool`, `reqwest::Client` — build once, clone (cheap, `Arc` inside), share via `State`.
- **Resilience layer stack** (outbound HTTP clients and shared services): `ServiceBuilder::new().layer(TimeoutLayer).layer(RateLimitLayer).layer(ConcurrencyLimitLayer).layer(LoadShedLayer).layer(RetryLayer).service(client)`. Name each layer explicitly — `LoadShedLayer` sheds excess load, `ConcurrencyLimitLayer` caps in-flight requests, `RateLimitLayer` bounds request rate, `RetryLayer` retries classified transient errors. Combining `LoadShedLayer` + `ConcurrencyLimitLayer` produces proper backpressure instead of unbounded queueing.

# Rustdoc Discipline

Doc comments are compiled artifacts, not prose decoration. `cargo test --doc` builds and runs every fenced example, so a stale doc example fails CI the same way a stale unit test does.

## Which comment form

| Form | Documents | Placement |
|------|-----------|-----------|
| `///` | The item that follows | Above `pub fn`, `pub struct`, `pub enum`, `pub trait`, fields, variants |
| `//!` | The enclosing item | First lines of `lib.rs`, `main.rs`, or a `mod.rs` |

Write `//!` on `lib.rs` to answer "what is this crate for and where does a caller start". Write `///` on every public item to answer "what does this do, what does it take, what comes back".

## Required sections

Use these headings when they apply — clippy's `missing_errors_doc`, `missing_panics_doc`, and `missing_safety_doc` lints check for exactly these:

```rust
/// Parses a config file into a validated `Config`.
///
/// # Examples
///
/// ```
/// # use mycrate::Config;
/// let cfg = Config::parse("timeout = 30")?;
/// assert_eq!(cfg.timeout.as_secs(), 30);
/// # Ok::<(), mycrate::Error>(())
/// ```
///
/// # Errors
///
/// Returns [`Error::Syntax`] if the input is not valid TOML, and
/// [`Error::Validation`] if a field is out of range.
///
/// # Panics
///
/// Panics if called after `Config::freeze` has run on the same thread.
pub fn parse(input: &str) -> Result<Config, Error> { /* ... */ }
```

- `# Examples` — at minimum on every public entry point. Lines prefixed `#` are compiled but hidden from rendered output, which keeps imports and error plumbing out of the reader's way.
- `# Errors` — what each error variant means, not just "returns an error".
- `# Panics` — every reachable panic, including `unwrap` on an invariant the caller could violate.
- `# Safety` — mandatory on every `pub unsafe fn`: the invariants the caller must uphold.

## Enforcement

```rust
// lib.rs
#![deny(missing_docs)]
```

Workspace-wide, prefer the centralized form so members can't drift:

```toml
[workspace.lints.rust]
missing_docs = "deny"

[workspace.lints.rustdoc]
broken_intra_doc_links = "deny"
private_intra_doc_links = "warn"
```

`broken_intra_doc_links` is the one that pays for itself — `[`Config::parse`]` links silently rot on rename, and nothing else catches it.

## Verify

- `cargo doc --no-deps --all-features` emits zero warnings
- `cargo test --doc --all-features` passes
- Public items added in the change carry `///` with the sections that apply

Binary crates can skip `missing_docs`; library crates and any crate published to a registry should not.

# Toolchain and public interfaces

## Tooling

| Tool | Purpose |
|------|---------|
| `cargo` | Build, dep management, script runner |
| `clippy` | Lint (`cargo clippy --workspace --all-targets -- -D warnings`) |
| `rustfmt` | Formatter (`cargo fmt --all`) |
| `cargo-nextest` | Test runner |
| `cargo-deny` | License + advisory + duplicate-dep checks |
| `cargo-machete` | Find unused dependencies |

- Pin `rust-toolchain.toml` per repo so every contributor and CI uses the same compiler.
- `cargo update -p <crate>` for single-package upgrades. `cargo update` rewrites everything — avoid in PR diffs.
- `Cargo.lock` goes in version control for binaries *and* libraries (modern guidance; reproducibility wins).
- `cargo install <crate>` from a registry or git source no-ops silently when the installed version matches — it prints "package is already installed" and keeps the old binary; pass `--force` in install scripts. `cargo install --path .` always rebuilds and replaces regardless of `--force`. Either way, certify the **installed** artifact (`which <bin>` + version/behavior probe), not `target/release/<bin>` — the two can diverge when a stale env override points tests at the wrong one.
- **A crate's default feature set can encode a runtime ABI floor**, and a version bump can move it. For a crate that loads a system library dynamically, the API-level feature is the contract demanded of the `.so` at load time; compilation never opens that library, so a green build proves nothing and the failure arrives as a version rejection at first use. Pin `default-features = false` plus the explicit API-level feature the installed runtime provides, add the target features back by name, and assert the compiled-against version constant in a test. A floor is a minimum — a newer runtime still serves the older table.


## Workspaces

Multi-crate projects use a workspace with layered crates. Dependencies point inward only.

```
Cargo.toml                  # [workspace] members + [workspace.dependencies]
crates/
  protocol/    # Shared types, no deps on other workspace crates
  storage/     # Persistence, depends on protocol
  service/    # Business logic, depends on protocol + storage
  cli/        # Binary, depends on everything
```

- Centralize versions in `[workspace.dependencies]`, reference as `foo = { workspace = true }` in members.
- Keep the leaf-most crate (`protocol` / types) dependency-free so every other crate can depend on it without cycles.
- Feature flags belong on the crate that introduces the dependency, not re-exported through the workspace root.
- **Library crates expose one stable facade**: a thin `lib.rs` with a `//!` purpose doc and `pub use` re-exports — one import path per concept, internals free to reorganize without breaking callers.
- **`pub` alone does not prove an item is externally reachable.** Reachability runs through the re-export graph: a `pub` item inside a private module that is never re-exported is free to change, while the same item surfaced through a `pub use` at the crate root is not — even though its containing module stays private. (A `pub(crate)` item cannot be re-exported *outside* the crate: `pub use` on one is `E0364`, while `pub(crate) use` compiles.) Trace the facade before calling a reorganization internal. On a library crate with a published baseline, `cargo semver-checks` settles it mechanically.
- **Defining a `macro_rules!` or proc macro, or handling paths, process output, or on-disk state?** Load [macros-and-os-boundaries.md](./macros-and-os-boundaries.md) — `$crate` resolution, single-interpolation of `$x:expr`, `$t:tt` precedence, item-name collisions across invocations, `syn::Error` over panic, non-UTF-8 `Path`/`OsStr`, and write-then-rename. These type-check cleanly and fail on a caller's machine.
- **Document public items at the point of exposure.** `///` on every public item (purpose, params, return, plus `# Examples` / `# Errors` / `# Panics` / `# Safety` where they apply); `//!` for modules and crates. Doc examples compile and run under `cargo test --doc`, so they are regression tests, not decoration. Enforce with `#![deny(missing_docs)]` on library crates; see [rustdoc.md](./rustdoc.md).
- **Feature gates must error, never silently degrade.** If runtime config requests a capability the binary wasn't compiled with (e.g. `device = "gpu"` on a non-CUDA build), fail at startup — silent fallback diverges from operator config unnoticed.
- **Centralize lints at the workspace root** with `[workspace.lints.*]` — every member crate inherits the same ruleset, no per-crate `#![deny(...)]` drift:

  ```toml
  [workspace.lints.clippy]
  all = { level = "warn", priority = -1 }
  pedantic = { level = "warn", priority = -1 }
  ```

  Each member crate opts in with `[lints] workspace = true`.


## Build Profiles

When tuning Cargo build profiles (release LTO, release-dbg symbols, release-min for distributable binaries) or adding dev-machine speedups (mold linker, `target-cpu=native`, share-generics), load [build-profiles.md](./build-profiles.md).


## CI

General CI design lives with the `ia-infrastructure-engineer` agent. For Rust-specific callouts (`rustsec/audit-check`, `cargo-llvm-cov`, `Swatinem/rust-cache`, `taiki-e/install-action`, matrix coverage guidance, doc-test step), load [ci-pipeline.md](./ci-pipeline.md).