# RunPod diagnostics and lifecycle

Read this reference after a Pod is created, when startup appears stuck, during
an approved interactive diagnosis, and before cleanup. It uses independent
signals so an agent does not confuse capacity, image pull, container restart,
application failure, proxy behavior, or storage mismatch.

Official Pod management, connection, proxy, billing, and storage pages were
directly fetched on 2026-08-24. Re-check current provider behavior before paid
work.

## Start with the verified resource contract

Do not diagnose application startup until the provisioning checklist passes.
A Pod with the wrong GPU, image, network volume, mount, ports, or mode is an
incorrect allocation, not a slow application.

Set two lifetimes:

- a provider-side automatic termination time that survives loss of the local
  process;
- a local orchestration deadline that leaves time to collect logs/evidence and
  delete the Pod cleanly.

The current CLI documents `--terminate-after`; the live GraphQL input exposes
`terminateAfter`. Verify the current interface before use and re-read the
resulting control.

## Use a multi-signal startup matrix

| Signal | What it can prove | What it cannot prove alone |
|---|---|---|
| Pod desired/runtime state | Scheduler/control-plane view | Container health or model progress |
| System logs | Pull/start/stop/restart lifecycle events | Application root cause |
| Container logs | Application stdout/stderr | Correct volume/GPU realization |
| Port mapping/proxy | Declared route and external response | Local application health when proxy fails |
| Local health endpoint | Process readiness inside container | External route/auth correctness |
| Volume identity | Intended persistent storage attached | Application consumed correct files |
| Durable phase/failure marker | Last completed app stage and bounded error | Provider lifecycle before app starts |
| Object timestamps/sizes/hashes | Real persistent side effects | Overall workload success |
| GPU telemetry/CUDA preflight | Runtime can execute on the device | Model/framework acceptance |

Classify only after at least one provider signal and one workload/storage signal
agree.

## Distinguish common startup states

### Cold pull or scheduler delay

Likely when system logs show pull/start progression, no restart cadence exists,
the resource contract is correct, and container logs have not begun. Large
private images can take many minutes. Keep the lifetime bound, communicate the
elapsed time, and continue monitoring without inventing an application cause.

### Restart loop

Likely when system start events repeat and container logs repeat the same early
exception. Capture the exact latest error and verify mount permissions,
entrypoint/arguments, closed-stdin behavior, required runtime packages, and
mode configuration. Do not switch GPUs unless CUDA/architecture evidence points
there.

Issue #69 example: repeated `start container ...: begin` looked like a long
pull, but later container logs repeatedly showed a permission failure at the
intended persistent mount. Safe-field inspection then proved the network volume
was null. Waiting alone would not have fixed it; checking logs and storage did.

### Control-plane lag

Possible when `runtime` or ports remain null while durable markers or objects
continue changing. Treat the provider view as one signal and use the durable
timestamps to show real progress. Do not claim success until the application
acceptance artifact exists.

### Application or framework failure

Use the narrowest durable stage/error. Preserve:

1. exact bounded symptom/exception;
2. root cause when proven;
3. focused passing check after the correction;
4. dead ends ruled out and why.

Do not collapse model loading, simulator reset, compilation, inference,
artifact writing, and gateway readiness into one generic “container failed.”

## Keep diagnostics secret-safe

Use a safe-field allowlist. Never print:

- full environment arrays/maps;
- Pod-scoped API keys or provider API keys;
- registry passwords/tokens;
- model-hub or object-storage credentials;
- private capability URLs/tokens;
- complete authenticated GraphQL/REST objects that may embed any of the above.

Persist small JSON phase/failure records with atomic replacement, bounded
messages, UTC timestamps, and credential-shape redaction. Keep tracebacks in a
private transient channel; commit only sanitized evidence.

If a credential appears in logs, stop further exposure, clean transient copies,
recommend rotation, and follow the operator's explicit security decision. Do
not weaken unrelated budget, lifetime, cleanup, or production gates.

## Use one bounded interactive iteration when justified

Interactive work is useful after the immutable image reaches the workload but
repeated builds/pulls would dominate each application-layer fix.

Before connecting:

- obtain approval for one Pod and a fixed deadline;
- verify exact image/GPU/volume/mode and automatic termination;
- keep the committed worktree authoritative;
- decide how commands and evidence survive an unreliable SSH session.

During diagnosis:

- reproduce one precise failing stage;
- make the smallest source overlay from reviewed local changes;
- run a focused check, then the real acceptance boundary if safe;
- record every runtime dependency or packaging seam that the image must gain;
- avoid changing provider resources unless separately approved.

After diagnosis:

- implement and test the fix locally;
- build a new immutable image once;
- run a fresh Pod from the exact digest with overlays and diagnostic bypasses
  removed;
- delete the interactive Pod and temporary transfer objects.

Issue #69 used one interactive Pod to isolate a direct-environment batching
seam, a missing runtime compiler, and a Pillow-to-ImageIO artifact boundary.
The overlay episode established the fixes, but only the later digest run with
default compilation established deployment readiness.

## Validate the immutable workload in layers

Use workload-appropriate gates, commonly:

1. **Runtime preflight:** exact device, driver/runtime/framework versions,
   architecture support, memory, and a minimal device operation.
2. **Data/model/config identity:** immutable revisions, required files, sizes,
   hashes, offline behavior, and correct mount/staging paths.
3. **Real acceptance run:** deterministic input/seed where possible, bounded
   step/time limit, authoritative success signal, latency and peak-memory data.
4. **Artifact contract:** expected records/frames/files, schema validation,
   independent hash after download.
5. **Service readiness:** local health before public route.
6. **Isolation:** root/foreign scope denied and authorized scope accepted.
7. **Cancellation:** separate noncanonical run reaches active state, cancels
   cooperatively, returns an explicit cancelled result, and becomes ready again.

Do not hide first-use compilation in aggregate latency. Report mean/p95/max and
identify warmup or first-compile outliers when they materially dominate.

## Test HTTP proxy and connection paths

The current official CLI documents HTTP URLs as:

```text
https://<pod-id>-<port>.proxy.runpod.net
```

Declare required ports at creation. Validate in this order:

1. process is listening on the expected container address/port;
2. localhost health succeeds inside the Pod;
3. provider port mapping matches the request;
4. proxy URL responds;
5. application authentication and scope isolation behave correctly.

A proxy timeout/HTTP error does not prove the local process is down. A local
success does not prove the public proxy or authentication is correct. Test both.

SSH and web-terminal behavior varies by connection mode and image. Verify the
current official connection guide. Do not assume SCP, SSH port forwarding, or a
remote command argument works through every RunPod SSH path; use the provider's
documented transfer/connection method and test it before relying on it for the
only copy of evidence.

## Collect evidence before termination

Download only the artifacts needed to support the claim:

- safe resource identity and timestamps;
- runtime/device preflight;
- sanitized phase/failure records;
- acceptance result and metrics;
- media or output artifact;
- proxy/cancellation response summary;
- cleanup and cost record.

Verify schemas and hashes locally. Store durable public evidence where the
project contract requires it; remove temporary private prefixes after download
and verification. Preserve persistent model/data volumes unless deletion was
explicitly authorized.

## Account for cost honestly

Capture balance/current spend immediately before and after the bounded block,
then query billing with an explicit time window when records are available.
State whether a number is:

- provider-attributed Pod billing;
- an upper bound from hourly price and lifetime; or
- an observed balance-window delta.

An observed-window delta can include failed allocations, image pulls,
persistent storage, and billing lag. Do not attribute it solely to the final
successful Pod unless provider billing proves that attribution.

Official billing guidance says compute and storage consume prepaid balance;
storage charges continue after compute stops/terminates; and insufficient funds
can stop or terminate workloads and eventually endanger data. Keep critical
data backed up outside RunPod.

## Cleanup checklist

- [ ] final evidence downloaded, validated, and independently hashed;
- [ ] temporary transfer/object prefixes removed and re-listed empty;
- [ ] interactive/debug templates or secrets removed if created for the block;
- [ ] Pod deleted/terminated through a current official interface;
- [ ] `runpodctl pod list --all` or an authoritative API list shows no
  unintended Pods;
- [ ] expected persistent volumes remain and their continuing cost is stated;
- [ ] post-block balance/current spend and billing window recorded;
- [ ] production-disabled mode remains unchanged;
- [ ] production enablement is reported as a separate step awaiting approval.

Termination is destructive for data outside a network volume. Resolve and
export the exact targets first. Never delete a network volume just because its
Pod is finished.

## Official sources

- [Manage Pods and Pod logs](https://docs.runpod.io/pods/manage-pods)
- [Connect to a Pod](https://docs.runpod.io/pods/connect-to-a-pod)
- [`runpodctl pod` and proxy URL](https://docs.runpod.io/runpodctl/reference/runpodctl-pod)
- [Pod REST list/read fields](https://docs.runpod.io/api-reference/pods/GET/pods)
- [Network volumes](https://docs.runpod.io/storage/network-volumes)
- [Pod billing history](https://docs.runpod.io/api-reference/billing/GET/billing/pods)
- [Billing overview](https://docs.runpod.io/accounts-billing/billing)

All were directly fetched on 2026-08-24. Dated incident findings come from
Robium issue #69 and learnings/2026-08-24-vla-pick-and-place.md rather than
being represented as permanent RunPod guarantees.

# RunPod provisioning: from workload contract to verified Pod

Read this reference before any paid allocation. It separates current upstream
interfaces from Robium's dated provider observations and makes the post-create
resource, not the submitted command, the source of truth.

Official RunPod CLI, Pod REST, GraphQL, storage, and billing pages were directly
fetched on 2026-08-24. Re-check them before paid work because flags, schemas,
stock, prices, and product behavior change.

## Define the complete contract

Write down all required fields before looking for a machine:

| Concern | Required decision |
|---|---|
| Workload | Command/mode and one acceptance test |
| Compute | CPU or exact minimum GPU architecture, VRAM, count, CUDA support |
| Image | Registry path pinned by immutable digest; architecture/platform |
| Registry | Provider-side credential ID; never the credential value |
| Storage | Network-volume ID or Pod volume, size, mount path, persistence |
| Placement | Cloud tier, datacenter/country, volume locality, public IP |
| Network | HTTP/TCP ports and workload authentication |
| Safety | Production-disabled mode, maximum hourly price, total budget |
| Lifetime | Provider termination control and local monitoring deadline |
| Evidence | Durable phase/failure paths and artifacts to download/hash |

Do not let a convenient template silently fill a safety-critical blank.

## Run read-only inventory first

The current official CLI documents these commands:

```bash
runpodctl user
runpodctl pod list --all
runpodctl network-volume list
runpodctl gpu list --include-unavailable
runpodctl datacenter list
```

Use `runpodctl billing pods` with an explicit RFC3339 window when historical
cost attribution is needed. Inspect `runpodctl <group> --help` before relying
on a flag in automation.

Extract only safe fields:

- balance, current spend per hour, and spend limit;
- Pod ID, name, status, image, cost per hour, GPU ID/count, datacenter, ports,
  network-volume ID, mount path, and termination time;
- volume ID, name, size, and datacenter;
- GPU ID, display name, memory, cloud support, stock, and price;
- datacenter ID and GPU availability.

Do not print email, API keys, complete environment maps, registry secrets,
capability tokens, or full authenticated resource objects. A provider object
that looks like metadata can include injected environment values.

## Select compute and storage together

Use this order:

1. Filter GPUs by the workload's hard architecture, VRAM, CUDA, and count
   floor.
2. Filter by allowed cloud tier and maximum price.
3. If a network volume is required, restrict placement to its datacenter.
4. Compare live stock at the resulting datacenter(s).
5. Rank compatible candidates by cost and availability; keep incompatible and
   unavailable candidates in separate lists.

Official network-volume guidance says Pod network volumes are available in
Secure Cloud, GPU choices depend on volume location, and attachment occurs
during deployment. A volume cannot be attached or detached from an existing
Pod without deleting that Pod. The volume exists independently after compute
termination and continues accruing storage cost.

When no compatible GPU exists in the volume's datacenter, the choices are:

- wait for stock;
- use another compatible GPU already colocated;
- create and populate a new volume in another datacenter, if separately
  approved and feasible;
- remove the persistent-volume requirement only if the workload genuinely does
  not need it.

Trying GPU names in sequence without this intersection wastes time and can
select a machine that cannot see the required data.

## Stop at the paid-compute gate

Present one bounded candidate before creation:

```text
GPU ID/count:
cloud tier/datacenter:
network volume/mount:
image digest/registry ID:
ports and disabled-production mode:
hourly price and maximum duration:
acceptance test:
cleanup plan:
```

Stop when credentials, balance, licensing, image access, volume/data, stock, or
required infrastructure is absent. Operator approval applies to this resource
and lifetime. A rejected create or incorrect Pod does not automatically
authorize another paid attempt.

## Choose an interface by contract coverage

| Interface | Good fit | Required caution |
|---|---|---|
| Console | One-off human deployment and visual log access | Record/re-read the resulting exact fields |
| `runpodctl` | Inventory and straightforward Pod lifecycle | Verify current help and post-create volume/image/termination fields |
| REST | Structured automation against documented v1 resources | Live inventory identifiers can outrun generated enums/clients |
| GraphQL | Exact live schema fields or gaps in another interface | Query only safe fields; schema and docs can differ |
| SDK | Repeated automation with typed/reusable code | Pin/version it and still verify the created object |

The current CLI reference documents Pod creation fields including image,
template, GPU ID/count, cloud type, datacenter IDs, ports, environment,
network-volume ID, mount path, registry-auth ID, CUDA floor, and automatic
termination. The current REST create API documents `networkVolumeId`,
`volumeInGb`, `volumeMountPath`, and registry auth. The live GraphQL schema
documents the same core placement/lifetime fields on its on-demand input.

No interface deserves implicit trust. Prefer the simplest one that expresses
the entire contract, then verify the result.

## Use immutable images and explicit modes

- Pin deployable images by digest. A mutable tag may identify the build for
  humans, but it is not evidence of which bytes ran.
- Keep credentials in provider secrets/registry-auth resources. Do not bake
  private model data or tokens into a public image.
- Make the container entrypoint noninteractive and deterministic. Prompting for
  paths or licenses in a headless Pod is a startup defect.
- Include runtime compilers/system headers when the framework performs JIT
  compilation; a builder-only compiler does not satisfy the runtime contract.
- Persist a small sanitized phase marker before expensive stages and a bounded
  failure marker on exception.
- Set production-disabled mode explicitly. Omission is not a safe default when
  the image has multiple startup modes.

## Network-volume contract

Use one mount owner. When a network volume owns the workload mount, avoid also
requesting a default Pod volume at the same path. Issue #69's successful
GraphQL allocation used:

```graphql
mutation CreatePod($input: PodFindAndDeployOnDemandInput!) {
  podFindAndDeployOnDemand(input: $input) {
    id
    imageName
    networkVolumeId
    volumeInGb
    volumeMountPath
    desiredStatus
    costPerHr
  }
}
```

Its input selected the exact live GPU ID and supplied `networkVolumeId`,
`volumeInGb: 0`, and the intended `volumeMountPath`. This shape was checked
against the live GraphQL schema and worked on 2026-08-24. It is not a promise
that every workload needs zero Pod volume or GraphQL; it is evidence that
storage ownership must be explicit and the result must be checked.

Before startup monitoring, query a Pod representation that includes the
network volume and verify:

- expected network-volume ID is non-null;
- returned volume datacenter matches the allocated machine;
- mount path is exact;
- Pod/default volume size does not shadow or conflict with the mount;
- expected durable files are readable through a safe channel.

If any check fails, preserve safe evidence and delete the incorrect Pod. Do not
wait for application logs from a Pod that cannot meet its storage contract.

## Issue #69 interface mismatch evidence

Observed on 2026-08-24 with `runpodctl` v2.8 and the then-current APIs:

1. A template/CLI create accepted the requested network volume but the returned
   Pod reported `networkVolume: null`. Once the image started, it repeatedly
   failed to create diagnostics under the intended mount.
2. A second CLI create with explicit network-volume and mount flags again
   returned a null attachment and was deleted before startup.
3. Direct REST rejected the live inventory's exact “Server Edition” GPU ID
   because the request schema exposed a shorter enum name.
4. Direct official GraphQL creation expressed the exact inventory GPU and
   network-volume contract; the response and follow-up read confirmed the
   attachment, and the workload passed.

Generalize this to one rule: accepted arguments, templates, client-side enums,
and HTTP success do not prove provider-side realization. Re-read the resource.

## Exact post-create checklist

Before counting startup time, compare:

- [ ] one new Pod ID exists; no accidental duplicate exists;
- [ ] image resolves to the approved digest;
- [ ] GPU ID/count, cloud tier, datacenter, CUDA filter, and hourly cost match;
- [ ] registry-auth ID is correct, without printing its secret;
- [ ] network-volume ID, volume datacenter, mount path, and volume size match;
- [ ] ports/public-IP/global-network settings match;
- [ ] mode flags preserve production-disabled state;
- [ ] provider termination control and local deadline are active;
- [ ] durable evidence paths are on persistent storage when required.

Only after this checklist passes should application startup monitoring begin.

## Official sources

- [RunPod CLI overview](https://docs.runpod.io/runpodctl/overview)
- [`runpodctl pod`](https://docs.runpod.io/runpodctl/reference/runpodctl-pod)
- [`runpodctl user`](https://docs.runpod.io/runpodctl/reference/runpodctl-user)
- [`runpodctl gpu`](https://docs.runpod.io/runpodctl/reference/runpodctl-gpu)
- [`runpodctl datacenter`](https://docs.runpod.io/runpodctl/reference/runpodctl-datacenter)
- [Pod REST create API](https://docs.runpod.io/api-reference/pods/POST/pods)
- [GraphQL Pod management](https://docs.runpod.io/sdks/graphql/manage-pods)
- [Live GraphQL schema](https://graphql-spec.dev.runpod.io/)
- [Network volumes](https://docs.runpod.io/storage/network-volumes)
- [Billing](https://docs.runpod.io/accounts-billing/billing)

All were directly fetched on 2026-08-24. Issue-specific behavior comes from
Robium issue #69 and its dated learning/evidence records, not from upstream
guarantees.