# Code-to-Workload (C2W) — agent reference

The `dr` CLI subcommands referenced here (`dr artifact create`, `dr artifact code init`, `dr artifact code sync`, `dr artifact code versions`, `dr artifact code checkout`, `dr artifact build create`, `dr artifact build logs`, `dr artifact lock`, `dr workload create`, `dr workload get`, `dr workload logs`) ship in **dr CLI v0.2.74+** behind the `DATAROBOT_CLI_FEATURE_WORKLOAD=true` feature flag. Each step below also lists the raw HTTP fallback so the agent can drop down to the REST endpoints when the CLI isn't installed or when a flag is unset.

## When to reach for C2W

The user has source code but no published image. They cannot reach a registry DataRobot can pull from (no local Docker; no public registry account; the org admin hasn't pre-configured private-registry credentials). Image-pull credentials are NOT yet acceptable at workload creation, so C2W is the workaround: the platform builds the image and pushes it to DataRobot's internal registry, which workloads can pull from by default.

Don't use C2W when the user already has an image in an accessible registry — that's strictly more steps. Use the bring-your-own-image flow in SKILL.md section 1.

## Prerequisites the agent must surface

- `ENABLE_WORKLOAD_API_CONTAINERS=true` on the org (admin-set feature flag). If absent, `POST /artifacts/{id}/builds` returns a feature-flag error; the agent should fall back to bring-your-own-image or surface the gap to the user.
- `DATAROBOT_CLI_FEATURE_WORKLOAD=true` exported client-side, plus `DATAROBOT_ENDPOINT` and `DATAROBOT_API_TOKEN` already set.
- The `dr` CLI installed at **v0.2.74 or newer** (`https://github.com/datarobot-oss/cli`). Verify with `dr --version`. The `artifact` and `workload` namespaces are hidden until the feature flag is set, so a `dr --help` that doesn't list them just means the flag is missing.
- An Execution Environment with `sourceDockerImageUri` — used as the base image for the generated Dockerfile. See the next section for how to find one.

## Finding an Execution Environment

The C2W artifact spec needs `executionEnvironmentId` and `executionEnvironmentVersionId`. Discover via:

```shell
curl -sS "${DATAROBOT_ENDPOINT}/executionEnvironments/?limit=10" \
  -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" | jq '.data[] | {id, name, programmingLanguage, useCases, latestSuccessfulVersion: .latestSuccessfulVersion.id}'
```

The endpoint accepts these narrowing filters (all optional):

- **`useCases`** — one of `customModel | notebook | gpu | customApplication | sparkApplication | customJob`. The `GeneratedDockerfile` schema in the public spec does **not** constrain which `useCases` an EE must have to work with C2W (it only requires the EE to resolve to a base Docker image), and the upstream tutorial doesn't specify either. So use this filter only to narrow if the user has stated which surface they're targeting. Otherwise filter by `programmingLanguage` and `name` instead.
- **`searchFor`** — substring search on the EE's name + description.
- **`isPublic`** — boolean; restricts to platform-provided or user-created environments.

**Response shape:** envelope `{count, totalCount, data, next, previous}`; each `data[]` record has `id`, `name`, `programmingLanguage`, `isPublic`, `useCases`, `description`, `latestVersion`, `latestSuccessfulVersion`. **Use `latestSuccessfulVersion.id` for the EE version id** — `latestVersion` may point at a failed build. For more versions per EE: `GET /executionEnvironments/{id}/versions/?limit=10`.

> **Heads up — this endpoint requires the "Custom Environment" read permission** (separate from `Admin API` access). A regular user without it gets `403 {"message": "You do not have read permission for Custom Environment"}` even with `isPublic=true`. If the user hits this 403, ask them for the EE id + version id directly (their admin can provide them) rather than guess.

## Artifact spec with `imageBuildConfig`

The artifact created for a C2W flow is `draft` with `imageUri: "placeholder:latest"` — the build replaces it. The new fields versus a bring-your-own-image artifact:

```json
{
  "name": "<artifact-name>",
  "type": "service",
  "spec": {
    "containerGroups": [{
      "containers": [{
        "name": "primary",
        "imageUri": "placeholder:latest",
        "primary": true,
        "port": 8080,
        "imageBuildConfig": {
          "dockerfile": {
            "source": "generated",                            // or "provided"
            "executionEnvironmentId": "<EE_ID>",              // required when source=generated
            "executionEnvironmentVersionId": "<EE_VERSION_ID>",
            "entrypoint": ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"]
          }
        },
        "readinessProbe": {
          "path": "/version", "port": 8080,
          "initialDelaySeconds": 10, "periodSeconds": 10,
          "timeoutSeconds": 5, "failureThreshold": 6, "scheme": "HTTP"
        }
      }]
    }]
  }
}
```

Create it. CLI (v0.2.74+):

```shell
dr artifact create --spec-file /tmp/spec.json --output-format json
```

Raw fallback:

```shell
curl -sS -X POST "${DATAROBOT_ENDPOINT}/artifacts/" \
  -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \
  -H "Content-Type: application/json" --data @/tmp/spec.json
```

## Linking a project directory + syncing source

`dr artifact code init <artifact_id>` writes a `.wapi/` directory in the project root that tracks which artifact, catalog, and version this directory is bound to (conceptually similar to `.git/`). `dr artifact code sync` then:

1. Zips the project directory (respects `.dockerignore`).
2. Uploads the zip to the Files API as a new catalog version.
3. Waits for the catalog version to finish processing.
4. PATCHes the artifact's container spec to set `codeRef.datarobot.catalogId` and `codeRef.datarobot.catalogVersionId`.

`codeRef` on the container looks like:

```json
{"codeRef": {"datarobot": {"catalogId": "<id>", "catalogVersionId": "<vid>"}}}
```

If the CLI isn't available, the agent can reproduce sync manually: `POST /api/v2/files/fromFile/` with the zipped project, then `PATCH /artifacts/{id}/` with the container's `codeRef` set to the returned catalog id + version id.

## Triggering and watching a build

CLI (v0.2.74+) — when run from a directory linked via `dr artifact code init`, the artifact id is read from `.wapi/config.json` and can be omitted:

```shell
dr artifact build create                 # uses linked artifact
dr artifact build create <artifact_id>   # explicit
```

Raw fallback (empty body — `codeRef` on the artifact already tells the build system where to find the source):

```shell
curl -sS -X POST "${DATAROBOT_ENDPOINT}/artifacts/${ARTIFACT_ID}/builds" \
  -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \
  -H "Content-Type: application/json" -d '{}'
```

Response: `202 Accepted` with `{"buildIds": ["<build_id>", ...]}`.

Poll with `python scripts/wait_for_build.py <artifact_id> <build_id>` (enforces the BUILT-vs-COMPLETED distinction below). The CLI's `dr artifact build get <build_id>` returns the current status if the agent wants a one-shot check rather than blocking polling.

**Build status progression — `BUILT` is NOT terminal-success:**

```
pending → in-progress → BUILT → COMPLETED       (or → FAILED)
```

- `BUILT` means the image was built locally on the build host but **has NOT been pushed to the registry yet**.
- `COMPLETED` means the image is built AND pushed to the registry — **only then is it deployable**.
- Scheduling a workload on an artifact whose build is `BUILT` (not yet `COMPLETED`) returns `422 runtime_image_uri ... None` because the registry can't resolve the imageUri yet.
- The gap between `BUILT` and `COMPLETED` can be **seconds to minutes** for large images.

So: **wait for `COMPLETED` specifically. Never trust `BUILT` as a green-light.** `wait_for_build.py` enforces this — `BUILT` keeps polling, only `COMPLETED` exits success.

C2W flows have also been observed reporting lowercase `pending` / `in-progress` / `completed` / `failed`. The poller's `.upper()` normalization treats `completed` and `COMPLETED` as equivalent terminal-success.

For real-time build logs, use `dr artifact build logs <build_id>` (v0.2.74+) or raw `GET /artifacts/{id}/builds/{bid}/logs` which returns **plain text** (not JSON) — the Docker build output. Read it when the user wants to see why a build failed.

After `COMPLETED`, the artifact's `imageUri` is populated automatically. Re-`GET` the artifact to confirm and surface the new image reference. Do not PATCH `imageUri` manually.

> **Known race condition (RAPTOR-17673):** even after `COMPLETED`, the image can briefly be unschedulable while the registry catches up — workload create returns `422 runtime_image_uri ... None`. If you hit this, wait a few seconds and retry the workload create. Platform-side fix in flight.

## `dockerfile.source` modes

- `generated` (default): the platform detects the project type (Python + uv lockfile is the documented case) and generates a Dockerfile using the Execution Environment's `sourceDockerImageUri` as the base. Installs dependencies from the lockfile and runs the `entrypoint` from `imageBuildConfig.dockerfile.entrypoint`. Recommended when the user has a standard project layout and no Dockerfile.
- `provided`: the user includes a `Dockerfile` at the project root. The build uses that file directly. Use when generated builds don't fit — custom system packages, multi-stage builds, non-Python projects.

The agent should default to `generated` unless the user explicitly asks otherwise or the project structure obviously requires it.

## Iteration loop

User edits source → `dr artifact code sync` → `dr artifact build create` → wait for `COMPLETED` → **redeploy the running workload onto the new build.**

`code sync` + `build create` keep the **same artifact ID** and only advance its `imageUri`; a running workload does **not** auto-adopt the rebuild until you redeploy. Redeploy the same draft with a rolling `PATCH /workloads/{wid}/settings/` (re-send the runtime body — even unchanged values roll it onto the latest `COMPLETED` build), or `POST /workloads/{wid}/replacement/` onto the same draft; switching to a *different* artifact is always a replacement. Full same-draft redeploy matrix and preconditions: `references/lifecycle-flows.md`.

Do **not** PATCH the artifact spec (env/probes) *between* `build create` and `COMPLETED` — it clobbers the pending `imageUri` auto-populate and you redeploy on the old image. Make spec edits before the build, or after `COMPLETED` with a fresh `GET`.

Each `dr artifact code sync` creates a new catalog version. Each build produces a new image. The artifact tracks the current image. `dr artifact code versions` lists the catalog versions for an artifact (i.e. its code history); `dr artifact code checkout <version_id>` downloads a previous version into `.wapi/.checkouts/` for read-only inspection or rollback.

## Locking for production

Once a draft artifact builds cleanly and the workload runs the way the user wants, `dr artifact lock <artifact_id>` (v0.2.74+) promotes the draft to **locked** — name, description, and spec become immutable, the artifact gets a version number, and it can never be deleted or unlocked. The CLI is the equivalent of `PATCH /artifacts/{id}/ {"status": "locked"}` and validates build completeness server-side: every container built from source must have its code uploaded and a build `COMPLETED`, otherwise the lock is rejected with a message naming the gap.

## Failure modes the agent should recognize

| Symptom | Likely cause | Action |
|---|---|---|
| `POST /builds/` returns a feature-flag error | `ENABLE_WORKLOAD_API_CONTAINERS=false` on the org | Surface to user; fall back to bring-your-own-image if they have an alternative |
| Build status `failed` with "lock file mismatch" in logs | `pyproject.toml` updated but `uv.lock` wasn't regenerated | User runs `uv lock` locally, `dr artifact code sync` again, new build |
| Build status `failed` with "unreachable base image" in logs | EE's `sourceDockerImageUri` not pullable from the build host | Try a different EE, or report to admin |
| Build status `failed` with missing-dependency error | `pyproject.toml` doesn't list a required package | User adds dependency, `uv lock`, sync, rebuild |
| Build status stuck on `in-progress` past 5 min for small projects | Build queue contention or platform-side delay | Continue polling; check `/builds/{bid}/logs/` for output progress |
| Artifact `imageUri` still `"placeholder:latest"` after build `completed` | Build succeeded but artifact write didn't propagate (rare) | Re-`GET` the artifact; if still placeholder, file a platform bug |
| `POST /workloads/` returns `422 runtime_image_uri ... None` after `COMPLETED` | Race condition (RAPTOR-17673): registry hasn't caught up post-push | Wait a few seconds and retry. Don't treat `BUILT` as deployable — that's the most common cause of this 422 |

## State map

| Where | What |
|---|---|
| Artifact `imageUri` | Set to `"placeholder:latest"` on create; populated by the build on success |
| Artifact `imageBuildConfig` | Persists across rebuilds; the build instruction set |
| Artifact `codeRef` | Pointer to the catalog version with source code; updated each `dr artifact code sync` |
| Catalog versions | Immutable snapshots of synced source; listed by `dr artifact code versions` |
| Builds | `POST /artifacts/{id}/builds/` produces one; listed by `GET /artifacts/{id}/builds/` |
| Workload | References artifact by `artifactId`; runs whatever image the artifact currently points at |

## Cleanup sequence

When the user is done experimenting and asks the agent to tear it all down (CLI v0.2.74+ in parens, raw REST also works):

1. `dr workload stop <wid>` (`POST /workloads/{wid}/stop`)
2. `dr workload delete <wid>` (`DELETE /workloads/{wid}`)
3. `dr artifact delete <aid>` (`DELETE /artifacts/{aid}`) — **only drafts can be deleted**; locked artifacts are permanent
4. Remove `.wapi/` from the project directory

Order matters: stop before delete on the workload; delete the workload before the artifact since the workload references it.

# Common error patterns

When a workload fails, the specific failure mode points at the fix. This is the lookup table the `diagnose_workload.py` script uses internally; agents that drill in manually should match symptoms here.

## `CrashLoopBackOff`

```
state: waiting   reason: CrashLoopBackOff   restartCount: 5+
```

The container starts then exits non-zero. **Pull application logs** (`/otel/workload/{id}/logs/`) — the app is throwing during startup. Common causes:

- Missing required env var
- Bad config (database URL, API key, …)
- Missing dependency in the image
- App listening on the wrong port (port doesn't match the artifact's `port` field)
- Failed connection to a backing service (DB, cache, upstream API)

### Special case — `exec format error`

```
state: waiting   reason: CrashLoopBackOff
log line: exec format error
       (or: exec /entrypoint: no such file or directory  even though the file exists)
```

The image is the wrong CPU architecture. DataRobot's worker nodes run **linux/amd64** only. An ARM64 image (the default when you `docker build` on Apple Silicon) crash-loops immediately. Fix:

```bash
docker buildx build --platform linux/amd64 -t <registry>/<image>:<tag> --push .
# Or multi-arch (Mac dev + DataRobot prod from one tag):
docker buildx build --platform linux/amd64,linux/arm64 -t <registry>/<image>:<tag> --push .
```

Verify before referencing: `docker buildx imagetools inspect <registry>/<image>:<tag>` — `linux/amd64` must be present in the manifest.

Then update the artifact's `imageUri` (PATCH for drafts; clone + PATCH + lock for locked artifacts) and roll out via `POST /workloads/{id}/replacement/`.

## `ImagePullBackOff` / `ErrImagePull`

```
state: waiting   reason: ImagePullBackOff
message: Failed to pull image "myregistry/myapp:v1": ...
```

- Wrong image URI (typo in tag or registry)
- Private registry without credentials configured
- Tag that doesn't exist on the registry

Verify the image is pullable from a fresh machine outside DataRobot before referencing it. Check tag spelling.

## `OOMKilled`

```
state: terminated   reason: OOMKilled   exitCode: 137
```

Container exceeded its memory limit. Bump `memory` in `runtime.containerGroups[0].containers[0].resourceAllocation` via `PATCH /workloads/{id}/settings/`, or pick a larger compute bundle.

## Probe failures (`ContainersReady = False`)

```
condition: ContainersReady = False
reason: ReadinessProbe failed
```

- Probe path/port doesn't match what the app actually exposes
- App is slow to start — bump `initialDelaySeconds` on the probe (default 10s for readiness, 30s for liveness is often not enough for cold-start)
- App's health endpoint returns non-2xx — fix the app or point the probe at a working path

## Pending pod (`PodScheduled = False`)

```
phase: pending
condition: PodScheduled = False
```

K8s can't place the pod. Usually:

- Requested resources/bundle has no current capacity on the cluster
- The bundle ID is invalid (typo, or removed from the catalog) — re-run `GET /mlops/compute/bundles/` and pick a valid one

Try a smaller bundle or wait for capacity.

## Terminated with non-OOM reason

Look at `exitCode` and `message`. Useful exit codes:

| Exit code | Likely cause |
|---|---|
| `0` | Clean exit (rare for services) |
| `1` | Generic app error |
| `2` | Misuse of shell builtins / argparse |
| `126` | Command found but not executable |
| `127` | Command not found |
| `137` | SIGKILL (often `OOMKilled` — confirm with `reason`) |
| `139` | Segfault |
| `143` | SIGTERM (graceful shutdown signal received) |

Exit codes >128 indicate the process was killed by a signal (signal number = exit code − 128).

## Port not listening

Symptom: workload reaches `running` but external requests time out, or readiness probe fails with `connection refused`.

The container started, but the app inside isn't listening on the `port` you set in the artifact. Common causes:

- Image defaults to port 80 (nginx, httpd) but you set `port: 8000`. Fix by configuring the image via env var (`PORT`, `LISTEN_PORT`) or by overriding `entrypoint`.
- App is binding to `127.0.0.1` instead of `0.0.0.0` (only accepts loopback connections). Reconfigure the app.

Port 80 (and any port < 1024) is privileged on Linux. DataRobot runs containers as non-root, so privileged ports are rejected at the API level — the artifact spec must use port ≥ 1024.

## Image architecture mismatch

Covered above under `exec format error`. The signature is:

- Build was on Apple Silicon (`uname -m` = `arm64`) without `--platform linux/amd64`
- Image manifest lacks an `amd64` entry — `docker buildx imagetools inspect <ref>` shows only `arm64`

## `POST /workloads/` returns `422 runtime_image_uri ... None`

The build hasn't pushed the image to the registry yet, so the workload can't be scheduled.

Two causes (in order of likelihood):

1. **The build is still `BUILT`, not `COMPLETED`** — the most common case. `BUILT` means the image was built locally on the build host but **has NOT been pushed to the registry yet**. `COMPLETED` is the only state where the image is deployable. Sequence: `PENDING` → `IN_PROGRESS` → `BUILT` → `COMPLETED`. The gap between `BUILT` and `COMPLETED` can be seconds to minutes for large images. Fix: wait for `COMPLETED` specifically (`scripts/wait_for_build.py` does this).
2. **Race condition (RAPTOR-17673)** — even after `COMPLETED`, the registry can briefly lag and the image isn't yet resolvable. Fix: wait a few seconds and retry the workload create. Platform-side fix in flight.

If the build status is `FAILED`, the workload create will also 422 — but in that case the fix is to investigate the build, not retry the workload create. See the C2W reference's failure-modes section.

## Diagnostic decision tree

When you don't know which pattern applies:

1. **Is the container ever `running`?** Check `proton.statusDetails.replicas[].containers[].status`.
   - **Never `running`** → it's a K8s / image-level issue. Check the container `reason` (`ImagePullBackOff`, `CrashLoopBackOff` with no log lines, etc.).
   - **Briefly `running` then died** → it's an app-level issue. Pull `/otel/workload/{id}/logs/`.
2. **Is `restartCount > 0`?** That's `CrashLoopBackOff` — the app keeps crashing after start. Always pull logs.
3. **Are any conditions `false`?** Find the first false condition and act on it (PodScheduled → scheduling; ContainersReady → probes/app; Ready → ContainersReady is upstream).
4. **No specific signal anywhere?** Pull the latest events — the platform's perspective often has the answer the runtime view doesn't.

If steps 1-4 don't yield a specific cause, the issue is in the application code — report logs to the user, don't guess.

## Web UI served through the endpoint behaves wrong

If the workload runs but a **browser-facing web app** served through
`dr workload endpoint <id>` misbehaves — 404s on assets/API, redirects to the
DataRobot login (`…?next=%2F`), `401 {"message":"Invalid API key"}` on API
calls, a login that never sticks, `403 "XSRF cookie does not match"`, or a
browser Basic-auth "Sign in" modal — the cause is the edge gateway, not a crash.
Key tell: **a failing request that does NOT appear in `dr workload logs` was
rejected by the edge before reaching the container** (auth/`Authorization`
issue). See `references/web-uis-behind-the-edge.md` for the full symptom→fix
table; the short version is: make the app sub-path aware, disable the app's own
auth and CSRF, and let the DataRobot edge authenticate.

# Artifact lifecycle — rules the spec doesn't state

The SKILL.md section 4 has the operational summary and example code. This reference holds **only the behavioral rules** that aren't visible from the spec alone (`POST /artifacts/`, `PATCH /artifacts/{id}/`, etc. shapes are in the spec).

## Lifecycle states and transitions

```
create  →  iterate (PATCH while draft)  →  lock  →  rolling replacement
              ↳ status=draft                ↳ status=locked, immutable
                                              clones create new drafts
```

- Artifacts start in `draft` when created.
- While `draft`: PATCH applies in place.
- When `locked`: artifact becomes immutable. Any edit requires `POST /artifacts/{id}/clone/` (produces a new draft in the same artifact repository), then PATCH on the clone.
- Once locked, to deploy changes you trigger a **rolling replacement** on the workload (`POST /workloads/{wid}/replacement/`). Promote is the alternative for the in-place draft→locked case.

## Replacement preconditions — and the draft same-artifact exception

`POST /workloads/{id}/replacement/` enforces two preconditions the spec doesn't spell out:

1. **Status must match.** **HTTP 400** `{"detail": "Artifact status mismatch: ..."}` unless the candidate's status matches the running artifact's (draft↔draft, locked↔locked).
2. **Same-artifact rule — drafts exempt.** Passing the *current* `artifactId` returns **HTTP 422** `{"detail": ["Cannot replace with the same artifact — candidate artifact ID matches current artifact."]}` for **locked** artifacts. **Drafts are exempt: same-artifact replacement is allowed when the artifact is a draft** — so the C2W rebuild-then-replace loop works.

Neither rule is in the spec's path docs. There is no `dr workload replacement` CLI subcommand; replacement is REST-only.

### Applying a change to the SAME draft a workload runs

`code sync` + `build create` (or a spec PATCH) keep the same artifact ID; a running workload does **not** auto-adopt the change — trigger a redeploy. Options, all rolling (zero-downtime at `replicaCount`/`minCount` ≥ 2; a single replica has a brief gap):

| Goal | Do this |
|---|---|
| roll onto the latest build / spec (works on **any** cluster version) | `PATCH /workloads/{id}/settings/` — re-send the runtime body (same shape `GET /workloads/{id}/settings/` returns; even unchanged values trigger the roll). It re-reads the artifact's current spec + its latest `COMPLETED` build's `imageUri`. Returns `202`. |
| same, but want explicit warmup / rollback-window controls | `POST /replacement/` onto the same draft |
| lock the running draft in place | `POST /workloads/{id}/promote/` (no restart) |
| switch to a different / newly-locked artifact | `POST /replacement/` onto the other artifact ID |
| locked → new content | clone → patch the draft → lock the new draft → `POST /replacement/` onto the clone |

## Promote — in-place lock without restart

`POST /workloads/{wid}/promote/` is the only way to transition a workload from "running a draft" to "running a locked production version" **without** a rolling restart:

- Artifact `status` flips draft → locked (becomes immutable).
- Workload's `artifactId` keeps pointing at the same artifact (now locked).
- Running pods are NOT restarted. Traffic uninterrupted.

If you also need a rolling *restart* to apply new env vars from a recent PATCH, roll the workload onto the (same) draft's current spec with `PATCH /workloads/{wid}/settings/` — re-send the runtime body (even unchanged values trigger the rolling redeploy). Same-artifact `POST /replacement/` also works for drafts. The intent split: promote = "the running version IS production"; replacement = "deploy a *different* artifact" (or the same draft); settings-PATCH = "restart onto the same artifact's latest spec/build".

## PATCH on multi-container artifacts replaces the whole `containerGroups` array

If your artifact has multiple containers and you only want to change one, **fetch the full `spec` first, modify only the target container in place, and send the entire array back**. Sending one container will silently drop the others. The spec describes the schema shape but doesn't warn about this replacement-semantics gotcha.

Also: don't include `spec.type` in PATCH bodies — it's a read-only discriminator that the `UpdateArtifactRequest` write model rejects.

## Server-side image builds

If the artifact was created with `imageBuildConfig` referencing source code in DataRobot Files, the platform can build the image. Triggered with `POST /artifacts/{id}/builds/`; poll via `scripts/wait_for_build.py`. Two non-spec behaviors:

- On success, the platform **populates the artifact's `imageUri` automatically**. Re-`GET` the artifact to see it. Do **not** set `imageUri` by hand — a manual `PATCH` of it returns `422 {"detail": "Image URI '...' is not permitted on this cluster."}` (only build-produced images are allowed). If both `imageBuildConfig` and `imageUri` are supplied at create time, the build overwrites `imageUri` on completion.
- **Do not PATCH the artifact spec while a build is in progress.** A spec PATCH is a whole-spec read-modify-write, so it sends back the *pre-build* `imageUri` and clobbers the completion's auto-populate — the artifact keeps pointing at the **old** image and the next deploy silently runs stale code. Sequence spec edits (env/probes) *before* `build create`, or *after* `COMPLETED` with a fresh `GET` so the PATCH carries the new `imageUri`. After `COMPLETED`, confirm `imageUri` advanced before redeploying.
- Status sequence: `PENDING` → `IN_PROGRESS` → `BUILT` → `COMPLETED` (or → `FAILED`). **`BUILT` is intermediate** — image built locally but not yet pushed to the registry. Only `COMPLETED` is deployable; scheduling a workload on a `BUILT` artifact returns `422 runtime_image_uri ... None`. `wait_for_build.py` waits for `COMPLETED` specifically.
- Only drafts can build. Builds for locked artifacts can't be triggered or deleted.

## Rolling replacement — non-idempotent, 404-after-completion

- **Not idempotent.** Calling `POST /workloads/{id}/replacement/` while one is in progress queues a second swap. Always check via `GET /workloads/{id}/replacement/` (or `scripts/wait_for_replacement.py`) before retrying.
- **`GET /workloads/{id}/replacement/` returns 404** when no active replacement exists — body: `{"detail": "There is no active replacement for this workload."}`. Treat as "no replacement in progress", not as an error. The polling script handles this case explicitly.
- On `failed`, the workload reverts to the old artifact. Diagnose the candidate's pods via `diagnose_workload.py` before retrying.

## Replacement history

`GET /workloads/{id}/history/` returns the chronological list of past replacements — who, when, which strategy. Useful for audit ("which artifact version was running on 2026-04-15?") and rollback ("the previous artifact ID was X — replace back to that").

# Schema reference (non-spec content)

The public OpenAPI spec at `${DATAROBOT_ENDPOINT}/openapi.yaml` is the source of truth for all schemas and endpoints. **The spec is ~5 MB — never load it whole into agent context.** Save once and extract targeted slices with `yq`:

```bash
curl -sS "${DATAROBOT_ENDPOINT}/openapi.yaml" -o /tmp/wapi-spec.yaml
yq '.components.schemas.CreateWorkloadRequest' /tmp/wapi-spec.yaml     # schema body
yq '.paths."/workloads/{workloadId}/".patch'    /tmp/wapi-spec.yaml     # endpoint params
yq '.components.schemas | keys | .[]' /tmp/wapi-spec.yaml | grep -i otel   # discover names
```

If `yq` isn't available, fall back to Python — but only `print()` the specific key (`spec["components"]["schemas"]["X"]`), never the parsed `spec` dict itself.

This file holds only the things the spec **doesn't** document: authorization quirks, runtime constraints not enforced at the schema level, and aggregate tables that would otherwise require repeated grepping.

## Org-set scaling limits — authorization

`maxConcurrentWorkloads` and `maxWorkloadReplicas` exist on three schemas in the spec — `OrganizationRetrieve`, `OrganizationUserResponse`, `UserRetrieveResponse` — but the endpoints that return them (`GET /organizations/{id}/`, `GET /organizations/{id}/users/{uid}/`, `GET /users/{uid}/`) all require **Admin API access** and return `403 {"message": "You do not have Admin API access permissions"}` for normal users, even for self-lookup on `/users/{uid}/`.

The only path a regular user has is **`GET /account/info/`**, which returns the **already-resolved effective limits** in a `limits` block:

```json
{"limits": {"maxConcurrentWorkloads": 50, "maxWorkloadReplicas": 3}}
```

Or run `python scripts/check_limits.py`. Value `0` means unlimited; any non-zero is enforced. Exceeding either limit on `POST /workloads/`, `PATCH /workloads/{id}/settings/`, or autoscaling `maxCount` returns **HTTP 403** with body `{"detail": "Requested replicas (N) exceeds the maximum allowed (M)."}`. Both fields were added in spec v2.46.

## Public-spec path-key prefix quirk

The published spec at `https://docs.datarobot.com/en/docs/api/reference/public-api/openapi.yaml` aggregates multiple internal specs and is internally inconsistent about path-key prefixing. Runtime URLs are unaffected because `${DATAROBOT_ENDPOINT}` already includes `/api/v2`, but **spec lookups** need to know:

| Path namespace | Keyed in spec as | Example |
|---|---|---|
| Workloads + artifacts | **with** `/api/v2/` | `/api/v2/workloads/{workload_id}/protons/{proton_id}/statusDetails` |
| OTEL (workload telemetry) | **with** `/api/v2/`, and **templated** | `/api/v2/otel/{entityType}/{entityId}/logs/` (`{entityType}` = literal `workload`) |
| Credentials | **with** `/api/v2/` | `/api/v2/credentials/` |
| Compute bundles | **with** `/api/v2/` | `/api/v2/mlops/compute/bundles/` |

When grepping `spec["paths"]`, try both shapes if the first miss. Runtime calls are always `${DATAROBOT_ENDPOINT}/<rest of path>` regardless.

## Credential types and `key` field names

Used in `environmentVars` entries shaped as `{"source": "dr-credential", "name": "<env var>", "drCredentialId": "<id>", "key": "<key below>"}`. This table aggregates fields the agent would otherwise have to look up by grepping each `*Credentials` schema individually.

| `credentialType` | Available `key` field names |
|---|---|
| `s3` | `awsAccessKeyId`, `awsSecretAccessKey`, `awsSessionToken` |
| `basic` | `user`, `password` |
| `api_token` | `apiToken` |
| `bearer` | `token` |
| `oauth` | `token`, `refreshToken` |
| `gcp` | `gcpKey` |
| `azure_service_principal` | `azureTenantId`, `clientId`, `clientSecret` |
| `azure` | `azureConnectionString` |
| `databricks_access_token_account` | `databricksAccessToken` |
| `snowflake_key_pair_user_account` | `privateKeyStr`, `passphrase`, `user` |

For any credential type not listed: fetch the spec and look up `<Type>Credentials` (e.g. `S3Credentials`, `BasicCredentials`, `OAuthCredentials`) — the schema's properties are the valid `key` values.

## Schemas where the read model and write model diverge

The spec defines these but the naming/divergence is non-obvious:

- **Artifacts:** the read body is `ArtifactFormatted`; the PATCH write body is `UpdateArtifactRequest` and **does NOT accept `spec.type`** (that's a read-only discriminator). The `MultiContainerArtifactSpec` schema covers the `spec` object both ways.
- **Artifact creation:** there is **no** `CreateArtifactRequest` schema. Artifacts are created either inline via `CreateWorkloadRequest.artifact`, or by cloning via `ArtifactCloneRequest` (note the word order — not `CloneArtifactRequest`).
- **Replacement:** `POST` body is `StartReplacementRequest`; the optional `config` block is `ReplacementConfig`. Only `strategy: "rolling"` is supported.
- **Image builds:** read body is `ImageBuildFormatted`; build config on the artifact is `ImageBuildConfig`. Success status can be either `BUILT` or `COMPLETED` depending on platform version — treat both as terminal-success.

# Status vocabulary — agent action mapping

Enum values are documented in the OpenAPI spec — confirm exact values with `spec["components"]["schemas"]["WorkloadStatus"]` and similar. This file holds **only the agent-action mapping** for each value: what to *do* when a workload (or proton, container, replica) is in a given state. That mapping isn't in the spec.

## Workload status → next step

| Status | Next step |
|---|---|
| `submitted` | Wait. If stuck > 1 min, check `GET /workloads/{id}/events/` for scheduling issues |
| `provisioning` / `launching` | Wait. If stuck > 5 min, drill into `GET /workloads/{id}/protons/{pid}/statusDetails/` |
| `running` | Healthy — check telemetry if asked |
| `updating` | A rolling redeploy is in progress (settings change or artifact replacement); transitions back to `running` once new replica passes readiness |
| `suspended` / `interrupted` | Platform-paused — check events for the cause |
| `stopping` / `stopped` | If unintended, `POST /workloads/{id}/start/` |
| `errored` | Recoverable startup failure — run `scripts/diagnose_workload.py` and fix via section 1 (settings) or section 4 (artifact) |
| `failed` / `terminated` | Unrecoverable — delete and recreate after fixing root cause |

Happy path: `submitted` → `provisioning` → `launching` → `running`. Only `running` and `stopped` are stable.

## Proton roles

A "proton" is one deployment instance (one artifact + one runtime config) on a workload.

- `active` — the currently-serving deployment. The default choice for diagnostics.
- `candidate` — only present during a rolling artifact replacement. If the replacement is what's failing, debug the `candidate`, not the `active`.

If no proton has the `active` role (rare, only during initial provisioning), pick the one with the most recent `createdAt`.

## Container / replica status → smoking gun

In `proton.statusDetails`, the per-pod detail. Read in this triage order:

1. **`replicas[*].containers[*].status` + `restartCount`** is the headline.
   - `waiting` + non-zero restarts → container can't start → pull logs.
   - `terminated` → ran and died → check `reason` (`OOMKilled`, exit code) → fix or pull logs.
2. **`replicas[*].conditions[*]`** — any condition with `value: false` is a smoking gun.
   - `PodScheduled: false` → scheduling failure (resources / bundle).
   - `ContainersReady: false` + `Ready: false` → probe failures or container not ready.
3. **`overallStatus.summary`** — DataRobot's human-readable interpretation. Useful for one-line user-facing diagnoses.

For symptom → fix mapping (CrashLoopBackOff, OOMKilled, etc.), see `references/common-error-patterns.md`.

## Build status

Sequence: `PENDING` → `IN_PROGRESS` → `BUILT` → `COMPLETED` (or → `FAILED`). Lowercase variants (`pending` / `in-progress` / `completed` / `failed`) are also returned by some flows — normalize to uppercase before comparing.

- `PENDING` / `IN_PROGRESS` — keep polling.
- **`BUILT` — image built locally but NOT yet pushed to the registry. NOT deployable.** A workload scheduled on an artifact at `BUILT` returns `422 runtime_image_uri ... None`. Keep polling. The gap from `BUILT` to `COMPLETED` can be seconds to minutes for large images.
- `COMPLETED` — built AND pushed; only now is the image deployable.
- `FAILED` — terminal failure. Pull `/artifacts/{id}/builds/{bid}/logs/` for the cause.

The `wait_for_build.py` script enforces this: only `COMPLETED` exits success, `BUILT` keeps polling.

## Replacement status

- `candidate-warming` / `switching` — in progress, keep polling.
- `completed` — terminal success.
- `failed` — terminal failure; workload reverted to old artifact. Diagnose the candidate before retrying.

The endpoint also returns **404** when no active replacement exists — that's "no replacement in progress", not an error. See `references/lifecycle-flows.md` for the full semantics.

## Importance levels — scheduling priority

`importance` controls scheduling priority and eviction behavior under cluster contention:

- `low` — dev, exploration, throwaway workloads. Most likely to be evicted/deprioritized.
- `moderate` — internal tools, non-critical services.
- `high` — production services.
- `critical` — production services that must not be evicted.

At scale (many replicas), use `high` or `critical` to reduce eviction risk.

# Serving a web UI (and its backend) through the workload endpoint

Applies whenever a workload **exposes a port that serves a browser-facing web
app** — a UI plus its own backend/API/WebSocket — and users reach it through
`dr workload endpoint <id>`. This is framework-agnostic (Jupyter, Streamlit,
Gradio, a SPA + REST/gRPC-web backend, Shiny, a plain Flask/Express app, …).
It is NOT needed for headless services called machine-to-machine.

## How the DataRobot edge gateway serves the endpoint

`dr workload endpoint <id>` returns a URL under a **path prefix**, e.g.

```
https://app.datarobot.com/api/v2/endpoints/workloads/<id>/
```

Four behaviors of that edge gateway drive everything below. Verify them for the
target cluster, but they held on MTSaaS as of this writing:

1. **The prefix is STRIPPED inbound.** The browser requests
   `…/workloads/<id>/lab`; the container receives `/lab`. Confirm from
   `dr workload logs` — the app logs the *stripped* path.
2. **The gateway authenticates access.** A user must be logged into DataRobot
   to reach the endpoint at all; an unauthenticated request is met with the
   platform's own challenge (you may see a browser Basic-auth "Sign in" modal
   if the DataRobot session is missing/expired). The edge is the auth gate.
3. **The `Authorization` header is consumed by the platform.** Because the
   endpoint lives under `/api/v2/`, the edge treats an inbound `Authorization`
   header as a *DataRobot* API key. An app (or app frontend) that sends
   `Authorization: Bearer …`/`token …` to its own backend gets
   `401 {"message": "Invalid API key"}` **from the edge — the request never
   reaches the container** (it won't appear in `dr workload logs`).
4. **Responses are NOT rewritten.** The edge does not re-add the prefix to the
   app's redirect `Location` headers, HTML, or asset URLs; and it does not
   inject `X-Forwarded-Prefix` you can rely on. What the app emits is what the
   browser gets.

**WebSockets are supported** — `wss://…/workloads/<id>/…` upgrades pass through
to the container. Real-time UIs work; no special handling beyond the sub-path
rules below.

## The four things a web app must do

### 1. Be prefix-aware — the "prefix shim"

This is the non-obvious one; it cost the most time to figure out, so understand
the shape before writing code.

**The bind.** Every URL the app emits — asset tags, API/XHR calls, the
WebSocket URL, and HTTP redirect `Location`s — must carry the endpoint prefix,
or the browser resolves it against the origin root and it escapes the workload
(→ 404, or a redirect that lands on the DataRobot app login). But the edge
delivers requests with the prefix **stripped** and does **not** rewrite
responses. So the app has two seemingly contradictory needs:

- **emit** URLs *with* the prefix (so the browser stays inside the workload), yet
- **match** inbound requests that arrive *without* it.

A thin **prefix shim** reconciles the two: tell the app its external mount point
is the prefix (fixes outbound), and route on the stripped path the app actually
received (fixes inbound).

**Derive the prefix from `WORKLOAD_ID` — don't pass it in.** The endpoint path is
always `/api/v2/endpoints/workloads/<workload-id>`, and DataRobot **auto-injects
the managed env var `WORKLOAD_ID`** into every workload container. Build the
prefix from it at startup — no extra env var to plumb through, and it stays
correct across rebuilds/replacements (the workload ID never changes):

```python
import os

PREFIX = f"/api/v2/endpoints/workloads/{os.environ['WORKLOAD_ID']}"  # no trailing slash
```

(That's the same path `dr workload endpoint <id>` prints, minus scheme/host — no
need to fetch or hardcode it.)

> **Caveat — proton-id URL paths need an explicit override.** The
> `WORKLOAD_ID`-derived prefix only matches the workload-id endpoint
> (`/api/v2/endpoints/workloads/<workload-id>/`). The same workload is also
> addressable by a **proton-id** path (`/protons/<proton-id>/…` — e.g. how
> internal routing/health probes reach it), and that prefix is different *and*
> changes on every replacement (each roll creates a new proton). The proton ID
> is **not** injected into the container env, so it can't be derived at startup
> like `WORKLOAD_ID` is. If you actually need to serve under the proton-id path,
> set the base-path prefix **explicitly** via your own env var
> (e.g. `WORKLOAD_BASE_PATH`) instead of deriving it — and update it whenever the
> proton changes. For the normal browser-facing workload-id endpoint, the
> `WORKLOAD_ID` derivation above is all you need.

**Preferred: use the framework's mount setting (no custom code).** Most WSGI /
ASGI apps already implement exactly this via `SCRIPT_NAME` / `root_path`:

```python
# WSGI (Flask/Django/Bottle/...). The edge already stripped the prefix, so
# PATH_INFO is the in-mount path; SCRIPT_NAME tells the app its external mount,
# and the framework prepends it to url_for()/redirect()/static URLs.
def prefix_shim(app):
    def wrapped(environ, start_response):
        environ["SCRIPT_NAME"] = PREFIX  # outbound URLs + redirects now carry PREFIX
        return app(
            environ, start_response
        )  # route on PATH_INFO (already stripped) = matches

    return wrapped


application = prefix_shim(application)
```

- **ASGI** (FastAPI/Starlette): the same idea is built in — run
  `uvicorn --root-path "/api/v2/endpoints/workloads/$WORKLOAD_ID"` (or set
  `root_path` on the app from the env var). Starlette prepends `root_path` to
  `url_for`/redirects and routes on the received path.
- **Streamlit / Shiny / SPA**: use the base-path option
  (`--server.baseUrlPath`, `server.rootUrl`, `<base href>`), no middleware.

**Fallback: frameworks that couple routing to the base-path (Tornado / Jupyter).**
Here setting the base-path makes the server expect the prefix to be *present in
the request path*, so the stripped inbound requests 404. Set the base-path to
the prefix (for outbound) **and** re-add the prefix to the inbound request path
before routing:

```python
# Tornado/Jupyter: base_url is set to PREFIX (outbound). This shim re-adds the
# stripped prefix to the inbound path so base_url-mounted handlers match.
# Idempotent: a request that already carries the prefix (e.g. an in-cluster
# probe hitting the prefixed path) passes untouched.
_orig = Application.find_handler


def find_handler(self, request, **kw):
    p = request.path or "/"
    if p != PREFIX and not p.startswith(PREFIX + "/"):
        request.path = PREFIX + p
        request.uri = request.path + (f"?{request.query}" if request.query else "")
    return _orig(self, request, **kw)


Application.find_handler = find_handler
```

A reverse proxy baked into the image (nginx `sub_filter`/rewrite, Traefik
`StripPrefix`/`AddPrefix`) can play the same role. The shim also lets **health
probes** hit either the bare or prefixed path (see #4).

**Redirects specifically.** With the mount configured, framework-issued
redirects already include the prefix. The trap is app code that hardcodes
*root-relative* redirects or links (`redirect("/home")`, `href="/x"`): those
bypass the mount and escape the prefix. Fix them to use the framework's URL
builder / relative links, or rewrite stray outbound `Location`s as a last
resort:

```python
# Prepend PREFIX to any root-relative Location the app emits (WSGI).
# Guarded so links the framework already prefixed aren't doubled.
def fix_location(app):
    def wrapped(environ, start_response):
        def sr(status, headers, exc=None):
            headers = [
                (
                    k,
                    PREFIX + v
                    if k.lower() == "location"
                    and v.startswith("/")
                    and v != PREFIX
                    and not v.startswith(PREFIX + "/")
                    else v,
                )
                for k, v in headers
            ]
            return start_response(status, headers, exc)

        return app(environ, sr)

    return wrapped
```

### 2. Disable the app's built-in auth — let the edge be the gate

The edge already requires a DataRobot login to reach the endpoint (behavior
#2). Running the app's own token/password/cookie auth on top is redundant AND
actively breaks here:

- a bearer token the frontend sends is hijacked by the edge (behavior #3);
- the app's own login **cookies often don't round-trip** through the gateway,
  so the session never sticks (login POST returns a redirect, then every page
  bounces back to the login screen).

So configure the app to **trust the proxy / allow unauthenticated access**, and
rely on the DataRobot edge for authentication. Concretely: turn off token auth
(so no bearer header is ever sent), turn off password/login, and enable
anonymous access. Framework examples: Jupyter
`ServerApp.token=""` + `allow_unauthenticated_access=True`; Streamlit has no
auth by default (fine); a FastAPI/Express app should skip its auth middleware
on this deployment.

> **Security — confirm the gate before disabling app auth.** Only disable the
> app's auth once you have confirmed the endpoint genuinely requires a
> DataRobot login (open the URL in a private window with no DataRobot session;
> you should be blocked). If confirmed, access is gated by the DataRobot login
> and RBAC on the workload. If NOT (endpoint publicly reachable), keeping the
> app's auth is required — but then you must solve behaviors #3/#4 another way
> (e.g. cookie-only auth with unique cookie names and no `Authorization`
> header). Never leave a code-executing app both reachable and unauthenticated.

### 3. Neutralize shared-origin cookie/CSRF collisions

The app is served from the same origin as the DataRobot app (e.g.
`app.datarobot.com`), which sets its own cookies (commonly `_xsrf`,
session cookies). A same-named cookie from the app collides — the server may
read the platform's value and fail its CSRF check (e.g. Jupyter's
`403 "XSRF cookie does not match POST argument"`).

- Prefer **disabling the app's CSRF check** when app auth is already disabled
  and access is edge-gated (there is no app session to forge).
- Renaming the app's CSRF cookie is often NOT viable: compiled frontends
  frequently hard-code the cookie name (e.g. JupyterLab reads `_xsrf` to build
  its `X-XSRFToken` header), so a rename blinds the frontend and every API call
  then fails the check. Test before relying on a rename.

### 4. Probe a path that exists WITHOUT auth

Health probes hit the container directly (not through the edge). Once app auth
is disabled, login routes may disappear (e.g. `/login` → 404), so a probe
pointed at them fails and the workload never goes ready. Point
`readinessProbe`/`livenessProbe` at a lightweight, always-available endpoint
(a dedicated `/healthz`, or the app's status route). With the inbound shim in
place, an unprefixed probe path works because the shim re-adds the prefix.

## Diagnostic playbook (symptom → cause → fix)

Always cross-check `dr workload logs <id>`: **if a failing request does NOT
appear in the pod logs, the edge rejected it before the container** (an
edge/auth problem — behaviors #2/#3); if it appears with a status code, it's an
app problem (sub-path/CSRF).

| Symptom in the browser | Root cause | Fix |
|---|---|---|
| After login you land on the DataRobot app login, URL `…?next=%2F` | App emitted a root-relative redirect (`Location: /`) that escaped the prefix | Set base-path to the prefix + inbound shim (#1) |
| UI shell loads but assets/API 404; pod otherwise healthy | Proxy strips prefix; app base-path is `/` so generated URLs miss the prefix | #1 (base-path + shim) |
| `401 {"message":"Invalid API key"}` on API/XHR; those requests **absent** from pod logs | Edge hijacked the app's `Authorization` header (#3) | Disable app token auth so no bearer header is sent (#2) |
| Login "succeeds" (302) but every page bounces back to login | App login cookie not round-tripping through the edge | Disable app auth, trust the edge (#2) |
| Browser-native username/password "Sign in" modal | Edge Basic challenge — no DataRobot session in the browser | Log into DataRobot first; the edge, not the app, is prompting |
| `403 "XSRF cookie does not match"` on POST/login | `_xsrf` (or other) cookie collides with the DataRobot app's on the shared origin | Disable app CSRF check (#3) |
| Workload never leaves `launching`; probe 404/401 on a login path | Probe points at a route that no longer exists (auth disabled) or is edge-gated | Point probes at an unauthenticated app path (#4) |

## Minimal checklist for "serve my web app through the endpoint"

1. Build the prefix at startup from the auto-injected `WORKLOAD_ID`:
   `/api/v2/endpoints/workloads/$WORKLOAD_ID` (no extra env var to pass).
2. Set the app's base-path/root-path to that prefix; add an inbound
   prefix-restoring shim (or a reverse-proxy rewrite) if the framework couples
   inbound routing to the base-path.
3. Disable the app's own authentication and enable anonymous access — after
   confirming the endpoint requires a DataRobot login.
4. Disable the app's CSRF check (or verify a cookie rename doesn't break the
   frontend).
5. Point liveness/readiness probes at an unauthenticated path.
6. Verify end to end: open the endpoint while logged into DataRobot; the UI
   loads under the prefix, API calls return 2xx (visible in `dr workload logs`),
   and any WebSocket connects.

# Copyright (c) 2026 DataRobot, Inc. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Print the current user's effective DataRobot workload scaling limits.

Reads DATAROBOT_ENDPOINT (must include /api/v2) and DATAROBOT_API_TOKEN from
the environment.  Limits are set by the org admin; a value of 0 means
unlimited.

Usage:
    python check_limits.py [--json]
"""

from __future__ import annotations

import argparse
import json
import os
import sys
from typing import Any, cast

import httpx


def fetch_limits(base: str, headers: dict[str, str]) -> dict[str, Any]:
    """GET /account/info/, return the limits block plus user/org ids for context."""
    r = httpx.get(f"{base}/account/info/", headers=headers, timeout=15)
    r.raise_for_status()
    data = cast(dict[str, Any], r.json())
    return {
        "uid": data.get("uid"),
        "email": data.get("email"),
        "orgId": data.get("orgId"),
        "tenantId": data.get("tenantId"),
        "limits": data.get("limits") or {},
    }


def main() -> int:
    p = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    p.add_argument("--json", action="store_true", help="Machine-readable JSON output")
    args = p.parse_args()

    base = os.environ.get("DATAROBOT_ENDPOINT", "").rstrip("/")
    token = os.environ.get("DATAROBOT_API_TOKEN", "")
    if not base or not token:
        print(
            "DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN must be set.", file=sys.stderr
        )
        return 1
    headers = {"Authorization": f"Bearer {token}"}

    info = fetch_limits(base, headers)
    if args.json:
        print(json.dumps(info, indent=2, default=str))
        return 0

    limits = info["limits"]
    mcw = limits.get("maxConcurrentWorkloads")
    mwr = limits.get("maxWorkloadReplicas")

    def fmt(val: Any) -> str:
        if val is None:
            return "not set"
        if val == 0:
            return "unlimited (0)"
        return str(val)

    print(f"User:      {info.get('email')} ({info.get('uid')})")
    print(f"Org ID:    {info.get('orgId')}")
    print(f"Tenant:    {info.get('tenantId')}")
    print()
    print(f"  maxConcurrentWorkloads: {fmt(mcw)}")
    print(f"  maxWorkloadReplicas:    {fmt(mwr)}")
    print()
    print(
        "These are the effective limits as resolved server-side. "
        "Limits are admin-set; users cannot change them. "
        "Exceeding either returns HTTP 403 from POST /workloads/ create or "
        "PATCH /workloads/{id}/settings/ scale. "
        "The org-level and per-user-in-org endpoints exist in the spec but "
        "require Admin API access — `/account/info/` (above) is the only path "
        "a regular user has."
    )
    return 0


if __name__ == "__main__":
    sys.exit(main())

# Copyright (c) 2026 DataRobot, Inc. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Run the full 5-step debug flow against a DataRobot workload and print a
structured diagnosis: status, lifecycle-event highlights, proton K8s detail,
and a recommended next step.

Reads DATAROBOT_ENDPOINT (must include /api/v2) and DATAROBOT_API_TOKEN from
the environment.

Usage:
    python diagnose_workload.py <workload_id>
"""

from __future__ import annotations

import argparse
import json
import os
import sys
from typing import Any, cast

import httpx

CONSOLE_URL = "https://app.datarobot.com/console-nextgen/workloads/{wid}/overview"
FLAG_EVENT_REASON_KEYWORDS = ("failed", "error", "kill", "oom", "backoff", "evict")

Headers = dict[str, str]
Json = dict[str, Any]


def get_workload(base: str, headers: Headers, wid: str) -> Json:
    r = httpx.get(f"{base}/workloads/{wid}/", headers=headers, timeout=30)
    r.raise_for_status()
    return cast(Json, r.json())


def get_events(base: str, headers: Headers, wid: str, limit: int = 20) -> list[Json]:
    r = httpx.get(
        f"{base}/workloads/{wid}/events/",
        headers=headers,
        params={"limit": limit},
        timeout=30,
    )
    if r.status_code == 404:
        return []
    r.raise_for_status()
    return cast(list[Json], r.json().get("data", []))


def list_protons(base: str, headers: Headers, wid: str) -> list[Json]:
    r = httpx.get(f"{base}/workloads/{wid}/protons/", headers=headers, timeout=30)
    r.raise_for_status()
    return cast(list[Json], r.json().get("data", []))


def get_proton_status_details(
    base: str, headers: Headers, wid: str, pid: str
) -> Json | None:
    r = httpx.get(
        f"{base}/workloads/{wid}/protons/{pid}/statusDetails/",
        headers=headers,
        timeout=30,
    )
    if r.status_code == 204 or not r.text:
        return None
    r.raise_for_status()
    return cast(Json, r.json())


def diagnose(base: str, headers: Headers, wid: str) -> Json:
    """Returns a dict with: status, summary, evidence, recommended_next_step."""
    w = get_workload(base, headers, wid)
    status = w.get("status", "unknown")
    status_details = w.get("statusDetails") or {}
    log_tail = status_details.get("logTail", []) or []

    findings: list[str] = []
    evidence: str | None = None

    # Step 1 — scan logTail for obvious signals
    keywords = (
        "error",
        "exception",
        "traceback",
        "killed",
        "permission denied",
        "connection refused",
        "exec format error",
    )
    for line in log_tail[-30:]:
        lower = (line or "").lower()
        for kw in keywords:
            if kw in lower:
                evidence = line.strip()
                findings.append(
                    f"logTail line matched keyword {kw!r}: {evidence[:200]}"
                )
                break
        if evidence:
            break

    # Step 2 — flag noteworthy lifecycle events
    flagged_events: list[str] = []
    for ev in get_events(base, headers, wid, limit=30):
        ev_type = (ev.get("type") or "").lower()
        ev_reason = (ev.get("reason") or "").lower()
        if ev_type == "warning" or any(
            k in ev_reason for k in FLAG_EVENT_REASON_KEYWORDS
        ):
            flagged_events.append(
                f"  [{ev.get('timestamp', '')[:19]}] {ev.get('type', '?')} "
                f"{ev.get('reason', '?')}: {(ev.get('message') or '')[:200]}"
            )
            if not evidence:
                evidence = ev.get("message") or f"{ev.get('reason')}"

    # Step 3/4 — drill into the active (or most recent) proton
    proton_summary: str | None = None
    proton_detail_summary: str | None = None
    protons = list_protons(base, headers, wid)
    if protons:
        target = next(
            (p for p in protons if p.get("role") == "active"),
            max(protons, key=lambda p: cast(str, p.get("createdAt", ""))),
        )
        proton_summary = (
            f"{len(protons)} proton(s); using {target.get('role', '?')} {target['id']}"
        )
        detail = get_proton_status_details(base, headers, wid, target["id"])
        if detail is None:
            proton_detail_summary = (
                "statusDetails returned 204 — proton still initializing"
            )
        else:
            overall = detail.get("overallStatus", {}) or {}
            replicas = detail.get("replicas", []) or []
            lines = [
                f"overallStatus.state = {overall.get('state', '?')}",
                f"overallStatus.summary = {(overall.get('summary') or '')[:200]}",
                f"replicas: {len(replicas)}",
            ]
            for rep in replicas[:3]:
                for c in rep.get("containers", []) or []:
                    lines.append(
                        f"  container {c.get('name', '?')}: status={c.get('status')} "
                        f"ready={c.get('ready')} restarts={c.get('restartCount')} "
                        f"image={(c.get('image') or '')[:80]}"
                    )
                for cond in rep.get("conditions", []) or []:
                    met = cond.get("value", cond.get("met"))
                    mark = "OK" if met else "--"
                    lines.append(f"  [{mark}] {cond.get('type', '?')}")
            proton_detail_summary = "\n".join(lines)
            if not evidence and overall.get("summary"):
                evidence = overall["summary"]

    # Recommendation
    if status == "running":
        recommendation = "Workload is running — nothing to debug here; pull telemetry if you need to look at request behavior."
    elif status in ("submitted", "provisioning", "launching"):
        recommendation = (
            "Workload is still coming up. Wait 30–60 seconds and re-run. "
            "If stuck > 5 min, check events + proton statusDetails (above)."
        )
    elif status == "errored":
        if not evidence:
            recommendation = (
                "errored with no specific signal in logTail/events. Pull application logs "
                "(`datarobot-workload-telemetry`: GET /otel/workload/<id>/logs/) for the underlying cause."
            )
        else:
            recommendation = (
                "errored. Fix the root cause flagged above. For image/spec/env-var changes use "
                "`datarobot-workload-artifacts`; for replicas/memory/bundle changes use "
                "`datarobot-workload-management`."
            )
    elif status in ("stopping", "stopped"):
        recommendation = "Workload is shutting down or stopped. Start it via POST /workloads/<id>/start/ if unintended."
    else:
        recommendation = f"Unhandled status {status!r}; check events above and the console URL below."

    return {
        "workload_id": wid,
        "status": status,
        "logTail_findings": findings,
        "events_flagged": flagged_events,
        "proton_summary": proton_summary,
        "proton_detail": proton_detail_summary,
        "evidence": evidence,
        "recommendation": recommendation,
    }


def print_report(d: Json) -> None:
    print(f"Workload {d['workload_id']} — Diagnosis")
    print(f"  Status:           {d['status']}")
    if d["logTail_findings"]:
        print("  logTail signals:")
        for line in d["logTail_findings"]:
            print(f"    - {line}")
    if d["events_flagged"]:
        print(f"  Flagged events ({len(d['events_flagged'])}):")
        for line in d["events_flagged"][:5]:
            print(line)
    if d["proton_summary"]:
        print(f"  Proton:           {d['proton_summary']}")
    if d["proton_detail"]:
        print("  Proton detail:")
        for line in d["proton_detail"].split("\n"):
            print(f"    {line}")
    if d["evidence"]:
        print(f"  Evidence:         {d['evidence'][:300]}")
    print(f"  Recommendation:   {d['recommendation']}")
    print(f"  Console:          {CONSOLE_URL.format(wid=d['workload_id'])}")


def main() -> int:
    p = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    p.add_argument("workload_id")
    p.add_argument(
        "--json",
        action="store_true",
        help="Emit raw JSON instead of human-readable text",
    )
    args = p.parse_args()

    base = os.environ.get("DATAROBOT_ENDPOINT", "").rstrip("/")
    token = os.environ.get("DATAROBOT_API_TOKEN", "")
    if not base or not token:
        print(
            "DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN must be set.", file=sys.stderr
        )
        return 1
    headers = {"Authorization": f"Bearer {token}"}

    diag = diagnose(base, headers, args.workload_id)
    if args.json:
        print(json.dumps(diag, indent=2, default=str))
    else:
        print_report(diag)
    return 0


if __name__ == "__main__":
    sys.exit(main())

# Copyright (c) 2026 DataRobot, Inc. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Poll a server-side artifact image build until the image is deployable.

IMPORTANT: only COMPLETED means the image is pushed to the registry and
deployable. BUILT is an intermediate state — the image has been built
locally but NOT yet pushed.  Scheduling a workload on a BUILT artifact
returns `422 runtime_image_uri ... None` because the registry can't
resolve the imageUri yet.  This script waits for COMPLETED specifically;
BUILT keeps polling.

Build progression: pending → in-progress → BUILT (built, not yet pushed)
→ COMPLETED (pushed, deployable). Lowercase variants (`completed`,
`failed`) from the C2W flow are normalized to uppercase before
comparison.

Reads DATAROBOT_ENDPOINT (must include /api/v2) and DATAROBOT_API_TOKEN
from the environment.  Exits 0 on COMPLETED, 2 on FAILED (prints last
2KB of build logs to stderr), 3 on timeout.

Usage:
    python wait_for_build.py <artifact_id> <build_id> [--timeout SECONDS] [--interval SECONDS]
"""

from __future__ import annotations

import argparse
import os
import sys
import time
from typing import Any, cast

import httpx

# Only COMPLETED means the image is pushed to the registry and deployable.
# BUILT is intermediate (built locally, not yet pushed) — keep polling.
SUCCESS = {"COMPLETED"}
FAILURE = {"FAILED"}

Headers = dict[str, str]
Json = dict[str, Any]


def wait_for_build(
    base: str,
    headers: Headers,
    artifact_id: str,
    build_id: str,
    timeout: int,
    interval: int,
) -> Json:
    deadline = time.time() + timeout
    last_status: str | None = None
    while time.time() < deadline:
        r = httpx.get(
            f"{base}/artifacts/{artifact_id}/builds/{build_id}/",
            headers=headers,
            timeout=30,
        )
        r.raise_for_status()
        b = cast(Json, r.json())
        status = (b.get("status") or "").upper()
        if status != last_status:
            print(
                f"[{int(time.time() - (deadline - timeout)):>4}s] build status: {status}",
                flush=True,
            )
            last_status = status
        if status in SUCCESS:
            return b
        if status in FAILURE:
            logs = httpx.get(
                f"{base}/artifacts/{artifact_id}/builds/{build_id}/logs/",
                headers=headers,
                timeout=30,
            ).text
            print(f"--- last 2KB of build logs ---\n{logs[-2000:]}", file=sys.stderr)
            raise RuntimeError(f"Build {build_id} FAILED")
        time.sleep(interval)
    raise TimeoutError(
        f"Build {build_id} did not finish within {timeout}s (last status: {last_status})"
    )


def main() -> int:
    p = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    p.add_argument("artifact_id")
    p.add_argument("build_id")
    p.add_argument(
        "--timeout",
        type=int,
        default=1800,
        help="Total poll budget in seconds (default: 1800)",
    )
    p.add_argument(
        "--interval",
        type=int,
        default=15,
        help="Poll interval in seconds (default: 15)",
    )
    args = p.parse_args()

    base = os.environ.get("DATAROBOT_ENDPOINT", "").rstrip("/")
    token = os.environ.get("DATAROBOT_API_TOKEN", "")
    if not base or not token:
        print(
            "DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN must be set.", file=sys.stderr
        )
        return 1
    headers = {"Authorization": f"Bearer {token}"}

    try:
        b = wait_for_build(
            base, headers, args.artifact_id, args.build_id, args.timeout, args.interval
        )
    except RuntimeError as e:
        print(f"FAILED: {e}", file=sys.stderr)
        return 2
    except TimeoutError as e:
        print(f"TIMEOUT: {e}", file=sys.stderr)
        return 3

    print(
        f"\nBUILD COMPLETED — image pushed to the registry. "
        f"Artifact's imageUri is now populated; GET /artifacts/{args.artifact_id}/ to confirm "
        f"before scheduling a workload."
    )
    return 0


if __name__ == "__main__":
    sys.exit(main())

# Copyright (c) 2026 DataRobot, Inc. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Poll a workload's active artifact replacement until it completes or fails.

GET /workloads/{id}/replacement/ returns 404 when there is no active
replacement.  This script treats 404 mid-poll as "completed and cleared" (the
platform removes the record after settling) and returns the last-seen status
to the caller.  If 404 is the very first response (no replacement was ever
started), the script exits with a clear "no active replacement" message.

Reads DATAROBOT_ENDPOINT (must include /api/v2) and DATAROBOT_API_TOKEN from
the environment.  Exits 0 on completed, 2 on failed, 3 on timeout, 4 if no
replacement was active at start.

Usage:
    python wait_for_replacement.py <workload_id> [--timeout SECONDS] [--interval SECONDS]
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import time
from typing import Any, cast

import httpx

Headers = dict[str, str]
Json = dict[str, Any]


def wait_for_replacement(
    base: str,
    headers: Headers,
    wid: str,
    timeout: int,
    interval: int,
) -> tuple[str, Json | None]:
    """Returns (outcome, last_seen_record).  outcome ∈ {completed, failed, gone, timeout}."""
    deadline = time.time() + timeout
    last: Json | None = None
    while time.time() < deadline:
        r = httpx.get(
            f"{base}/workloads/{wid}/replacement/", headers=headers, timeout=30
        )
        if r.status_code == 404:
            return ("gone" if last is None else "completed"), last
        r.raise_for_status()
        last = cast(Json, r.json())
        status = last.get("status")
        print(
            f"[{int(time.time() - (deadline - timeout)):>4}s] replacement status: {status}",
            flush=True,
        )
        if status == "completed":
            return "completed", last
        if status == "failed":
            return "failed", last
        time.sleep(interval)
    return "timeout", last


def main() -> int:
    p = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    p.add_argument("workload_id")
    p.add_argument(
        "--timeout",
        type=int,
        default=1200,
        help="Total poll budget in seconds (default: 1200)",
    )
    p.add_argument(
        "--interval",
        type=int,
        default=20,
        help="Poll interval in seconds (default: 20)",
    )
    args = p.parse_args()

    base = os.environ.get("DATAROBOT_ENDPOINT", "").rstrip("/")
    token = os.environ.get("DATAROBOT_API_TOKEN", "")
    if not base or not token:
        print(
            "DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN must be set.", file=sys.stderr
        )
        return 1
    headers = {"Authorization": f"Bearer {token}"}

    outcome, last = wait_for_replacement(
        base, headers, args.workload_id, args.timeout, args.interval
    )

    if outcome == "gone":
        print(
            f"No active replacement on workload {args.workload_id} (404 at start). "
            f"Did you intend to POST /workloads/{args.workload_id}/replacement/ first?"
        )
        return 4
    if outcome == "completed":
        print("Replacement completed.")
        if last:
            print(json.dumps(last, indent=2, default=str)[:600])
        return 0
    if outcome == "failed":
        print(
            f"Replacement FAILED — workload reverted to the old artifact. "
            f"Diagnose with: python ../../datarobot-workload-debug/scripts/diagnose_workload.py {args.workload_id}",
            file=sys.stderr,
        )
        if last:
            print(json.dumps(last, indent=2, default=str)[:600], file=sys.stderr)
        return 2
    # timeout
    print(
        f"TIMEOUT — replacement on {args.workload_id} did not settle within {args.timeout}s.",
        file=sys.stderr,
    )
    if last:
        print(f"Last seen: {json.dumps(last, default=str)[:300]}", file=sys.stderr)
    return 3


if __name__ == "__main__":
    sys.exit(main())

# Copyright (c) 2026 DataRobot, Inc. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Poll a DataRobot workload until it reaches the `running` status.

Reads DATAROBOT_ENDPOINT (must include /api/v2) and DATAROBOT_API_TOKEN from
the environment.  Returns exit 0 once the workload is running, exit 2 if it
enters a terminal failure state, exit 3 on timeout.

Usage:
    python wait_for_running.py <workload_id> [--timeout SECONDS] [--interval SECONDS]
"""

from __future__ import annotations

import argparse
import json
import os
import sys
import time
from typing import Any, cast

import httpx

TERMINAL_FAILURES = ("errored", "failed", "terminated")


def wait_for_running(
    base: str,
    headers: dict[str, str],
    workload_id: str,
    timeout: int,
    interval: int,
) -> dict[str, Any]:
    deadline = time.time() + timeout
    last_status: str | None = None
    while time.time() < deadline:
        r = httpx.get(f"{base}/workloads/{workload_id}/", headers=headers, timeout=30)
        r.raise_for_status()
        w = cast(dict[str, Any], r.json())
        status = w.get("status")
        if status != last_status:
            print(
                f"[{int(time.time() - (deadline - timeout)):>4}s] status: {status}",
                flush=True,
            )
            last_status = status
        if status == "running":
            return w
        if status in TERMINAL_FAILURES:
            raise RuntimeError(
                f"Workload {workload_id} entered terminal state {status!r}. "
                f"statusDetails={json.dumps(w.get('statusDetails'), default=str)[:500]}"
            )
        time.sleep(interval)
    raise TimeoutError(
        f"Workload {workload_id} did not reach 'running' within {timeout}s "
        f"(last status: {last_status})"
    )


def main() -> int:
    p = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    p.add_argument("workload_id")
    p.add_argument(
        "--timeout",
        type=int,
        default=300,
        help="Total poll budget in seconds (default: 300)",
    )
    p.add_argument(
        "--interval",
        type=int,
        default=10,
        help="Poll interval in seconds (default: 10)",
    )
    args = p.parse_args()

    base = os.environ.get("DATAROBOT_ENDPOINT", "").rstrip("/")
    token = os.environ.get("DATAROBOT_API_TOKEN", "")
    if not base or not token:
        print(
            "DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN must be set.", file=sys.stderr
        )
        return 1
    headers = {"Authorization": f"Bearer {token}"}

    try:
        w = wait_for_running(
            base, headers, args.workload_id, args.timeout, args.interval
        )
    except RuntimeError as e:
        print(f"FAILED: {e}", file=sys.stderr)
        return 2
    except TimeoutError as e:
        print(f"TIMEOUT: {e}", file=sys.stderr)
        return 3

    print(f"\nRUNNING. endpoint: {w.get('endpoint')}")
    return 0


if __name__ == "__main__":
    sys.exit(main())