Run container images as managed, autoscalable services on DataRobot. One skill, four jobs — pick the section by user intent:
\nDATAROBOT_ENDPOINT (must end in /api/v2) and DATAROBOT_API_TOKEN must be set. Run datarobot-setup if not. Auth header: Authorization: Bearer ${DATAROBOT_API_TOKEN}. The Workload API is not in the datarobot Python SDK — call REST directly.
Transport. Examples use Python httpx (pip install httpx). The API is plain HTTP, so equivalent calls work via curl or the pulumi-datarobot Pulumi provider declaratively. The skill teaches the model; transport is interchangeable.
Runnable Python in scripts/ (this skill's folder). Each uses httpx and reads DATAROBOT_ENDPOINT + DATAROBOT_API_TOKEN:
wait_for_running.py <workload_id> — poll until running; exit 2 on terminal failure, 3 on timeoutdiagnose_workload.py <workload_id> — run the 5-step debug flow, print a structured diagnosis (--json for machine-readable)wait_for_build.py <artifact_id> <build_id> — poll a server-side image build; dumps last 2KB of logs on FAILEDwait_for_replacement.py <workload_id> — poll a rolling replacement; handles the 404-when-cleared casecheck_limits.py — print the user's effective org-set scaling limits via /account/info/SKILL.md is the operational core; occasional detail lives in references/:
status-vocabulary.md — workload + proton status enums and transitionscommon-error-patterns.md — CrashLoopBackOff / ImagePullBackOff / OOMKilled / probe / exec-format / pendingschema-reference.md — schemas to look up, credential-type→key maps, public-spec path quirkslifecycle-flows.md — artifact draft→lock→prod rules, replacement preconditions, redeploy matrix, imageUri gotchascode-to-workload.md — deploy from source: dr CLI, codeRef, Execution Environments, iterate-rebuild loopweb-uis-behind-the-edge.md — browser-facing web app through the endpoint: prefix stripping, auth gate, Authorization hijack, shim, CSRF, WebSocketsAt ${DATAROBOT_ENDPOINT}/openapi.yaml. ~5 MB — never dump it whole. Save once, then slice with yq (or print() only the specific key in Python):
curl -sS \"${DATAROBOT_ENDPOINT}/openapi.yaml\" -o /tmp/wapi-spec.yaml\nyq '.components.schemas.CreateWorkloadRequest' /tmp/wapi-spec.yaml\nyq '.components.schemas | keys | .[]' /tmp/wapi-spec.yaml | grep -i workload # discover\nAll workload paths are keyed with the /api/v2/ prefix — see references/schema-reference.md.
# spec.yaml — JSON also accepted; spec is sent verbatim\nname: my-api-service\nimportance: low\nartifact:\n name: my-api-service-artifact\n spec:\n type: service\n containerGroups:\n - name: default\n containers:\n - name: main\n imageUri: ghcr.io/org/my-app:latest\n port: 8000\n primary: true\n readinessProbe: {path: /readyz, port: 8000, initialDelaySeconds: 10}\n livenessProbe: {path: /healthz, port: 8000, initialDelaySeconds: 30}\nruntime:\n containerGroups:\n - name: default # must match artifact.spec.containerGroups[].name (above)\n replicaCount: 1\n containers:\n - name: main\n resourceAllocation: {cpu: 1, memory: \"512MB\"}\ndr workload create --spec-file spec.yaml # v0.2.74+; 4xx: 400=schema/limit, 403=cap (run check_limits.py), 409=name conflict\ndr workload get <workload_id> # or `dr workload status` — poll until status=running\nLifecycle one-liners (v0.2.74+): dr workload {stop|start|delete|endpoint|list} <id>.
Raw fallback when CLI unavailable: httpx.post(f\"{base}/workloads/\", headers=headers, json=spec) + r.raise_for_status() + r.json()[\"id\"]. Then python scripts/wait_for_running.py <workload_id>.
Critical gotchas:
\nimportance: low/moderate/high/critical; type: service (default) or nim. Exactly one container per group has primary: true.cpu is cores (float OK). memory accepts decimal string (\"512MB\", units B/KB/MB/GB) or byte integer; Kubernetes binary suffixes (Mi/Gi) NOT supported.port MUST be >= 1024. The container must actually listen on it (set via image env vars or entrypoint).exec format error. Build with docker buildx build --platform linux/amd64,linux/arm64 -t <ref> --push ..submitted → provisioning → launching → running (happy path); updating during rolling redeploys; errored recoverable; failed/terminated unrecoverable. Full table in references/status-vocabulary.md.If the container serves a web app (UI + its own backend/API/WebSocket) opened in a browser via dr workload endpoint <id> (not a headless service), the DataRobot edge gateway serves it under a path prefix and: strips the prefix inbound (no outbound rewrite — the app must be sub-path aware); is the auth gate (DataRobot login required) and hijacks the Authorization header (→ 401 {\"message\":\"Invalid API key\"}, never reaching the container); passes WebSockets through. Winning pattern: set the app's base-path to the prefix + re-add it inbound (derive it from the injected WORKLOAD_ID), disable the app's own auth (trust the edge), disable CSRF, probe an unauthenticated path. Full guidance, shim code, and per-symptom diagnostics: references/web-uis-behind-the-edge.md.
| User intent | \nEndpoint | \nEffect | \n
|---|---|---|
| Rename / redescribe / change importance | \nPATCH /workloads/{id}/ | \nMetadata only — no restart | \n
| Change replicas / resources / autoscaling on the same artifact | \nPATCH /workloads/{id}/settings/ | \nTriggers rolling redeploy | \n
| Deploy a different artifact (new image / version) | \nPOST /workloads/{id}/replacement/ | \nRolling swap — see section 4 | \n
PATCH /workloads/{wid}/settings/ with full body shape — use exactly one of replicaCount or autoscaling. Read settings first via GET /workloads/{wid}/settings/, then PATCH back:
httpx.patch(\n f\"{base}/workloads/{wid}/settings/\",\n headers=headers,\n json={\n \"runtime\": {\n \"containerGroups\": [\n {\n \"name\": \"default\",\n \"replicaCount\": 3,\n \"containers\": [\n {\n \"name\": \"main\",\n \"resourceAllocation\": {\"cpu\": 2, \"memory\": \"1GB\"},\n }\n ],\n # OR: \"autoscaling\": {\"enabled\": True, \"policies\": [{\n # \"scalingMetric\": \"cpuAverageUtilization\",\n # \"target\": 70, \"minCount\": 1, \"maxCount\": 10}]}\n }\n ]\n }\n },\n)\nValid scalingMetric values: cpuAverageUtilization, httpRequestsConcurrency, gpuCacheUtilization, gpuRequestQueueDepth, or a custom NIM metric. Settings updates are rolling; zero-downtime only with replicaCount >= 2 (or autoscaling minCount >= 2).
Two admin-set caps: maxConcurrentWorkloads and maxWorkloadReplicas. Value 0 = unlimited; users can't change them. Read via GET /account/info/ — response includes {\"limits\": {\"maxConcurrentWorkloads\": N, \"maxWorkloadReplicas\": M}} (or python scripts/check_limits.py). The spec's /users/{uid}/ and /organizations/{id}/ paths require Admin API access. Exceeding either limit returns HTTP 403 with {\"detail\": \"Requested replicas (N) exceeds the maximum allowed (M).\"} — check limits first, then propose the max allowed or flag that admin help is needed.
resourceAllocation only accepts cpu, memory, gpu (count). There is NO gpuType or gpuMemory field. To target a GPU model / VRAM size: GET /mlops/compute/bundles/ lists bundles (cpu.small, gpu.l4.small, gpu.a10g.medium); pass via \"resourceBundles\": [\"gpu.l4.small\"] (a list, but exactly ONE bundle allowed) under the container group. When a bundle is set, CPU/memory in resourceAllocation are ignored — the bundle defines them.
DataRobot credentials are stored centrally and injected into environmentVars by reference:
\"environmentVars\": [\n {\"name\": \"PLAIN_VAR\", \"value\": \"literal-value\"},\n {\"source\": \"dr-credential\", \"name\": \"AWS_ACCESS_KEY_ID\",\n \"drCredentialId\": \"<credential-id>\", \"key\": \"awsAccessKeyId\"},\n]\nWorkflow: GET /credentials/?limit=50 → note the credential's credentialType → look up the valid key field names for that type in references/schema-reference.md (covers s3, basic, api_token, bearer, oauth, gcp, azure_*, databricks_*, snowflake_*, …).
Provide artifactId instead of the inline artifact block. The containerGroups[].name and containers[].name in runtime must match what the artifact defines.
python scripts/diagnose_workload.py <workload_id>\nRuns all 5 steps below, prints a structured report (status / logTail signals / flagged events / proton K8s detail / evidence / recommended next step / console URL). --json for machine-readable. If Evidence is empty, pull application logs via section 3 — don't guess from status alone.
The script encapsulates this; use the model below for ambiguous output or one-off calls.
\nGET /workloads/{id}/ — status, statusDetails.logTail (~30 lines; scan for error/exception/traceback/killed/permission denied/connection refused), statusDetails.conditions. Guard statusDetails — it's null during submitted/provisioning.GET /workloads/{id}/events/ — flag type: Warning or reason with Failed/Error/Kill/OOM; the last Warning before errored is usually the trigger.GET /workloads/{id}/protons/ — pick role: \"active\" (or the candidate during a rolling replacement; else newest createdAt).GET /workloads/{id}/protons/{pid}/statusDetails/ — 204 while initializing (not an error). Read replicas[*].containers[*].status+restartCount → replicas[*].conditions[*] (any value:false) → overallStatus.summary.Common patterns (CrashLoopBackOff, ImagePullBackOff, OOMKilled, probe/pending, exec format error) and fixes: references/common-error-patterns.md.
Workload {id} — Diagnosis\n- Status: {current}\n- Root cause: {one sentence}\n- Evidence: {the specific logTail line, condition, container reason, or event}\n- Recommended fix: {actionable next step — section 1 (settings), section 4 (artifact), or app code}\n- Console: https://app.datarobot.com/console-nextgen/workloads/{id}/overview\n| Stream | \nEndpoint | \nNeeds app instrumentation? | \n
|---|---|---|
| Logs | \n/otel/workload/{id}/logs/ | \nNo — auto from stdout/stderr | \n
| Traces | \n/otel/workload/{id}/traces/ | \nYes (OTEL spans) | \n
| Metrics | \n/otel/workload/{id}/metrics/autocollectedValues/ | \nPartially | \n
| Service stats | \n/workloads/{id}/stats/ | \nNo — DataRobot edge proxy | \n
| Replacement history | \n/workloads/{id}/history/ | \nNo — platform | \n
| Lifecycle events | \n/workloads/{id}/events/ | \nNo — platform | \n
Always check r.status_code before .json(): 401 = bad token; 404 = workload not found; 429 = rate limited (exponential backoff). All list endpoints accept limit + offset.
dr workload logs <wid> --level error --limit 100 # v0.2.74+; --follow streams; --output-format json\n--level is an EXACT severity match (not a threshold). For substring filtering on the message body, or proton-scoped logs (find proton IDs in section 2), drop to REST — dr workload logs doesn't expose those filters:
r = httpx.get(\n f\"{base}/otel/workload/{wid}/logs/\",\n headers=headers,\n params=[\n (\"searchKeys\", \"proton_id\"),\n (\"searchValues\", pid),\n (\"searchKeys\", \"level\"),\n (\"searchValues\", \"error\"),\n ],\n)\nsearchKeys / searchValues are positional parallel lists — pass a list of tuples to httpx (dict can't repeat keys). includes=<substring> does case-sensitive substring filtering on the message body.
traces = httpx.get(f\"{base}/otel/workload/{wid}/traces/\", headers=headers).json()[\n \"data\"\n]\n# summary: traceId, rootSpanName, rootServiceName, duration (NANOSECONDS), spansCount, errorSpansCount\ntrace_id = next(\n (t[\"traceId\"] for t in traces if t.get(\"errorSpansCount\", 0) > 0),\n traces[0][\"traceId\"],\n)\ntrace = httpx.get(\n f\"{base}/otel/workload/{wid}/traces/{trace_id}/\", headers=headers\n).json()\n\n\n\n
durationis NANOSECONDS on summaries AND spans. Divide by 1,000,000 for ms before display. Emptydata= app isn't instrumented; direct the user to wire up OTEL.
Convert before display: bytes→MB (/1024**2), nanocores→cores (/1_000_000), percentage already %.
stats = httpx.get(f\"{base}/workloads/{wid}/stats/\", headers=headers).json()\n# {\"period\": {...}, \"metrics\": {totalRequests, serverErrors, userErrors, slowRequests,\n# responseTime, requestsPerMinute, concurrentRequests, *ErrorRate}}. /workloads/stats/ = aggregate.\n\n\nDestructive:
\nDELETE /workloads/{id}/stats/?metricName=<name>zeroes a metric's history — only on explicit request.
Logs: timestamp | level | message, ERROR/CRITICAL first. Traces: table sorted by errors desc then recency. Metrics: apply unit conversion before display. Service stats one-liner: \"{totalRequests} requests, {totalErrorRate*100:.2f}% errors, {responseTime:.1f} ms avg, {requestsPerMinute} req/min.\" Empty data → say why (not running, not instrumented, empty window), don't just \"no data\".
An artifact is the immutable-after-lock definition of what a workload runs (image, port, env vars, probes). A workload is the running instance + its runtime (replicas, resources, autoscaling). Resources do NOT belong on the artifact.
\nFind the running artifact (workload[\"artifactId\"]), check artifact[\"status\"]. A running workload does not auto-adopt a rebuild until you redeploy.
PATCH /workloads/{id}/settings/: re-send the runtime body (even unchanged values trigger a rolling 202 redeploy that re-reads the current spec + latest COMPLETED build). Zero-downtime at ≥2 replicas. (POST /replacement/ onto the same draft also works.)POST /replacement/ onto the other artifact ID. Locked in-place edit: clone → PATCH clone → lock → replace onto the clone.Lock: dr artifact lock <id> (= PATCH /artifacts/{id}/ {\"status\":\"locked\"}). Promote (POST /workloads/{wid}/promote/, 200) locks the running draft in place, no restart. Runtime-only changes (replicas/resources/autoscaling) → PATCH /settings/; a PATCH to the artifact doesn't affect live workloads until you redeploy.
Preconditions (status-match, same-artifact rule) and the full redeploy matrix: references/lifecycle-flows.md.
The artifact's imageUri must point at a registry DataRobot can pull from (image-pull creds aren't accepted at workload creation yet). Two paths:
docker buildx ... --platform linux/amd64, push, set imageUri. Default flow.dr artifact code init + sync, then dr artifact build create builds server-side, pushes to DataRobot's internal registry, and populates imageUri. Full flow in references/code-to-workload.md.Poll builds with python scripts/wait_for_build.py <artifact_id> <build_id>; only drafts build. imageUri is build-managed — never PATCH it by hand (422 \"not permitted on this cluster\"), and never PATCH the spec mid-build (a whole-spec write clobbers the pending build image → redeploys the old one). Sequence spec edits before build create or after COMPLETED.
\n\nC2W is preview / feature-flagged —
\nENABLE_WORKLOAD_API_CONTAINERS=true(org) +DATAROBOT_CLI_FEATURE_WORKLOAD=true(client).
httpx.post(\n f\"{base}/workloads/{wid}/replacement/\",\n headers=headers,\n json={\n \"artifactId\": new_artifact_id,\n \"strategy\": \"rolling\", # only \"rolling\" supported\n \"config\": {\"warmupDurationMinutes\": 2, \"keepOldVersionMinutes\": 5}, # optional\n # \"runtime\": {...} # optional; same shape as PATCH /settings/\n },\n)\nMonitor with python scripts/wait_for_replacement.py <workload_id>. Preconditions: status must match (draft↔draft / locked↔locked, else 400); same-artifact replacement 422s for locked but works for drafts — to roll the same draft without replacement use PATCH /settings/. Not idempotent (a second POST queues another swap); GET .../replacement/ 404 = none in progress; DELETE to cancel. Detail in references/lifecycle-flows.md.
datarobot-setup — install SDK, configure auth, set env varsdatarobot-app-framework-cicd — declarative artifact + workload management via Pulumi and CI/CDdatarobot-external-agent-monitoring — instrument arbitrary agent code with OTEL → DataRobot