name: kermt-monitor\ndescription: Check progress for a detached KERMT run (pretrain, finetune, or any kermt_run_detached invocation). Reads run.json, queries docker for container state, tails the pretrain/finetune log, and parses progress lines (epoch, step, val loss).\nlicense: Apache-2.0\ncompatibility: Requires docker and jq. Designed for Claude Code, Codex, and Nemotron.\nmetadata:\nowner: evax@nvidia.com\nclassification: atomic-skill\nrisk_tier: skill
\nCompanion skill for any KERMT workflow that runs detached: the three pretrain\nskills (kermt-continue-pretrain, kermt-pretrain-scratch,\nkermt-add-cmim-pretrain) plus kermt-finetune. kermt-infer and\nkermt-embed run blocking by default and don't need this skill, but if a\nuser launches them detached on purpose the monitor still works (the\nworkflow-dispatch in step 4 handles unknown workflows by tailing the\nmost-recent log file in the run dir). Reads the run directory's run.json,\nqueries docker for the container's state, surfaces the latest progress,\nand either tails or follows the log.
None. This skill only reads disk + queries docker; no GPU compute.
\nOne of:
\n<run-dir> — a positional argument pointing at the directory containing\nrun.json (e.g. runs/continue-pretrain_2026-05-17T10-23Z). Preferred.--container <name-or-id> — direct container reference; the skill still\nreads run.json from the run dir referenced inside the container's\ninspect output if available, but works degraded-mode without it.Optional:
\n--lines N — number of trailing log lines to print (default 50).--follow — stream docker logs -f until ^C. Useful for \"watch the\nloss\". Without it, the skill is one-shot and exits.--json — emit a structured status report instead of human-readable text.\nUseful when the parent agent wants to take downstream action.Let RUN_DIR=$1 (or whatever path the user supplies).
Locate the manifest.
\nMANIFEST=$RUN_DIR/run.json\nRefuse to proceed if it doesn't exist; surface a helpful message\npointing the user at the run-dir convention (runs/<workflow>_<ts>/).
Parse the manifest (Python helper):
\nworkflow=$(jq -r .workflow $MANIFEST)\ncontainer_name=... # not directly in run.json today; the skill that\n # launched stored it in run.json under\n # container.name during launch (see below note).\nlogs_dir=$(jq -r .logs_dir $MANIFEST)\nimage_tag=$(jq -r .container.image_tag $MANIFEST)\nstarted_at=$(jq -r .started_at $MANIFEST)\nQuery docker for container state.
\ndocker ps --filter \"name=$container_name\" --format \\\n '{{.ID}}\\t{{.Status}}\\t{{.CreatedAt}}'\nIf absent, fall back to docker inspect $container_name --format '{{.State.Status}} (exit {{.State.ExitCode}})' to see whether the\ncontainer exited (ok or failed) or was removed (--rm after exit).
Find the live log file.
\ncase \"$workflow\" in\n continue-pretrain|pretrain-scratch) LOG=$logs_dir/pretrain_ddp.log ;;\n finetune) LOG=$logs_dir/finetune.log ;;\n *) LOG=$(ls -1t $logs_dir/*.log 2>/dev/null | head -n 1) ;;\nesac\nThe manifest's workflow field disambiguates pretrain (pretrain_ddp.log)\nfrom finetune (finetune.log). Other workflows fall back to the\nmost-recently-modified .log in $logs_dir.
Show the latest progress.
\ntail -n $LINES $LOG for the raw recent output.Current epoch: 12/100 step: 4523/9000 val_loss: 0.832 (best 0.821 @ step 4100)\nWall-clock: 1h 23m since started_at; ETA ~6h remaining.\nFinal test-metrics block (finetune, on completion). If workflow is\nfinetune AND the container has exited cleanly (State.Status=exited,\nExitCode=0) AND $RUN_DIR/ckpt/fold_*/test_result.csv exists, parse it\nand emit a per-task metric table:
Final test metrics (per task):\n Target MAE\n HLM_clearance 0.187\n RLM_clearance 0.213\n MDR1-MDCK_efflux 0.241\n solubility_pH6.8 0.156\nThe metric column matches args_applied.metric (mae for regression, auc\nfor classification, etc.). For multi-fold or ensemble runs, average across\nfolds/models and note ± std if std > 0. Skip silently if no\ntest_result.csv exists (run incomplete or no test split was emitted).
If --follow, stream live logs.
docker logs -f $container_name\nWraps until ^C.
\nStop / cleanup hints (printed at end of one-shot mode):
\nTo stop: docker stop $container_name\nTo remove: docker rm $container_name\nTo re-run: `$(jq -r .cmd_replay $MANIFEST)`\nrun.json, never touch the\ncontainer's checkpoint dir. The monitor only inspects.docker stop; if they ask to abandon, leave it\nrunning and just exit.The run.json schema as currently written does not yet include the launched\ncontainer name — kermt_run_detached prints it to stdout but the runner\nscript doesn't capture it into run.json. The monitor falls back to a\nfilesystem-based lookup: list runs/<workflow>_*/ directories and match by\nmtime; or accept --container <name> explicitly. Follow-up: have the\nlaunching skill record container name into run.json before exiting.
KERMT continue-pretrain · runs/continue-pretrain_2026-05-17T10-23Z\n Container : kermt-continue-pretrain-… (Up 1 hour, status: running)\n Image : kermt:latest@sha256:…\n Repo : 2fe00f9 (clean)\n Started : 2026-05-17T10:23:14Z (1h 23m ago)\n Workflow : continue-pretrain, pretrain_mode=hybrid, world_size=2\n\n Latest log (last 50 lines from $LOG):\n [Epoch 12/100] step 4523/9000 loss 0.832 lr 1.2e-4\n [val] step 4100 val_loss 0.821 (new best)\n ...\n\n Progress: epoch 12/100, ~12% done. ETA ~6h.\n TensorBoard: tensorboard --logdir $RUN_DIR/logs/tb\n Replay command: $(jq -r .cmd_replay $RUN_DIR/run.json)\n