Claude Cursor Skill

deepstream-import-vision-model

Use this skill to bring a supported object-detection vision model from HuggingFace or NVIDIA NGC into an NVIDIA DeepStream pipeline with end-to-end automation: ONNX download, SafeTensors export, TRT engine build, custom nvinfer bbox parser, multi-stream benchmark, and PDF report.

LLM Mart · 0 points · 4 views 0 listing impressions 0 install-command copies
Download nvidia-skills-skills_deepstream-import-vision-model-d8519c5.zip · 147 KB

Install

skills CLI npx skills add https://github.com/NVIDIA/skills/tree/main/skills/deepstream-import-vision-model
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install nvidia-skills@llmmart
Git git clone https://github.com/NVIDIA/skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole nvidia/skills collection as a plugin from our marketplace. Git is the plain clone.

README

DeepStream Import Vision Model

Automated end-to-end pipeline: HuggingFace model → TensorRT engine → DeepStream multi-stream benchmark → PDF report.

Overview

This self-contained skill uses four phase-specific reference documents. Together they automate the full model bringup workflow for NVIDIA DeepStream, from downloading a model on HuggingFace or NVIDIA NGC to a publication-ready benchmark report.

Supported input formats: ONNX (direct), SafeTensors (auto-exported via torch.onnx.export).

Current scope: object detection models only. Classification, segmentation, pose estimation, and other vision tasks are not yet supported — the pipeline fails fast if a non-detection architecture is detected in config.json.

Prerequisites

Host: only Docker + the NVIDIA driver. Everything else runs inside the DeepStream container — DeepStream, TensorRT/trtexec, the Python export venv (torch/onnx/onnxruntime), wkhtmltopdf, deepstream-app/gst-launch-1.0 — and is bootstrapped by setup.sh. Nothing is installed on the host. Runs identically on Linux and Windows (Docker Desktop + WSL2 backend, required for GPU).

  • Docker — Docker Desktop with the WSL2 backend on Windows; Docker Engine + NVIDIA Container Toolkit on Linux.
  • NVIDIA GPU + driver (on Windows, the WSL2 GPU driver — no host CUDA/TensorRT/DeepStream needed).
  • docker pull nvcr.io/nvidia/deepstream:9.1-triton-multiarch, then run setup.sh through the container.

See references/windows.md for the cross-platform run model and the per-shell docker run mount token.

Installation

bash <path-to-deepstream-import-vision-model>/install.sh --target <your-project-path>

Preview what will be installed first with --dry-run:

bash <path-to-deepstream-import-vision-model>/install.sh --target <your-project-path> --dry-run

Where <path-to-deepstream-import-vision-model> is the location of this skill in your repo, e.g.:

  • In team-mind-hub: team-skills/deepstream-sdk/deepstream-import-vision-model
  • In ds-copilot: team-skills/deepstream-sdk/deepstream-import-vision-model (same path)

The script copies the complete skill into the target project for Claude Code, Codex, and Cursor. Re-running it safely refreshes an existing installation.

Usage

Claude Code:

Use deepstream-import-vision-model to run this model: https://huggingface.co/onnx-community/yolov8n

Codex:

Use $deepstream-import-vision-model to deploy and benchmark https://huggingface.co/onnx-community/yolov8n

Cursor:

@deepstream-import-vision-model run this model: https://huggingface.co/onnx-community/yolov8n

The skill runs the full pipeline autonomously — no manual steps required.

Pipeline Steps

Step Phase reference Action
1–3 references/model-acquire.md Browse HF repo, download ONNX or export SafeTensors
4–5 references/engine-build.md Build dynamic TRT engine, run trtexec benchmarks
6–7 references/pipeline-run.md Custom bbox parser, DeepStream single + multi-stream
8 references/report-generation.md 5 charts, HTML report, PDF

Output Structure

Per-model outputs are written to models/<model_name>/ in your project:

models/<model_name>/
  model/          ONNX file(s)
  parser/         Custom nvinfer bbox parser (.cpp, .so)
  config/         nvinfer config, DS app config, labels.txt
  scripts/        Model-specific run helpers
  benchmarks/     TRT engines, trtexec logs
  reports/        benchmark_report.md / .html / .pdf + charts/
  samples/        Output videos, test frames, KITTI detections

Files in this package

deepstream-import-vision-model/
├── SKILL.md                    Top-level skill definition
├── README.md                   This file
├── references/                 Phase-specific runbooks
├── scripts/                    Utility scripts by pipeline phase
└── tests/                      Installer and script regression tests

Skill manifest

DeepStream Import Vision Model

When this skill is active, read the relevant reference document before starting each phase. Do not rely on memory — reference documents contain exact script paths, bash variable conventions, log filename contracts, and critical parsing rules.

Current scope: Object detection models only. Fail fast on classification, segmentation, or other architectures detected in config.json.

Model choice — always offer two options

Before preflight, browsing, downloads, or file creation, present exactly these two choices. Do not start with only an open-ended model-source prompt. If the user's request already clearly selects a model, confirm the matching choice instead of asking redundantly.

1. Default model (recommended)

Use the validated Hugging Face RT-DETR model:

model_id: PekingU/rtdetr_r50vd
source: huggingface
task: object-detection
precision_preference: fp16

2. Custom object-detection model

Ask for one supported source:

  • Hugging Face model ID (organization/model) or full model URL.
  • NVIDIA NGC catalog model URL including its version.

Explain that the skill currently rejects classification, segmentation, and other non-detection architectures after inspecting config.json. Do not invent or silently substitute a model when the custom source is missing or unsupported.

For a dry run, present the same two choices and simulate discovery, build, benchmark, and report stages without browsing, downloading, launching Docker, writing files, or starting processes.

Pipeline Overview

Step Phase Reference What it does
1–3 Model Acquire references/model-acquire.md Browse HF/NGC, detect format, download ONNX or export SafeTensors
4–5 Engine Build references/engine-build.md Build dynamic TRT engine, run trtexec BS=1 and BS=MAX_BS
6–7 DS Pipeline references/pipeline-run.md Custom bbox parser, nvinfer config, single-stream + multi-stream benchmarks
8 Report references/report-generation.md 5 charts, HTML, PDF benchmark report

Run the full pipeline autonomously without pausing for confirmation at each step.

Runs entirely through Docker (no host packages)

Every step runs INSIDE the DeepStream container. The host needs only Docker + the NVIDIA driver — no host python/venv/torch/trtexec/make/wkhtmltopdf. This works identically on Linux and Windows (Docker Desktop + WSL2 backend, required for --gpus). The per-shell bind-mount token is the only OS difference — -v "$PWD":/work (bash), -v "${PWD}:/work" (PowerShell), -v "%cd%:/work" (cmd); full guide in references/windows.md. All venv/ONNX/ engine/parser/config/report artifacts live under the mounted working root and persist between the ephemeral --rm containers.

Pre-flight — bootstrap + verify (through the container)

1. One-time bootstrap — builds build/.venv_optimum (torch/onnx/onnxruntime/report deps; the venv name is historical, optimum is no longer used) + installs wkhtmltopdf, all in-container. From the working root:

docker run --rm -it --gpus all --shm-size=16g -v "$PWD":/work -w /work \
  --entrypoint bash nvcr.io/nvidia/deepstream:9.1-triton-multiarch \
  .claude/skills/deepstream-import-vision-model/setup.sh

2. Preflight — GPU + venv + trtexec, run THROUGH the container (container-mode auto-detects):

docker run --rm --gpus all -v "$PWD":/work -w /work \
  --entrypoint bash nvcr.io/nvidia/deepstream:9.1-triton-multiarch \
  .claude/skills/deepstream-import-vision-model/scripts/preflight.sh   # proceed only on PASS

Every subsequent phase runs the same way — issue the model's commands via docker run … --entrypoint bash … -lc '<commands>' (or the .claude/skills/deepstream-import-vision-model/scripts/dsrun.sh wrapper: bash .claude/skills/deepstream-import-vision-model/scripts/dsrun.sh '<in-container command>'), using PY=build/.venv_optimum/bin/python and trtexec at /usr/src/tensorrt/bin/trtexec inside the container. deepstream-app, gst-launch-1.0, and /opt/nvidia/deepstream/… sample paths all exist in the image. TensorRT build+runtime share one image, so there is no version skew (the concern the old "build on the host" rule tried to avoid — see references/engine-build.md). sample_720p.mp4 ships in the image; set DS_VIDEO only to override.

Mandatory Output Structure

Create once MODEL_NAME is known (Step 1). Never dump files flat.

models/{model_name}/
  model/           <- ONNX file(s)
  parser/          <- .cpp, Makefile, .so
  config/          <- nvinfer config, ds-app config, labels.txt
  scripts/         <- run helper scripts
  benchmarks/
    engines/       <- _dynamic_b{MAX_BS}.engine, timing.cache, build logs
    b1/            <- trtexec BS=1 log
    b{MAX_BS}/     <- trtexec BS=MAX_BS log
    ds/            <- DS benchmark logs
  reports/         <- benchmark_report.md, .html, .pdf, benchmark_data.json
    charts/        <- chart_*.png (5 charts)
  samples/         <- output .mp4 or .ogv (theoraenc fallback), test frames
    kitti_output/  <- KITTI detection .txt files
mkdir -p models/$MODEL_NAME/{model,parser,config,scripts,benchmarks/engines,benchmarks/ds,reports/charts,samples/kitti_output}

Critical Rules

  1. Engine naming — always {model}_dynamic_b{MAX_BS}.engine. Never bare model_dynamic.engine.
  2. batch_size == num_streams — in DS runs, batch-size and stream count are always equal.
  3. Log filenames are fixed — trtexec_b1.log, trtexec_b${MAX_BS}.log, ds_s${N}_run1.log, ds_s${N}_run2.log. No timestamps. Report generation reads exact paths.
  4. Parser zero-init — always NvDsInferObjectDetectionInfo obj = {};. Required for DS 9.1 OBB support; bare obj; leaves rotation_angle uninitialized, causing tilted bounding boxes.
  5. KITTI validation gate — do NOT proceed to Step 7 if KITTI frame count is zero or detection rate < 90%.
  6. Shared venv — build/.venv_optimum reused across all models. Never create per-model venvs.
  7. trtexec --noDataTransfers — GPU-only compute matches DeepStream's GPU-to-GPU data flow.
  8. Report HTML+PDF — always use .claude/skills/deepstream-import-vision-model/scripts/report/md-to-html-pdf.py. Never write a custom HTML generator or call wkhtmltopdf directly.
  9. Object detection only — reject non-detection architectures from config.json before building anything.
  10. Encoder fallback (MANDATORY) — x264enc and openh264enc are prohibited. On NVENC-unavailable systems, use theoraenc + oggmux (LGPL; ships in gst-plugins-base; output is .ogv). If theoraenc/oggmux are absent, skip video creation (DS_SINGLE_STREAM_MODE=skipped). Report which mode was used: nvv4l2h264enc / theoraenc-fallback / skipped.
  11. Video source (MANDATORY) — default is always sample_720p.mp4 (1280×720). Never autonomously substitute sample_1080p_h264.mp4 or any other file. Only use a different video when the user explicitly provides a path (via DS_VIDEO env var or script argument).

Examples

Default model, end to end. Bootstrap once, then run the full pipeline:

docker run --rm -it --gpus all --shm-size=16g -v "$PWD":/work -w /work \
  --entrypoint bash nvcr.io/nvidia/deepstream:9.1-triton-multiarch \
  .claude/skills/deepstream-import-vision-model/setup.sh
# then: "Use deepstream-import-vision-model to run PekingU/rtdetr_r50vd"

SafeTensors model with no published ONNX. Step 2b exports it first; the wrapper reports which backend produced the graph and fails loudly if the batch dimension was baked in:

bash .claude/skills/deepstream-import-vision-model/scripts/model/safetensors-to-onnx.sh \
  models/$MODEL_NAME/hf_model models/$MODEL_NAME/onnx_export/
#   [export] backend=dynamo
#   [export] dynamo produced a static batch dimension; trying the next backend
#   [export] backend=legacy-torchscript
#   [export] pixel_values shape=['batch', 3, 640, 640]

Pin a Hub revision for a reproducible build — any exporter flag passes straight through:

bash .claude/skills/deepstream-import-vision-model/scripts/model/safetensors-to-onnx.sh \
  PekingU/rtdetr_r50vd models/rtdetr/onnx_export --revision <commit-sha> --opset 18

Pipeline Timing

Wrap every step:

STEP_START=$(date +%s.%N)
# ... step commands ...
STEP_END=$(date +%s.%N)
STEP_DURATION=$(python3 -c "print(round($STEP_END - $STEP_START, 2))")   # bc is not in the container; python3 always is
echo "[Step N] completed in ${STEP_DURATION}s"

Track PIPELINE_START (before Step 1) and PIPELINE_END (after Step 8). Report all durations in the benchmark report.

Report Output (MANDATORY — all 3 formats)

  1. benchmark_report.md — markdown source (12 mandatory sections)
  2. benchmark_report.html — styled HTML (charts base64-inlined, no local file access)
  3. benchmark_report_{model_name}.pdf — via md-to-html-pdf.py; verify charts are embedded by counting data:image/png occurrences in the HTML output: grep -o 'data:image/png' benchmark_report.html | wc -l should equal 5

Run charts and report scripts with the shared venv active: source build/.venv_optimum/bin/activate.

Reference Documents

IMPORTANT: Read the relevant reference before starting each phase. Do NOT generate code from memory.

Document Use When
references/model-acquire.md Steps 1–3: HF/NGC URL parsing, format detection, ONNX download, SafeTensors export, label extraction
references/engine-build.md Steps 4–5: trtexec engine build, benchmarks, PEAK_GPU_STREAMS derivation, iterative scaling
references/pipeline-run.md Steps 6–7: custom bbox parser, nvinfer config, single-stream validation, KITTI dump, multi-stream benchmark
references/report-generation.md Step 8: benchmark_data.json, 5 charts, 12-section markdown report, HTML + PDF

Scripts

Installed into .claude/skills/deepstream-import-vision-model/scripts/ by install.sh.

Script Phase Purpose
model/hf-list-files.sh 1–3 List HuggingFace repo files
model/hf-download-config.sh 1–3 Download config.json from HF
model/ngc-list-files.sh 1–3 List NGC model files
model/ngc-download.sh 1–3 Download NGC model archive
model/safetensors-to-onnx.sh 1–3 Export SafeTensors → ONNX via torch.onnx.export (wrapper)
model/safetensors_to_onnx.py 1–3 The exporter — dynamo backend, TorchScript fallback, verifies dynamic batch
model/inspect-onnx.py 1–5 Inspect ONNX input/output shapes
model/make-static-batch-onnx.py 4–5 Bake batch dim into ONNX
model/cleanup.sh Any Remove staging dirs, preserve shared venv
engine/benchmark-trtexec.sh 4–5 Run trtexec with standard flags
deepstream/ds-single-stream.sh 6–7 Single-stream visual validation (NVENC primary; theoraenc+oggmux fallback; skip if neither)
deepstream/ds-sweep.sh 6–7 2-phase batch size sweep
deepstream/benchmark-ds.sh 6–7 Fixed-stream DS benchmark
deepstream/ds-kitti-dump.sh 6–7 KITTI detection dump via deepstream-app
deepstream/ds-perf-run.sh 7 Step 7c two-run benchmark — wraps deepstream-app with enable-perf-measurement=1, writes fixed-name log for the report parser
deepstream/extract-frame.sh 6–7 Extract sample frames from output video (.mp4 NVENC path or .ogv theoraenc fallback)
report/generate-benchmark-charts.py 8 Generate 5 benchmark PNG charts
report/md-to-html-pdf.py 8 Markdown → styled HTML → PDF (canonical benchmark report path)
report/md-to-pdf.sh Any Markdown → PDF via pandoc/pdflatex — for design docs and references only, NOT for benchmark reports (use md-to-html-pdf.py for those)
report/report-style.css 8 CSS for HTML report
report/render-mermaid-for-pdf.py 8 Mermaid diagram → PNG
report/mermaid-puppeteer.json 8 Vetted Puppeteer config for Mermaid (sandboxed; non-root)
report/mermaid-puppeteer-root.json 8 Vetted Puppeteer config for Mermaid (used when running as root)

Quick Error Reference

Error Fix
Tilted/diagonal bounding boxes Parser struct not zero-initialized — use NvDsInferObjectDetectionInfo obj = {};
Zero KITTI files gie-kitti-output-dir not read by nvinfer — use ds-kitti-dump.sh (wraps deepstream-app)
Engine rebuilds every DS run model-engine-file path wrong — check relative path from config/ dir
setDimensions negative dims Add infer-dims=3;H;W to nvinfer config for dynamic ONNX models
--memPoolSize workspace 0.03 MiB Use M suffix not MiB — e.g. --memPoolSize=workspace:32768M
ForeignNode build failure (DETR) Run onnxsim — see references/engine-build.md. Not reproduced on TRT 10.16 with either export backend
ONNX has a static batch dim Both export backends specialized it — see the gotchas in references/model-acquire.md
Zero detections Wrong net-scale-factor — check model family table in references/pipeline-run.md
No module named 'pyservicemaker' Install into venv: pip install /opt/nvidia/deepstream/.../pyservicemaker*.whl
Files (skills)
  • agents
    • openai.yaml 384 B
      Loading…
  • evals
    • evals.json 24.1 KB
      Loading…
  • references
    • engine-build.md 16.6 KB
      Loading…
    • model-acquire.md 24.7 KB
      Loading…
    • pipeline-run.md 27.2 KB
      Loading…
    • README.md 774 B
      Loading…
    • report-generation.md 26.8 KB
      Loading…
    • windows.md 3.3 KB
      Loading…
  • scripts
    • deepstream
      • benchmark-ds.sh 3.6 KB
        Loading…
      • ds-kitti-dump.sh 4.5 KB
        Loading…
      • ds-perf-run.sh 4.8 KB
        Loading…
      • ds-single-stream.sh 4.9 KB
        Loading…
      • ds-sweep.sh 12.3 KB
        Loading…
      • extract-frame.sh 1.9 KB
        Loading…
    • engine
      • benchmark-trtexec.sh 2.8 KB
        Loading…
    • model
      • cleanup.sh 2.7 KB
        Loading…
      • config-to-labels.py 4.1 KB
        Loading…
      • hf-download-config.sh 2.2 KB
        Loading…
      • hf-list-files.sh 4.4 KB
        Loading…
      • inspect-onnx.py 3.7 KB
        Loading…
      • make-static-batch-onnx.py 2.7 KB
        Loading…
      • ngc-download.sh 3.8 KB
        Loading…
      • ngc-list-files.sh 2.7 KB
        Loading…
      • resolve-engine.sh 1.8 KB
        Loading…
      • safetensors-to-onnx.sh 3.6 KB
        Loading…
      • safetensors_to_onnx.py 10 KB
        Loading…
    • report
      • generate-benchmark-charts.py 10.8 KB
        Loading…
      • latex-pdf-wrap.tex 1.6 KB · in bundle
      • md-to-html-pdf.py 7.1 KB
        Loading…
      • md-to-pdf.sh 2.2 KB
        Loading…
      • mermaid-puppeteer-root.json 86 B
        Loading…
      • mermaid-puppeteer.json 42 B
        Loading…
      • pandoc-wrap-tables.lua 2.2 KB · in bundle
      • render-mermaid-for-pdf.py 7.4 KB
        Loading…
      • report-style.css 1.9 KB · in bundle
    • dsrun.sh 1.4 KB
      Loading…
    • preflight.sh 4.2 KB
      Loading…
    • requirements.txt 2.6 KB
      Loading…
  • tests
    • README.md 3 KB
      Loading…
    • test_evals_contract.py 10.5 KB
      Loading…
    • test_hardened_scripts.py 13.3 KB
      Loading…
    • test_onnx_export_contract.py 8.9 KB
      Loading…
    • test_powershell_installer.py 9.2 KB
      Loading…
    • test_skill_model_intake.py 2 KB
      Loading…
    • __init__.py 137 B
      Loading…
  • .gitattributes 640 B · in bundle
  • .gitignore 46 B · in bundle
  • BENCHMARK.md 4.9 KB
    Loading…
  • CHANGELOG.md 18.1 KB
    Loading…
  • install.ps1 5.4 KB · in bundle
  • install.sh 6.3 KB
    Loading…
  • README.md 4 KB
    # DeepStream Import Vision Model
    
    Automated end-to-end pipeline: HuggingFace model → TensorRT engine → DeepStream
    multi-stream benchmark → PDF report.
    
    ## Overview
    
    This self-contained skill uses four phase-specific reference documents. Together they automate the full model bringup workflow for NVIDIA DeepStream, from
    downloading a model on HuggingFace or NVIDIA NGC to a publication-ready benchmark report.
    
    Supported input formats: ONNX (direct), SafeTensors (auto-exported via `torch.onnx.export`).
    
    **Current scope:** object detection models only. Classification, segmentation, pose estimation, and other vision tasks are not yet supported — the pipeline fails fast if a non-detection architecture is detected in `config.json`.
    
    ## Prerequisites
    
    **Host: only Docker + the NVIDIA driver.** Everything else runs **inside the DeepStream container** —
    DeepStream, TensorRT/`trtexec`, the Python export venv (`torch`/`onnx`/`onnxruntime`), `wkhtmltopdf`,
    `deepstream-app`/`gst-launch-1.0` — and is bootstrapped by `setup.sh`. Nothing is installed on the host.
    Runs identically on Linux and **Windows** (Docker Desktop + WSL2 backend, required for GPU).
    
    - **Docker** — Docker Desktop with the WSL2 backend on Windows; Docker Engine + NVIDIA Container
      Toolkit on Linux.
    - **NVIDIA GPU + driver** (on Windows, the WSL2 GPU driver — no host CUDA/TensorRT/DeepStream needed).
    - `docker pull nvcr.io/nvidia/deepstream:9.1-triton-multiarch`, then run `setup.sh` through the container.
    
    See **[references/windows.md](references/windows.md)** for the cross-platform run model and the
    per-shell `docker run` mount token.
    
    ## Installation
    
    ```bash
    bash <path-to-deepstream-import-vision-model>/install.sh --target <your-project-path>
    ```
    
    Preview what will be installed first with `--dry-run`:
    
    ```bash
    bash <path-to-deepstream-import-vision-model>/install.sh --target <your-project-path> --dry-run
    ```
    
    Where `<path-to-deepstream-import-vision-model>` is the location of this skill in your repo, e.g.:
    - In **team-mind-hub**: `team-skills/deepstream-sdk/deepstream-import-vision-model`
    - In **ds-copilot**: `team-skills/deepstream-sdk/deepstream-import-vision-model` (same path)
    
    The script copies the complete skill into the target project for Claude Code, Codex, and Cursor. Re-running it safely refreshes an existing installation.
    
    ## Usage
    
    **Claude Code:**
    ```text
    Use deepstream-import-vision-model to run this model: https://huggingface.co/onnx-community/yolov8n
    ```
    
    **Codex:**
    ```text
    Use $deepstream-import-vision-model to deploy and benchmark https://huggingface.co/onnx-community/yolov8n
    ```
    
    **Cursor:**
    ```text
    @deepstream-import-vision-model run this model: https://huggingface.co/onnx-community/yolov8n
    ```
    
    The skill runs the full pipeline autonomously — no manual steps required.
    
    ## Pipeline Steps
    
    | Step | Phase reference | Action |
    |------|-----------|--------|
    | 1–3 | `references/model-acquire.md` | Browse HF repo, download ONNX or export SafeTensors |
    | 4–5 | `references/engine-build.md` | Build dynamic TRT engine, run trtexec benchmarks |
    | 6–7 | `references/pipeline-run.md` | Custom bbox parser, DeepStream single + multi-stream |
    | 8 | `references/report-generation.md` | 5 charts, HTML report, PDF |
    
    ## Output Structure
    
    Per-model outputs are written to `models/<model_name>/` in your project:
    
    ```text
    models/<model_name>/
      model/          ONNX file(s)
      parser/         Custom nvinfer bbox parser (.cpp, .so)
      config/         nvinfer config, DS app config, labels.txt
      scripts/        Model-specific run helpers
      benchmarks/     TRT engines, trtexec logs
      reports/        benchmark_report.md / .html / .pdf + charts/
      samples/        Output videos, test frames, KITTI detections
    ```
    
    ## Files in this package
    
    ```text
    deepstream-import-vision-model/
    ├── SKILL.md                    Top-level skill definition
    ├── README.md                   This file
    ├── references/                 Phase-specific runbooks
    ├── scripts/                    Utility scripts by pipeline phase
    └── tests/                      Installer and script regression tests
    ```
    
  • setup.sh 3.6 KB
    Loading…
  • skill-card.md 4.5 KB
    Loading…
  • SKILL.md 14.2 KB
    ---
    name: deepstream-import-vision-model
    description: >
      Use this skill to bring a supported object-detection vision model from HuggingFace or NVIDIA NGC into
      an NVIDIA DeepStream pipeline with end-to-end automation: ONNX download,
      SafeTensors export, TRT engine build, custom nvinfer bbox parser, multi-stream
      benchmark, and PDF report. Object detection models only.
    license: CC-BY-4.0 AND Apache-2.0
    metadata:
      author: "Tushar Khinvasara <tkhinvasara@nvidia.com>"
      owner: "Tushar Khinvasara <tkhinvasara@nvidia.com>"
      service: "deepstream"
      version: "1.5.2"
      reviewed: "2026-08-04"
      team: deepstream-sdk
      tags:
        - deepstream
        - tensorrt
        - object-detection
        - import-vision-model
      languages:
        - bash
        - python
        - cpp
      domain: computer-vision
    ---
    
    # DeepStream Import Vision Model
    
    When this skill is active, **read the relevant reference document before starting each phase**. Do not rely on memory — reference documents contain exact script paths, bash variable conventions, log filename contracts, and critical parsing rules.
    
    **Current scope:** Object detection models only. Fail fast on classification, segmentation, or other architectures detected in `config.json`.
    
    ## Model choice — always offer two options
    
    Before preflight, browsing, downloads, or file creation, present exactly these two choices. Do not
    start with only an open-ended model-source prompt. If the user's request already clearly selects a
    model, confirm the matching choice instead of asking redundantly.
    
    ### 1. Default model (recommended)
    
    Use the validated Hugging Face RT-DETR model:
    
    ```yaml
    model_id: PekingU/rtdetr_r50vd
    source: huggingface
    task: object-detection
    precision_preference: fp16
    ```
    
    ### 2. Custom object-detection model
    
    Ask for one supported source:
    
    - Hugging Face model ID (`organization/model`) or full model URL.
    - NVIDIA NGC catalog model URL including its version.
    
    Explain that the skill currently rejects classification, segmentation, and other non-detection
    architectures after inspecting `config.json`. Do not invent or silently substitute a model when the
    custom source is missing or unsupported.
    
    For a dry run, present the same two choices and simulate discovery, build, benchmark, and report
    stages without browsing, downloading, launching Docker, writing files, or starting processes.
    
    ## Pipeline Overview
    
    | Step | Phase | Reference | What it does |
    |------|-------|-----------|--------------|
    | 1–3 | Model Acquire | [references/model-acquire.md](references/model-acquire.md) | Browse HF/NGC, detect format, download ONNX or export SafeTensors |
    | 4–5 | Engine Build  | [references/engine-build.md](references/engine-build.md) | Build dynamic TRT engine, run trtexec BS=1 and BS=MAX_BS |
    | 6–7 | DS Pipeline   | [references/pipeline-run.md](references/pipeline-run.md) | Custom bbox parser, nvinfer config, single-stream + multi-stream benchmarks |
    | 8   | Report        | [references/report-generation.md](references/report-generation.md) | 5 charts, HTML, PDF benchmark report |
    
    Run the full pipeline autonomously without pausing for confirmation at each step.
    
    ## Runs entirely through Docker (no host packages)
    
    **Every step runs INSIDE the DeepStream container.** The host needs only **Docker + the NVIDIA
    driver** — no host python/venv/torch/trtexec/make/wkhtmltopdf. This works identically on Linux and
    **Windows** (Docker Desktop + WSL2 backend, required for `--gpus`). The per-shell bind-mount
    token is the only OS difference — `-v "$PWD":/work` (bash), `-v "${PWD}:/work"` (PowerShell),
    `-v "%cd%:/work"` (cmd); full guide in [references/windows.md](references/windows.md). All venv/ONNX/
    engine/parser/config/report artifacts live under the mounted working root and persist between the
    ephemeral `--rm` containers.
    
    ## Pre-flight — bootstrap + verify (through the container)
    
    **1. One-time bootstrap** — builds `build/.venv_optimum` (torch/onnx/onnxruntime/report deps; the
    venv name is historical, optimum is no longer used) +
    installs `wkhtmltopdf`, all in-container. From the working root:
    ```bash
    docker run --rm -it --gpus all --shm-size=16g -v "$PWD":/work -w /work \
      --entrypoint bash nvcr.io/nvidia/deepstream:9.1-triton-multiarch \
      .claude/skills/deepstream-import-vision-model/setup.sh
    ```
    
    **2. Preflight** — GPU + venv + trtexec, run THROUGH the container (container-mode auto-detects):
    ```bash
    docker run --rm --gpus all -v "$PWD":/work -w /work \
      --entrypoint bash nvcr.io/nvidia/deepstream:9.1-triton-multiarch \
      .claude/skills/deepstream-import-vision-model/scripts/preflight.sh   # proceed only on PASS
    ```
    
    **Every subsequent phase runs the same way** — issue the model's commands via
    `docker run … --entrypoint bash … -lc '<commands>'` (or the
    `.claude/skills/deepstream-import-vision-model/scripts/dsrun.sh` wrapper:
    `bash .claude/skills/deepstream-import-vision-model/scripts/dsrun.sh '<in-container command>'`),
    using `PY=build/.venv_optimum/bin/python` and
    `trtexec` at `/usr/src/tensorrt/bin/trtexec` inside the container. `deepstream-app`,
    `gst-launch-1.0`, and `/opt/nvidia/deepstream/…` sample paths all exist **in** the image.
    TensorRT build+runtime share one image, so there is **no version skew** (the concern the old
    "build on the host" rule tried to avoid — see [references/engine-build.md](references/engine-build.md)).
    `sample_720p.mp4` ships in the image; set `DS_VIDEO` only to override.
    
    ## Mandatory Output Structure
    
    Create once `MODEL_NAME` is known (Step 1). Never dump files flat.
    
    ```
    models/{model_name}/
      model/           <- ONNX file(s)
      parser/          <- .cpp, Makefile, .so
      config/          <- nvinfer config, ds-app config, labels.txt
      scripts/         <- run helper scripts
      benchmarks/
        engines/       <- _dynamic_b{MAX_BS}.engine, timing.cache, build logs
        b1/            <- trtexec BS=1 log
        b{MAX_BS}/     <- trtexec BS=MAX_BS log
        ds/            <- DS benchmark logs
      reports/         <- benchmark_report.md, .html, .pdf, benchmark_data.json
        charts/        <- chart_*.png (5 charts)
      samples/         <- output .mp4 or .ogv (theoraenc fallback), test frames
        kitti_output/  <- KITTI detection .txt files
    ```
    
    ```bash
    mkdir -p models/$MODEL_NAME/{model,parser,config,scripts,benchmarks/engines,benchmarks/ds,reports/charts,samples/kitti_output}
    ```
    
    ## Critical Rules
    
    1. **Engine naming** — always `{model}_dynamic_b{MAX_BS}.engine`. Never bare `model_dynamic.engine`.
    2. **batch_size == num_streams** — in DS runs, `batch-size` and stream count are always equal.
    3. **Log filenames are fixed** — `trtexec_b1.log`, `trtexec_b${MAX_BS}.log`, `ds_s${N}_run1.log`, `ds_s${N}_run2.log`. No timestamps. Report generation reads exact paths.
    4. **Parser zero-init** — always `NvDsInferObjectDetectionInfo obj = {};`. Required for DS 9.1 OBB support; bare `obj;` leaves `rotation_angle` uninitialized, causing tilted bounding boxes.
    5. **KITTI validation gate** — do NOT proceed to Step 7 if KITTI frame count is zero or detection rate < 90%.
    6. **Shared venv** — `build/.venv_optimum` reused across all models. Never create per-model venvs.
    7. **trtexec `--noDataTransfers`** — GPU-only compute matches DeepStream's GPU-to-GPU data flow.
    8. **Report HTML+PDF** — always use `.claude/skills/deepstream-import-vision-model/scripts/report/md-to-html-pdf.py`. Never write a custom HTML generator or call `wkhtmltopdf` directly.
    9. **Object detection only** — reject non-detection architectures from `config.json` before building anything.
    10. **Encoder fallback (MANDATORY)** — `x264enc` and `openh264enc` are **prohibited**. On NVENC-unavailable systems, use `theoraenc + oggmux` (LGPL; ships in gst-plugins-base; output is `.ogv`). If `theoraenc`/`oggmux` are absent, skip video creation (`DS_SINGLE_STREAM_MODE=skipped`). Report which mode was used: `nvv4l2h264enc` / `theoraenc-fallback` / `skipped`.
    11. **Video source (MANDATORY)** — default is always `sample_720p.mp4` (1280×720). Never autonomously substitute `sample_1080p_h264.mp4` or any other file. Only use a different video when the user explicitly provides a path (via `DS_VIDEO` env var or script argument).
    
    ## Examples
    
    **Default model, end to end.** Bootstrap once, then run the full pipeline:
    
    ```bash
    docker run --rm -it --gpus all --shm-size=16g -v "$PWD":/work -w /work \
      --entrypoint bash nvcr.io/nvidia/deepstream:9.1-triton-multiarch \
      .claude/skills/deepstream-import-vision-model/setup.sh
    # then: "Use deepstream-import-vision-model to run PekingU/rtdetr_r50vd"
    ```
    
    **SafeTensors model with no published ONNX.** Step 2b exports it first; the wrapper reports which
    backend produced the graph and fails loudly if the batch dimension was baked in:
    
    ```bash
    bash .claude/skills/deepstream-import-vision-model/scripts/model/safetensors-to-onnx.sh \
      models/$MODEL_NAME/hf_model models/$MODEL_NAME/onnx_export/
    #   [export] backend=dynamo
    #   [export] dynamo produced a static batch dimension; trying the next backend
    #   [export] backend=legacy-torchscript
    #   [export] pixel_values shape=['batch', 3, 640, 640]
    ```
    
    **Pin a Hub revision** for a reproducible build — any exporter flag passes straight through:
    
    ```bash
    bash .claude/skills/deepstream-import-vision-model/scripts/model/safetensors-to-onnx.sh \
      PekingU/rtdetr_r50vd models/rtdetr/onnx_export --revision <commit-sha> --opset 18
    ```
    
    ## Pipeline Timing
    
    Wrap every step:
    
    ```bash
    STEP_START=$(date +%s.%N)
    # ... step commands ...
    STEP_END=$(date +%s.%N)
    STEP_DURATION=$(python3 -c "print(round($STEP_END - $STEP_START, 2))")   # bc is not in the container; python3 always is
    echo "[Step N] completed in ${STEP_DURATION}s"
    ```
    
    Track `PIPELINE_START` (before Step 1) and `PIPELINE_END` (after Step 8). Report all durations in the benchmark report.
    
    ## Report Output (MANDATORY — all 3 formats)
    
    1. `benchmark_report.md` — markdown source (12 mandatory sections)
    2. `benchmark_report.html` — styled HTML (charts base64-inlined, no local file access)
    3. `benchmark_report_{model_name}.pdf` — via `md-to-html-pdf.py`; verify charts are embedded by counting `data:image/png` occurrences in the HTML output: `grep -o 'data:image/png' benchmark_report.html | wc -l` should equal 5
    
    Run charts and report scripts with the shared venv active: `source build/.venv_optimum/bin/activate`.
    
    ## Reference Documents
    
    **IMPORTANT**: Read the relevant reference before starting each phase. Do NOT generate code from memory.
    
    | Document | Use When |
    |----------|----------|
    | [references/model-acquire.md](references/model-acquire.md) | Steps 1–3: HF/NGC URL parsing, format detection, ONNX download, SafeTensors export, label extraction |
    | [references/engine-build.md](references/engine-build.md) | Steps 4–5: trtexec engine build, benchmarks, PEAK_GPU_STREAMS derivation, iterative scaling |
    | [references/pipeline-run.md](references/pipeline-run.md) | Steps 6–7: custom bbox parser, nvinfer config, single-stream validation, KITTI dump, multi-stream benchmark |
    | [references/report-generation.md](references/report-generation.md) | Step 8: benchmark_data.json, 5 charts, 12-section markdown report, HTML + PDF |
    
    ## Scripts
    
    Installed into `.claude/skills/deepstream-import-vision-model/scripts/` by `install.sh`.
    
    | Script | Phase | Purpose |
    |--------|-------|---------|
    | `model/hf-list-files.sh` | 1–3 | List HuggingFace repo files |
    | `model/hf-download-config.sh` | 1–3 | Download config.json from HF |
    | `model/ngc-list-files.sh` | 1–3 | List NGC model files |
    | `model/ngc-download.sh` | 1–3 | Download NGC model archive |
    | `model/safetensors-to-onnx.sh` | 1–3 | Export SafeTensors → ONNX via `torch.onnx.export` (wrapper) |
    | `model/safetensors_to_onnx.py` | 1–3 | The exporter — dynamo backend, TorchScript fallback, verifies dynamic batch |
    | `model/inspect-onnx.py` | 1–5 | Inspect ONNX input/output shapes |
    | `model/make-static-batch-onnx.py` | 4–5 | Bake batch dim into ONNX |
    | `model/cleanup.sh` | Any | Remove staging dirs, preserve shared venv |
    | `engine/benchmark-trtexec.sh` | 4–5 | Run trtexec with standard flags |
    | `deepstream/ds-single-stream.sh` | 6–7 | Single-stream visual validation (NVENC primary; theoraenc+oggmux fallback; skip if neither) |
    | `deepstream/ds-sweep.sh` | 6–7 | 2-phase batch size sweep |
    | `deepstream/benchmark-ds.sh` | 6–7 | Fixed-stream DS benchmark |
    | `deepstream/ds-kitti-dump.sh` | 6–7 | KITTI detection dump via deepstream-app |
    | `deepstream/ds-perf-run.sh` | 7 | Step 7c two-run benchmark — wraps `deepstream-app` with `enable-perf-measurement=1`, writes fixed-name log for the report parser |
    | `deepstream/extract-frame.sh` | 6–7 | Extract sample frames from output video (`.mp4` NVENC path or `.ogv` theoraenc fallback) |
    | `report/generate-benchmark-charts.py` | 8 | Generate 5 benchmark PNG charts |
    | `report/md-to-html-pdf.py` | 8 | Markdown → styled HTML → PDF (canonical benchmark report path) |
    | `report/md-to-pdf.sh` | Any | Markdown → PDF via pandoc/pdflatex — for design docs and references only, NOT for benchmark reports (use md-to-html-pdf.py for those) |
    | `report/report-style.css` | 8 | CSS for HTML report |
    | `report/render-mermaid-for-pdf.py` | 8 | Mermaid diagram → PNG |
    | `report/mermaid-puppeteer.json` | 8 | Vetted Puppeteer config for Mermaid (sandboxed; non-root) |
    | `report/mermaid-puppeteer-root.json` | 8 | Vetted Puppeteer config for Mermaid (used when running as root) |
    
    ## Quick Error Reference
    
    | Error | Fix |
    |-------|-----|
    | Tilted/diagonal bounding boxes | Parser struct not zero-initialized — use `NvDsInferObjectDetectionInfo obj = {};` |
    | Zero KITTI files | `gie-kitti-output-dir` not read by nvinfer — use `ds-kitti-dump.sh` (wraps `deepstream-app`) |
    | Engine rebuilds every DS run | `model-engine-file` path wrong — check relative path from `config/` dir |
    | `setDimensions` negative dims | Add `infer-dims=3;H;W` to nvinfer config for dynamic ONNX models |
    | `--memPoolSize` workspace 0.03 MiB | Use `M` suffix not `MiB` — e.g. `--memPoolSize=workspace:32768M` |
    | ForeignNode build failure (DETR) | Run `onnxsim` — see references/engine-build.md. Not reproduced on TRT 10.16 with either export backend |
    | ONNX has a static batch dim | Both export backends specialized it — see the gotchas in references/model-acquire.md |
    | Zero detections | Wrong `net-scale-factor` — check model family table in references/pipeline-run.md |
    | `No module named 'pyservicemaker'` | Install into venv: `pip install /opt/nvidia/deepstream/.../pyservicemaker*.whl` |
    
  • skill.oms.sig 16.1 KB · in bundle

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.