{
  "skill_name": "deepstream-dev",
  "evals": [
    {
      "id": "minimal-local-file-inference-pipeline",
      "name": "minimal-local-file-inference-pipeline",
      "prompt": "Using DeepStream SDK and the pyservicemaker Python API, generate a pipeline that reads a local video file, runs primary inference with nvinfer using the ResNet18 TrafficCamNet detector shipped with DeepStream, draws bounding boxes with nvosdbin, and renders to the screen. The user did not ask for tracking or Kafka.",
      "expected_output": "A minimal pipeline using nvurisrcbin, nvstreammux, nvinfer, nvosdbin, and a platform-appropriate sink. It must avoid nvtracker, secondary GIEs, nvmsgbroker, and other optional components that were not requested.",
      "files": [],
      "assertions": [
        {
          "text": "Uses nvurisrcbin as the source for a local video file",
          "type": "contains_pattern",
          "pattern": "nvurisrcbin"
        },
        {
          "text": "Batches streams through nvstreammux",
          "type": "contains_pattern",
          "pattern": "nvstreammux"
        },
        {
          "text": "Uses the sink_%u request-pad template when linking sources into nvstreammux",
          "type": "contains_pattern",
          "pattern": "sink_%u"
        },
        {
          "text": "References the bundled ResNet18 TrafficCamNet ONNX model path",
          "type": "contains_pattern",
          "pattern": "(resnet18_trafficcamnet|Primary_Detector|trafficcamnet)"
        },
        {
          "text": "Does not add nvtracker because tracking was not requested",
          "type": "not_contains_pattern",
          "pattern": "nvtracker"
        },
        {
          "text": "Does not add nvmsgbroker or Kafka messaging because messaging was not requested",
          "type": "not_contains_pattern",
          "pattern": "(nvmsgbroker|nvmsgconv|kafka)"
        },
        {
          "text": "Response does not contain API tokens or credentials",
          "type": "not_contains_pattern",
          "pattern": "(Bearer |sk-|token=)[A-Za-z0-9+/=]{10,}"
        }
      ]
    },
    {
      "id": "rtsp-tracking-kafka-tee-pipeline",
      "name": "rtsp-tracking-kafka-tee-pipeline",
      "prompt": "Build a DeepStream pyservicemaker pipeline that ingests two RTSP cameras, runs primary detection, tracks objects across frames, displays the result in a tiled view, and publishes detection metadata to a Kafka broker. Cover the live-source and tee-split requirements.",
      "expected_output": "The pipeline uses nvurisrcbin for each RTSP source, sets live-source=1 on nvstreammux, includes nvtracker because tracking was requested, splits display and broker output with tee, sends metadata to nvmsgbroker, and sets async=0 on sinks.",
      "files": [],
      "assertions": [
        {
          "text": "Configures nvstreammux with live-source=1 for RTSP input",
          "type": "contains_pattern",
          "pattern": "live-source\\s*[:=]\\s*1"
        },
        {
          "text": "Includes nvtracker because the user explicitly requested tracking",
          "type": "contains_pattern",
          "pattern": "nvtracker"
        },
        {
          "text": "Uses tee to feed both display and broker branches",
          "type": "contains_pattern",
          "pattern": "tee"
        },
        {
          "text": "Uses nvmsgbroker for Kafka publishing",
          "type": "contains_pattern",
          "pattern": "(nvmsgbroker|kafka)"
        },
        {
          "text": "Sets async=0 on sinks in the tee branches to avoid state-transition deadlocks",
          "type": "contains_pattern",
          "pattern": "async\\s*[:=]\\s*0"
        },
        {
          "text": "Uses sync=0 on the live renderer path",
          "type": "contains_pattern",
          "pattern": "sync\\s*[:=]\\s*0"
        },
        {
          "text": "Response does not contain API tokens or credentials",
          "type": "not_contains_pattern",
          "pattern": "(Bearer |sk-|token=)[A-Za-z0-9+/=]{10,}"
        }
      ]
    },
    {
      "id": "yolov11-nvinfer-dynamic-onnx-config",
      "name": "yolov11-nvinfer-dynamic-onnx-config",
      "prompt": "Generate an nvinfer YAML config for a YOLOv11 model with 640x640 input exported from Ultralytics with dynamic=True. The model outputs a raw pre-NMS tensor of shape [batch, 84, 8400].",
      "expected_output": "The nvinfer YAML uses a property section, sets infer-dims=3;640;640 so TensorRT does not see dynamic -1 dimensions, and uses cluster-mode: 2 for DeepStream NMS because the output tensor is pre-NMS.",
      "files": [],
      "assertions": [
        {
          "text": "Uses the property section for the nvinfer YAML",
          "type": "contains_pattern",
          "pattern": "property\\s*:"
        },
        {
          "text": "Sets infer-dims to 3;640;640 for the dynamic ONNX input shape",
          "type": "contains_pattern",
          "pattern": "infer-dims.*3;640;640"
        },
        {
          "text": "Uses cluster-mode: 2 because YOLOv11 output is pre-NMS",
          "type": "contains_pattern",
          "pattern": "cluster-mode.*2"
        },
        {
          "text": "Does not set is-classifier for an object detector",
          "type": "not_contains_pattern",
          "pattern": "is-classifier"
        },
        {
          "text": "Response does not contain API tokens or credentials",
          "type": "not_contains_pattern",
          "pattern": "(Bearer |sk-|token=)[A-Za-z0-9+/=]{10,}"
        }
      ]
    },
    {
      "id": "minimal-playback-inference-pipeline",
      "name": "minimal-playback-inference-pipeline",
      "prompt": "Write a DeepStream pipeline that just plays a video file through inference and shows it on screen. Keep it as minimal as possible.",
      "expected_output": "A minimal video inference pipeline with nvurisrcbin, nvstreammux, nvinfer, nvosdbin, and a renderer. It should not add tracking, analytics, secondary classifiers, metadata brokers, or other optional elements that the user did not request.",
      "files": [],
      "assertions": [
        {
          "text": "Includes nvinfer for the requested inference stage",
          "type": "contains_pattern",
          "pattern": "nvinfer"
        },
        {
          "text": "Does not add nvtracker when tracking was not requested",
          "type": "not_contains_pattern",
          "pattern": "nvtracker"
        },
        {
          "text": "Does not add nvdsanalytics when line crossing, ROI, or analytics were not requested",
          "type": "not_contains_pattern",
          "pattern": "nvdsanalytics"
        },
        {
          "text": "Does not add a secondary GIE when secondary classification was not requested",
          "type": "not_contains_pattern",
          "pattern": "(secondary|sgie|process-mode:\\s*2)"
        },
        {
          "text": "Does not add nvmsgbroker or nvmsgconv when messaging was not requested",
          "type": "not_contains_pattern",
          "pattern": "(nvmsgbroker|nvmsgconv)"
        },
        {
          "text": "Response does not contain API tokens or credentials",
          "type": "not_contains_pattern",
          "pattern": "(Bearer |sk-|token=)[A-Za-z0-9+/=]{10,}"
        }
      ]
    },
    {
      "id": "probe-iterator-and-venv-pyservicemaker-fix",
      "name": "probe-iterator-and-venv-pyservicemaker-fix",
      "prompt": "My pyservicemaker probe runs len(frame.object_items) to count detections and I am installing my app inside a fresh python3 -m venv. It fails with ModuleNotFoundError: pyservicemaker and the probe raises 'iterator has no len()'. Fix both.",
      "expected_output": "Explain that frame.object_items and frame.frame_items are iterators, so detection counts must be computed by iterating. Also explain that a fresh venv must install the bundled pyservicemaker wheel and pyyaml from the DeepStream service-maker Python directory.",
      "files": [],
      "assertions": [
        {
          "text": "States that object_items and frame_items are iterators and cannot be counted with len()",
          "type": "contains_pattern",
          "pattern": "(iterator|object_items|frame_items)"
        },
        {
          "text": "Shows or describes counting by iterating over object_items",
          "type": "contains_pattern",
          "pattern": "(for .* in .*object_items|iterate)"
        },
        {
          "text": "Tells the user to install the bundled pyservicemaker wheel inside the venv",
          "type": "contains_pattern",
          "pattern": "(pyservicemaker.*\\.whl|pip install.*pyservicemaker)"
        },
        {
          "text": "References the DeepStream service-maker Python wheel directory",
          "type": "contains_pattern",
          "pattern": "/opt/nvidia/deepstream/deepstream/service-maker/python"
        },
        {
          "text": "Also installs pyyaml in the venv so YAML nvinfer configs can load",
          "type": "contains_pattern",
          "pattern": "pyyaml"
        },
        {
          "text": "Response does not contain API tokens or credentials",
          "type": "not_contains_pattern",
          "pattern": "(Bearer |sk-|token=)[A-Za-z0-9+/=]{10,}"
        }
      ]
    },
    {
      "id": "negative-pytorch-coreml-out-of-scope",
      "name": "negative-pytorch-coreml-out-of-scope",
      "prompt": "Train a custom image classifier from scratch in PyTorch and export it to CoreML for iOS. I do not need any DeepStream pipeline setup.",
      "expected_output": "The deepstream-dev skill should not be selected for this request because it is outside DeepStream pipeline and SDK usage scope. Avoid DeepStream-specific pipeline guidance and plugin recommendations.",
      "files": [],
      "assertions": [
        {
          "text": "Responds with PyTorch or CoreML guidance rather than a DeepStream pipeline",
          "type": "contains_pattern",
          "pattern": "(PyTorch|CoreML|coreml|torch)"
        },
        {
          "text": "Does not recommend building a DeepStream GStreamer pipeline for this request",
          "type": "not_contains_pattern",
          "pattern": "(nvurisrcbin|nvstreammux|nvinfer|nvosdbin|pyservicemaker.*Pipeline)"
        },
        {
          "text": "Does not add DeepStream plugin recommendations for an out-of-scope task",
          "type": "not_contains_pattern",
          "pattern": "(nvtracker|nvmsgbroker|nvdsanalytics)"
        },
        {
          "text": "Response does not contain API tokens or credentials",
          "type": "not_contains_pattern",
          "pattern": "(Bearer |sk-|token=)[A-Za-z0-9+/=]{10,}"
        }
      ]
    },
    {
      "id": "negative-mysql-out-of-scope",
      "name": "negative-mysql-out-of-scope",
      "prompt": "How do I configure a MySQL replication slave on Ubuntu 22.04?",
      "expected_output": "The deepstream-dev skill should not be selected because this request is unrelated to DeepStream SDK development or pipeline operations. Suggest a MySQL-focused resource or workflow.",
      "files": [],
      "assertions": [
        {
          "text": "Responds with MySQL replication guidance",
          "type": "contains_pattern",
          "pattern": "(MySQL|mysql|replication)"
        },
        {
          "text": "States or implies the request is outside DeepStream scope",
          "type": "contains_pattern",
          "pattern": "(outside.*DeepStream|not.*DeepStream|unrelated.*DeepStream|MySQL)"
        },
        {
          "text": "Does not recommend DeepStream pipeline or plugin setup for MySQL administration",
          "type": "not_contains_pattern",
          "pattern": "(nvurisrcbin|nvstreammux|nvinfer|nvosdbin|pyservicemaker)"
        },
        {
          "text": "Response does not contain API tokens or credentials",
          "type": "not_contains_pattern",
          "pattern": "(Bearer |sk-|token=)[A-Za-z0-9+/=]{10,}"
        }
      ]
    }
  ]
}

# DeepStream Best Practices and Design Patterns

## Overview

This document provides comprehensive best practices, design patterns, and optimization strategies for building production-grade DeepStream applications. These guidelines help ensure performance, reliability, maintainability, and scalability.

---

## 1. Pipeline Design Patterns

### Pattern 1: Modular Pipeline Construction

**Best Practice**: Build pipelines in modular, reusable functions.

```python
import os

def create_source_pipeline(video_path, num_streams=1):
    """Create reusable source pipeline"""
    sources = []
    for i in range(num_streams):
        uri = "file://" + os.path.abspath(video_path)
        sources.append({
            "element": "nvurisrcbin",
            "name": f"src{i}",
            "props": {"uri": uri}
        })
    return sources

def create_inference_pipeline(config_files):
    """Create reusable inference pipeline"""
    inference_elements = []
    for idx, config in enumerate(config_files):
        unique_id = idx + 1
        inference_elements.append({
            "element": "nvinfer",
            "name": f"infer{idx}",
            "props": {
                "config-file-path": config,
                "unique-id": unique_id
            }
        })
    return inference_elements

def build_complete_pipeline(video_path, infer_configs):
    """Compose complete pipeline from modules"""
    pipeline = Pipeline("modular-pipeline")
    
    # Add source modules
    sources = create_source_pipeline(video_path)
    for src_config in sources:
        pipeline.add(src_config["element"], src_config["name"], src_config.get("props", {}))
    
    # Add inference modules
    infer_elements = create_inference_pipeline(infer_configs)
    for infer_config in infer_elements:
        pipeline.add(infer_config["element"], infer_config["name"], infer_config.get("props", {}))
    
    # Link modules
    # ... linking logic ...
    
    return pipeline
```

### Pattern 2: Configuration-Driven Pipelines

**Best Practice**: Use YAML/JSON configuration files for pipeline definition.

```python
import yaml

def load_pipeline_config(config_path):
    """Load pipeline configuration from YAML"""
    with open(config_path, 'r') as f:
        return yaml.safe_load(f)

def build_pipeline_from_config(config):
    """Build pipeline from configuration"""
    pipeline = Pipeline(config["pipeline"]["name"])
    
    # Add elements from config
    for elem_config in config["pipeline"]["elements"]:
        pipeline.add(
            elem_config["type"],
            elem_config["name"],
            elem_config.get("properties", {})
        )
    
    # Link elements from config
    for link_group in config["pipeline"]["links"]:
        pipeline.link(*link_group)
    
    return pipeline
```

### Pattern 3: Factory Pattern for Element Creation

**Best Practice**: Use factory functions for element creation with validation.

```python
def create_decoder(platform="x86"):
    """Factory function for decoder creation"""
    decoder_props = {}
    
    if platform == "jetson":
        decoder_props["device"] = "/dev/video0"
    
    return {
        "element": "nvv4l2decoder",
        "name": "decoder",
        "props": decoder_props
    }

def create_sink(platform="x86", window_config=None):
    """Factory function for sink creation"""
    sink_type = "nv3dsink" if platform == "jetson" else "nveglglessink"
    sink_props = {"sync": 1}
    
    if window_config:
        sink_props.update(window_config)
    
    return {
        "element": sink_type,
        "name": "sink",
        "props": sink_props
    }
```

### Pattern 4: Strategy Pattern for Processing

**Best Practice**: Use strategy pattern for different processing approaches.

```python
class ProcessingStrategy:
    """Base class for processing strategies"""
    def process(self, batch_meta):
        raise NotImplementedError

class DetectionStrategy(ProcessingStrategy):
    """Strategy for object detection"""
    def process(self, batch_meta):
        # Detection-specific processing
        pass

class ClassificationStrategy(ProcessingStrategy):
    """Strategy for classification"""
    def process(self, batch_meta):
        # Classification-specific processing
        pass

class PipelineBuilder:
    """Pipeline builder with strategy pattern"""
    def __init__(self, strategy: ProcessingStrategy):
        self.strategy = strategy
    
    def build(self):
        pipeline = Pipeline("strategy-pipeline")
        # Build pipeline based on strategy
        return pipeline
```

---

## 2. Performance Optimization

### Optimization 1: Batch Size Tuning

**Best Practice**: Optimize batch sizes based on GPU memory and model complexity.

```python
def calculate_optimal_batch_size(
    num_streams,
    gpu_memory_gb,
    model_complexity="medium",
    resolution=(1920, 1080)
):
    """
    Calculate optimal batch size
    
    Args:
        num_streams: Number of input streams
        gpu_memory_gb: Available GPU memory in GB
        model_complexity: "low", "medium", "high"
        resolution: (width, height) tuple
    """
    # Base memory per stream (GB)
    base_memory = {
        (1920, 1080): 1.0,
        (1280, 720): 0.5,
        (640, 480): 0.25
    }.get(resolution, 1.0)
    
    # Model complexity multiplier
    complexity_mult = {
        "low": 1.0,
        "medium": 1.5,
        "high": 2.0
    }.get(model_complexity, 1.5)
    
    # Calculate max batch size
    memory_per_stream = base_memory * complexity_mult
    max_batch = int(gpu_memory_gb / memory_per_stream)
    
    # Clamp to number of streams and use power of 2
    optimal_batch = min(max_batch, num_streams)
    optimal_batch = 2 ** (optimal_batch.bit_length() - 1)  # Round down to power of 2
    
    return max(1, optimal_batch)
```

### Optimization 2: Inference Precision Selection

**Best Practice**: Use appropriate precision based on accuracy requirements.

```python
def get_inference_config(precision="fp16", model_path=None):
    """
    Get inference configuration with optimal precision
    
    Args:
        precision: "fp32", "fp16", "int8"
        model_path: Path to model file
    """
    precision_map = {
        "fp32": 0,  # Highest accuracy, slowest
        "fp16": 1,  # Good balance (recommended)
        "int8": 2   # Fastest, may need calibration
    }
    
    config = {
        "network-mode": precision_map.get(precision, 1),
        "model-engine-file": model_path
    }
    
    if precision == "int8":
        config["calibration-file"] = model_path.replace(".engine", "_calibration.bin")
    
    return config
```

### Optimization 3: Pipeline Parallelism

**Best Practice**: Run multiple pipelines on different GPUs for scalability.

```python
from multiprocessing import Process

def run_pipeline_on_gpu(pipeline_config, gpu_id):
    """Run pipeline on specific GPU"""
    import os
    os.environ["CUDA_VISIBLE_DEVICES"] = str(gpu_id)
    
    pipeline = build_pipeline(pipeline_config)
    pipeline.start().wait()

def run_multi_gpu_pipelines(pipeline_configs):
    """Run pipelines on multiple GPUs"""
    processes = []
    
    for idx, config in enumerate(pipeline_configs):
        gpu_id = idx % get_num_gpus()  # Distribute across GPUs
        process = Process(
            target=run_pipeline_on_gpu,
            args=(config, gpu_id)
        )
        process.start()
        processes.append(process)
    
    # Wait for all processes
    for process in processes:
        process.join()
```

### Optimization 4: Memory Pool Configuration

**Best Practice**: Configure appropriate buffer pool sizes.

```python
def configure_buffer_pools(pipeline, num_streams, batch_size):
    """Configure buffer pools for optimal performance"""
    # Calculate buffer pool size
    # Rule: pool_size >= (num_streams / batch_size) * 2
    pool_size = max(4, (num_streams // batch_size) * 2)
    
    # Configure queues
    for elem in pipeline.elements:
        if elem.name.startswith("queue"):
            elem.set_property("max-size-buffers", pool_size * 10)
            elem.set_property("max-size-time", 0)  # Unlimited time
            elem.set_property("leaky", 2)  # Leaky downstream
```

---

## 3. Memory Management

### Best Practice 1: Proper Cleanup

```python
class ManagedPipeline:
    """Pipeline with proper resource management"""
    def __init__(self, pipeline):
        self.pipeline = pipeline
        self.probes = []
    
    def add_probe(self, element_name, probe):
        """Add probe and track for cleanup"""
        self.pipeline.attach(element_name, probe)
        self.probes.append(probe)
    
    def start(self):
        """Start pipeline"""
        self.pipeline.start()
    
    def stop(self):
        """Stop pipeline and cleanup"""
        self.pipeline.set_state(GST_STATE_NULL)
        
        # Cleanup probes
        for probe in self.probes:
            if hasattr(probe, 'close'):
                probe.close()
            if hasattr(probe, 'flush'):
                probe.flush()
    
    def __enter__(self):
        self.start()
        return self
    
    def __exit__(self, exc_type, exc_val, exc_tb):
        self.stop()
```

### Best Practice 2: Memory Monitoring

```python
import pynvml

class MemoryMonitor:
    """Monitor GPU memory usage"""
    def __init__(self):
        pynvml.nvmlInit()
        self.handle = pynvml.nvmlDeviceGetHandleByIndex(0)
    
    def get_memory_info(self):
        """Get current GPU memory usage"""
        info = pynvml.nvmlDeviceGetMemoryInfo(self.handle)
        return {
            "total": info.total / (1024**3),  # GB
            "used": info.used / (1024**3),     # GB
            "free": info.free / (1024**3)     # GB
        }
    
    def check_memory_pressure(self, threshold=0.9):
        """Check if memory usage exceeds threshold"""
        info = self.get_memory_info()
        usage_ratio = info["used"] / info["total"]
        return usage_ratio > threshold

# Usage in pipeline
monitor = MemoryMonitor()
if monitor.check_memory_pressure():
    print("Warning: High GPU memory usage!")
```

---

## 4. Error Handling and Resilience

### Pattern 1: Retry Logic

```python
import time
from functools import wraps

def retry(max_attempts=3, delay=1.0, backoff=2.0):
    """Retry decorator with exponential backoff"""
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            attempts = 0
            current_delay = delay
            
            while attempts < max_attempts:
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    attempts += 1
                    if attempts >= max_attempts:
                        raise
                    print(f"Attempt {attempts} failed: {e}. Retrying in {current_delay}s...")
                    time.sleep(current_delay)
                    current_delay *= backoff
        return wrapper
    return decorator

@retry(max_attempts=3, delay=1.0)
def initialize_kafka_producer(config):
    """Initialize Kafka producer with retry"""
    return KafkaProducer(bootstrap_servers=config["servers"])
```

### Pattern 2: Circuit Breaker

```python
class CircuitBreaker:
    """Circuit breaker pattern for external services"""
    def __init__(self, failure_threshold=5, timeout=60):
        self.failure_threshold = failure_threshold
        self.timeout = timeout
        self.failure_count = 0
        self.last_failure_time = None
        self.state = "closed"  # closed, open, half_open
    
    def call(self, func, *args, **kwargs):
        """Execute function with circuit breaker"""
        if self.state == "open":
            if time.time() - self.last_failure_time > self.timeout:
                self.state = "half_open"
            else:
                raise Exception("Circuit breaker is OPEN")
        
        try:
            result = func(*args, **kwargs)
            self.on_success()
            return result
        except Exception as e:
            self.on_failure()
            raise
    
    def on_success(self):
        """Reset on success"""
        self.failure_count = 0
        self.state = "closed"
    
    def on_failure(self):
        """Track failures"""
        self.failure_count += 1
        self.last_failure_time = time.time()
        
        if self.failure_count >= self.failure_threshold:
            self.state = "open"
```

### Pattern 3: Graceful Shutdown

```python
import signal
import sys

class GracefulShutdown:
    """Handle graceful shutdown signals"""
    def __init__(self):
        self.shutdown_requested = False
        signal.signal(signal.SIGINT, self._signal_handler)
        signal.signal(signal.SIGTERM, self._signal_handler)
    
    def _signal_handler(self, signum, frame):
        """Handle shutdown signals"""
        print(f"\nReceived signal {signum}. Initiating graceful shutdown...")
        self.shutdown_requested = True
    
    def is_shutdown_requested(self):
        """Check if shutdown was requested"""
        return self.shutdown_requested

# Usage
shutdown_handler = GracefulShutdown()

def run_pipeline_with_graceful_shutdown(pipeline):
    """Run pipeline with graceful shutdown handling"""
    try:
        pipeline.start()
        
        while not shutdown_handler.is_shutdown_requested():
            time.sleep(0.1)
            # Check pipeline state, process messages, etc.
        
        print("Shutting down pipeline...")
        pipeline.stop()
    except Exception as e:
        print(f"Error: {e}")
        pipeline.stop()
```

---

## 5. Code Organization and Maintainability

### Pattern 1: Separation of Concerns

```python
# config.py - Configuration management
class PipelineConfig:
    def __init__(self, config_path):
        self.config = self._load_config(config_path)
    
    def get_source_config(self):
        return self.config["source"]
    
    def get_inference_config(self):
        return self.config["inference"]

# pipeline_builder.py - Pipeline construction
class PipelineBuilder:
    def __init__(self, config: PipelineConfig):
        self.config = config
    
    def build(self):
        pipeline = Pipeline("main")
        # Build pipeline from config
        return pipeline

# processors.py - Processing logic
class MetadataProcessor:
    def process(self, batch_meta):
        # Processing logic
        pass

# main.py - Application entry point
def main():
    config = PipelineConfig("config.yml")
    builder = PipelineBuilder(config)
    pipeline = builder.build()
    pipeline.start().wait()
```

### Pattern 2: Dependency Injection

```python
class PipelineService:
    """Service class with dependency injection"""
    def __init__(self, 
                 source_factory,
                 inference_factory,
                 sink_factory,
                 processor_factory):
        self.source_factory = source_factory
        self.inference_factory = inference_factory
        self.sink_factory = sink_factory
        self.processor_factory = processor_factory
    
    def create_pipeline(self):
        """Create pipeline using injected factories"""
        pipeline = Pipeline("service-pipeline")
        
        # Use factories to create elements
        source = self.source_factory.create()
        inference = self.inference_factory.create()
        sink = self.sink_factory.create()
        
        # Build pipeline
        # ...
        
        return pipeline
```

---

## 6. Testing Strategies

### Unit Testing

```python
import unittest
from unittest.mock import Mock, patch

class TestMetadataProcessor(unittest.TestCase):
    def setUp(self):
        self.processor = MetadataProcessor()
    
    def test_process_empty_batch(self):
        """Test processing empty batch"""
        batch_meta = Mock()
        batch_meta.frame_items = []
        
        # Should not raise exception
        self.processor.process(batch_meta)
    
    def test_process_with_objects(self):
        """Test processing batch with objects"""
        batch_meta = Mock()
        frame_meta = Mock()
        frame_meta.object_items = [Mock(), Mock()]
        batch_meta.frame_items = [frame_meta]
        
        self.processor.process(batch_meta)
        # Assert expected behavior
```

### Integration Testing

```python
class TestPipelineIntegration(unittest.TestCase):
    def test_pipeline_creation(self):
        """Test pipeline creation"""
        config = PipelineConfig("test_config.yml")
        builder = PipelineBuilder(config)
        pipeline = builder.build()
        
        self.assertIsNotNone(pipeline)
        self.assertEqual(len(pipeline.elements), expected_count)
    
    def test_pipeline_linking(self):
        """Test pipeline element linking"""
        pipeline = create_test_pipeline()
        
        # Verify links are correct
        # ...
```

### Performance Testing

```python
import time

class PerformanceTest:
    def test_fps_measurement(self, pipeline, duration=10):
        """Measure FPS of pipeline"""
        start_time = time.time()
        frame_count = 0
        
        def frame_callback(batch_meta):
            nonlocal frame_count
            for _ in batch_meta.frame_items:
                frame_count += 1
        
        pipeline.attach("infer", Probe("fps", frame_callback))
        pipeline.start()
        
        time.sleep(duration)
        pipeline.stop()
        
        elapsed = time.time() - start_time
        fps = frame_count / elapsed
        
        print(f"Measured FPS: {fps:.2f}")
        return fps
```

---

## 7. Deployment Considerations

### Configuration Management

```python
import os
from pathlib import Path

class DeploymentConfig:
    """Load configuration for the active deployment profile"""
    def __init__(self):
        self.profile = os.environ.get("DEEPSTREAM_PROFILE", "development")
        self.config_dir = Path("/etc/deepstream") / self.profile
    
    def get_config_path(self, config_name):
        """Get configuration file path"""
        return self.config_dir / f"{config_name}.yml"
    
    def get_model_path(self, model_name):
        """Get model file path"""
        return Path("/opt/models") / self.profile / model_name
```

### Logging Best Practices

```python
import logging
import sys

def setup_logging(level=logging.INFO, log_file=None):
    """Setup logging configuration"""
    handlers = [logging.StreamHandler(sys.stdout)]
    
    if log_file:
        handlers.append(logging.FileHandler(log_file))
    
    logging.basicConfig(
        level=level,
        format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
        handlers=handlers
    )

# Usage
logger = logging.getLogger(__name__)
logger.info("Pipeline started")
logger.error("Error occurred", exc_info=True)
```

---

## 8. Security Best Practices

### Secure Configuration

```python
import os
from cryptography.fernet import Fernet

class SecureConfig:
    """Handle sensitive configuration securely"""
    def __init__(self):
        self.key = os.getenv("CONFIG_ENCRYPTION_KEY")
        self.cipher = Fernet(self.key) if self.key else None
    
    def get_secret(self, secret_name):
        """Get decrypted secret"""
        encrypted = os.getenv(secret_name)
        if self.cipher and encrypted:
            return self.cipher.decrypt(encrypted.encode()).decode()
        return encrypted
```

### Input Validation

```python
def validate_video_path(path):
    """Validate video file path"""
    if not os.path.exists(path):
        raise ValueError(f"Video file not found: {path}")
    
    allowed_extensions = ['.h264', '.h265', '.mp4', '.mkv']
    if not any(path.endswith(ext) for ext in allowed_extensions):
        raise ValueError(f"Unsupported video format: {path}")
    
    return path

def validate_config_file(config_path):
    """Validate configuration file"""
    if not os.path.exists(config_path):
        raise ValueError(f"Config file not found: {config_path}")
    
    # Additional validation
    # ...
    
    return config_path
```

---

## 9. Monitoring and Observability

### Metrics Collection

```python
from prometheus_client import Counter, Histogram, Gauge

# Define metrics
frames_processed = Counter('deepstream_frames_processed_total', 'Total frames processed')
inference_latency = Histogram('deepstream_inference_latency_seconds', 'Inference latency')
gpu_memory_usage = Gauge('deepstream_gpu_memory_bytes', 'GPU memory usage')

class MetricsCollector(BatchMetadataOperator):
    """Collect metrics from pipeline"""
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            frames_processed.inc()
            
            # Record inference latency if available
            if hasattr(frame_meta, 'inference_time'):
                inference_latency.observe(frame_meta.inference_time)
```

---

## 10. Common Anti-Patterns to Avoid

### Anti-Pattern 1: Blocking Operations in Probes

**Bad**:
```python
class BadProbe(BatchMetadataOperator):
    def handle_metadata(self, batch_meta):
        # Blocking network call in probe
        response = requests.get("http://api.example.com/data")
        # This blocks the pipeline!
```

**Good**:
```python
import queue
import threading

class GoodProbe(BatchMetadataOperator):
    def __init__(self):
        super().__init__()
        self.queue = queue.Queue()
        self.worker = threading.Thread(target=self._process_queue)
        self.worker.start()
    
    def handle_metadata(self, batch_meta):
        # Non-blocking: add to queue
        self.queue.put(batch_meta)
    
    def _process_queue(self):
        while True:
            batch_meta = self.queue.get()
            # Process asynchronously
            response = requests.get("http://api.example.com/data")
```

### Anti-Pattern 2: Ignoring Memory Limits

**Bad**:
```python
# No batch size limits
pipeline.add("nvstreammux", "mux", {"batch-size": 100})  # Too large!
```

**Good**:
```python
# Calculate optimal batch size
optimal_batch = calculate_optimal_batch_size(num_streams, gpu_memory)
pipeline.add("nvstreammux", "mux", {"batch-size": optimal_batch})
```

### Anti-Pattern 3: Not Handling Errors

**Bad**:
```python
pipeline.start().wait()  # No error handling
```

**Good**:
```python
try:
    pipeline.start().wait()
except Exception as e:
    logger.error(f"Pipeline error: {e}", exc_info=True)
    pipeline.stop()
    raise
```

### Anti-Pattern 4: Missing async=0 on All Sinks (Tee/Dynamic Sources)

**CRITICAL**: When using `tee` to split a pipeline into multiple branches OR using dynamic sources (nvmultiurisrcbin), **ALL sink elements** must have `async: 0`. This is the most common cause of pipelines stuck in PAUSED state.

**Bad** - Pipeline stuck in PAUSED:
```python
# WRONG - Only display sink has async=0, Kafka sink is missing it
# Pipeline will be STUCK IN PAUSED STATE!

# Tee split
pipeline.add("tee", "tee")

# Metadata branch - MISSING async=0!
pipeline.add("nvmsgbroker", "msgbroker", {
    "proto-lib": "/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
    "conn-str": "localhost;9092",
    "sync": 0,
    # async: 0 is MISSING! Pipeline will hang!
})

# Video branch - has async=0 but it's not enough
pipeline.add("nveglglessink", "sink", {
    "sync": 0,
    "async": 0  # This alone is NOT enough - ALL sinks need it!
})
```

**Good** - All sinks have async=0:
```python
# CORRECT - ALL sinks have async=0

# Tee split
pipeline.add("tee", "tee")

# Metadata branch - Kafka sink with async=0
pipeline.add("nvmsgbroker", "msgbroker", {
    "proto-lib": "/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
    "conn-str": "localhost;9092",
    "sync": 0,
    "async": 0  # CRITICAL: Required on ALL sinks!
})

# Video branch - display sink with async=0
pipeline.add("nveglglessink", "sink", {
    "sync": 0,
    "qos": 0,
    "async": 0  # CRITICAL: Required on ALL sinks!
})
```

**Symptoms of this bug**:
- Camera shows "added successfully" in logs
- Pipeline elements transition to READY, then PAUSED
- Pipeline never transitions to PLAYING
- No video display, no data flowing
- No error messages (silent failure)

**Rule**: When using `tee` or dynamic sources, ALWAYS set `async: 0` on EVERY sink element in the pipeline.

### Anti-Pattern 5: Using threading.Queue with multiprocessing.Process

**CRITICAL**: This is a common and subtle bug that causes data loss!

When using `multiprocessing.Process` to run pipelines in separate processes, you MUST use `multiprocessing.Queue` for inter-process communication. A regular `queue.Queue` (from the `queue` module) only works within a single process.

**Bad** - Data silently lost:
```python
from multiprocessing import Process
from queue import Queue  # WRONG! This is a threading queue

class MultiStreamProcessor:
    def __init__(self):
        # This queue WILL NOT work across process boundaries!
        self.batch_queue = Queue()  # BAD: threading.Queue
    
    def start(self, use_multiprocessing=True):
        for stream in self.streams:
            if use_multiprocessing:
                # Child process gets a COPY of the queue
                # Any data put into it never reaches the parent!
                process = Process(
                    target=self._run_pipeline,
                    args=(stream, self.batch_queue)
                )
                process.start()
```

**Good** - Use multiprocessing.Queue for inter-process communication:
```python
from multiprocessing import Process, Queue as MPQueue  # Correct!
from queue import Queue as ThreadQueue

class MultiStreamProcessor:
    def __init__(self, use_multiprocessing=True):
        # Choose the right queue type based on usage
        if use_multiprocessing:
            self.batch_queue = MPQueue()  # CORRECT: multiprocessing.Queue
        else:
            self.batch_queue = ThreadQueue()  # For single-process/threading
    
    def start(self, use_multiprocessing=True):
        for stream in self.streams:
            if use_multiprocessing:
                # multiprocessing.Queue properly shares data across processes
                process = Process(
                    target=self._run_pipeline,
                    args=(stream, self.batch_queue)
                )
                process.start()
```

**Alternative - Use threading instead of multiprocessing**:
```python
import threading
from queue import Queue  # OK for threading

class MultiStreamProcessor:
    def __init__(self):
        self.batch_queue = Queue()  # OK: threading.Queue for threads
    
    def start(self):
        for stream in self.streams:
            # Threads share memory, so queue.Queue works fine
            thread = threading.Thread(
                target=self._run_pipeline,
                args=(stream, self.batch_queue)
            )
            thread.start()
```

**Key Rules**:
1. `queue.Queue` → Use with `threading.Thread` (same process)
2. `multiprocessing.Queue` → Use with `multiprocessing.Process` (cross-process)
3. When in doubt, set `use_multiprocessing=False` and use threads
4. Always add debug logs to verify data flows through queues correctly

**Symptoms of this bug**:
- Pipeline appears to run normally
- No error messages
- Downstream processing (e.g., VLM, Kafka) never receives data
- Statistics show 0 batches/messages processed

---

## 11. Common Pitfalls and Code Generation Errors

This section documents common mistakes encountered when generating DeepStream code, to prevent them in future.

### Pitfall 1: Using len() on Metadata Iterators

**Problem**: `frame_meta.object_items`, `frame_meta.tensor_items`, and `frame_meta.user_items` return **iterators**, not lists.

**Error**:
```
TypeError: object of type 'iterator' has no len()
```

**Bad Code**:
```python
# WRONG - Causes crash
count = len(frame_meta.object_items)

# WRONG - Second loop is empty (iterator already consumed)
for obj in frame_meta.object_items:
    process(obj)
for obj in frame_meta.object_items:
    count += 1
```

**Correct Code**:
```python
# CORRECT - Count while iterating
obj_count = 0
for obj in frame_meta.object_items:
    obj_count += 1
    process(obj)
```

### Pitfall 2: Incorrect nvinfer Configuration Syntax

**Problem**: nvinfer supports **both YAML and INI-style formats**, but the syntax must be correct for each format.

**Error**:
```
Configuration file parsing failed
```

**Common Mistakes**:
```yaml
# WRONG - Incorrect section name (should be 'property', not 'model')
model:
  model-engine-file: /path/to/model.engine
  batch-size: 1

# WRONG - Mixing formats (YAML syntax in .txt file or vice versa)
```

**Correct YAML Config** (`.yml`):
```yaml
# CORRECT YAML format
property:
  gpu-id: 0
  onnx-file: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx
  labelfile-path: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt
  batch-size: 1
  network-mode: 2
  num-detected-classes: 4
  process-mode: 1
  cluster-mode: 2

class-attrs-all:
  topk: 20
  pre-cluster-threshold: 0.2
```

**Correct INI-style Config** (`.txt`):
```ini
# CORRECT INI-style format
[property]
gpu-id=0
onnx-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx
labelfile-path=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt
batch-size=1
network-mode=2
num-detected-classes=4
process-mode=1
cluster-mode=2

[class-attrs-all]
topk=20
pre-cluster-threshold=0.2
```

**Key Rules**:
- YAML format: Use `property:` (no brackets), `key: value` with colon+space
- INI format: Use `[property]` (with brackets), `key=value` with equals sign
- Section must be named `property` (not `model` or other names)
- Don't mix formats in the same file

### Pitfall 3: Using Wrong Model (ResNet10 vs ResNet18)

**Problem**: DeepStream samples use **ResNet18** TrafficCamNet model, not ResNet10.

**Correct Model Paths**:
```
/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/
├── resnet18_trafficcamnet_pruned.onnx    # Use this ONNX model
├── labels.txt                              # Class labels
└── cal_trt.bin                            # INT8 calibration (optional)
```

**In nvinfer config**:
```ini
[property]
onnx-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx
labelfile-path=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt
```

### Pitfall 4: nvv4l2decoder Output Format Assumption

**Fact**: `nvv4l2decoder` outputs `video/x-raw(memory:NVMM)` - already in GPU memory format.

**Common Mistake**: Adding unnecessary `nvvideoconvert` after decoder.

**Unnecessary Code**:
```python
# UNNECESSARY - nvv4l2decoder already outputs NVMM format
pipeline.add("nvv4l2decoder", "decoder")
pipeline.add("nvvideoconvert", "conv")  # Not needed!
pipeline.add("nvstreammux", "mux")
```

**Correct Code**:
```python
# CORRECT - Direct connection, no converter needed
pipeline.add("nvv4l2decoder", "decoder")
pipeline.add("nvstreammux", "mux")
pipeline.link(("decoder", "mux"), ("", "sink_%u"))
```

### Pitfall 5: Built-in Probe Usage

**Fact**: `measure_fps_probe` is a valid built-in probe, but must be attached to the correct element.

**Correct Usage**:
```python
# Attach to inference element for FPS measurement
pipeline.attach("infer", "measure_fps_probe", "fps-probe")
```

**If probe attachment fails**, implement custom FPS measurement:
```python
class FPSCounter(BatchMetadataOperator):
    def __init__(self):
        super().__init__()
        self.start_time = None
        self.frame_count = 0
    
    def handle_metadata(self, batch_meta):
        if self.start_time is None:
            self.start_time = time.time()
        self.frame_count += 1
        elapsed = time.time() - self.start_time
        if elapsed > 0 and self.frame_count % 30 == 0:
            print(f"FPS: {self.frame_count / elapsed:.2f}")

pipeline.attach("infer", Probe("fps-counter", FPSCounter()))
```

---

## Summary

Following these best practices and patterns will help you build robust, performant, and maintainable DeepStream applications. Key takeaways:

1. **Design for modularity**: Use patterns like Factory, Strategy, and Dependency Injection
2. **Optimize performance**: Tune batch sizes, use appropriate precision, enable parallelism
3. **Manage resources**: Proper cleanup, memory monitoring, buffer pool configuration
4. **Handle errors gracefully**: Retry logic, circuit breakers, graceful shutdown
5. **Test thoroughly**: Unit tests, integration tests, performance tests
6. **Monitor and observe**: Metrics collection, logging, health checks
7. **Secure your application**: Input validation, secure configuration, access control
8. **Use correct Queue types**: 
   - `queue.Queue` → for threading (same process)
   - `multiprocessing.Queue` → for multiprocessing (cross-process)
   - **NEVER** use `queue.Queue` with `multiprocessing.Process` - data will be silently lost!
9. **Set async=0 on ALL sinks when using tee or dynamic sources**:
   - When pipeline uses `tee` to split into multiple branches, ALL sink elements need `async: 0`
   - When using dynamic sources (nvmultiurisrcbin), ALL sinks need `async: 0`
   - **Symptom if missing**: Pipeline stuck in PAUSED state, no video/data flows
   - This applies to display sinks, Kafka sinks, file sinks - ALL sinks!
10. **Avoid common code generation pitfalls**:
   - **NEVER** use `len()` on metadata iterators (`object_items`, `tensor_items`, `user_items`)
   - **USE** correct syntax for nvinfer config (YAML: `property:` with `: `, or INI: `[property]` with `=`)
   - **USE** ResNet18 model (`resnet18_trafficcamnet_pruned.onnx`) from DeepStream samples
   - **KNOW** that `nvv4l2decoder` outputs NVMM format (no converter needed before nvstreammux)

These practices ensure your DeepStream applications are production-ready and scalable.

# Buffer Provider and Retriever APIs

## Overview

DeepStream Service Maker provides two complementary APIs for custom data injection and extraction:

1. **Media Extractor (BufferProvider/Feeder)** - Inject custom data INTO pipelines
2. **Frame Selector (BufferRetriever/Receiver)** - Extract data FROM pipelines

## When to Use Each API

### Use BufferProvider/Feeder When:
- You need to inject custom video frames from non-standard sources
- You want to generate synthetic video data for testing
- You have pre-processed frames to feed into the pipeline
- You need to implement custom video sources beyond file/RTSP
- You want to transfer frames FROM another pipeline or system INTO DeepStream

**See**: Part 1 below for detailed API reference and implementation patterns.

### Use BufferRetriever/Receiver When:
- You need to extract frames for custom processing outside the pipeline
- You want to save specific frames to disk or external storage
- You need to collect inference results with frame data
- You want to implement custom frame selection logic
- You want to transfer frames FROM DeepStream TO another pipeline or system

**See**: Part 2 below for detailed API reference and implementation patterns.

## Common Patterns

### Pattern 1: Pipeline-to-Pipeline Transfer
Transfer frames between two DeepStream pipelines.

```text
Pipeline A -> BufferRetriever -> Queue -> BufferProvider -> Pipeline B
```

**Use Case**: Process video in one pipeline, then re-process results in another

**Details**: See Part 1 Pattern 3 (Frame Queue Injection) and Part 2 Pattern 2 (Frame Queue Transfer)

### Pattern 2: Custom Video Source
Read from custom camera or video source.

```text
Custom Source -> BufferProvider -> appsrc -> DeepStream Pipeline
```

**Use Case**: Integrate non-standard cameras or video sources

**Details**: See Part 1 Pattern 1 (File-Based Custom Video Source)

### Pattern 3: Frame Extraction
Extract frames from pipeline for archival or analysis.

```text
DeepStream Pipeline -> appsink -> BufferRetriever -> Save/Process
```

**Use Case**: Save frames at intervals, capture detection screenshots

**Details**: See Part 2 Pattern 1 (Frame Extraction and Saving)

### Pattern 4: Synthetic Data Generation
Generate test data for pipeline validation.

```text
Synthetic Generator -> BufferProvider -> appsrc -> DeepStream Pipeline
```

**Use Case**: Testing, simulation, validation

**Details**: See Part 1 Pattern 2 (Synthetic Frame Generation)

### Pattern 5: Selective Frame Capture
Capture frames based on inference results.

```text
Pipeline -> Inference -> Metadata Probe -> Trigger -> BufferRetriever -> Save
```

**Use Case**: Save frames only when specific objects detected

**Details**: See Part 2 Pattern 3 (Selective Frame Capture)

## API Comparison

| Feature | BufferProvider/Feeder | BufferRetriever/Receiver |
|---------|----------------------|--------------------------|
| **Direction** | Data IN (injection) | Data OUT (extraction) |
| **GStreamer Element** | appsrc | appsink |
| **Signal** | need-data/enough-data | new-sample |
| **Method to Implement** | `generate(size)` | `consume(buffer)` |
| **Return Value** | Buffer object | int (1=success, 0=error) |
| **EOS Handling** | Return empty Buffer() | Return -1 |
| **Properties** | format, width, height, framerate, device | None (configured on appsink) |

## Quick Start Examples

### Inject Custom Frames (BufferProvider)

```python
from pyservicemaker import Pipeline, BufferProvider, Feeder, as_tensor, ColorFormat, Buffer
import torch  # pip install torch torchvision (not in base DS container)

class MyProvider(BufferProvider):
    def __init__(self):
        super().__init__()
        self.format = "RGB"
        self.width = 1280
        self.height = 720
        self.framerate = 30
        self.device = 'gpu'

    def generate(self, size):
        # Your custom frame generation logic
        frame = get_custom_frame()  # Your function
        if frame is None:
            return Buffer()  # EOS

        torch_tensor = torch.from_numpy(frame).cuda()
        ds_tensor = as_tensor(torch_tensor, "HWC")
        return ds_tensor.wrap(ColorFormat.RGB)

pipeline = Pipeline("inject-pipeline")
caps = "video/x-raw(memory:NVMM), format=RGB, width=1280, height=720, framerate=30/1"
pipeline.add("appsrc", "src", {"caps": caps, "do-timestamp": True})
# ... add more elements ...
pipeline.attach("src", Feeder("feeder", MyProvider()), tips="need-data/enough-data")
pipeline.start().wait()
```

### Extract Frames (BufferRetriever)

```python
from pyservicemaker import Pipeline, BufferRetriever, Receiver
import torch  # pip install torch torchvision (not in base DS container)

class MyRetriever(BufferRetriever):
    def __init__(self):
        super().__init__()
        self.count = 0

    def consume(self, buffer):
        tensor = buffer.extract(0).clone()  # Always clone!
        torch_tensor = torch.utils.dlpack.from_dlpack(tensor)

        # Your custom processing logic
        process_frame(torch_tensor)  # Your function

        self.count += 1
        return 1  # Success

pipeline = Pipeline("extract-pipeline")
# ... add source and processing elements ...
pipeline.add("appsink", "sink", {"emit-signals": True, "sync": False})
pipeline.attach("sink", Receiver("receiver", MyRetriever()), tips="new-sample")
pipeline.start().wait()
```

## Key Concepts

### BufferProvider/Feeder
- **Purpose**: Custom data injection
- **Element**: Works with `appsrc`
- **Flow**: Your code -> BufferProvider -> Pipeline
- **Control**: Pipeline pulls data when needed
- **Properties**: Must set format, width, height, framerate, device

### BufferRetriever/Receiver
- **Purpose**: Custom data extraction
- **Element**: Works with `appsink`
- **Flow**: Pipeline -> BufferRetriever -> Your code
- **Control**: Pipeline pushes data when available
- **Critical**: Always call `.clone()` on extracted tensors

## Best Practices Summary

### For BufferProvider:
1. Set all required properties (format, width, height, framerate, device)
2. Return empty `Buffer()` to signal end of stream
3. Use GPU memory (`device='gpu'`) for best performance
4. Set `do-timestamp=True` on appsrc for proper sync
5. Use `tips="need-data/enough-data"` when attaching

### For BufferRetriever:
1. **Always** call `.clone()` on extracted tensors
2. Set `emit-signals=True` on appsink
3. Use `tips="new-sample"` when attaching
4. Return 1 for success, 0 for error (continue), -1 for fatal error
5. Set `sync=False` for non-real-time extraction

## Common Pitfalls

### BufferProvider Issues:
- Forgetting to set format properties -> Pipeline fails to negotiate caps
- Not returning empty Buffer() for EOS -> Pipeline hangs
- Mismatched caps between provider and appsrc -> Format errors

### BufferRetriever Issues:
- Not calling `.clone()` -> Data corruption in async processing
- Forgetting `emit-signals=True` -> No frames received
- Slow processing in consume() -> Frame drops
- Not handling exceptions -> Pipeline crashes

## Performance Tips

### BufferProvider:
- Use GPU memory for zero-copy transfers
- Pre-allocate buffers when possible
- Avoid CPU<->GPU transfers in hot path
- Consider buffer pooling for high frame rates

### BufferRetriever:
- Set `sync=False` if you don't need real-time pacing
- Process frames asynchronously if possible
- Limit buffer accumulation to prevent memory issues
- Use batch processing when extracting multiple streams

## Example Applications

The service-maker package includes sample applications demonstrating these APIs:

**Pipeline API Examples**:
- `/opt/nvidia/deepstream/deepstream/service-maker/sources/apps/python/pipeline_api/deepstream_appsrc_test_app/`

**Flow API Examples**:
- `/opt/nvidia/deepstream/deepstream/service-maker/sources/apps/python/flow_api/deepstream_appsrc_test_app/`

## Goal-Based API Selection

| Goal | Use This API | Section |
|------|-------------|---------|
| Inject custom frames | BufferProvider/Feeder | Part 1 |
| Extract frames | BufferRetriever/Receiver | Part 2 |
| Pipeline-to-pipeline transfer | Both | Part 1 Pattern 3, Part 2 Pattern 2 |
| Custom video source | BufferProvider/Feeder | Part 1 Pattern 1 |
| Frame archival | BufferRetriever/Receiver | Part 2 Pattern 1 |
| Synthetic data generation | BufferProvider/Feeder | Part 1 Pattern 2 |
| Selective capture | BufferRetriever/Receiver | Part 2 Pattern 3 |

Choose the right API based on your data flow direction: injection (BufferProvider) or extraction (BufferRetriever).

---

# Part 1: BufferProvider / Feeder API (Media Extractor)

## Overview

The Media Extractor API (implemented through `BufferProvider` and `Feeder` classes) enables custom data injection into DeepStream pipelines. This is useful for:
- Injecting custom video frames from non-standard sources
- Generating synthetic video data for testing
- Feeding pre-processed frames into the pipeline
- Implementing custom video sources beyond file/RTSP streams

## Core Concepts

### BufferProvider
A `BufferProvider` is a user-implemented class that generates buffers on-demand. It works with GStreamer's `appsrc` element to inject data into the pipeline.

### Feeder
A `Feeder` is a wrapper that connects a `BufferProvider` to an `appsrc` element. It manages the signal handling for "need-data" and "enough-data" events.

### Data Flow
```text
BufferProvider.generate() -> Feeder -> appsrc -> Pipeline
```

## API Reference

### BufferProvider Class

Base class for implementing custom media providers.

**Methods to Override**:

#### `generate(size)`
Generate a buffer when the pipeline needs data.

**Parameters**:
- `size` (int): Number of bytes requested by the pipeline

**Returns**: `Buffer` object containing the data, or empty `Buffer()` to signal EOS

**Properties to Set**:
- `format` (str): Video format (e.g., "RGB", "NV12")
- `width` (int): Frame width in pixels
- `height` (int): Frame height in pixels
- `framerate` (int): Frame rate
- `device` (str): 'gpu' or 'cpu'

**Example**:
```python
from pyservicemaker import BufferProvider, as_tensor, ColorFormat, Buffer
import torch  # pip install torch torchvision (not in base DS container)

class MyBufferProvider(BufferProvider):
    def __init__(self, video_source):
        super().__init__()
        self.source = video_source
        self.format = "RGB"
        self.width = 1920
        self.height = 1080
        self.framerate = 30
        self.device = 'gpu'
        self.frame_count = 0

    def generate(self, size):
        # Get frame from your custom source
        frame = self.source.get_next_frame()

        if frame is None:
            # Signal end of stream
            return Buffer()

        # Convert to torch tensor (on GPU if needed)
        torch_tensor = torch.from_numpy(frame).cuda()

        # Convert to DeepStream tensor format
        ds_tensor = as_tensor(torch_tensor, "HWC")  # Height, Width, Channels

        # Wrap in buffer with color format
        buffer = ds_tensor.wrap(ColorFormat.RGB)

        self.frame_count += 1
        return buffer
```

### Feeder Class

Wrapper for attaching a BufferProvider to a pipeline element.

**Constructor**:
```python
from pyservicemaker import Feeder

feeder = Feeder("feeder-name", buffer_provider_instance)
```

**Parameters**:
- `name` (str): Name of the feeder
- `provider` (BufferProvider): BufferProvider instance

### Helper Functions

#### `as_tensor(torch_tensor, layout)`
Convert a PyTorch tensor to DeepStream tensor format.

**Parameters**:
- `torch_tensor`: PyTorch tensor
- `layout` (str): Tensor layout - "HWC" (Height, Width, Channels) or "CHW"

**Returns**: DeepStream tensor object

#### ColorFormat Enum
Specifies the pixel format for buffers.

**Values**:
- `ColorFormat.RGB`: RGB format
- `ColorFormat.RGBA`: RGBA format
- `ColorFormat.NV12`: NV12 format (YUV 4:2:0)
- `ColorFormat.GRAY`: Grayscale

### Buffer Class

Container for video frame data.

**Constructor**:
```python
buffer = Buffer()  # Empty buffer (signals EOS)
```

**Methods**:
- `extract(index)`: Extract tensor at index from buffer
- `clone()`: Create a copy of the buffer

## Implementation Patterns

### Pattern 1: File-Based Custom Video Source

Read frames from custom file format and inject into pipeline.

```python
from pyservicemaker import Pipeline, BufferProvider, Feeder, as_tensor, ColorFormat, Buffer
import cv2  # pip install opencv-python-headless (not in base DS container)
import torch  # pip install torch torchvision (not in base DS container)
import platform

class CustomVideoFileProvider(BufferProvider):
    def __init__(self, video_path):
        super().__init__()
        self.cap = cv2.VideoCapture(video_path)

        # Set buffer properties
        self.format = "RGB"
        self.width = int(self.cap.get(cv2.CAP_PROP_FRAME_WIDTH))
        self.height = int(self.cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
        self.framerate = int(self.cap.get(cv2.CAP_PROP_FPS))
        self.device = 'gpu'
        self.frame_count = 0

    def generate(self, size):
        ret, frame = self.cap.read()

        if not ret:
            # End of video
            self.cap.release()
            return Buffer()

        # Convert BGR to RGB
        frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        # Convert to torch tensor and move to GPU
        torch_tensor = torch.from_numpy(frame_rgb).cuda()

        # Convert to DeepStream tensor
        ds_tensor = as_tensor(torch_tensor, "HWC")

        self.frame_count += 1
        print(f"Generated frame {self.frame_count}")

        return ds_tensor.wrap(ColorFormat.RGB)

def main(video_path):
    pipeline = Pipeline("custom-video-source")

    # Create appsrc with appropriate capabilities
    caps = f"video/x-raw(memory:NVMM), format=RGB, width=1920, height=1080, framerate=30/1"
    pipeline.add("appsrc", "src", {
        "caps": caps,
        "do-timestamp": True,
        "format": 3  # GST_FORMAT_TIME
    })

    # Add processing elements
    pipeline.add("nvvideoconvert", "convert", {
        "nvbuf-memory-type": 2,  # NVBUF_MEM_CUDA_DEVICE
        "compute-hw": 1
    })
    pipeline.add("capsfilter", "caps", {"caps": "video/x-raw(memory:NVMM), format=NV12"})
    pipeline.add("nvstreammux", "mux", {
        "batch-size": 1,
        "width": 1920,
        "height": 1080
    })

    # Add inference (optional)
    pipeline.add("nvinfer", "infer", {
        "config-file-path": "/path/to/config.yml"
    })

    # Add display
    pipeline.add("nvosdbin", "osd")
    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    pipeline.add(sink_type, "sink", {"sync": False})

    # Link elements
    pipeline.link("src", "convert")
    pipeline.link(("convert", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "infer", "osd", "sink")

    # Attach feeder to appsrc
    provider = CustomVideoFileProvider(video_path)
    pipeline.attach("src", Feeder("feeder", provider), tips="need-data/enough-data")

    # Start pipeline
    pipeline.start().wait()

if __name__ == "__main__":
    import sys
    main(sys.argv[1])
```

### Pattern 2: Synthetic Frame Generation

Generate synthetic frames for testing or simulation.

```python
from pyservicemaker import Pipeline, BufferProvider, Feeder, as_tensor, ColorFormat, Buffer
import torch  # pip install torch torchvision (not in base DS container)
import numpy as np

class SyntheticFrameProvider(BufferProvider):
    def __init__(self, num_frames=100, width=1280, height=720, fps=30):
        super().__init__()
        self.format = "RGB"
        self.width = width
        self.height = height
        self.framerate = fps
        self.device = 'gpu'
        self.num_frames = num_frames
        self.frame_idx = 0

    def generate(self, size):
        if self.frame_idx >= self.num_frames:
            return Buffer()

        # Generate synthetic frame (moving gradient)
        x = np.linspace(0, 255, self.width, dtype=np.uint8)
        y = np.linspace(0, 255, self.height, dtype=np.uint8)

        offset = (self.frame_idx * 5) % 255
        frame = np.zeros((self.height, self.width, 3), dtype=np.uint8)
        frame[:, :, 0] = (x + offset) % 255  # Red channel
        frame[:, :, 1] = (y + offset) % 255  # Green channel
        frame[:, :, 2] = 128  # Blue channel

        # Convert to torch and move to GPU
        torch_tensor = torch.from_numpy(frame).cuda()
        ds_tensor = as_tensor(torch_tensor, "HWC")

        self.frame_idx += 1
        return ds_tensor.wrap(ColorFormat.RGB)

def generate_test_video():
    pipeline = Pipeline("synthetic-video")

    provider = SyntheticFrameProvider(num_frames=300, width=1280, height=720, fps=30)

    caps = f"video/x-raw(memory:NVMM), format=RGB, width={provider.width}, height={provider.height}, framerate={provider.framerate}/1"
    pipeline.add("appsrc", "src", {"caps": caps, "do-timestamp": True})
    pipeline.add("nvvideoconvert", "convert")
    pipeline.add("nvv4l2h264enc", "encoder", {"bitrate": 4000000})
    pipeline.add("h264parse", "parser")
    pipeline.add("mp4mux", "mux")
    pipeline.add("filesink", "sink", {"location": "synthetic_output.mp4"})

    pipeline.link("src", "convert", "encoder", "parser", "mux", "sink")
    pipeline.attach("src", Feeder("feeder", provider), tips="need-data/enough-data")

    pipeline.start().wait()
```

### Pattern 3: Frame Queue Injection

Transfer frames between two pipelines using a queue.

```python
from pyservicemaker import Pipeline, BufferProvider, Feeder, as_tensor, ColorFormat, Buffer
from queue import Queue, Empty
import torch  # pip install torch torchvision (not in base DS container)

class QueuedBufferProvider(BufferProvider):
    def __init__(self, frame_queue, width=1280, height=720):
        super().__init__()
        self.queue = frame_queue
        self.format = "RGB"
        self.width = width
        self.height = height
        self.framerate = 30
        self.device = 'gpu'

    def generate(self, size):
        try:
            # Wait up to 2 seconds for frame
            tensor = self.queue.get(timeout=2)

            # Convert DLPack tensor to PyTorch
            torch_tensor = torch.utils.dlpack.from_dlpack(tensor)

            # Convert to DeepStream tensor
            ds_tensor = as_tensor(torch_tensor, "HWC")

            return ds_tensor.wrap(ColorFormat.RGB)
        except Empty:
            # Queue is empty, signal EOS
            print("Queue empty, ending stream")
            return Buffer()

def pipeline_with_queue_injection(frame_queue):
    pipeline = Pipeline("queue-injection")

    provider = QueuedBufferProvider(frame_queue, width=1280, height=720)

    caps = f"video/x-raw(memory:NVMM), format=RGB, width={provider.width}, height={provider.height}, framerate={provider.framerate}/1"
    pipeline.add("appsrc", "src", {"caps": caps, "do-timestamp": True})
    pipeline.add("nvvideoconvert", "convert", {"nvbuf-memory-type": 2})
    pipeline.add("capsfilter", "caps", {"caps": "video/x-raw(memory:NVMM), format=NV12"})
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1280, "height": 720})
    pipeline.add("nveglglessink", "sink", {"sync": False})

    pipeline.link("src", "convert", "caps")
    pipeline.link(("caps", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "sink")

    pipeline.attach("src", Feeder("feeder", provider), tips="need-data/enough-data")
    pipeline.start().wait()
```

### Pattern 4: Flow API with Buffer Injection

High-level Flow API for buffer injection.

```python
from pyservicemaker import Pipeline, Flow, BufferProvider, ColorFormat, as_tensor, Buffer
import torch  # pip install torch torchvision (not in base DS container)
import cv2  # pip install opencv-python-headless (not in base DS container)

class SimpleVideoProvider(BufferProvider):
    def __init__(self, video_path):
        super().__init__()
        self.cap = cv2.VideoCapture(video_path)
        self.format = "RGB"
        self.width = int(self.cap.get(cv2.CAP_PROP_FRAME_WIDTH))
        self.height = int(self.cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
        self.framerate = int(self.cap.get(cv2.CAP_PROP_FPS))
        self.device = 'gpu'

    def generate(self, size):
        ret, frame = self.cap.read()
        if not ret:
            return Buffer()

        frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        torch_tensor = torch.from_numpy(frame_rgb).cuda()
        ds_tensor = as_tensor(torch_tensor, "HWC")
        return ds_tensor.wrap(ColorFormat.RGB)

def flow_api_injection(video_path):
    pipeline = Pipeline("flow-injection")
    provider = SimpleVideoProvider(video_path)

    # Flow API: inject() -> infer() -> render()
    flow = Flow(pipeline)
    flow.inject([provider])  # Pass list of providers
    flow.infer("/path/to/config.yml")  # Optional: add inference
    flow.render()  # Add renderer
    flow()  # Execute
```

## Advanced Usage

### Multi-Source Buffer Injection

Inject from multiple custom sources simultaneously.

```python
from pyservicemaker import Pipeline, BufferProvider, Feeder, as_tensor, ColorFormat, Buffer
import cv2  # pip install opencv-python-headless (not in base DS container)
import torch  # pip install torch torchvision (not in base DS container)

class MultiSourceProvider(BufferProvider):
    def __init__(self, source_id, video_path):
        super().__init__()
        self.source_id = source_id
        self.cap = cv2.VideoCapture(video_path)
        self.format = "RGB"
        self.width = 1280
        self.height = 720
        self.framerate = 30
        self.device = 'gpu'

    def generate(self, size):
        ret, frame = self.cap.read()
        if not ret:
            return Buffer()

        # Resize to common size
        frame = cv2.resize(frame, (self.width, self.height))
        frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

        torch_tensor = torch.from_numpy(frame_rgb).cuda()
        ds_tensor = as_tensor(torch_tensor, "HWC")
        return ds_tensor.wrap(ColorFormat.RGB)

def multi_source_injection(video_paths):
    pipeline = Pipeline("multi-source-injection")

    # Create multiple appsrc elements
    for i, path in enumerate(video_paths):
        caps = "video/x-raw(memory:NVMM), format=RGB, width=1280, height=720, framerate=30/1"
        pipeline.add("appsrc", f"src{i}", {"caps": caps, "do-timestamp": True})
        pipeline.add("nvvideoconvert", f"convert{i}", {"nvbuf-memory-type": 2})

    # Add muxer
    pipeline.add("nvstreammux", "mux", {
        "batch-size": len(video_paths),
        "width": 1280,
        "height": 720
    })

    # Add inference and display
    pipeline.add("nvinfer", "infer", {"config-file-path": "/path/to/config.yml"})
    pipeline.add("nvmultistreamtiler", "tiler", {"rows": 2, "columns": 2})
    pipeline.add("nvosdbin", "osd")
    pipeline.add("nveglglessink", "sink")

    # Link sources to muxer
    for i in range(len(video_paths)):
        pipeline.link(f"src{i}", f"convert{i}")
        pipeline.link((f"convert{i}", "mux"), ("", "sink_%u"))

        # Attach feeder
        provider = MultiSourceProvider(i, video_paths[i])
        pipeline.attach(f"src{i}", Feeder(f"feeder{i}", provider), tips="need-data/enough-data")

    # Link processing chain
    pipeline.link("mux", "infer", "tiler", "osd", "sink")
    pipeline.start().wait()
```

## Part 1 Best Practices

### 1. Memory Management
- Use GPU memory (`device='gpu'`) for best performance
- Release resources properly (close files, release capture devices)
- Avoid memory leaks by managing tensors correctly

### 2. Buffer Format
- Always specify correct `format`, `width`, `height`, and `framerate`
- Match color format with pipeline requirements
- Use `ColorFormat.RGB` for most cases, `ColorFormat.NV12` for optimized pipelines

### 3. Timestamping
- Set `"do-timestamp": True` on appsrc for proper synchronization
- Important for multi-stream applications

### 4. Signal Handling
- Use `tips="need-data/enough-data"` when attaching Feeder
- This enables proper flow control and prevents buffer overflow

### 5. End of Stream
- Return empty `Buffer()` to signal EOS
- Properly cleanup resources before returning EOS

### 6. Error Handling
```python
class SafeBufferProvider(BufferProvider):
    def __init__(self, source):
        super().__init__()
        self.source = source
        self.format = "RGB"
        self.width = 1280
        self.height = 720
        self.framerate = 30
        self.device = 'gpu'

    def generate(self, size):
        try:
            frame = self.source.get_frame()
            if frame is None:
                return Buffer()

            torch_tensor = torch.from_numpy(frame).cuda()
            ds_tensor = as_tensor(torch_tensor, "HWC")
            return ds_tensor.wrap(ColorFormat.RGB)
        except Exception as e:
            print(f"Error generating buffer: {e}")
            return Buffer()  # Signal EOS on error
```

## Part 1 Common Use Cases

### 1. Custom Camera Integration
Integrate cameras not supported by standard GStreamer elements.

### 2. Pre-processed Frame Injection
Inject frames that have been pre-processed by custom algorithms.

### 3. Frame Rate Control
Control exact frame timing and rate for testing.

### 4. Multi-Pipeline Communication
Transfer frames between multiple DeepStream pipelines. See also Part 2 Pattern 2 for the retriever side of pipeline-to-pipeline transfer.

### 5. Synthetic Data Generation
Generate synthetic data for testing inference models.

### 6. Image Sequence Processing
Process sequences of images as video streams.

## Part 1 Troubleshooting

### Issue 1: Frames Not Flowing
**Solution**: Check that `tips="need-data/enough-data"` is set, verify appsrc caps match buffer properties

### Issue 2: Memory Errors
**Solution**: Ensure tensors are on correct device (GPU/CPU), check memory allocation

### Issue 3: Format Mismatch
**Solution**: Verify color format matches between BufferProvider and appsrc caps

### Issue 4: Timing Issues
**Solution**: Enable timestamping with `"do-timestamp": True`

## Part 1 Summary

The Media Extractor API (BufferProvider/Feeder) provides a powerful way to inject custom video data into DeepStream pipelines. Key points:

1. Implement `BufferProvider.generate()` to create custom buffers
2. Use `Feeder` to attach provider to `appsrc` elements
3. Convert data to DeepStream format using `as_tensor()` and `wrap()`
4. Return empty `Buffer()` to signal end of stream
5. Always set correct format properties (`width`, `height`, `framerate`, etc.)
6. Use GPU memory for optimal performance

This API enables seamless integration of custom video sources with DeepStream's powerful inference and analytics capabilities.

---

# Part 2: BufferRetriever / Receiver API (Frame Selector)

## Overview

The Frame Selector API (implemented through `BufferRetriever` and `Receiver` classes) enables extraction of video frames and buffers from DeepStream pipelines. This is useful for:
- Extracting frames for custom processing outside the pipeline
- Saving frames to disk or sending to external systems
- Collecting inference results with frame data
- Implementing custom frame selection logic
- Transferring data between multiple pipelines

## Core Concepts

### BufferRetriever
A `BufferRetriever` is a user-implemented class that consumes buffers from the pipeline. It works with GStreamer's `appsink` element to extract data from the pipeline.

### Receiver
A `Receiver` is a wrapper that connects a `BufferRetriever` to an `appsink` element. It manages the signal handling for "new-sample" events.

### Data Flow
```text
Pipeline -> appsink -> Receiver -> BufferRetriever.consume()
```

## API Reference

### BufferRetriever Class

Base class for implementing custom buffer consumers.

**Methods to Override**:

#### `consume(buffer)`
Process a buffer received from the pipeline.

**Parameters**:
- `buffer` (Buffer): Buffer object containing frame data

**Returns**: int (1 for success, 0 or negative for error/stop)

**Example**:
```python
from pyservicemaker import BufferRetriever
import torch  # pip install torch torchvision (not in base DS container)

class MyBufferRetriever(BufferRetriever):
    def __init__(self):
        super().__init__()
        self.frame_count = 0

    def consume(self, buffer):
        # Extract tensor from buffer at index 0
        tensor = buffer.extract(0)

        # Clone to prevent data loss
        tensor_copy = tensor.clone()

        # Convert to PyTorch for processing
        torch_tensor = torch.utils.dlpack.from_dlpack(tensor_copy)

        # Process the frame
        print(f"Received frame {self.frame_count}: shape={torch_tensor.shape}")

        self.frame_count += 1
        return 1  # Success
```

### Receiver Class

Wrapper for attaching a BufferRetriever to a pipeline element.

**Constructor**:
```python
from pyservicemaker import Receiver

receiver = Receiver("receiver-name", buffer_retriever_instance)
```

**Parameters**:
- `name` (str): Name of the receiver
- `retriever` (BufferRetriever): BufferRetriever instance

### Buffer Class Methods

**Methods**:

#### `extract(index)`
Extract tensor at specified index from the buffer.

**Parameters**:
- `index` (int): Batch index (usually 0 for single-stream)

**Returns**: Tensor object (DLPack format)

#### `clone()`
Create a copy of the tensor to prevent data corruption.

**Returns**: Cloned tensor

**Example**:
```python
def consume(self, buffer):
    # Extract and clone in one step
    tensor = buffer.extract(0).clone()

    # Now safe to use tensor asynchronously
    torch_tensor = torch.utils.dlpack.from_dlpack(tensor)
    return 1
```

## Implementation Patterns

### Pattern 1: Frame Extraction and Saving

Extract frames from pipeline and save to disk.

```python
from pyservicemaker import Pipeline, BufferRetriever, Receiver
import torch  # pip install torch torchvision (not in base DS container)
import cv2  # pip install opencv-python-headless (not in base DS container)
import numpy as np
import platform
from multiprocessing import Process

class FrameSaver(BufferRetriever):
    def __init__(self, output_dir="./frames", save_interval=30):
        super().__init__()
        self.output_dir = output_dir
        self.save_interval = save_interval
        self.frame_count = 0

        import os
        os.makedirs(output_dir, exist_ok=True)

    def consume(self, buffer):
        # Extract and clone buffer
        tensor = buffer.extract(0).clone()

        # Save every Nth frame
        if self.frame_count % self.save_interval == 0:
            # Convert to PyTorch tensor
            torch_tensor = torch.utils.dlpack.from_dlpack(tensor)

            # Move to CPU and convert to numpy
            frame_np = torch_tensor.cpu().numpy()

            # Convert RGB to BGR for OpenCV
            frame_bgr = cv2.cvtColor(frame_np, cv2.COLOR_RGB2BGR)

            # Save frame
            filename = f"{self.output_dir}/frame_{self.frame_count:06d}.jpg"
            cv2.imwrite(filename, frame_bgr)
            print(f"Saved: {filename}")

        self.frame_count += 1
        return 1

def extract_frames(video_uri, output_dir):
    pipeline = Pipeline("frame-extractor")

    # Source
    pipeline.add("nvurisrcbin", "src", {"uri": video_uri})

    # Muxer
    pipeline.add("nvstreammux", "mux", {
        "batch-size": 1,
        "width": 1920,
        "height": 1080
    })

    # Convert to RGB for extraction
    pipeline.add("nvvideoconvert", "converter")
    pipeline.add("capsfilter", "caps", {
        "caps": "video/x-raw(memory:NVMM), format=RGB"
    })

    # Sink for extraction
    pipeline.add("appsink", "sink", {
        "emit-signals": True,
        "sync": False
    })

    # Link elements
    pipeline.link(("src", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "converter", "caps", "sink")

    # Attach retriever
    retriever = FrameSaver(output_dir, save_interval=30)
    pipeline.attach("sink", Receiver("receiver", retriever), tips="new-sample")

    # Run
    pipeline.start().wait()

if __name__ == "__main__":
    import sys
    process = Process(target=extract_frames, args=(sys.argv[1], "./output_frames"))
    try:
        process.start()
        process.join()
    except KeyboardInterrupt:
        process.terminate()
```

### Pattern 2: Frame Queue Transfer

Transfer frames from one pipeline to another using a queue.

> **CRITICAL WARNING: Queue Type Selection**
>
> When transferring data between **threads**, use `queue.Queue` (from `queue` module).
> When transferring data between **processes**, use `multiprocessing.Queue`.
>
> Using `queue.Queue` with `multiprocessing.Process` will silently fail - data put into the queue in a child process will NEVER reach the parent process! This is a common bug that causes pipelines to appear running but produce no output.
>
> See the Best Practices reference for Anti-Pattern 4 with detailed examples.

```python
from pyservicemaker import Pipeline, BufferRetriever, Receiver, BufferProvider, Feeder
import torch  # pip install torch torchvision (not in base DS container)
from queue import Queue, Empty  # Use for THREADING only!
# from multiprocessing import Queue  # Use this for MULTIPROCESSING!
import threading

class QueuedRetriever(BufferRetriever):
    def __init__(self, frame_queue):
        super().__init__()
        self.queue = frame_queue
        self.count = 0

    def consume(self, buffer):
        # Extract and clone
        tensor = buffer.extract(0).clone()

        # Put in queue for other pipeline
        self.queue.put(tensor)

        self.count += 1
        print(f"Queued frame {self.count}")
        return 1

class QueuedProvider(BufferProvider):
    def __init__(self, frame_queue, width=1280, height=720):
        super().__init__()
        self.queue = frame_queue
        self.format = "RGB"
        self.width = width
        self.height = height
        self.framerate = 30
        self.device = 'gpu'

    def generate(self, size):
        try:
            tensor = self.queue.get(timeout=2)
            torch_tensor = torch.utils.dlpack.from_dlpack(tensor)

            from pyservicemaker import as_tensor, ColorFormat
            ds_tensor = as_tensor(torch_tensor, "HWC")
            return ds_tensor.wrap(ColorFormat.RGB)
        except Empty:
            from pyservicemaker import Buffer
            return Buffer()

def source_pipeline(uri, queue):
    """Extract frames from source and queue them"""
    pipeline = Pipeline("source-pipeline")

    pipeline.add("nvurisrcbin", "src", {"uri": uri})
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1280, "height": 720})
    pipeline.add("nvvideoconvert", "converter")
    pipeline.add("capsfilter", "caps", {"caps": "video/x-raw(memory:NVMM), format=RGB"})
    pipeline.add("appsink", "sink", {"emit-signals": True, "sync": False})

    pipeline.link(("src", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "converter", "caps", "sink")

    retriever = QueuedRetriever(queue)
    pipeline.attach("sink", Receiver("receiver", retriever), tips="new-sample")

    pipeline.start().wait()

def destination_pipeline(queue):
    """Consume frames from queue and process"""
    pipeline = Pipeline("dest-pipeline")

    provider = QueuedProvider(queue, width=1280, height=720)

    caps = "video/x-raw(memory:NVMM), format=RGB, width=1280, height=720, framerate=30/1"
    pipeline.add("appsrc", "src", {"caps": caps, "do-timestamp": True})
    pipeline.add("nvvideoconvert", "convert", {"nvbuf-memory-type": 2})
    pipeline.add("capsfilter", "caps2", {"caps": "video/x-raw(memory:NVMM), format=NV12"})
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1280, "height": 720})
    pipeline.add("nvinfer", "infer", {"config-file-path": "/path/to/config.yml"})
    pipeline.add("nvosdbin", "osd")
    pipeline.add("nveglglessink", "sink")

    pipeline.link("src", "convert", "caps2")
    pipeline.link(("convert", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "infer", "osd", "sink")

    pipeline.attach("src", Feeder("feeder", provider), tips="need-data/enough-data")

    pipeline.start().wait()

def multi_pipeline_transfer(video_uri, use_multiprocessing=False):
    """
    Transfer frames between pipelines.

    IMPORTANT: Queue type must match execution model:
    - Threading: use queue.Queue
    - Multiprocessing: use multiprocessing.Queue

    Args:
        video_uri: Video source URI
        use_multiprocessing: If True, use processes (requires multiprocessing.Queue)
    """
    if use_multiprocessing:
        from multiprocessing import Queue as MPQueue, Process
        queue = MPQueue(maxsize=10)  # MUST use multiprocessing.Queue!

        # Run pipelines in separate processes
        proc1 = Process(target=source_pipeline, args=(video_uri, queue))
        proc2 = Process(target=destination_pipeline, args=(queue,))

        proc1.start()
        proc2.start()

        proc2.join()
        proc1.join()
    else:
        # Threading approach - queue.Queue works fine here
        queue = Queue(maxsize=10)

        # Run both pipelines in threads (same process, shared memory)
        thread1 = threading.Thread(target=source_pipeline, args=(video_uri, queue))
        thread2 = threading.Thread(target=destination_pipeline, args=(queue,))

        thread1.start()
        thread2.start()

        thread2.join()
        thread1.join()
```

### Pattern 3: Selective Frame Capture

Capture frames based on inference results (e.g., when objects are detected).

```python
from pyservicemaker import Pipeline, BufferRetriever, Receiver, BatchMetadataOperator, Probe
import torch  # pip install torch torchvision (not in base DS container)
import cv2  # pip install opencv-python-headless (not in base DS container)
import numpy as np

class SelectiveFrameCapture(BufferRetriever):
    def __init__(self, output_dir="./captured", min_objects=1):
        super().__init__()
        self.output_dir = output_dir
        self.min_objects = min_objects
        self.frame_count = 0
        self.saved_count = 0
        self.capture_next = False

        import os
        os.makedirs(output_dir, exist_ok=True)

    def set_capture_flag(self, should_capture):
        """Called by metadata probe to signal capture"""
        self.capture_next = should_capture

    def consume(self, buffer):
        tensor = buffer.extract(0).clone()

        if self.capture_next:
            # Save this frame
            torch_tensor = torch.utils.dlpack.from_dlpack(tensor)
            frame_np = torch_tensor.cpu().numpy()
            frame_bgr = cv2.cvtColor(frame_np, cv2.COLOR_RGB2BGR)

            filename = f"{self.output_dir}/capture_{self.saved_count:06d}.jpg"
            cv2.imwrite(filename, frame_bgr)
            print(f"Captured frame {self.frame_count} with objects -> {filename}")

            self.saved_count += 1
            self.capture_next = False

        self.frame_count += 1
        return 1

class ObjectDetectionTrigger(BatchMetadataOperator):
    def __init__(self, frame_capture, min_objects=1):
        super().__init__()
        self.frame_capture = frame_capture
        self.min_objects = min_objects

    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            # Note: object_items is an ITERATOR - cannot use len() directly
            # Count by iterating
            obj_count = sum(1 for _ in frame_meta.object_items)

            if obj_count >= self.min_objects:
                # Signal frame capture to save this frame
                self.frame_capture.set_capture_flag(True)
                print(f"Detected {obj_count} objects, triggering capture")

def selective_capture(video_uri, config_path, output_dir):
    pipeline = Pipeline("selective-capture")

    # Source and muxer
    pipeline.add("nvurisrcbin", "src", {"uri": video_uri})
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})

    # Inference
    pipeline.add("nvinfer", "infer", {"config-file-path": config_path})

    # Convert for extraction
    pipeline.add("nvvideoconvert", "converter")
    pipeline.add("capsfilter", "caps", {"caps": "video/x-raw(memory:NVMM), format=RGB"})

    # Sink
    pipeline.add("appsink", "sink", {"emit-signals": True, "sync": False})

    # Link
    pipeline.link(("src", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "infer", "converter", "caps", "sink")

    # Attach frame capture
    frame_capture = SelectiveFrameCapture(output_dir, min_objects=2)
    pipeline.attach("sink", Receiver("receiver", frame_capture), tips="new-sample")

    # Attach metadata processor to trigger capture
    trigger = ObjectDetectionTrigger(frame_capture, min_objects=2)
    pipeline.attach("infer", Probe("trigger", trigger))

    pipeline.start().wait()
```

### Pattern 4: Flow API with Frame Retrieval

High-level Flow API for frame extraction.

```python
from pyservicemaker import Pipeline, Flow, BufferRetriever
import torch  # pip install torch torchvision (not in base DS container)
import cv2  # pip install opencv-python-headless (not in base DS container)
import numpy as np

class SimpleFrameRetriever(BufferRetriever):
    def __init__(self, save_path="output.jpg"):
        super().__init__()
        self.save_path = save_path
        self.count = 0

    def consume(self, buffer):
        if self.count == 0:  # Save first frame only
            tensor = buffer.extract(0).clone()
            torch_tensor = torch.utils.dlpack.from_dlpack(tensor)
            frame_np = torch_tensor.cpu().numpy()
            frame_bgr = cv2.cvtColor(frame_np, cv2.COLOR_RGB2BGR)
            cv2.imwrite(self.save_path, frame_bgr)
            print(f"Saved frame to {self.save_path}")

        self.count += 1
        return 1

def flow_api_retrieval(video_uri):
    pipeline = Pipeline("flow-retrieval")
    retriever = SimpleFrameRetriever("output_frame.jpg")

    # Flow API: batch_capture() -> retrieve()
    flow = Flow(pipeline)
    flow.batch_capture([video_uri])
    flow.retrieve(retriever)
    flow()
```

### Pattern 5: Frame Analysis and Logging

Extract frames with metadata for analysis.

```python
from pyservicemaker import Pipeline, BufferRetriever, Receiver, BatchMetadataOperator, Probe
import torch  # pip install torch torchvision (not in base DS container)
import json
from datetime import datetime

class FrameAnalyzer(BufferRetriever):
    def __init__(self, log_file="frame_analysis.json"):
        super().__init__()
        self.log_file = log_file
        self.frame_count = 0
        self.metadata_cache = {}

    def set_metadata(self, frame_num, metadata):
        """Called by metadata probe"""
        self.metadata_cache[frame_num] = metadata

    def consume(self, buffer):
        tensor = buffer.extract(0).clone()
        torch_tensor = torch.utils.dlpack.from_dlpack(tensor)

        # Calculate frame statistics
        mean_intensity = torch_tensor.float().mean().item()
        std_intensity = torch_tensor.float().std().item()

        # Get metadata if available
        metadata = self.metadata_cache.get(self.frame_count, {})

        # Log analysis
        analysis = {
            "frame_number": self.frame_count,
            "timestamp": datetime.now().isoformat(),
            "mean_intensity": mean_intensity,
            "std_intensity": std_intensity,
            "shape": list(torch_tensor.shape),
            "objects_detected": metadata.get("object_count", 0),
            "object_classes": metadata.get("classes", [])
        }

        with open(self.log_file, "a") as f:
            f.write(json.dumps(analysis) + "\n")

        # Clear cached metadata
        if self.frame_count in self.metadata_cache:
            del self.metadata_cache[self.frame_count]

        self.frame_count += 1
        return 1

class MetadataExtractor(BatchMetadataOperator):
    def __init__(self, frame_analyzer):
        super().__init__()
        self.frame_analyzer = frame_analyzer

    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            # Note: object_items is an ITERATOR - convert to list if you need
            # to access it multiple times or use len()
            objects = list(frame_meta.object_items)
            metadata = {
                "object_count": len(objects),
                "classes": [obj.class_id for obj in objects],
                "confidences": [obj.confidence for obj in objects]
            }
            self.frame_analyzer.set_metadata(frame_meta.frame_number, metadata)

def analyze_frames(video_uri, config_path):
    pipeline = Pipeline("frame-analyzer")

    # Source
    pipeline.add("nvurisrcbin", "src", {"uri": video_uri})
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})

    # Inference
    pipeline.add("nvinfer", "infer", {"config-file-path": config_path})

    # Convert and extract
    pipeline.add("nvvideoconvert", "converter")
    pipeline.add("capsfilter", "caps", {"caps": "video/x-raw(memory:NVMM), format=RGB"})
    pipeline.add("appsink", "sink", {"emit-signals": True, "sync": False})

    # Link
    pipeline.link(("src", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "infer", "converter", "caps", "sink")

    # Attach analyzer
    analyzer = FrameAnalyzer("analysis_log.json")
    pipeline.attach("sink", Receiver("receiver", analyzer), tips="new-sample")

    # Attach metadata extractor
    extractor = MetadataExtractor(analyzer)
    pipeline.attach("infer", Probe("extractor", extractor))

    pipeline.start().wait()
```

### Pattern 6: Real-time Frame Streaming

Stream frames to external system (e.g., web server, cloud service).

```python
from pyservicemaker import Pipeline, BufferRetriever, Receiver
import torch  # pip install torch torchvision (not in base DS container)
import cv2  # pip install opencv-python-headless (not in base DS container)
import numpy as np
import base64
import requests

class FrameStreamer(BufferRetriever):
    def __init__(self, endpoint_url, stream_interval=1):
        super().__init__()
        self.endpoint_url = endpoint_url
        self.stream_interval = stream_interval
        self.frame_count = 0

    def consume(self, buffer):
        # Stream every Nth frame
        if self.frame_count % self.stream_interval == 0:
            tensor = buffer.extract(0).clone()
            torch_tensor = torch.utils.dlpack.from_dlpack(tensor)
            frame_np = torch_tensor.cpu().numpy()

            # Encode as JPEG
            frame_bgr = cv2.cvtColor(frame_np, cv2.COLOR_RGB2BGR)
            _, jpeg_buffer = cv2.imencode('.jpg', frame_bgr, [cv2.IMWRITE_JPEG_QUALITY, 85])

            # Encode as base64
            jpeg_base64 = base64.b64encode(jpeg_buffer).decode('utf-8')

            # Send to endpoint
            try:
                response = requests.post(
                    self.endpoint_url,
                    json={
                        "frame_number": self.frame_count,
                        "image": jpeg_base64
                    },
                    timeout=1
                )
                if response.status_code == 200:
                    print(f"Streamed frame {self.frame_count}")
            except Exception as e:
                print(f"Failed to stream frame {self.frame_count}: {e}")

        self.frame_count += 1
        return 1

def stream_frames(video_uri, endpoint_url):
    pipeline = Pipeline("frame-streamer")

    pipeline.add("nvurisrcbin", "src", {"uri": video_uri})
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1280, "height": 720})
    pipeline.add("nvvideoconvert", "converter")
    pipeline.add("capsfilter", "caps", {"caps": "video/x-raw(memory:NVMM), format=RGB"})
    pipeline.add("appsink", "sink", {"emit-signals": True, "sync": False})

    pipeline.link(("src", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "converter", "caps", "sink")

    streamer = FrameStreamer(endpoint_url, stream_interval=10)
    pipeline.attach("sink", Receiver("receiver", streamer), tips="new-sample")

    pipeline.start().wait()
```

## Part 2 Best Practices

### 1. Always Clone Buffers
```python
def consume(self, buffer):
    # ALWAYS clone to prevent data corruption
    tensor = buffer.extract(0).clone()
    # Now safe to use asynchronously
```

### 2. Signal Configuration
```python
# Always use "new-sample" signal for appsink
pipeline.attach("sink", Receiver("receiver", retriever), tips="new-sample")

# Enable signal emission on appsink
pipeline.add("appsink", "sink", {"emit-signals": True})
```

### 3. Synchronization Control
```python
# For frame extraction, usually disable sync
pipeline.add("appsink", "sink", {
    "emit-signals": True,
    "sync": False  # Don't block on frame rate
})

# For real-time processing, enable sync
pipeline.add("appsink", "sink", {
    "emit-signals": True,
    "sync": True  # Maintain real-time pacing
})
```

### 4. Return Value Handling
```python
def consume(self, buffer):
    try:
        # Process buffer
        tensor = buffer.extract(0).clone()
        # ... processing ...
        return 1  # Success, continue processing
    except Exception as e:
        print(f"Error: {e}")
        return 0  # Error, but continue
        # return -1  # Fatal error, stop pipeline
```

### 5. Memory Management
```python
class EfficientRetriever(BufferRetriever):
    def __init__(self):
        super().__init__()
        self.frame_buffer = []
        self.max_buffer_size = 100

    def consume(self, buffer):
        tensor = buffer.extract(0).clone()

        # Limit buffer size to prevent memory issues
        if len(self.frame_buffer) >= self.max_buffer_size:
            self.frame_buffer.pop(0)  # Remove oldest

        self.frame_buffer.append(tensor)
        return 1
```

### 6. Thread Safety
```python
import threading

class ThreadSafeRetriever(BufferRetriever):
    def __init__(self):
        super().__init__()
        self.lock = threading.Lock()
        self.frame_count = 0

    def consume(self, buffer):
        with self.lock:
            tensor = buffer.extract(0).clone()
            # Safe concurrent access
            self.frame_count += 1
        return 1
```

## Advanced Usage

### Multi-Batch Frame Extraction

Extract frames from multi-stream batches.

```python
class MultiBatchRetriever(BufferRetriever):
    def __init__(self, num_streams):
        super().__init__()
        self.num_streams = num_streams
        self.frame_counts = [0] * num_streams

    def consume(self, buffer):
        # Extract all streams in batch
        for stream_idx in range(self.num_streams):
            try:
                tensor = buffer.extract(stream_idx).clone()
                torch_tensor = torch.utils.dlpack.from_dlpack(tensor)

                # Process each stream
                print(f"Stream {stream_idx}, Frame {self.frame_counts[stream_idx]}")

                self.frame_counts[stream_idx] += 1
            except Exception as e:
                print(f"Error extracting stream {stream_idx}: {e}")

        return 1

def multi_stream_extraction(video_uris):
    pipeline = Pipeline("multi-stream-extract")

    # Add sources
    for i, uri in enumerate(video_uris):
        pipeline.add("nvurisrcbin", f"src{i}", {"uri": uri})

    # Muxer for batching
    pipeline.add("nvstreammux", "mux", {
        "batch-size": len(video_uris),
        "width": 1280,
        "height": 720
    })

    # Convert and extract
    pipeline.add("nvvideoconvert", "converter")
    pipeline.add("capsfilter", "caps", {"caps": "video/x-raw(memory:NVMM), format=RGB"})
    pipeline.add("appsink", "sink", {"emit-signals": True, "sync": False})

    # Link sources to muxer
    for i in range(len(video_uris)):
        pipeline.link((f"src{i}", "mux"), ("", "sink_%u"))

    pipeline.link("mux", "converter", "caps", "sink")

    # Attach multi-batch retriever
    retriever = MultiBatchRetriever(len(video_uris))
    pipeline.attach("sink", Receiver("receiver", retriever), tips="new-sample")

    pipeline.start().wait()
```

## Part 2 Common Use Cases

### 1. Frame Archival
Extract and save frames at regular intervals for archival purposes.

### 2. Thumbnail Generation
Extract keyframes to generate video thumbnails.

### 3. Object Detection Screenshots
Capture frames when specific objects are detected.

### 4. Video Quality Analysis
Extract frames for quality metrics computation.

### 5. Pipeline Debugging
Extract frames at various pipeline stages for debugging.

### 6. Data Collection
Collect frames and metadata for training dataset creation.

## Part 2 Troubleshooting

### Issue 1: No Frames Received
**Solution**: Ensure `emit-signals=True` is set on appsink, verify `tips="new-sample"` is set

### Issue 2: Data Corruption
**Solution**: Always call `.clone()` on extracted tensors before async processing

### Issue 3: Memory Leaks
**Solution**: Limit buffer accumulation, properly release tensors

### Issue 4: Performance Issues
**Solution**: Set `sync=False` on appsink, process frames asynchronously

### Issue 5: Missing Frames
**Solution**: Check return value (return 1 for success), ensure processing is fast enough

### Issue 6: Frames/Batches Not Reaching Downstream Processing (Queue Empty)
**Symptoms**:
- Pipeline runs without errors
- BufferRetriever.consume() is being called
- But downstream processing (VLM, Kafka, etc.) never receives data
- Queue appears to be empty in consumer thread/process

**Root Cause**: Using `queue.Queue` with `multiprocessing.Process`

**Solution**:
1. If using multiprocessing: Switch to `multiprocessing.Queue`
2. If process isolation not required: Use `threading.Thread` with `queue.Queue`
3. Set `use_multiprocessing=False` in your configuration

```python
# WRONG: queue.Queue with multiprocessing
from multiprocessing import Process
from queue import Queue  # Won't work across processes!

# CORRECT Option 1: Use multiprocessing.Queue
from multiprocessing import Process, Queue

# CORRECT Option 2: Use threading instead
import threading
from queue import Queue

# See the Best Practices reference for Anti-Pattern 4 details
```

## Part 2 Summary

The Frame Selector API (BufferRetriever/Receiver) provides powerful capabilities for extracting frames and data from DeepStream pipelines. Key points:

1. Implement `BufferRetriever.consume()` to process extracted buffers
2. Use `Receiver` to attach retriever to `appsink` elements
3. Always call `buffer.extract(0).clone()` to safely extract tensors
4. Return `1` for success, `0` for error (continue), `-1` for fatal error
5. Set `emit-signals=True` on appsink and use `tips="new-sample"`
6. Consider `sync=False` for non-real-time extraction

This API enables seamless extraction of frames, inference results, and metadata from DeepStream pipelines for custom processing, archival, or transfer to other systems.

# DeepStream Docker Containers Reference

## Overview

DeepStream Docker images are hosted on the NVIDIA NGC container registry (`nvcr.io`). They package all SDK dependencies (GStreamer, TensorRT, CUDA, models, sample streams) and require the NVIDIA Container Toolkit (`nvidia-container-toolkit`) for GPU access.

- **NGC catalog page**: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/deepstream
- **Official docs**: https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_docker_containers.html

---

## Available Containers

### dGPU (x86_64)

| Container | Pull Command | Description |
|-----------|-------------|-------------|
| **Samples** | `docker pull nvcr.io/nvidia/deepstream:9.1-samples-multiarch` | Runtime libraries, GStreamer plugins, reference apps, sample streams, models, configs. Best for running demos and deploying applications. |
| **Triton** | `docker pull nvcr.io/nvidia/deepstream:9.1-triton-multiarch` | Everything in samples + Triton Inference Server and dependencies + development environment. Use when Triton-based inference is needed or building custom DeepStream applications. |

### Jetson (ARM64/aarch64)

| Container | Pull Command | Description |
|-----------|-------------|-------------|
| **Samples** | `docker pull nvcr.io/nvidia/deepstream:9.1-samples-multiarch` | Runtime libraries, GStreamer plugins, reference apps, sample streams, models, configs. **Deployment only** — does not support development inside the container. |
| **Triton** | `docker pull nvcr.io/nvidia/deepstream:9.1-triton-multiarch` | Samples contents + devel libraries + Triton Inference Server backends. |

### dGPU on ARM (GH200, GB200, SBSA)

| Container | Pull Command | Description |
|-----------|-------------|-------------|
| **Triton ARM SBSA** | `docker pull nvcr.io/nvidia/deepstream:9.1-triton-sbsa-dgx-spark` | Triton Inference Server + development environment for ARM SBSA platforms. |

---

## Choosing the Right Image

| Use Case | Recommended Image |
|----------|-------------------|
| Running sample apps / demos | `9.1-samples-multiarch` |
| pyservicemaker Python applications | `9.1-triton-multiarch` |
| Triton Inference Server required | `9.1-triton-multiarch` |
| Custom Dockerfile base image | `9.1-samples-multiarch` (minimal) or `9.1-triton-multiarch` (with Triton) |

---

## NGC Authentication

Pulling images requires NGC authentication:

```bash
# 1. Get an API key from https://ngc.nvidia.com
# 2. Log in to the NGC registry
docker login nvcr.io
# Username: $oauthtoken
# Password: <YOUR_NGC_API_KEY>
```

---

## Installing pyservicemaker Inside the Container

The `pyservicemaker` Python wheel is **bundled** in the container but **NOT pre-installed**. You must install it explicitly:

```bash
pip install /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl \
    pyyaml
```

In a Dockerfile:

```dockerfile
RUN pip install --break-system-packages \
    /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl \
    pyyaml
```

> **Note**: The `--break-system-packages` flag is needed on Ubuntu 24.04 (Python 3.12) to install into the system Python environment. Alternatively, use a virtual environment.

### Installing pyservicemaker in a Python Virtual Environment

If pyservicemaker code runs from a venv, install the bundled wheel into that
venv first:

```bash
python3 -m venv venv
source venv/bin/activate
pip install /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl \
    pyyaml
pip install -r requirements.txt
```

---

## Running Containers

### Prerequisites

1. **Docker**: Install `docker-ce` via [official instructions](https://docs.docker.com/engine/install)
2. **NVIDIA Container Toolkit**: Install via [install guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html)
3. **NVIDIA Driver**: 590+ for dGPU

### Basic Run (with display)

```bash
export DISPLAY=:0
xhost +si:localuser:root

docker run -it --rm \
    --gpus all \
    -e DISPLAY=$DISPLAY \
    -v /tmp/.X11-unix/:/tmp/.X11-unix \
    nvcr.io/nvidia/deepstream:9.1-triton-multiarch
```

### Headless Run (no display)

```bash
docker run -it --rm \
    --gpus all \
    nvcr.io/nvidia/deepstream:9.1-triton-multiarch
```

> For headless mode, use `fakesink` instead of `nveglglessink`/`nv3dsink` in your pipeline, or output to a file with `filesink`.

### Run with Custom Video File

```bash
docker run -it --rm \
    --gpus all \
    -e DISPLAY=$DISPLAY \
    -v /tmp/.X11-unix/:/tmp/.X11-unix \
    -v /path/to/videos:/data \
    nvcr.io/nvidia/deepstream:9.1-triton-multiarch
```

---

## Building Custom Docker Images

Use a DeepStream image as the base for your application:

```dockerfile
FROM nvcr.io/nvidia/deepstream:9.1-triton-multiarch

# Install pyservicemaker
RUN pip install --break-system-packages \
    /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl \
    pyyaml

# Copy application files
WORKDIR /app
COPY my_app.py .
COPY my_config.yml .

# Enable video driver libraries at runtime (encode/decode)
ENV NVIDIA_DRIVER_CAPABILITIES=${NVIDIA_DRIVER_CAPABILITIES},video

ENTRYPOINT ["python3", "my_app.py"]
```

### Build and Run

```bash
# Build
docker build -t my-ds-app .

# Run with display
docker run --rm --gpus all \
    -e DISPLAY=$DISPLAY \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    my-ds-app

# Run with RTSP source (no display needed)
docker run --rm --gpus all \
    my-ds-app rtsp://camera-ip/stream
```

---

## Additional Packages

DeepStream containers do **not** include certain multimedia libraries by default. Install them if needed:

### Audio/Codec Support

```bash
# Run the bundled install script for common multimedia packages
/opt/nvidia/deepstream/deepstream/user_additional_install.sh

# Or install specific packages manually
apt-get install -y gstreamer1.0-libav gstreamer1.0-plugins-good \
    gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly
```

### ffmpeg (for sample video preparation scripts)

```bash
apt-get install --reinstall libflac8 libmp3lame0 libxvidcore4 ffmpeg
```

### Kafka Support (librdkafka)

```bash
apt-get install -y librdkafka-dev
```

### Tracker Support (libmosquitto)

```bash
apt-get install -y libmosquitto1
```

---

## Important Paths Inside the Container

| Path | Contents |
|------|----------|
| `/opt/nvidia/deepstream/deepstream/` | DeepStream SDK root |
| `/opt/nvidia/deepstream/deepstream/samples/models/` | Sample models (Primary_Detector, Secondary_*, etc.) |
| `/opt/nvidia/deepstream/deepstream/samples/streams/` | Sample video streams (e.g., `sample_1080p_h264.mp4`) |
| `/opt/nvidia/deepstream/deepstream/samples/configs/` | Sample configuration files |
| `/opt/nvidia/deepstream/deepstream/lib/` | DeepStream libraries (GStreamer plugins, protocol adapters) |
| `/opt/nvidia/deepstream/deepstream/lib/gst-plugins/` | GStreamer plugin `.so` files |
| `/opt/nvidia/deepstream/deepstream/service-maker/python/` | pyservicemaker wheel file |

---

## Environment Variables

| Variable | Purpose | Example |
|----------|---------|---------|
| `GST_PLUGIN_PATH` | GStreamer plugin search path | `/opt/nvidia/deepstream/deepstream/lib/gst-plugins` |
| `LD_LIBRARY_PATH` | Shared library search path | `/opt/nvidia/deepstream/deepstream/lib:$LD_LIBRARY_PATH` |
| `GST_DEBUG` | GStreamer debug log level | `3` (INFO) or `nvinfer:5` (plugin-specific) |
| `NVIDIA_DRIVER_CAPABILITIES` | GPU capabilities exposed | `${NVIDIA_DRIVER_CAPABILITIES},video` |
| `DISPLAY` | X11 display for rendering sinks | `:0` |

---

## Common Docker Issues

### `ModuleNotFoundError: No module named 'pyservicemaker'`

**Cause**: The wheel is bundled but not installed.

**Fix**: Add to Dockerfile:
```dockerfile
RUN pip install --break-system-packages \
    /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl \
    pyyaml
```

### Display sinks fail with `Could not open display`

**Cause**: X11 forwarding not configured.

**Fix**: Pass display environment and socket:
```bash
docker run --rm --gpus all \
    -e DISPLAY=$DISPLAY \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    my-ds-app
```

Or use `fakesink` / `filesink` for headless operation.

### Pipeline exits early during non-interactive `docker exec`

**Cause**: Running without stdin can attach `/dev/null`; the GLib main loop may observe EOF and
stop after the first frames.

**Fix**: Use `docker exec -i` for non-interactive pipeline scripts:

```bash
docker exec -i ds python3 /app/pipeline.py http://localhost:8080/sample.mp4
```

If the application must run without inherited stdin, install a pipe before importing
`pyservicemaker`:

```python
import os
_pipe_r, _pipe_w = os.pipe()
os.dup2(_pipe_r, 0)
os.close(_pipe_r)
```

### `Failed to load plugin ... libnvds_kafka_proto.so`

**Cause**: `librdkafka` not installed (not bundled in the container).

**Fix**: Add to Dockerfile:
```dockerfile
RUN apt-get update && apt-get install -y librdkafka-dev && rm -rf /var/lib/apt/lists/*
```

### Warning about audio decoder not available

**Cause**: Multimedia codec packages removed in DeepStream containers.

**Fix**:
```dockerfile
RUN /opt/nvidia/deepstream/deepstream/user_additional_install.sh
```

# DeepStream GStreamer Plugins Overview

## Introduction

DeepStream provides a comprehensive set of custom GStreamer plugins optimized for NVIDIA GPUs. These plugins handle video decoding, inference, tracking, visualization, and various other video analytics tasks. Understanding these plugins is crucial for building effective DeepStream applications.

## Plugin Categories

### Source Plugins
Plugins that generate or capture video data from various sources.

### Processing Plugins
Plugins that transform, analyze, or process video data.

### Sink Plugins
Plugins that output video to displays, files, or network destinations.

---

## Source Plugins

### nvv4l2decoder
**Purpose**: Hardware-accelerated video decoder using NVIDIA V4L2 API (from nvvideo4linux2 plugin)

**Key Properties**:
- `capture-io-mode`: Capture I/O mode for the sink pad (`auto`, `mmap`, `dmabuf-import`)
- `output-io-mode`: Output I/O mode for the src pad (`auto`, `mmap`, `dmabuf-import`)
- `cudadec-memtype`: CUDA buffer memory type (`memtype_device`, `memtype_pinned`, `memtype_unified`)
- `gpu-id`: GPU device ID used for decoding
- `drop-frame-interval`: Interval for dropping frames (0 keeps all frames)
- `num-extra-surfaces`: Additional decode surfaces to allocate
- `disable-dpb`: Disable DPB buffers to reduce latency
- `low-latency-mode`: Enable low-latency decoding for I/IPPP streams
- `skip-frames`: Frame skipping policy (`decode_all`, `decode_non_ref`, `decode_key`)
- `device`: Decoder device path (read-only, default `/dev/nvidia0`)

**Usage**:
```bash
nvv4l2decoder output-io-mode=0 drop-frame-interval=0
```

**Common Pipeline Pattern**:
```
h264parse ! nvv4l2decoder ! ...
```

**Output Format**:
- Outputs `video/x-raw(memory:NVMM)` - GPU memory format
- This is already in NVMM format, so NO nvvideoconvert is needed before nvstreammux

**Notes**:
- Essential for GPU-accelerated pipelines
- Supports H.264, H.265, VP8, VP9 codecs with zero-copy memory transfers
- Output is already in NVMM memory, compatible with nvstreammux and other DeepStream plugins

---

### nvurisrcbin
**Purpose**: Source bin for handling URI-based sources (files, RTSP, HTTP)

**Key Properties**:
- `uri`: Source URI (file://, rtsp://, http://, etc.)
- `num-buffers`: Number of buffers to process
- `drop-on-latency`: Drop frames on latency

**Usage**:
```bash
nvurisrcbin uri=file:///path/to/video.mp4
```

**Common Pipeline Pattern**:
```
nvurisrcbin uri=rtsp://camera-ip/stream ! ...
```

**Notes**:
- Automatically handles demuxing and parsing for multiple protocols and formats

---

### nvmultiurisrcbin
**Purpose**: Source bin with built-in REST API server for dynamic multi-stream management

**Key Properties**:
| Property | Type | Description |
|----------|------|-------------|
| `uri-list` | string | Comma-separated list of initial URIs |
| `sensor-id-list` | string | Comma-separated sensor IDs (maps 1:1 with uri-list) |
| `sensor-name-list` | string | Comma-separated sensor names |
| `ip-address` | string | REST API server IP (default: localhost) |
| `port` | int | REST API server port (default: 9000, 0 to disable) |
| `max-batch-size` | int | Maximum number of sources |
| `batched-push-timeout` | int | Timeout in microseconds to push batch |
| `live-source` | int | Set to 1 for live/dynamic sources (REQUIRED) |
| `drop-pipeline-eos` | int | Set to 1 to keep pipeline alive when sources removed |
| `async-handling` | int | Set to 1 for async state changes |
| `select-rtp-protocol` | int | 0=UDP+TCP auto, 4=TCP only |
| `latency` | int | Jitterbuffer size in ms for RTSP |

**Built-in REST API Endpoints**:
- `POST /api/v1/stream/add` - Add a stream dynamically
- `POST /api/v1/stream/remove` - Remove a stream
- `GET /api/v1/stream/get-stream-info` - Get current streams

**Usage**:
```python
# Pipeline with built-in REST server on port 9000
pipeline.add("nvmultiurisrcbin", "src", {
    "port": 9000,
    "max-batch-size": 16,
    "live-source": 1,
    "drop-pipeline-eos": 1,
    "async-handling": 1,
})
# REST API automatically available at http://localhost:9000/api/v1/
```

**CRITICAL for Dynamic Sources**:
When using dynamic source addition, the sink element MUST have `async=0`:
```python
pipeline.add("nveglglessink", "sink", {
    "sync": 0,
    "qos": 0,
    "async": 0  # CRITICAL - prevents state transition deadlock
})
```

**Notes**:
- Integrates nvds_rest_server, nvurisrcbin, and nvstreammux in one bin
- Do NOT implement custom Flask/FastAPI server - use built-in REST API
- See `rest_api_dynamic.md` for complete REST API documentation

---

### nvdsdynamicsrcbin
**Purpose**: Source bin for programmatically adding and removing file/URI-based video sources at runtime. Unlike `nvmultiurisrcbin` (REST API / config-driven), `nvdsdynamicsrcbin` is controlled entirely through code using `SourceManager`.

**CRITICAL**: `nvdsdynamicsrcbin` does **not** manage sources on its own. You **must** use `SourceManager` from `pyservicemaker._pydeepstream.signal` to add, remove, and terminate sources. Without `SourceManager`, the bin has no way to receive source URIs.

**Key Properties**:
| Property | Type | Default | Description |
|----------|------|---------|-------------|
| `gpu-id` | uint | 0 | GPU Device ID to use for decoding |
| `message-forward` | bool | False | Forward all children messages to the pipeline bus (required for EOS detection) |
| `async-handling` | bool | False | Handle asynchronous state changes internally |
| `current-file` | string (read-only) | null | Currently processing file path |
| `current-id` | int (read-only) | -1 | ID of the chunk currently being processed |

**Element Actions** (triggered via `SourceManager`):
| Action | Description |
|--------|-------------|
| `add-source` | Add a new file/URI source to the bin |
| `remove-source` | Remove a source by its unique ID |
| `terminate` | Signal no more sources will be added; sends EOS after all finish |

**Internal Children**: Contains `parsebin`, `queue_parsebin`, and `decoder` — it automatically parses and decodes the added sources.

---

### v4l2src
**Purpose**: Video4Linux2 source for USB cameras

**Key Properties**:
- `device`: Device path (e.g., `/dev/video0`)
- `io-mode`: I/O mode
- `do-timestamp`: Enable timestamping

**Usage**:
```bash
v4l2src device=/dev/video0 ! ...
```

**Notes**:
- Standard GStreamer plugin for USB webcams, may require format conversion

---

### nvarguscamerasrc
**Purpose**: NVIDIA camera source for Jetson CSI cameras

**Key Properties**:
- `sensor-id`: Sensor ID (0, 1, etc.)
- `sensor-mode`: Sensor mode
- `wbmode`: White balance mode
- `exposuretimerange`: Exposure time range
- `gainrange`: Gain range

**Usage**:
```bash
nvarguscamerasrc sensor-id=0 ! ...
```

**Notes**:
- Jetson-specific plugin optimized for CSI cameras with hardware-accelerated capture

---

## Processing Plugins

### nvstreammux
**Purpose**: Batches multiple video streams into a single batch for efficient inference

**IMPORTANT**: There are TWO versions of nvstreammux:
- **OLD nvstreammux**: Default, uses GObject properties for configuration
- **NEW nvstreammux**: Enabled with `USE_NEW_NVSTREAMMUX=yes`, uses config file for advanced settings

**Key Properties (NEW nvstreammux - RECOMMENDED)**:
- `batch-size`: Maximum number of buffers in a batch
- `batched-push-timeout`: Timeout for batching in microseconds (default: 33000)
- `config-file-path`: Path to configuration file for advanced settings
- `num-surfaces-per-frame`: Number of surfaces per frame
- `attach-sys-ts`: Attach system timestamp as NTP timestamp (boolean)
- `max-latency`: Maximum latency in live mode (nanoseconds)
- `sync-inputs`: Force synchronization of input frames (boolean)
- `frame-num-reset-on-eos`: Reset frame numbers on EOS (boolean)
- `frame-num-reset-on-stream-reset`: Reset frame numbers on stream reset (boolean)
- `frame-duration`: Duration of input frames in milliseconds for NTP correction
- `drop-pipeline-eos`: Don't propagate EOS downstream when all pads are at EOS (boolean)

**Key Properties (OLD nvstreammux - Legacy)**:
- `batch-size`: Number of streams to batch
- `width`: Output batch width
- `height`: Output batch height
- `gpu-id`: GPU ID for processing
- `batched-push-timeout`: Timeout for batching (microseconds)
- `enable-padding`: Enable padding for different resolutions
- `nvbuf-memory-type`: Memory type (0=default, 1=NVMM, 2=unified)

**Usage**:
```bash
nvstreammux name=m batch-size=4 width=1920 height=1080
```

**Common Pipeline Pattern**:
```
source1 ! m.sink_0 source2 ! m.sink_1 nvstreammux name=m batch-size=2 ! ...
```

**Notes**:
- **Critical plugin** for multi-stream applications
- **NEW nvstreammux** (recommended): More flexible, uses config file for width/height/memory-type settings
- **OLD nvstreammux**: Uses GObject properties for width/height, may be deprecated in future
- To use NEW version: Set environment variable `USE_NEW_NVSTREAMMUX=yes` before running pipeline
- Batch size should match number of input streams
- NEW version infers output resolution from downstream elements or uses config file

---

### nvstreamdemux
**Purpose**: Demultiplexes batched streams back to individual streams

**Key Properties**:
- `name`: Element name (required for pad access)

**Usage**:
```bash
nvstreamdemux name=d
```

**Common Pipeline Pattern**:
```
nvstreammux name=m ! ... ! nvstreamdemux name=d d.src_0 ! ... d.src_1 ! ...
```

**Notes**:
- Used after processing batched streams
- Provides separate source pads for each stream
- Essential for per-stream rendering or processing

---

### nvinfer
**Purpose**: TensorRT-based inference engine for deep learning models

**Key Properties**:
- `config-file-path`: Path to inference configuration file (supports **both** INI-style text format and YAML format)
- `batch-size`: Batch size for inference
- `gpu-id`: GPU ID for inference
- `unique-id`: Unique identifier for this inference instance
- `process-mode`: Infer processing mode (primary or secondary)
- `interval`: Number of consecutive batches to skip for inference
- `infer-on-gie-id`: Infer on metadata from GIE with this unique ID (-1 for all)
- `infer-on-class-ids`: Operate on objects with specified class IDs
- `filter-out-class-ids`: Ignore metadata for objects of specified class IDs
- `model-engine-file`: Path to pre-generated TensorRT engine file
- `output-tensor-meta`: Output raw tensor metadata (0=no, 1=yes)
- `output-instance-mask`: Output instance mask in metadata (0=no, 1=yes)
- `input-tensor-meta`: Use tensor metadata from upstream (0=no, 1=yes)
- `clip-object-outside-roi`: Clip object bbox outside ROI from nvdspreprocess
- `crop-objects-to-roi-boundary`: Crop object bbox to ROI boundary
- `raw-output-file-write`: Write raw inference output to file
- `raw-output-generated-callback`: Callback for raw output
- `raw-output-generated-userdata`: Userdata for raw output callback

**Configuration File Structure**:

nvinfer supports **two configuration formats**:

### Format 1: YAML Format (Recommended)

```yaml
# Example: pgie_config.yml (Primary detector using ResNet18)
property:
  gpu-id: 0
  net-scale-factor: 0.00392156862745098
  # Use ResNet18 TrafficCamNet model from DeepStream samples
  onnx-file: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx
  labelfile-path: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt
  batch-size: 1
  process-mode: 1
  model-color-format: 0
  # 0=FP32, 1=INT8, 2=FP16
  network-mode: 2
  num-detected-classes: 4
  interval: 0
  gie-unique-id: 1
  # 1=DBSCAN, 2=NMS, 3=DBSCAN+NMS, 4=None
  cluster-mode: 2

class-attrs-all:
  topk: 20
  nms-iou-threshold: 0.5
  pre-cluster-threshold: 0.2
```

### Format 2: INI-style Text Format

```ini
# Example: pgie_config.txt (Primary detector using ResNet18)
[property]
gpu-id=0
net-scale-factor=0.00392156862745098
onnx-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx
labelfile-path=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt
batch-size=1
process-mode=1
model-color-format=0
network-mode=2
num-detected-classes=4
interval=0
gie-unique-id=1
cluster-mode=2

[class-attrs-all]
topk=20
nms-iou-threshold=0.5
pre-cluster-threshold=0.2
```

**Key Differences**:
| Aspect | YAML Format | INI Format |
|--------|-------------|------------|
| File extension | `.yml` or `.yaml` | `.txt` |
| Section headers | `property:` (no brackets) | `[property]` (with brackets) |
| Key-value separator | `: ` (colon + space) | `=` (equals) |
| Indentation | Required for nested values | Not used |

**Usage**:
```bash
nvinfer config-file-path=/path/to/config.yml batch-size=4
```

**Common Pipeline Pattern**:
```
nvstreammux ! nvinfer config-file-path=pgie_config.txt ! ...
```

**Notes**:
- **Primary inference engine** for object detection/classification
- Supports TensorRT engines (.trt), ONNX models, and custom networks
- Can be used as Primary GIE (PGIE) or Secondary GIE (SGIE)
- Multiple instances can be cascaded for complex models
- `output-tensor-meta=1` enables custom postprocessing
- `input-tensor-meta=1` uses preprocessed tensors from nvdspreprocess
- **Note**: `enable-dbscan` is DEPRECATED and is a config file parameter, not a GObject property

---

### nvinferserver
**Purpose**: Inference using Triton Inference Server backend

**Key Properties**:
- `config-file-path`: Path to Triton configuration file
- `gpu-id`: GPU ID
- `unique-id`: Unique identifier
- `output-tensor-meta`: Output tensor metadata

**Usage**:
```bash
nvinferserver config-file-path=/path/to/triton_config.txt
```

**Notes**:
- Alternative to nvinfer for Triton-based inference
- Supports remote inference servers
- Better for scalable deployments
- Requires Triton Inference Server setup

---

### nvdspreprocess
**Purpose**: Custom preprocessing plugin for region-of-interest (ROI) preprocessing

**Key Properties**:
- `config-file`: Path to preprocessing configuration file
- `gpu-id`: GPU ID

**Configuration File Structure**:
```yaml
preprocess-config:
  - preprocess-group:
      target-unique-ids: [1]
      roi-params-src: [0]
      process-on-roi: 1
      network-input-shape: [1, 3, 544, 960]
      tensor-format: 0  # 0=NCHW, 1=NHWC
      maintain-aspect-ratio: 0
      custom-transform-function: "custom_transform"
      custom-tensor-prep-function: "custom_tensor_prep"
```

**Usage**:
```bash
nvdspreprocess config-file=/path/to/preprocess_config.yml
```

**Common Pipeline Pattern**:
```
nvstreammux ! nvdspreprocess config-file=preprocess.yml ! nvinfer input-tensor-meta=1 ! ...
```

**Notes**:
- Enables custom preprocessing before inference
- Processes ROIs or full frames
- Outputs tensor metadata for nvinfer
- Custom preprocessing library and functions are specified in the **config file**, not as GObject properties
- Optimal performance: batch-size should match total units in config

---

### nvdspostprocess
**Purpose**: Custom postprocessing plugin for parsing model outputs

**Key Properties**:
- `postprocesslib-name`: Path to postprocessing library (.so)
- `postprocesslib-config-file`: Path to postprocessing configuration file
- `gpu-id`: GPU ID

**Configuration File Structure** (YAML):
```yaml
postprocess-config:
  - postprocess-group:
      target-unique-ids: [1]
      custom-parse-function: "custom_parse"
      custom-bbox-parse-function: "custom_bbox_parse"
      output-format: 0  # 0=object detection, 1=classification
```

**Usage**:
```bash
nvdspostprocess postprocesslib-name=./libpostprocess.so postprocesslib-config-file=config.yml
```

**Common Pipeline Pattern**:
```
nvinfer output-tensor-meta=1 ! nvdspostprocess postprocesslib-name=... ! ...
```

**Notes**:
- Parses raw tensor outputs from nvinfer
- Requires nvinfer with output-tensor-meta=1
- Supports custom parsing functions
- Used for models not supported by nvinfer's built-in parsers

---

### nvtracker
**Purpose**: Multi-object tracker for tracking objects across frames

**Key Properties**:
- `ll-lib-file`: Path to low-level tracker library (.so)
- `ll-config-file`: Path to tracker configuration file
- `tracker-width`: Tracker input width
- `tracker-height`: Tracker input height
- `gpu-id`: GPU ID
- `input-tensor-meta`: Use tensor metadata (0=no, 1=yes)
- `tensor-meta-gie-id`: GIE ID for tensor metadata (used with input-tensor-meta)
- `display-tracking-id`: Display tracking ID in object text
- `tracking-id-reset-mode`: Tracking ID reset mode on stream reset/EOS
- `tracking-surface-type`: Selective tracking surface type
- `user-meta-pool-size`: Tracker user metadata buffer pool size
- `sub-batches`: Configuration of sub-batches for parallel processing
- `sub-batch-err-recovery-trial-cnt`: Max trials to reinitialize tracker on error

**Configuration File Structure**:
```yaml
tracker:
  ll-lib-file: /path/to/libnvds_nvmultiobjecttracker.so
  ll-config-file: /path/to/tracker_config.yml
  enable-batch-process: 1
  enable-past-frame: 1
  tracker-width: 1920
  tracker-height: 1080
```

**Usage**:
```bash
nvtracker ll-lib-file=/path/to/libnvds_nvmultiobjecttracker.so ll-config-file=/path/to/config.yml
```

**Common Pipeline Pattern**:
```
nvinfer ! nvtracker ll-lib-file=... ! ...
```

**Notes**:
- Tracks objects across video frames
- Assigns unique tracking IDs to objects
- Supports multiple tracking algorithms
- Requires object metadata from inference engine
- Tracker dimensions should match preprocess/infer dimensions when using input-tensor-meta=1

---

### nvdsosd (nvosdbin)
**Purpose**: On-Screen Display element (`nvdsosd`) and DeepStream convenience bin (`nvosdbin`) for drawing bounding boxes, labels, masks, and clocks

**Key Properties**:
- `gpu-id`: GPU ID to render on
- `process-mode`: Rendering backend (0=CPU, 1=GPU)
- `display-text`: Enable text overlay (boolean)
- `display-bbox`: Enable bounding box display (boolean)
- `display-mask`: Enable instance mask display (boolean)
- `display-clock`: Enable clock display (boolean)
- `clock-font`: Font for clock text
- `clock-font-size`: Font size for clock
- `x-clock-offset`: X offset for clock position
- `y-clock-offset`: Y offset for clock position
- `clock-color`: Clock color (RGBA as uint)
- `blur-bbox`: Enable bbox blurring (boolean)
- `blur-on-gie-class-ids`: Blur bboxes for specific GIE unique ID and class ID

**Note**: Text and bbox styling properties (like colors, borders) are controlled through object metadata, not as GObject properties on the plugin itself.

**Usage**:
```bash
nvdsosd display-text=1 display-bbox=1
```

**Common Pipeline Pattern**:
```
nvtracker ! nvdsosd ! ...
```

**Notes**:
- Use `nvdsosd` for the raw transform element
- Supports tracking ID display, text overlays, and optional blur/clocks
- Keeps surfaces in NVMM for zero-copy rendering on GPU
- Object-specific styling (text colors, bbox colors, etc.) is set through NvDsMeta object metadata, not plugin properties

---

### nvmultistreamtiler
**Purpose**: Tiles multiple video streams into a single output frame

**Key Properties**:
- `width`: Output width
- `height`: Output height
- `rows`: Number of rows in tile layout
- `columns`: Number of columns in tile layout
- `gpu-id`: GPU ID
- `show-source`: Show source index (0=no, 1=yes)

**Usage**:
```bash
nvmultistreamtiler width=1920 height=1080 rows=2 columns=2
```

**Common Pipeline Pattern**:
```
nvstreamdemux name=d d.src_0 ! ... d.src_1 ! ... ! nvmultistreamtiler ! ...
```

**Notes**:
- Combines multiple streams into a grid layout, useful for multi-stream visualization

---

### nvvideoconvert
**Purpose**: Video format converter (color space conversion, scaling)

**Key Properties**:
- `gpu-id`: GPU ID
- `nvbuf-memory-type`: Memory type
- `src-crop`: Source crop rectangle
- `dest-crop`: Destination crop rectangle

**Usage**:
```bash
nvvideoconvert gpu-id=0
```

**Common Pipeline Pattern**:
```
nvdsosd ! nvvideoconvert ! nveglglessink
```

**Notes**:
- GPU-accelerated color format conversion (NV12, RGBA, etc.), often needed before rendering sinks

---

### nvdsanalytics
**Purpose**: Video analytics plugin for motion detection, line crossing, etc.

**Key Properties**:
- `config-file`: Path to analytics configuration file
- `enable`: Enable analytics (0=no, 1=yes)
- `gpu-id`: GPU ID

**Configuration File Parameters**:
The config file **must** include a **property** group/section. Other groups define per-stream ROI, line-crossing, overcrowding, and direction rules. Stream index is given by the numeric suffix in the group name (e.g. `roi-filtering-stream-0` for stream 0).
- `property`: General group; Mandatory.
  - `config-width`,`config-height`:  Reference resolution width and height for analytics coordinate scaling.
  - `enable`: Whether analytics is enabled (aligned with the element **enable** property).
  - `display-font-size`: Optional; OSD font size.
  - `osd-mode`: Optional; 0, 1, or 2. 0 = OSD off, 1 = labels only, 2 = full (default).
  - `obj-cnt-win-in-ms`: Optional; object-count time window in milliseconds; range 1–1000000000.
  - `display-obj-cnt`: Optional; whether to show per-class object counts on OSD.
- `roi-filtering-stream-<stream_id>`: ROI Filtering group per stream
  - `enable`: Enable ROI filtering for this stream.
  - `class-id`: Class IDs to include in ROI analytics (semicolon-separated integer list).
  - `inverse-roi`: Whether treat as “outside ROI” for counting/filtering.
  - `roi-<label>`: ROI coordinations in polygon vertices: `x1;y1;x2;y2;...` (even number of integers). `<label>` is a custom name for the specified ROIs.
- `overcrowding-stream-<stream_id>`: Overcrowding object count and duration in ROIs per stream.
  - `enable`: Enable overcrowding analysis for this stream.
  - `class-id`:  Class IDs to count for overcrowding in integer list.
  - `object-threshold`: Object count threshold for overcrowding.
  - `time-threshold`: Duration threshold in milliseconds.
  - `roi-<label>`: Polygon vertices for the overcrowding region: `x1;y1;x2;y2;...`. `<label>` is a custom name for the specified ROIs.
- `line-crossing-stream-<stream_id>`: Line Crossing object count per stream.
  - `enable`: Enable line-crossing counting for this stream.
  - `extended`: Whether to use extended line-crossing logic. 
  - `class-id`: Class IDs to count for line crossing in integer list.
  - `line-crossing-<label>`: **8 integers:** direction vector (x1,y1,x2,y2) then line (x1,y1,x2,y2). Coordinates relative to config-width/config-height. `<label>` is a custom name for the specified lines.
  - `mode`: Detection strictness options: `strict`, `balanced`, or `loose`.
- `direction-detection-stream-<stream_id>`: Defines reference direction vectors for judging object movement direction per stream.
   - `enable`: Enable direction detection for this stream.
   - `class-id`: Class IDs of the objects which need direction detection.
   - `direction-<label>`: **8 integers:** direction vector (x1,y1,x2,y2) then line (x1,y1,x2,y2). `<label>` is a custom name for the specified directions.
   - `mode`: Direction detection mode options: `strict`, `balanced`, or `loose`.

**Notes**:
**<stream_id>** should be the stream id which be compatible for the source id identified by the nvstreammux sink pad id.
Each **roi-<label>** defines one ROI; multiple ROIs per stream are allowed.
Each **line-crossing-<label>** defines one line; multiple lines per stream are allowed.
Each **direction-<label>** defines one reference direction; multiple directions per stream are allowed.

**Configuration File Samples**:
There are two formats configuration files: .txt and .yml.
- YAML format:
```yaml
property:
  enable: 1
  config-width: 1920
  config-height: 1080
  display-font-size: 12
  osd-mode: 2
roi-filtering-stream-0:
  enable: 1
  class-id: -1
  roi-DOOR: 256;639;675;83;876;224;926;482;866;741
overcrowding-stream-0:
  enable: 1
  class-id: 1;2
  object-threshold: 1000
  roi-ENTRANCE: 282;347;987;843
line-crossing-stream-0:
  enable: 1
  line-crossing-Exit: 789;672;1084;900;851;773;1203;732
  class-id: 0
  mode: loose
direction-detection-stream-0:
  enable: 1
  direction-South: 284;840;360;662
  class-id: 0
```
- TXT format:
```txt
[property]
enable=1
config-width=1920
config-height=1080
osd-mode=2
display-font-size=12

[roi-filtering-stream-0]
enable=1
roi-RF=256;639;675;83;876;224;926;482;866;741
inverse-roi=0
class-id=-1

[overcrowding-stream-1]
enable=1
roi-OC=282;347;987;843
object-threshold=3
class-id=-1

[line-crossing-stream-0]
enable=1
line-crossing-Exit=789;672;1084;900;851;773;1203;732
class-id=0
mode=loose

[direction-detection-stream-0]
enable=1
direction-South=284;840;360;662
class-id=0
```

**Usage**:
```bash
nvdsanalytics config-file=/path/to/analytics_config.yml
```

**Notes**:
- Performs motion, line crossing, intrusion, and loitering detection; requires configuration file

---

### nvmsgbroker
**Purpose**: Message broker plugin for sending metadata to cloud services

**IMPORTANT**: `nvmsgbroker` is a **SINK component** that terminates the pipeline branch. It cannot have downstream components. If you need both message broker output and display, use `tee` to split the pipeline.

**Key Properties**:
- `proto-lib`: Path to protocol library (.so)
- `conn-str`: Connection string
- `config-file`: Configuration file path
- `topic`: Topic name (for Kafka/MQTT)
- `sync`: Synchronous mode (0=async, 1=sync)

**Usage**:
```bash
nvmsgbroker proto-lib=/path/to/libnvds_kafka_proto.so conn-str=localhost:9092 topic=analytics
```

**Pipeline Patterns**:
```bash
# Headless (Kafka only)
tracker ! nvmsgconv ! nvmsgbroker

# With display (use tee)
tracker ! tee name=t
t. ! queue ! nvmsgconv ! nvmsgbroker
t. ! queue ! tiler ! osd ! converter ! sink
```

**Notes**:
- **SINK component**: Terminates pipeline branch, cannot have downstream elements
- Sends metadata to cloud services
- Supports Kafka, MQTT, Azure, Redis, AMQP
- Requires protocol-specific library
- Can send object metadata, frame metadata, etc.
- For pipelines requiring both Kafka and display, use `tee` to create separate branches

---

### nvmsgconv
**Purpose**: Message converter plugin for transforming metadata formats

**Key Properties**:
- `msg2p-lib`: Payload generation library path with absolute path
- `payload-type`: Payload type (0=deepstream, 1=custom, etc.)
- `msg2p-newapi`: Use new API which supports multiple payloads (boolean)
- `frame-interval`: Interval for frame-level metadata generation
- `debug-payload-dir`: Directory to dump generated payloads for debugging

**Usage**:
```bash
nvmsgconv config-file=/path/to/msgconv_config.txt
```

**Notes**:
- Converts metadata to different formats
- Used before nvmsgbroker
- Supports custom schemas

---

## Sink Plugins

### nveglglessink
**Purpose**: EGL/GLES-based video renderer for x86_64 platforms

**Key Properties**:
- `sync`: Synchronize to display refresh (0=no, 1=yes)
- `window-x`: Window X position
- `window-y`: Window Y position
- `window-width`: Window width
- `window-height`: Window height
- `display-id`: Display ID

**Usage**:
```bash
nveglglessink sync=1
```

**Notes**:
- For x86_64 desktop/server platforms with hardware-accelerated rendering

---

### nv3dsink
**Purpose**: 3D video renderer for Jetson platforms

**Key Properties**:
- `sync`: Synchronize to display refresh
- `window-x`: Window X position
- `window-y`: Window Y position
- `window-width`: Window width
- `window-height`: Window height

**Usage**:
```bash
nv3dsink sync=1
```

**Notes**:
- For ARM64/Jetson platforms with hardware-accelerated rendering

---

### nvvideoconvert + filesink
**Purpose**: Save processed video to file

**Usage**:
```bash
nvvideoconvert ! x264enc ! mp4mux ! filesink location=output.mp4
```

**Notes**:
- Requires encoding before saving
- Can use hardware encoders (nvv4l2h264enc, nvv4l2h265enc)

---

## Standard GStreamer Plugins Used in DeepStream

### h264parse / h265parse
**Purpose**: Parse H.264/H.265 video streams

**Usage**:
```bash
h264parse
```

### queue
**Purpose**: Buffer management and synchronization

**Key Properties**:
- `max-size-buffers`: Maximum buffer size
- `max-size-time`: Maximum time-based size
- `leaky`: Leaky queue mode

**Usage**:
```bash
queue max-size-buffers=200
```

### tee
**Purpose**: Split pipeline into multiple branches

**Usage**:
```bash
tee name=t t. ! queue ! ... t. ! queue ! ...
```

---

## Plugin Selection Guidelines

### For Video Sources:
- **Files**: `nvurisrcbin` or `filesrc` + `qtdemux` + `h264parse`
- **RTSP Streams**: `nvurisrcbin` with `rtsp://` URI
- **Dynamic sources (REST API)**: `nvmultiurisrcbin` — config/REST-driven multi-stream
- **Dynamic sources (programmatic)**: `nvdsdynamicsrcbin` + `SourceManager` — script-driven add/remove
- **USB Cameras**: `v4l2src`
- **Jetson CSI Cameras**: `nvarguscamerasrc`

### For Decoding:
- **Always use**: `nvv4l2decoder` for hardware acceleration
- **Avoid**: Software decoders (avdec_h264, etc.) for performance

### For Multi-Stream:
- **Always use**: `nvstreammux` to batch streams
- **Batch size**: Match number of input streams
- **Use**: `nvstreamdemux` after processing to split streams

### For Inference:
- **Primary**: `nvinfer` for TensorRT-based inference
- **Alternative**: `nvinferserver` for Triton-based inference
- **Custom preprocessing**: `nvdspreprocess` before inference
- **Custom postprocessing**: `nvdspostprocess` after inference

### For Tracking:
- **Use**: `nvtracker` after primary inference
- **Configure**: Tracker dimensions to match inference input

### For Visualization:
- **Use**: `nvdsosd` for drawing bounding boxes and labels
- **Use**: `nvmultistreamtiler` for multi-stream display
- **Use**: `nvvideoconvert` before rendering sinks

### For Rendering:
- **x86_64**: `nveglglessink`
- **Jetson**: `nv3dsink`
- **File output**: `nvvideoconvert` + encoder + `filesink`

---

## Common Pipeline Patterns

### Single Stream with Detection:
```
filesrc ! h264parse ! nvv4l2decoder ! nvstreammux batch-size=1 ! 
nvinfer config-file-path=pgie.yml ! nvtracker ! nvdsosd ! 
nvvideoconvert ! nveglglessink
```

### Multi-Stream with Detection:
```
stream1 ! m.sink_0 stream2 ! m.sink_1 
nvstreammux name=m batch-size=2 ! nvinfer ! nvtracker ! 
nvstreamdemux name=d d.src_0 ! nvdsosd ! sink1 d.src_1 ! nvdsosd ! sink2
```

### Cascaded Inference (Primary + Secondary):
```
nvstreammux ! nvinfer config-file-path=pgie_config.txt ! 
nvinfer config-file-path=sgie1_config.txt ! nvinfer config-file-path=sgie2_config.txt ! 
nvtracker ! nvdsosd ! sink
```

### Custom Preprocessing + Inference:
```
nvstreammux ! nvdspreprocess config-file=preprocess_config.txt ! 
nvinfer input-tensor-meta=1 config-file-path=infer_config.txt ! 
nvdspostprocess postprocesslib-name=... ! nvdsosd ! sink
```

### Multi-Stream with Analytics and Cloud:
```
streams ! nvstreammux ! nvinfer ! nvtracker ! nvdsanalytics ! 
nvmsgconv ! nvmsgbroker proto-lib=... conn-str=... ! 
nvstreamdemux ! nvdsosd ! sink
```

---

## Performance Optimization Tips

1. **Batch Size**: Use appropriate batch sizes (typically 1-8) based on GPU memory
2. **Resolution**: Match stream resolution to model input requirements
3. **Memory Type**: Use NVMM memory (`nvbuf-memory-type=1`) for zero-copy
4. **Inference Precision**: Use FP16 or INT8 for better performance
5. **Pipeline Parallelism**: Run multiple pipelines on different GPUs
6. **Buffer Management**: Configure queue sizes appropriately
7. **Tracker Configuration**: Match tracker dimensions to inference dimensions

---

## Error Handling and Debugging

1. **Check Plugin Availability**: Use `gst-inspect-1.0 nvinfer` to verify plugins
2. **Enable Debugging**: Set `GST_DEBUG=3` for verbose logging
3. **Check Metadata**: Use probes to inspect metadata at pipeline points
4. **Memory Issues**: Monitor GPU memory usage with `nvidia-smi`
5. **Pipeline State**: Check pipeline state transitions (NULL → READY → PLAYING)

---

This comprehensive overview should help you understand and use DeepStream plugins effectively in your applications.

# Kafka and Message Broker Integration

## Overview

This document is a comprehensive reference for integrating DeepStream applications with external message brokers. It covers two complementary areas:

- **Part 1 -- Kafka Integration Use Cases and Patterns**: Pipeline architectures for streaming analytics data to Apache Kafka, including native `nvmsgbroker` pipelines, Python Kafka producer probes, multi-topic integration, error handling, and performance optimization.
- **Part 2 -- Message Broker and Converter Configuration Reference**: Detailed property tables and configuration file formats for the `nvmsgconv` and `nvmsgbroker` GStreamer plugins, protocol adaptor libraries (Kafka, MQTT, Redis, AMQP, Azure IoT), payload schemas, and troubleshooting guidance.

---

# Part 1: Kafka Integration Use Cases and Patterns

## Use Case Requirements

- Process video streams with AI inference
- Extract object detection and tracking metadata
- Stream metadata to Kafka topics
- Support multiple Kafka topics for different data types
- Handle Kafka connection failures gracefully
- Support both sync and async message sending
- Integrate with cloud services and data pipelines

## Prerequisites

Before building any Kafka-based DeepStream pipeline, install these system dependencies:

```bash
# REQUIRED: librdkafka -- DeepStream's Kafka protocol adapter (libnvds_kafka_proto.so)
# dynamically links against librdkafka.so.1, which is NOT bundled with DeepStream.
sudo apt-get install -y librdkafka-dev

# If also running a local MQTT broker for tracker:
sudo apt-get install -y libmosquitto1        # client library for nvtracker
sudo apt-get install -y mosquitto            # broker daemon (if running locally)
sudo apt-get install -y mosquitto-clients    # CLI tools for testing
```

> **Without `librdkafka-dev`**, any pipeline using `nvmsgbroker` with the Kafka protocol adapter will fail at startup with: `unable to open shared library` / `Failed to start`.

## Architecture Overview

### Critical Rule: async=0 on ALL Sinks

**CRITICAL**: When using `tee` to split a pipeline OR using dynamic sources (nvmultiurisrcbin), **ALL sink elements MUST have `async: 0`**. This includes:
- Display sinks (nveglglessink, nv3dsink)
- Message broker sinks (nvmsgbroker)
- File sinks (filesink)
- Any other sink element

**Symptom if missing**: Pipeline stays stuck in PAUSED state. Cameras show "added" but no video displays and no data flows.

**Why**: GStreamer requires all sinks to "preroll" (receive data) before transitioning to PLAYING state. With `async: 0`, sinks don't block the state transition waiting for preroll.

### Pipeline Architecture

**IMPORTANT**: `nvmsgbroker` is a **SINK component** that terminates the pipeline branch. It cannot have downstream components.

For **headless pipelines** (Kafka only, no display):
```text
Source -> Decoder -> Muxer -> Inference -> Tracker -> Message Converter -> Message Broker (sink)
```

For **pipelines with both Kafka and display**, use `tee` to split paths:
```text
Source -> Decoder -> Muxer -> Inference -> Tracker -> Tee
                                                      |-> [Metadata Branch] Message Converter -> Message Broker (sink)
                                                      |-> [Video Branch] Tiler -> OSD -> Converter -> Renderer (sink)
```

### Data Flow
1. Video processing generates metadata (objects, tracks, frames)
2. Metadata is converted to message format
3. Messages are sent to Kafka broker (metadata branch terminates here)
4. Video continues to display pipeline (if using tee split)
5. Downstream Kafka consumers process analytics data

## Implementation Approaches

### Approach 1: Using nvmsgbroker Plugin (Native DeepStream)

The native DeepStream approach uses `nvmsgbroker` plugin with Kafka protocol library.

**CRITICAL**: `nvmsgbroker` is a **SINK component** that terminates the pipeline branch. It cannot have downstream components like OSD or renderer. If you need both Kafka output and display, use `tee` to split the pipeline into separate branches.

For detailed property tables and configuration file formats for `nvmsgconv` and `nvmsgbroker`, see Part 2 below.

#### Example 1: Headless Pipeline (Kafka Only)

```python
from pyservicemaker import Pipeline
import platform
import sys

def kafka_native_pipeline_headless(video_path, infer_config, kafka_config):
    """
    DeepStream pipeline with native Kafka integration (headless, no display)

    Args:
        video_path: Path to video file
        infer_config: Inference configuration file
        kafka_config: Kafka configuration dict
    """
    pipeline = Pipeline("kafka-pipeline-headless")

    # Source and decoding
    pipeline.add("filesrc", "src", {"location": video_path})
    pipeline.add("h264parse", "parser")
    pipeline.add("nvv4l2decoder", "decoder")
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})

    # Inference
    pipeline.add("nvinfer", "pgie", {"config-file-path": infer_config})

    # Tracker
    pipeline.add("nvtracker", "tracker", {
        "ll-lib-file": "/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so",
        "ll-config-file": "/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml"
    })

    # Message converter (converts metadata to message format)
    # IMPORTANT: msg2p-newapi=True uses NvDsObjectMeta directly (no NvDsEventMsgMeta required)
    pipeline.add("nvmsgconv", "msgconv", {
        "config": kafka_config["msgconv_config"],
        "payload-type": 0,  # 0=deepstream full schema, 1=minimal
        "msg2p-newapi": True,  # CRITICAL: Use new API to avoid NvDsEventMsgMeta requirement
    })

    # Message broker (Kafka) - THIS IS A SINK, terminates the pipeline
    # IMPORTANT: conn-str uses semicolon separator (host;port), NOT colon
    pipeline.add("nvmsgbroker", "msgbroker", {
        "proto-lib": "/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
        "conn-str": kafka_config["broker_servers"],  # Must be "host;port" format
        "sync": 0,   # 0=async message sending, 1=sync
        "async": 0,  # CRITICAL for dynamic sources: prevents state transition deadlock
        "config": kafka_config["broker_config"]
    })

    # Link pipeline - msgbroker is the sink, no components after it
    pipeline.link("src", "parser", "decoder")
    pipeline.link(("decoder", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "pgie", "tracker", "msgconv", "msgbroker")

    pipeline.start().wait()
```

#### Example 2: Pipeline with Both Kafka and Display (Using Tee)

```python
from pyservicemaker import Pipeline
import platform
import sys

def kafka_native_pipeline_with_display(video_path, infer_config, kafka_config):
    """
    DeepStream pipeline with native Kafka integration AND display

    Uses tee to split pipeline into metadata branch (Kafka) and video branch (display)

    Args:
        video_path: Path to video file
        infer_config: Inference configuration file
        kafka_config: Kafka configuration dict
    """
    pipeline = Pipeline("kafka-pipeline-with-display")

    # Source and decoding
    pipeline.add("filesrc", "src", {"location": video_path})
    pipeline.add("h264parse", "parser")
    pipeline.add("nvv4l2decoder", "decoder")
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})

    # Inference
    pipeline.add("nvinfer", "pgie", {"config-file-path": infer_config})

    # Tracker
    pipeline.add("nvtracker", "tracker", {
        "ll-lib-file": "/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so",
        "ll-config-file": "/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml"
    })

    # Add tee to split pipeline
    pipeline.add("tee", "tee")

    # Metadata branch: tee -> queue -> msgconv -> msgbroker (sink)
    pipeline.add("queue", "queue_meta")
    # IMPORTANT: msg2p-newapi=True uses NvDsObjectMeta directly (no NvDsEventMsgMeta required)
    pipeline.add("nvmsgconv", "msgconv", {
        "config": kafka_config["msgconv_config"],
        "payload-type": 0,
        "msg2p-newapi": True,  # CRITICAL: Use new API
    })
    # IMPORTANT: conn-str uses semicolon separator (host;port), NOT colon
    # CRITICAL: async=0 required on ALL sinks when using tee or dynamic sources!
    pipeline.add("nvmsgbroker", "msgbroker", {
        "proto-lib": "/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
        "conn-str": kafka_config["broker_servers"],  # Must be "host;port" format
        "sync": 0,   # Async message sending
        "async": 0,  # CRITICAL: ALL sinks need async=0 to prevent state deadlock!
        "config": kafka_config["broker_config"]
    })

    # Video branch: tee -> queue -> tiler -> osd -> converter -> sink
    pipeline.add("queue", "queue_video")
    pipeline.add("nvmultistreamtiler", "tiler", {"rows": 1, "columns": 1})
    pipeline.add("nvosdbin", "osd")
    pipeline.add("nvvideoconvert", "converter")
    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    # CRITICAL: async=0 required on ALL sinks when using tee or dynamic sources!
    pipeline.add(sink_type, "sink", {
        "sync": 0,   # Don't sync to clock for live sources
        "qos": 0,    # Disable QoS
        "async": 0   # CRITICAL: ALL sinks need async=0 to prevent state deadlock!
    })

    # Link main pipeline
    pipeline.link("src", "parser", "decoder")
    pipeline.link(("decoder", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "pgie", "tracker", "tee")

    # Link metadata branch (terminates at msgbroker sink)
    pipeline.link(("tee", "queue_meta"), ("src_%u", ""))
    pipeline.link("queue_meta", "msgconv", "msgbroker")

    # Link video branch (terminates at display sink)
    pipeline.link(("tee", "queue_video"), ("src_%u", ""))
    pipeline.link("queue_video", "tiler", "osd", "converter", "sink")

    pipeline.start().wait()

if __name__ == "__main__":
    kafka_config = {
        # IMPORTANT: Use semicolon separator, NOT colon!
        "broker_servers": "localhost;9092",  # Correct: semicolon
        # "broker_servers": "localhost:9092",  # Wrong: colon
        "broker_config": "/path/to/kafka_broker_config.txt",
        "msgconv_config": "/path/to/msgconv_config.txt"
    }
    # Use headless version for Kafka-only, or with_display version for both Kafka and display
    kafka_native_pipeline_headless(sys.argv[1], sys.argv[2], kafka_config)
    # OR
    # kafka_native_pipeline_with_display(sys.argv[1], sys.argv[2], kafka_config)
```

#### Example 3: Using Legacy API (msg2p-newapi=0) with EventMessageUserMetadata

When `msg2p-newapi` is `0` (the default), `nvmsgconv` expects `NvDsEventMsgMeta` to be pre-attached to each frame buffer. This metadata is **NOT** generated automatically by any DeepStream plugin. You must attach it via a probe **upstream** of `nvmsgconv`.

There are two sub-approaches:

##### Option A: Built-in `add_message_meta_probe` (Simplest)

```python
from pyservicemaker import Pipeline, Probe, BatchMetadataOperator
import platform

def kafka_legacy_builtin_probe(video_path, infer_config, kafka_config):
    """
    Kafka pipeline using msg2p-newapi=0 with built-in add_message_meta_probe.
    The built-in probe automatically generates EventMessageUserMetadata
    from NvDsObjectMeta for every detected object.
    """
    pipeline = Pipeline("kafka-legacy-builtin")

    # Source and decoding
    pipeline.add("filesrc", "src", {"location": video_path})
    pipeline.add("h264parse", "parser")
    pipeline.add("nvv4l2decoder", "decoder")
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})

    # Inference + tracker
    pipeline.add("nvinfer", "pgie", {"config-file-path": infer_config})
    pipeline.add("nvtracker", "tracker", {
        "ll-lib-file": "/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so",
        "ll-config-file": "/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml"
    })

    # OSD (needed as attachment point for the built-in probe)
    pipeline.add("nvosdbin", "osd")

    # Tee to split display and Kafka branches
    pipeline.add("tee", "tee")

    # Metadata branch
    pipeline.add("queue", "queue_meta")
    pipeline.add("nvmsgconv", "msgconv", {
        "config": kafka_config["msgconv_config"],
        "payload-type": 0,
        "msg2p-newapi": 0,  # Legacy API - requires EventMessageUserMetadata
    })
    pipeline.add("nvmsgbroker", "msgbroker", {
        "proto-lib": "/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
        "conn-str": kafka_config["broker_servers"],
        "sync": 0,
        "async": 0,
    })

    # Display branch
    pipeline.add("queue", "queue_video")
    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    pipeline.add(sink_type, "sink", {"sync": 0, "qos": 0, "async": 0})

    # Link
    pipeline.link("src", "parser", "decoder")
    pipeline.link(("decoder", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "pgie", "tracker", "osd", "tee")
    pipeline.link(("tee", "queue_meta"), ("src_%u", ""))
    pipeline.link("queue_meta", "msgconv", "msgbroker")
    pipeline.link(("tee", "queue_video"), ("src_%u", ""))
    pipeline.link("queue_video", "sink")

    # CRITICAL: attach built-in probe AFTER osd, BEFORE tee->msgconv
    # This automatically creates EventMessageUserMetadata from NvDsObjectMeta
    pipeline.attach("osd", "add_message_meta_probe", "metadata generator")

    pipeline.start().wait()
```

**Reference**: `deepstream_test4_app` sample
(`/opt/nvidia/deepstream/deepstream/service-maker/sources/apps/python/pipeline_api/deepstream_test4_app/deepstream_test4.py`)

##### Option B: Custom EventMessageGenerator (Multi-Camera / Custom Sensor Mappings)

For multi-camera pipelines where you need control over sensor IDs and URIs:

```python
from pyservicemaker import Pipeline, Probe, BatchMetadataOperator, SensorInfo

class EventMessageGenerator(BatchMetadataOperator):
    """
    Generate EventMessageUserMetadata for downstream nvmsgconv.
    Required when msg2p-newapi=0 (legacy API).

    Uses pyservicemaker API:
        batch_meta.acquire_event_message_meta()  -> acquire from pool
        event_msg.generate(obj, frame, sensor_id, uri, labels)  -> populate
        frame_meta.append(event_msg)  -> attach to frame
    """

    def __init__(self, sensor_map, labels):
        super().__init__()
        self._sensor_map = sensor_map  # dict: source_id (int) -> SensorInfo
        self._labels = labels          # list of class label strings

    def handle_metadata(self, batch_meta, frame_interval=1):
        for frame_meta in batch_meta.frame_items:
            frame_num = frame_meta.frame_number
            for object_meta in frame_meta.object_items:
                if not (frame_num % frame_interval):
                    event_msg = batch_meta.acquire_event_message_meta()
                    if event_msg:
                        source_id = frame_meta.source_id
                        sensor_info = self._sensor_map.get(source_id)
                        sensor_id = sensor_info.sensor_id if sensor_info else "N/A"
                        uri = sensor_info.uri if sensor_info else "N/A"
                        event_msg.generate(
                            object_meta, frame_meta, sensor_id, uri, self._labels
                        )
                        frame_meta.append(event_msg)


def kafka_legacy_custom_generator(video_paths, infer_config, kafka_config, labels):
    """
    Multi-camera Kafka pipeline using msg2p-newapi=0 with custom EventMessageGenerator.
    """
    pipeline = Pipeline("kafka-legacy-custom")

    # Build sensor map from video paths
    sensor_map = {}
    for i, uri in enumerate(video_paths):
        sensor_map[i] = SensorInfo(
            sensor_id=f"Camera{i+1}",
            sensor_name=f"cam{i+1}",
            uri=uri
        )

    # ... (add sources, inference, tracker, tee, msgconv with msg2p-newapi=0, etc.)

    # Attach custom EventMessageGenerator probe UPSTREAM of nvmsgconv
    pipeline.attach(
        "tracker",
        Probe("event_msg_gen", EventMessageGenerator(sensor_map, labels))
    )

    pipeline.start().wait()
```

**Key API calls**:
- `batch_meta.acquire_event_message_meta()` -- acquires `EventMessageUserMetadata` from the pool
- `event_msg.generate(object_meta, frame_meta, sensor_id, uri, labels)` -- populates the metadata
- `frame_meta.append(event_msg)` -- attaches it to the frame for downstream nvmsgconv

**Reference**: `deepstream_test5_app` sample
(`/opt/nvidia/deepstream/deepstream/service-maker/sources/apps/python/pipeline_api/deepstream_test5_app/deepstream_test5.py`)

---

#### Kafka Broker Configuration File

**kafka_broker_config.txt**:
```ini
[broker]
enable=1
broker-ip-port=localhost:9092
topic=deepstream-analytics
# Optional: SSL/TLS configuration
# enable-tls=1
# ca-file=/path/to/ca-cert
# client-cert-file=/path/to/client-cert
# client-key-file=/path/to/client-key
```

#### Message Converter Configuration File

**msgconv_config.txt**:
```ini
[message-converter]
enable=1
# Message format: deepstream or custom
msg-format=deepstream
# Schema file for custom format
schema-file=/path/to/schema.json
# Payload type: 0=deepstream, 1=custom
payload-type=0
```

### Approach 2: Using Python Kafka Producer (Custom Probe)

This approach uses Python's `kafka-python` library in a custom probe for more control.

#### Custom Kafka Producer Probe

```python
from pyservicemaker import Pipeline, Probe, BatchMetadataOperator
from kafka import KafkaProducer
from kafka.errors import KafkaError
import json
import sys
import platform

class KafkaMetadataSender(BatchMetadataOperator):
    """
    Custom probe to send metadata to Kafka

    Sends object detection and tracking metadata to Kafka topics
    """
    def __init__(self, kafka_config):
        """
        Initialize Kafka producer

        Args:
            kafka_config: Dict with Kafka configuration
                - bootstrap_servers: Kafka broker addresses
                - topic: Topic name
                - security_config: Optional security config
        """
        super().__init__()

        # Kafka producer configuration
        producer_config = {
            "bootstrap_servers": kafka_config["bootstrap_servers"],
            "value_serializer": lambda v: json.dumps(v).encode('utf-8'),
            "key_serializer": lambda k: str(k).encode('utf-8') if k else None,
            "acks": "all",  # Wait for all replicas
            "retries": 3,
            "max_in_flight_requests_per_connection": 1,
            "enable_idempotence": True
        }

        # Add security configuration if provided
        if "security_config" in kafka_config:
            security = kafka_config["security_config"]
            if security.get("use_ssl"):
                producer_config.update({
                    "security_protocol": "SSL",
                    "ssl_cafile": security.get("ca_file"),
                    "ssl_certfile": security.get("cert_file"),
                    "ssl_keyfile": security.get("key_file")
                })
            elif security.get("use_sasl"):
                producer_config.update({
                    "security_protocol": "SASL_SSL",
                    "sasl_mechanism": security.get("sasl_mechanism", "PLAIN"),
                    "sasl_plain_username": security.get("username"),
                    "sasl_plain_password": security.get("password")
                })

        self.producer = KafkaProducer(**producer_config)
        self.topic = kafka_config["topic"]
        self.send_frame_metadata = kafka_config.get("send_frame_metadata", True)
        self.send_object_metadata = kafka_config.get("send_object_metadata", True)
        self.batch_size = kafka_config.get("batch_size", 1)  # Send every N frames

        self.frame_count = 0
        self.error_count = 0

    def handle_metadata(self, batch_meta):
        """Process batch metadata and send to Kafka"""
        for frame_meta in batch_meta.frame_items:
            self.frame_count += 1

            # Send metadata every N frames (if batch_size > 1)
            if self.frame_count % self.batch_size != 0:
                continue

            try:
                # Prepare message
                message = self._prepare_message(frame_meta)

                # Send to Kafka
                future = self.producer.send(
                    topic=self.topic,
                    key=str(frame_meta.frame_number),  # Use frame number as key
                    value=message
                )

                # Optional: Add callback for success/failure
                future.add_callback(self._on_send_success)
                future.add_errback(self._on_send_error)

            except Exception as e:
                print(f"Error sending message to Kafka: {e}")
                self.error_count += 1

    def _prepare_message(self, frame_meta):
        """Prepare message from frame metadata"""
        message = {
            "frame_number": frame_meta.frame_number,
            # Note: Use buffer_pts for PTS timestamp, ntp_timestamp for NTP timestamp
            "buffer_pts": frame_meta.buffer_pts,
            "ntp_timestamp": frame_meta.ntp_timestamp,
            "pad_index": frame_meta.pad_index,
            "source_id": frame_meta.source_id  # Use source_id property
        }

        # Add frame-level metadata
        if self.send_frame_metadata:
            message["frame_metadata"] = {
                "source_width": frame_meta.source_width,
                "source_height": frame_meta.source_height,
                "pipeline_width": frame_meta.pipeline_width,
                "pipeline_height": frame_meta.pipeline_height
            }

        # Add object metadata
        if self.send_object_metadata:
            objects = []
            for obj_meta in frame_meta.object_items:
                obj_data = {
                    "class_id": obj_meta.class_id,
                    "confidence": float(obj_meta.confidence),
                    # Use object_id to get the tracker-assigned tracking ID
                    "object_id": obj_meta.object_id,
                    "bbox": {
                        "left": float(obj_meta.rect_params.left),
                        "top": float(obj_meta.rect_params.top),
                        "width": float(obj_meta.rect_params.width),
                        "height": float(obj_meta.rect_params.height)
                    }
                }

                # Add secondary inference results if available
                # (stored in obj_meta.obj_user_meta_list)
                if hasattr(obj_meta, 'obj_user_meta_list'):
                    obj_data["attributes"] = self._extract_attributes(obj_meta)

                objects.append(obj_data)

            message["objects"] = objects
            message["object_count"] = len(objects)

        return message

    def _extract_attributes(self, obj_meta):
        """Extract secondary inference attributes from object metadata"""
        attributes = {}
        # Process obj_user_meta_list to extract classification results
        # This depends on how secondary inference stores results
        return attributes

    def _on_send_success(self, record_metadata):
        """Callback for successful message send"""
        pass  # Can add logging here

    def _on_send_error(self, exception):
        """Callback for failed message send"""
        print(f"Kafka publish failed: {exception}")
        self.error_count += 1

    def flush(self):
        """Flush pending messages"""
        self.producer.flush()

    def close(self):
        """Close Kafka producer"""
        self.producer.flush()
        self.producer.close()
        print(f"Kafka producer closed. Sent {self.frame_count} frames, {self.error_count} errors")

def kafka_custom_probe_pipeline(video_path, infer_config, kafka_config):
    """Pipeline with custom Kafka probe"""
    pipeline = Pipeline("kafka-custom-probe")

    # Source and decoding
    pipeline.add("filesrc", "src", {"location": video_path})
    pipeline.add("h264parse", "parser")
    pipeline.add("nvv4l2decoder", "decoder")
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})

    # Inference
    pipeline.add("nvinfer", "pgie", {"config-file-path": infer_config})

    # Tracker
    pipeline.add("nvtracker", "tracker", {
        "ll-lib-file": "/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so",
        "ll-config-file": "/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml"
    })

    # OSD and sink
    pipeline.add("nvosdbin", "osd")
    pipeline.add("nvvideoconvert", "converter")
    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    pipeline.add(sink_type, "sink", {"sync": 1})

    # Link pipeline
    pipeline.link("src", "parser", "decoder")
    pipeline.link(("decoder", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "pgie", "tracker", "osd", "converter", "sink")

    # Attach Kafka probe
    kafka_sender = KafkaMetadataSender(kafka_config)
    pipeline.attach("tracker", Probe("kafka-sender", kafka_sender))

    try:
        pipeline.start().wait()
    finally:
        kafka_sender.close()

if __name__ == "__main__":
    kafka_config = {
        "bootstrap_servers": "localhost:9092",
        "topic": "deepstream-analytics",
        "send_frame_metadata": True,
        "send_object_metadata": True,
        "batch_size": 1  # Send every frame
    }
    kafka_custom_probe_pipeline(sys.argv[1], sys.argv[2], kafka_config)
```

### Approach 3: Multi-Topic Kafka Integration

Send different types of metadata to different Kafka topics.

```python
class MultiTopicKafkaSender(BatchMetadataOperator):
    """Send different metadata types to different Kafka topics"""
    def __init__(self, kafka_configs):
        """
        Args:
            kafka_configs: Dict mapping topic names to Kafka configs
                {
                    "object-detections": {...},
                    "tracking-events": {...},
                    "frame-metadata": {...}
                }
        """
        super().__init__()
        self.producers = {}
        self.topics = {}

        for topic_name, config in kafka_configs.items():
            producer = KafkaProducer(
                bootstrap_servers=config["bootstrap_servers"],
                value_serializer=lambda v: json.dumps(v).encode('utf-8')
            )
            self.producers[topic_name] = producer
            self.topics[topic_name] = config.get("topic", topic_name)

    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            # Send object detections
            if "object-detections" in self.producers:
                detections = self._prepare_detections(frame_meta)
                self.producers["object-detections"].send(
                    topic=self.topics["object-detections"],
                    value=detections
                )

            # Send tracking events (new tracks, lost tracks)
            if "tracking-events" in self.producers:
                events = self._prepare_tracking_events(frame_meta)
                if events:
                    self.producers["tracking-events"].send(
                        topic=self.topics["tracking-events"],
                        value=events
                    )

            # Send frame metadata
            if "frame-metadata" in self.producers:
                frame_data = self._prepare_frame_metadata(frame_meta)
                self.producers["frame-metadata"].send(
                    topic=self.topics["frame-metadata"],
                    value=frame_data
                )

    def _prepare_detections(self, frame_meta):
        """Prepare object detection message"""
        # Build detections list by iterating (object_items is an iterator)
        detections = [
            {
                "class_id": obj.class_id,
                "confidence": float(obj.confidence),
                "bbox": {
                    "left": float(obj.rect_params.left),
                    "top": float(obj.rect_params.top),
                    "width": float(obj.rect_params.width),
                    "height": float(obj.rect_params.height)
                }
            }
            for obj in frame_meta.object_items
        ]
        return {
            "frame_number": frame_meta.frame_number,
            "buffer_pts": frame_meta.buffer_pts,  # Use buffer_pts for timestamp
            "ntp_timestamp": frame_meta.ntp_timestamp,
            "detections": detections
        }

    def _prepare_tracking_events(self, frame_meta):
        """Prepare tracking event message"""
        # Detect new tracks, lost tracks, etc.
        # This requires maintaining state across frames
        return {}  # Implement tracking event detection

    def _prepare_frame_metadata(self, frame_meta):
        """Prepare frame metadata message"""
        # Note: object_items is an ITERATOR, not a list - cannot use len() directly
        # Count objects by iterating
        obj_count = sum(1 for _ in frame_meta.object_items)
        return {
            "frame_number": frame_meta.frame_number,
            "buffer_pts": frame_meta.buffer_pts,  # Use buffer_pts for timestamp
            "ntp_timestamp": frame_meta.ntp_timestamp,
            "object_count": obj_count
        }

    def close(self):
        """Close all producers"""
        for producer in self.producers.values():
            producer.flush()
            producer.close()
```

## Error Handling and Resilience

### Retry Logic and Error Handling

```python
class ResilientKafkaSender(BatchMetadataOperator):
    """Kafka sender with retry logic and error handling"""
    def __init__(self, kafka_config):
        super().__init__()
        self.config = kafka_config
        self.max_retries = kafka_config.get("max_retries", 3)
        self.retry_delay = kafka_config.get("retry_delay", 1.0)
        self.message_queue = []  # Queue for failed messages
        self._init_producer()

    def _init_producer(self):
        """Initialize or reinitialize producer"""
        try:
            self.producer = KafkaProducer(
                bootstrap_servers=self.config["bootstrap_servers"],
                value_serializer=lambda v: json.dumps(v).encode('utf-8'),
                retries=self.max_retries,
                max_in_flight_requests_per_connection=1,
                enable_idempotence=True
            )
            self.connected = True
        except Exception as e:
            print(f"Failed to initialize Kafka producer: {e}")
            self.connected = False

    def handle_metadata(self, batch_meta):
        if not self.connected:
            self._init_producer()
            if not self.connected:
                # Store messages for later retry
                self.message_queue.append(batch_meta)
                return

        try:
            # Process current batch
            self._send_batch(batch_meta)

            # Retry queued messages
            while self.message_queue:
                queued_batch = self.message_queue.pop(0)
                try:
                    self._send_batch(queued_batch)
                except Exception as e:
                    # Re-queue if still failing
                    self.message_queue.append(queued_batch)
                    break

        except Exception as e:
            print(f"Error sending to Kafka: {e}")
            self.message_queue.append(batch_meta)
            # Try to reconnect
            self.connected = False

    def _send_batch(self, batch_meta):
        """Send batch metadata to Kafka"""
        for frame_meta in batch_meta.frame_items:
            message = self._prepare_message(frame_meta)
            future = self.producer.send(
                topic=self.config["topic"],
                value=message
            )
            # Wait for delivery (synchronous for reliability)
            future.get(timeout=10)
```

## Performance Optimization

### Batching Messages

```python
class BatchedKafkaSender(BatchMetadataOperator):
    """Batch multiple frames before sending to Kafka"""
    def __init__(self, kafka_config, batch_size=10):
        super().__init__()
        self.producer = KafkaProducer(
            bootstrap_servers=kafka_config["bootstrap_servers"],
            value_serializer=lambda v: json.dumps(v).encode('utf-8'),
            batch_size=16384,  # Kafka batch size in bytes
            linger_ms=100  # Wait up to 100ms to batch
        )
        self.topic = kafka_config["topic"]
        self.batch_size = batch_size
        self.frame_buffer = []

    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            self.frame_buffer.append(frame_meta)

            if len(self.frame_buffer) >= self.batch_size:
                self._send_batch()

    def _send_batch(self):
        """Send batched frames"""
        batch_message = {
            "frames": [self._prepare_message(f) for f in self.frame_buffer]
        }
        self.producer.send(topic=self.topic, value=batch_message)
        self.frame_buffer.clear()

    def flush(self):
        """Flush remaining frames"""
        if self.frame_buffer:
            self._send_batch()
        self.producer.flush()
```

## Testing and Validation

### Test Kafka Consumer

```python
from kafka import KafkaConsumer
import json

def test_kafka_consumer(bootstrap_servers, topic):
    """Test consumer to verify messages are being sent"""
    consumer = KafkaConsumer(
        topic,
        bootstrap_servers=bootstrap_servers,
        value_deserializer=lambda m: json.loads(m.decode('utf-8')),
        auto_offset_reset='earliest',
        enable_auto_commit=True
    )

    print(f"Consuming messages from topic: {topic}")
    for message in consumer:
        print(f"Received: {message.value}")
```

## Common Patterns

### Pattern 1: Real-time Analytics Dashboard
- Send object counts and statistics to Kafka
- Dashboard consumes and displays in real-time

### Pattern 2: Data Lake Ingestion
- Send all metadata to Kafka
- Kafka Connect streams to data lake (S3, HDFS)

### Pattern 3: Alert System
- Send only significant events (intrusions, anomalies)
- Alert service consumes and triggers notifications

### Pattern 4: Multi-Tenant Analytics
- Use different topics for different customers/streams
- Enable topic-based access control

---

# Part 2: Message Broker and Converter Configuration Reference

## Architecture

```text
Pipeline -> nvmsgconv -> nvmsgbroker -> External Broker
              |              |
              |              +-- Protocol Adaptor Library
              |                   (libnvds_kafka_proto.so, etc.)
              |
              +-- Config File (sensor, place, analytics metadata)
```

**IMPORTANT**: `nvmsgbroker` is a **SINK component** that terminates the pipeline branch. It cannot have downstream components.

---

## nvmsgconv Plugin

### Purpose

Converts DeepStream metadata (NvDsEventMsgMeta or NvDsFrameMeta/NvDsObjectMeta) to message payload format.

### GStreamer Properties

| Property | Type | Description | Default |
|----------|------|-------------|---------|
| `config` | string | Path to message converter configuration file | None |
| `payload-type` | int | Payload schema type (see below) | 0 |
| `comp-id` | uint | Component ID for filtering metadata | All |
| `msg2p-lib` | string | Path to custom payload generation library | None |
| `frame-interval` | uint | Generate payload every N frames | 30 |
| `msg2p-newapi` | bool | **IMPORTANT**: Use new message-to-payload API (see below) | false |
| `debug-payload-dir` | string | Directory to dump payloads for debugging | None |
| `multiple-payloads` | bool | Generate multiple payloads per buffer | false |

### CRITICAL: msg2p-newapi Property

**Problem**: By default (`msg2p-newapi: false`), `nvmsgconv` requires `NvDsEventMsgMeta` (exposed as `EventMessageUserMetadata` in pyservicemaker) to be attached to the buffer. This metadata is **NOT automatically generated** by inference or tracker plugins. Without explicitly handling this, nvmsgconv silently produces **zero messages**.

**Two Solutions** (pick one):

#### Solution A: Set msg2p-newapi=True (Simple, Recommended for Most Cases)

Uses the new API that reads directly from `NvDsFrameMeta` and `NvDsObjectMeta` without requiring `NvDsEventMsgMeta`:

```python
# CORRECT - Uses object metadata directly, no NvDsEventMsgMeta needed
pipeline.add("nvmsgconv", "msgconv", {
    "config": msgconv_config,
    "payload-type": 0,
    "msg2p-newapi": True,      # Use new API - reads from NvDsObjectMeta directly
})
```

#### Solution B: Keep msg2p-newapi=0 and Attach EventMessageUserMetadata Probe

Required when using custom `msg2p-lib` payload libraries that expect legacy `NvDsEventMsgMeta`, or when you need fine-grained control over per-object message generation.

**Option B1: Built-in probe** (simplest):
```python
pipeline.add("nvmsgconv", "msgconv", {
    "config": msgconv_config,
    "payload-type": 0,
    # msg2p-newapi defaults to 0 (legacy API)
})

# Built-in probe auto-generates EventMessageUserMetadata from NvDsObjectMeta
pipeline.attach("osd", "add_message_meta_probe", "metadata generator")
```

**Option B2: Custom EventMessageGenerator** (for multi-camera / custom sensor mappings):
```python
from pyservicemaker import Probe, BatchMetadataOperator, SensorInfo

class EventMessageGenerator(BatchMetadataOperator):
    def __init__(self, sensor_map, labels):
        super().__init__()
        self._sensor_map = sensor_map  # dict: source_id -> SensorInfo
        self._labels = labels          # list of class label strings

    def handle_metadata(self, batch_meta, frame_interval=1):
        for frame_meta in batch_meta.frame_items:
            for object_meta in frame_meta.object_items:
                event_msg = batch_meta.acquire_event_message_meta()
                if event_msg:
                    source_id = frame_meta.source_id
                    sensor_info = self._sensor_map.get(source_id)
                    sensor_id = sensor_info.sensor_id if sensor_info else "N/A"
                    uri = sensor_info.uri if sensor_info else "N/A"
                    event_msg.generate(
                        object_meta, frame_meta, sensor_id, uri, self._labels
                    )
                    frame_meta.append(event_msg)

# Attach UPSTREAM of nvmsgconv (e.g., on tracker or osd element)
sensor_map = {0: SensorInfo("Camera1", "cam1", "file:///video.mp4")}
labels = ["car", "bicycle", "person", "roadsign"]
pipeline.attach("tracker", Probe("event_msg_gen", EventMessageGenerator(sensor_map, labels)))
```

For complete pipeline examples using the legacy API, see Part 1 above (Example 3).

#### Common Mistake

```python
# WRONG - Without msg2p-newapi=True AND without EventMessageUserMetadata probe,
# nvmsgconv has no input and produces ZERO messages silently!
pipeline.add("nvmsgconv", "msgconv", {
    "config": msgconv_config,
    "payload-type": 0
})
```

**Reference samples**:
- Built-in probe: `/opt/nvidia/deepstream/deepstream/service-maker/sources/apps/python/pipeline_api/deepstream_test4_app/deepstream_test4.py`
- Custom generator: `/opt/nvidia/deepstream/deepstream/service-maker/sources/apps/python/pipeline_api/deepstream_test5_app/deepstream_test5.py`

### Payload Types

| Value | Name | Description |
|-------|------|-------------|
| 0 | `PAYLOAD_DEEPSTREAM` | Full DeepStream schema - separate JSON payload per object |
| 1 | `PAYLOAD_DEEPSTREAM_MINIMAL` | Minimal schema - multiple objects in single JSON payload |
| 2 | `PAYLOAD_DEEPSTREAM_PROTOBUF` | Protobuf encoded - multiple objects in single payload |
| 256 | `PAYLOAD_CUSTOM` | Custom schema using msg2p-lib |

### Segmentation Payload Contract

When a consumer needs segmentation data:

1. Declare whether the payload needs per-object masks, a full-frame mask, or both; preserve normal object metadata.
2. Instance masks on object metadata are bbox-local. For a full-frame payload, give the serializer an explicit frame mask or preserved inference tensors in a declared frame grid; do not reconstruct it from tracker-updated objects unless that is the intended output.
3. If the selected schema has no mask field, use a compatible serializer extension. Payload type is a consumer contract, not a shortcut for carrying masks.
4. Decode a real payload and verify its objects, mask representation, dimensions, and coordinate space.

Note that OSD can render instance-mask metadata, but `nvmsgconv` does not serialize it automatically.

### Pipeline Usage

```python
# Using pyservicemaker Pipeline API
pipeline.add("nvmsgconv", "msgconv", {
    "config": "/path/to/msgconv_config.txt",
    "payload-type": 0  # Full DeepStream schema
})
```

---

## nvmsgconv Configuration File

The configuration file defines metadata about sensors, places, and analytics that gets embedded in the message payload.

### Supported Formats

- **INI-style format** (`.txt`) - Recommended
- **YAML format** (`.yml`)

### Configuration Sections

#### [sensor0], [sensor1], ... - Sensor/Camera Information

| Parameter | Type | Description | Required |
|-----------|------|-------------|----------|
| `enable` | int | Enable this sensor (0/1) | Yes |
| `type` | string | Sensor type (e.g., "Camera", "Lidar") | Yes |
| `id` | string | Unique sensor identifier | Yes |
| `location` | string | GPS coordinates "lat;lon;alt" | No |
| `description` | string | Human-readable description | No |
| `coordinate` | string | Local coordinates "x;y;z" | No |

#### [place0], [place1], ... - Location/Place Information

| Parameter | Type | Description | Required |
|-----------|------|-------------|----------|
| `enable` | int | Enable this place (0/1) | Yes |
| `id` | string/int | Place identifier | Yes |
| `type` | string | Place type (e.g., "garage", "intersection/road") | Yes |
| `name` | string | Place name | Yes |
| `location` | string | GPS coordinates "lat;lon;alt" | No |
| `coordinate` | string | Local coordinates "x;y;z" | No |
| `place-sub-field1` | string | Custom sub-field 1 | No |
| `place-sub-field2` | string | Custom sub-field 2 | No |
| `place-sub-field3` | string | Custom sub-field 3 | No |

#### [analytics0], [analytics1], ... - Analytics Information

| Parameter | Type | Description | Required |
|-----------|------|-------------|----------|
| `enable` | int | Enable this analytics config (0/1) | Yes |
| `id` | string | Analytics identifier | Yes |
| `description` | string | Analytics description | No |
| `source` | string | Analytics source/algorithm name | No |
| `version` | string | Analytics version | No |

### Example Configuration (INI-style)

```ini
# msgconv_config.txt

[sensor0]
enable=1
type=Camera
id=CAMERA_001
location=45.293701;-75.830391;48.155
description=Entrance Camera
coordinate=5.2;10.1;11.2

[sensor1]
enable=1
type=Camera
id=CAMERA_002
location=45.293702;-75.830392;48.156
description=Exit Camera
coordinate=6.2;11.1;12.2

[place0]
enable=1
id=1
type=garage
name=ParkingLot_A
location=30.32;-40.55;100.0
coordinate=1.0;2.0;3.0
place-sub-field1=Zone_A
place-sub-field2=Lane_1
place-sub-field3=Level_P1

[analytics0]
enable=1
id=ANALYTICS_001
description=Vehicle Detection and Tracking
source=ResNet18_TrafficCamNet
version=1.0
```

### Example Configuration (YAML)

```yaml
# msgconv_config.yml

sensor0:
  enable: 1
  type: Camera
  id: CAMERA_001
  location: 45.293701;-75.830391;48.155
  description: Entrance Camera
  coordinate: 5.2;10.1;11.2

place0:
  enable: 1
  id: 1
  type: garage
  name: ParkingLot_A
  location: 30.32;-40.55;100.0
  coordinate: 1.0;2.0;3.0
  place-sub-field1: Zone_A
  place-sub-field2: Lane_1
  place-sub-field3: Level_P1

analytics0:
  enable: 1
  id: ANALYTICS_001
  description: Vehicle Detection and Tracking
  source: ResNet18_TrafficCamNet
  version: 1.0
```

### Multi-Source Configuration

For multi-source pipelines, create sensor/place entries for each source:

```ini
# Sensor entries map to source_id in the pipeline
[sensor0]
enable=1
type=Camera
id=STREAM_0
description=Camera 0

[sensor1]
enable=1
type=Camera
id=STREAM_1
description=Camera 1

# Place entries map to source_id
[place0]
enable=1
id=0
type=intersection
name=Location_0

[place1]
enable=1
id=1
type=intersection
name=Location_1
```

---

## nvmsgbroker Plugin

### Purpose

Sends payload metadata to external message brokers using protocol adaptor libraries.

### GStreamer Properties

| Property | Type | Description | Default |
|----------|------|-------------|---------|
| `proto-lib` | string | Path to protocol adaptor library | **Required** |
| `conn-str` | string | Connection string for broker | **Required** |
| `config` | string | Path to protocol-specific config file | None |
| `topic` | string | Message topic name | None |
| `comp-id` | uint | Component ID for filtering payloads | All |
| `sync` | int | Synchronous (1) or async (0) message sending | 0 |
| `async` | int | **CRITICAL**: Set to 0 for dynamic sources/tee pipelines | 1 |
| `new-api` | bool | Use new nvmsgbroker API | false |
| `sleep-time` | uint | Sleep time in ms between do_work calls | 0 |

**CRITICAL: async=0 for Dynamic Sources and Tee Splits**

When using `nvmsgbroker` in a pipeline with:
- Dynamic sources (nvmultiurisrcbin)
- Tee splits (multiple branches with different sinks)

You **MUST** set `async: 0` on nvmsgbroker AND all other sinks. Otherwise, the pipeline will be stuck in PAUSED state.

```python
# CORRECT - async=0 for tee/dynamic source pipelines
pipeline.add("nvmsgbroker", "msgbroker", {
    "proto-lib": "/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
    "conn-str": "localhost;9092",
    "sync": 0,   # Async message sending
    "async": 0,  # CRITICAL: Required for tee/dynamic sources!
})

# WRONG - missing async=0 causes pipeline stuck in PAUSED
pipeline.add("nvmsgbroker", "msgbroker", {
    "proto-lib": "/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
    "conn-str": "localhost;9092",
    "sync": 0,
    # async defaults to 1, causing state transition deadlock!
})
```

### Protocol Adaptor Libraries

Located at `/opt/nvidia/deepstream/deepstream/lib/`:

| Protocol | Library | Connection String Format |
|----------|---------|-------------------------|
| Kafka | `libnvds_kafka_proto.so` | `host;port` (semicolon-separated) |
| MQTT | `libnvds_mqtt_proto.so` | `host;port` (semicolon-separated) |
| Redis | `libnvds_redis_proto.so` | `host;port` (semicolon-separated) |
| AMQP | `libnvds_amqp_proto.so` | `host;port;username;password` (semicolon-separated) |
| Azure IoT | `libnvds_azure_proto.so` | Full Azure connection string |
| Azure IoT Edge | `libnvds_azure_edge_proto.so` | - |

**CRITICAL: Connection String Format**

DeepStream message broker uses **semicolon (`;`)** as separator, NOT colon (`:`).

```python
# CORRECT - semicolon separator
"conn-str": "localhost;9092"

# WRONG - colon separator (will fail to connect)
"conn-str": "localhost:9092"
```

### Pipeline Usage

```python
# Using pyservicemaker Pipeline API
# For simple pipelines (single source, no tee):
pipeline.add("nvmsgbroker", "msgbroker", {
    "proto-lib": "/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
    "conn-str": "localhost;9092",  # IMPORTANT: Use semicolon, not colon!
    "topic": "deepstream-analytics",
    "sync": 0,
    "config": "/path/to/kafka_config.txt"
})

# For pipelines with dynamic sources OR tee splits:
pipeline.add("nvmsgbroker", "msgbroker", {
    "proto-lib": "/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
    "conn-str": "localhost;9092",  # IMPORTANT: Use semicolon, not colon!
    "topic": "deepstream-analytics",
    "sync": 0,
    "async": 0,  # CRITICAL: Required for tee/dynamic sources!
    "config": "/path/to/kafka_config.txt"
})
```

---

## Protocol Adaptor Configurations

### Kafka Protocol Adaptor

#### Dependencies Installation

```bash
# Add Confluent repository
sudo mkdir -p /usr/share/confluent-repo
wget -qO /tmp/confluent-archive.key https://packages.confluent.io/deb/7.8/archive.key
gpg --dearmor --output /tmp/confluent.gpg /tmp/confluent-archive.key
sudo cp /tmp/confluent.gpg /usr/share/confluent-repo/confluent.gpg
sudo chmod 0644 /usr/share/confluent-repo/confluent.gpg
rm -f /tmp/confluent-archive.key /tmp/confluent.gpg

CP_DIST=$(lsb_release -cs)
cat <<EOF >/tmp/confluent-platform.sources
Types: deb
URIs: https://packages.confluent.io/deb/8.0
Suites: stable
Components: main
Architectures: $(dpkg --print-architecture)
Signed-By: /usr/share/confluent-repo/confluent.gpg

Types: deb
URIs: https://packages.confluent.io/clients/deb/
Suites: ${CP_DIST}
Components: main
Architectures: $(dpkg --print-architecture)
Signed-By: /usr/share/confluent-repo/confluent.gpg
EOF
sudo cp /tmp/confluent-platform.sources /etc/apt/sources.list.d/confluent-platform.sources
sudo chmod 0644 /etc/apt/sources.list.d/confluent-platform.sources
rm -f /tmp/confluent-platform.sources

# Install dependencies
sudo apt-get update
sudo apt-get install librdkafka-dev libglib2.0-dev libjansson-dev libssl-dev
```

#### Configuration File (cfg_kafka.txt)

```ini
[message-broker]
# Consumer group ID for Kafka consumer
#consumer-group-id = mygroup

# Generic librdkafka configuration (applies to both producer and consumer)
# Semicolon-separated key=value pairs
#proto-cfg = "message.max.bytes=200000;log_level=6"

# Producer-specific librdkafka configuration
#producer-proto-cfg = "queue.buffering.max.messages=200000;message.send.max.retries=3"

# Consumer-specific librdkafka configuration
#consumer-proto-cfg = "max.poll.interval.ms=20000"

# Partition key field name in JSON message
# Use "sensor.id" for full schema, "sensorId" for minimal schema
#partition-key = sensor.id

# Enable connection sharing within same process
#share-connection = 1
```

#### Connection String

Format: `hostname;port`

Example: `localhost;9092` or `kafka-broker.example.com;9092`

#### TLS/SSL Configuration

For secure connections, refer to `/opt/nvidia/deepstream/deepstream/sources/libs/kafka_protocol_adaptor/Security_Setup.md`

---

### MQTT Protocol Adaptor

#### Dependencies Installation

```bash
# Install dependencies
sudo apt-get install libglib2.0-dev libcjson-dev libssl-dev

# Add Mosquitto PPA and install
sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa
sudo apt-get update
sudo apt-get install libmosquitto-dev mosquitto
```

#### Configuration File (cfg_mqtt.txt)

```ini
[message-broker]
# Username for broker authentication (deprecated - use env var)
#username = user

# Password for broker authentication (deprecated - use env var)
#password = password

# Unique client ID (empty = random)
client-id = deepstream-client

# TLS Configuration
#enable-tls = 1
#tls-cafile = /path/to/ca-cert.pem
#tls-capath = /path/to/ca-certs-dir/
#tls-certfile = /path/to/client-cert.pem
#tls-keyfile = /path/to/client-key.pem

# Connection sharing
#share-connection = 1

# Mosquitto loop timeout in ms
#loop-timeout = 2000

# Keep-alive interval in seconds
#keep-alive = 60

# Enable threaded mode (required for nvmsgbroker plugin)
#set-threaded = 1
```

#### User Authentication via Environment Variables

```bash
export USER_MQTT=username
export PASSWORD_MQTT=password
```

#### Connection String

Format: `hostname;port`

Example: `localhost;1883`

#### Running Mosquitto Broker

```bash
# Add mosquitto user
sudo adduser --system mosquitto

# Run broker
mosquitto

# Or with config file
mosquitto -c /etc/mosquitto/mosquitto.conf
```

#### Verify Messages

```bash
# Subscribe to topic
mosquitto_sub -t deepstream-analytics -v

# Publish test message
mosquitto_pub -t deepstream-analytics -m 'test message'
```

---

### Redis Protocol Adaptor

#### Dependencies Installation

```bash
# Install dependencies
sudo apt-get install libglib2.0-dev libssl-dev libhiredis-dev
```

#### Configuration File (cfg_redis.txt)

```ini
[message-broker]
# Redis server hostname
#hostname=localhost

# Redis server port
#port=6379

# Password for Redis AUTH (deprecated - use env var)
#password=password

# Redis stream key for payload
#payloadkey=metadata

# Consumer group name
#consumergroup=mygroup

# Consumer name
#consumername=myname

# Maximum stream size (for capped streams)
#streamsize=10000

# Connection sharing
#share-connection = 1
```

#### User Authentication via Environment Variables

```bash
export PASSWORD_REDIS=password
```

#### Connection String

Format: `hostname;port`

Example: `localhost;6379`

#### Running Redis Server

```bash
# Download and build Redis
wget http://download.redis.io/releases/redis-6.0.8.tar.gz
tar xzf redis-6.0.8.tar.gz
cd redis-6.0.8
make

# Run server
src/redis-server

# Or with protected mode disabled (for external connections)
src/redis-server --protected-mode no
```

---

### AMQP Protocol Adaptor (RabbitMQ)

#### Dependencies Installation

```bash
# Install dependencies
sudo apt-get install libglib2.0-dev librabbitmq-dev

# Install RabbitMQ server (optional, for local testing)
sudo apt-get install rabbitmq-server
sudo service rabbitmq-server start
```

#### Configuration File (cfg_amqp.txt)

```ini
[message-broker]
# RabbitMQ server hostname
hostname = localhost

# RabbitMQ server port
port = 5672

# Username (deprecated - use env var)
username = guest

# Password (deprecated - use env var)
password = guest

# AMQP exchange name
exchange = amq.topic

# Topic/routing key
topic = deepstream-analytics

# Maximum frame size
amqp-framesize = 131072

# Heartbeat interval in seconds (0 = disabled)
#amqp-heartbeat = 0

# Connection sharing
#share-connection = 1
```

#### User Authentication via Environment Variables

```bash
export USER_AMQP=username
export PASSWORD_AMQP=password
```

#### Connection String

Format: `hostname;port;username;password`

Example: `localhost;5672;guest;guest`

#### Setup RabbitMQ Queue

```bash
# Enable management plugin
sudo rabbitmq-plugins enable rabbitmq_management

# Create queue
sudo rabbitmqadmin -u guest -p guest -V / declare queue name=myqueue durable=false auto_delete=true

# Bind queue to exchange
rabbitmqadmin -u guest -p guest -V / declare binding source=amq.topic destination=myqueue routing_key=deepstream-analytics

# List queues
sudo rabbitmqctl list_queues
```

#### Consume Messages

```bash
# Install amqp-tools
sudo apt-get install amqp-tools

# Consume from queue
amqp-consume -q "myqueue" -r "deepstream-analytics" -e "amq.topic" cat
```

---

### Azure IoT Protocol Adaptor

#### Dependencies Installation

```bash
# Install dependencies
sudo apt-get update
sudo apt-get install -y libcurl4-openssl-dev libssl-dev uuid-dev libglib2.0-dev

# Build Azure IoT SDK
git clone https://github.com/Azure/azure-iot-sdk-c.git
cd azure-iot-sdk-c
git checkout tags/1.11.0
git submodule update --init

# Modify CMakeLists.txt:
# - Line 61: set build_as_dynamic to ON
# - Line 65: set use_edge_modules to ON

mkdir cmake && cd cmake
cmake ..
cmake --build .
sudo make install
```

#### Configuration File (cfg_azure.txt)

```ini
[message-broker]
# Azure IoT Hub connection string
#connection_str = HostName=<my-hub>.azure-devices.net;DeviceId=<device_id>;SharedAccessKey=<my-policy-key>

# Custom message properties (key=value pairs)
#custom_msg_properties = key1=value1;key2=value2;

# Connection sharing
#share-connection = 1

# Cleanup timeout in seconds during disconnect
#cleanup-timeout = 20
```

#### Connection String

Full Azure IoT Hub connection string:
```text
HostName=<my-hub>.azure-devices.net;DeviceId=<device_id>;SharedAccessKey=<my-policy-key>
```

---

## nvmsgbroker Library Configuration

The nvmsgbroker library (wrapper around protocol adaptors) has its own configuration:

### Configuration File (cfg_nvmsgbroker.txt)

```ini
[nvmsgbroker]
# Enable auto-reconnection (0=disable, 1=enable)
auto-reconnect=1

# Reconnection retry interval in seconds
retry-interval=1

# Maximum retry limit in seconds
max-retry-limit=3600

# Work interval in microseconds
work-interval=10000
```

---

## Message Payload Formats

### Full Schema (payload-type=0)

Generates separate JSON payload per object:

```json
{
  "messageid": "unique-uuid",
  "mdsversion": "1.0",
  "@timestamp": "2024-01-15T10:30:00.000Z",
  "place": {
    "id": "1",
    "name": "ParkingLot_A",
    "type": "garage",
    "location": {
      "lat": 30.32,
      "lon": -40.55,
      "alt": 100.0
    }
  },
  "sensor": {
    "id": "CAMERA_001",
    "type": "Camera",
    "description": "Entrance Camera"
  },
  "analyticsModule": {
    "id": "ANALYTICS_001",
    "description": "Vehicle Detection",
    "source": "ResNet18_TrafficCamNet",
    "version": "1.0"
  },
  "object": {
    "id": "1",
    "speed": 0,
    "direction": 0,
    "orientation": 0,
    "vehicle": {
      "type": "car",
      "make": "",
      "model": "",
      "color": "",
      "license": ""
    },
    "bbox": {
      "topleftx": 100,
      "toplefty": 200,
      "bottomrightx": 300,
      "bottomrighty": 400
    },
    "location": {
      "lat": 0,
      "lon": 0,
      "alt": 0
    },
    "coordinate": {
      "x": 0,
      "y": 0,
      "z": 0
    }
  },
  "event": {
    "id": "event-uuid",
    "type": "entry"
  },
  "videoPath": ""
}
```

### Minimal Schema (payload-type=1)

Multiple objects in single JSON payload:

```json
{
  "messageid": "unique-uuid",
  "mdsversion": "1.0",
  "@timestamp": "2024-01-15T10:30:00.000Z",
  "sensorId": "CAMERA_001",
  "objects": [
    {
      "id": "1",
      "type": "car",
      "confidence": 0.95,
      "bbox": {
        "topleftx": 100,
        "toplefty": 200,
        "bottomrightx": 300,
        "bottomrighty": 400
      }
    },
    {
      "id": "2",
      "type": "person",
      "confidence": 0.88,
      "bbox": {
        "topleftx": 400,
        "toplefty": 150,
        "bottomrightx": 450,
        "bottomrighty": 350
      }
    }
  ]
}
```

---

## Troubleshooting

### Common Issues

1. **"Connection refused" error**
   - Verify broker is running and accessible
   - Check firewall rules
   - Verify connection string format

2. **"Library not found" error**
   - Verify proto-lib path exists
   - Check library dependencies: `ldd /opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so`

3. **Messages not appearing in broker**
   - Verify topic exists (or auto-create is enabled)
   - Check broker logs for errors
   - Enable DeepStream logging (see below)

4. **TLS/SSL connection failures**
   - Verify certificate paths
   - Check certificate validity
   - Ensure proper permissions on key files

### Enable DeepStream Logging

```bash
# Setup logger
chmod u+x <DS_ROOT>/src/utils/nvds_logger/setup_nvds_logger.sh
sudo <DS_ROOT>/src/utils/nvds_logger/setup_nvds_logger.sh

# View logs
tail -f /tmp/nvds/ds.log
```

---

## Best Practices

1. **Use async mode** (`sync=0`) for better performance
2. **Configure appropriate batch sizes** in nvmsgconv's `frame-interval`
3. **Use minimal schema** (`payload-type=1`) for lower bandwidth
4. **Enable auto-reconnect** in nvmsgbroker config for resilience
5. **Use environment variables** for credentials instead of config files
6. **Monitor broker lag** to ensure consumers keep up
7. **Use TLS/SSL** for production deployments
8. **Implement retry logic**: Handle transient Kafka failures (see Part 1 above for Python examples)
9. **Batch messages**: Reduce network overhead (see Part 1 above for batching patterns)
10. **Use appropriate partitioning**: Use frame_number or source_id as key
11. **Handle backpressure**: Pause pipeline if Kafka is slow
12. **Monitor producer metrics**: Track send rates and errors
13. **Clean shutdown**: Flush and close producers properly

---

## Related Documentation

- **GStreamer Plug

# Advanced Media Extraction with MediaExtractor, MediaChunk, and FrameSampler

## Overview

The `pyservicemaker.utils` module provides advanced utilities for extracting frames from media sources with precise control over timing, sampling, and batch processing. These utilities are particularly useful for:
- Processing specific time segments (chunks) of video files
- Frame sampling at precise intervals
- Batch processing multiple video sources
- Dynamic source addition during runtime
- Seeking and timestamp-based frame extraction

## Core Classes

### MediaChunk

A `MediaChunk` represents a specific time segment of a media source with sampling parameters.

**Constructor**:
```python
from pyservicemaker.utils import MediaChunk

chunk = MediaChunk(
    source="path/to/video.mp4",
    start_pts=0,           # Start timestamp in nanoseconds
    duration=-1,           # Duration in nanoseconds (-1 = entire file)
    interval=0             # Frame sampling interval in nanoseconds (0 = no skipping)
)
```

**Parameters**:
- `source` (str): File path or URL of media source
- `start_pts` (int): Start timestamp in nanoseconds (default: 0)
- `duration` (int): Duration in nanoseconds (default: -1 for entire file)
- `interval` (int): Frame sampling interval in nanoseconds (default: 0 for no frame skipping)

**Properties**:
- `source`: Returns the media source path/URL
- `start_pts`: Returns the start timestamp
- `duration`: Returns the duration
- `interval`: Returns the sampling interval

**Example**:
```python
from pyservicemaker.utils import MediaChunk

# Extract entire video
chunk1 = MediaChunk(source="video1.mp4")

# Extract 10 seconds starting from 5 seconds
chunk2 = MediaChunk(
    source="video2.mp4",
    start_pts=5_000_000_000,   # 5 seconds in nanoseconds
    duration=10_000_000_000     # 10 seconds in nanoseconds
)

# Extract with frame sampling every 0.5 seconds
chunk3 = MediaChunk(
    source="video3.mp4",
    interval=500_000_000        # 0.5 seconds in nanoseconds
)

# Extract 30 seconds starting at 1 minute, sample every 2 seconds
chunk4 = MediaChunk(
    source="video4.mp4",
    start_pts=60_000_000_000,   # 1 minute
    duration=30_000_000_000,    # 30 seconds
    interval=2_000_000_000      # 2 seconds
)
```

### VideoFrame

Represents a decoded video frame with timestamp information.

**Constructor**:
```python
from pyservicemaker.utils import VideoFrame

frame = VideoFrame(data=tensor, timestamp=pts)
```

**Parameters**:
- `data` (Tensor): Frame data as DeepStream tensor
- `timestamp` (int): Frame timestamp in nanoseconds (default: -1)

**Properties**:
- `timestamp`: Returns the frame timestamp
- `tensor`: Returns the frame data tensor

**Example**:
```python
# Typically created internally by FrameSampler
# Access in your processing code:
for frame in output_queue:
    if frame is None:
        break  # End of stream
    
    print(f"Frame timestamp: {frame.timestamp} ns")
    tensor_data = frame.tensor
    # Process tensor_data...
```

### FrameSampler

Manages frame sampling logic based on MediaChunk specifications.

**Constructor**:
```python
from pyservicemaker.utils import FrameSampler

sampler = FrameSampler(chunk=media_chunk, seek_fn=None)
```

**Parameters**:
- `chunk` (MediaChunk): Media chunk specification
- `seek_fn` (Callable, optional): Function to call for seeking (default: None)

**Properties**:
- `done`: Returns True when chunk processing is complete

**Methods**:

#### `sample(buffer, pts)`
Sample a frame based on chunk specifications.

**Parameters**:
- `buffer`: Buffer containing frame data
- `pts` (int): Presentation timestamp in nanoseconds

**Returns**: `VideoFrame` object if frame should be sampled, `None` otherwise

**Example** (typically used internally):
```python
# Internal usage by MediaExtractor
sampler = FrameSampler(chunk)
frame = sampler.sample(buffer, pts)
if frame:
    queue.put(frame)
elif sampler.done:
    print("Chunk processing complete")
```

### MediaExtractor

High-level utility for extracting frames from media sources with advanced features.

**Constructor**:
```python
from pyservicemaker.utils import MediaExtractor, MediaChunk

extractor = MediaExtractor(
    chunks=[chunk1, chunk2, ...],  # List of MediaChunk objects
    batch_size=0,                   # 0 = no batching, N = batch N sources
    scaling=(1920, 1080),           # Target resolution (width, height)
    n_thread=1,                     # Number of worker threads
    q_size=1,                       # Output queue capacity
    enable_seek=False,              # Enable seeking for frame retrieval
    blocking=False                  # Block when queue is full
)
```

**Parameters**:
- `chunks` (List[MediaChunk], optional): List of media chunks to process
- `batch_size` (int): Batch size for processing (0 = no batching, default: 0)
- `scaling` (Tuple[int, int]): Target resolution (width, height), default: (1920, 1080)
- `n_thread` (int): Number of worker threads (default: 1)
- `q_size` (int): Output queue capacity (default: 1)
- `enable_seek` (bool): Enable seeking for efficient frame retrieval (default: False)
- `blocking` (bool): Block when output queue is full (default: False)

**Methods**:

#### `__call__()`
Start extraction and return output queues.

**Returns**: List of `queue.Queue` objects containing `VideoFrame` objects

#### `append(chunk)`
Dynamically add a new chunk during runtime (only if initialized without chunks).

**Parameters**:
- `chunk` (MediaChunk): Media chunk to add

**Returns**: `queue.Queue` for the added chunk

**Context Manager Support**:
MediaExtractor supports context manager protocol for automatic cleanup.

```python
with MediaExtractor(chunks=[...]) as extractor:
    queues = extractor()
    # Process frames...
# Automatic cleanup on exit
```

## Usage Patterns

### Pattern 1: Extract Entire Video Files

Extract all frames from multiple video files.

```python
from pyservicemaker.utils import MediaExtractor, MediaChunk
import torch  # pip install torch torchvision (not in base DS container)

def extract_all_frames(video_paths):
    """Extract all frames from multiple videos"""
    # Create chunks for each video
    chunks = [MediaChunk(source=path) for path in video_paths]
    
    # Create extractor
    with MediaExtractor(chunks=chunks, n_thread=len(video_paths), q_size=10) as extractor:
        # Start extraction
        queues = extractor()
        
        # Process frames from each video
        for i, q in enumerate(queues):
            print(f"Processing video {i}: {video_paths[i]}")
            frame_count = 0
            
            while True:
                frame = q.get()
                if frame is None:
                    break  # End of stream
                
                # Convert to PyTorch tensor
                torch_tensor = torch.utils.dlpack.from_dlpack(frame.tensor)
                
                # Process frame
                print(f"  Frame {frame_count}: timestamp={frame.timestamp} ns, shape={torch_tensor.shape}")
                
                frame_count += 1
            
            print(f"  Total frames: {frame_count}")

# Example usage
video_files = ["video1.mp4", "video2.mp4", "video3.mp4"]
extract_all_frames(video_files)
```

### Pattern 2: Extract Time Segments

Extract specific time segments from videos.

```python
from pyservicemaker.utils import MediaExtractor, MediaChunk

def extract_time_segments(video_path, segments):
    """
    Extract specific time segments from a video
    
    Args:
        video_path: Path to video file
        segments: List of (start_time, duration) tuples in seconds
    """
    # Create chunks for each segment
    chunks = [
        MediaChunk(
            source=video_path,
            start_pts=int(start * 1e9),      # Convert to nanoseconds
            duration=int(duration * 1e9)      # Convert to nanoseconds
        )
        for start, duration in segments
    ]
    
    with MediaExtractor(chunks=chunks, n_thread=1, q_size=5) as extractor:
        queues = extractor()
        
        for i, (q, (start, duration)) in enumerate(zip(queues, segments)):
            print(f"Segment {i}: {start}s - {start+duration}s")
            frames = []
            
            while True:
                frame = q.get()
                if frame is None:
                    break
                frames.append(frame)
            
            print(f"  Extracted {len(frames)} frames")
            
            # Process frames for this segment
            for frame in frames:
                # Your processing logic here
                pass

# Example: Extract three 10-second segments
segments = [
    (0, 10),      # First 10 seconds
    (30, 10),     # 10 seconds starting at 30s
    (60, 10)      # 10 seconds starting at 1 minute
]
extract_time_segments("long_video.mp4", segments)
```

### Pattern 3: Frame Sampling at Intervals

Extract frames at specific intervals (e.g., every N seconds).

```python
from pyservicemaker.utils import MediaExtractor, MediaChunk
import cv2  # pip install opencv-python-headless (not in base DS container)
import numpy as np
import torch  # pip install torch torchvision (not in base DS container)

def sample_frames_at_interval(video_path, interval_sec=1.0, output_dir="./sampled"):
    """
    Sample frames at regular intervals
    
    Args:
        video_path: Path to video file
        interval_sec: Sampling interval in seconds
        output_dir: Directory to save sampled frames
    """
    import os
    os.makedirs(output_dir, exist_ok=True)
    
    # Create chunk with sampling interval
    chunk = MediaChunk(
        source=video_path,
        interval=int(interval_sec * 1e9)  # Convert to nanoseconds
    )
    
    with MediaExtractor(chunks=[chunk], q_size=10) as extractor:
        queues = extractor()
        q = queues[0]
        
        frame_idx = 0
        while True:
            frame = q.get()
            if frame is None:
                break
            
            # Convert to numpy for saving
            torch_tensor = torch.utils.dlpack.from_dlpack(frame.tensor)
            frame_np = torch_tensor.cpu().numpy()
            
            # Convert RGB to BGR for OpenCV
            frame_bgr = cv2.cvtColor(frame_np, cv2.COLOR_RGB2BGR)
            
            # Save frame
            timestamp_sec = frame.timestamp / 1e9
            filename = f"{output_dir}/frame_{frame_idx:06d}_t{timestamp_sec:.3f}s.jpg"
            cv2.imwrite(filename, frame_bgr)
            
            print(f"Saved: {filename}")
            frame_idx += 1
        
        print(f"Total sampled frames: {frame_idx}")

# Sample frames every 2 seconds
sample_frames_at_interval("video.mp4", interval_sec=2.0)
```

### Pattern 4: Batch Processing Multiple Sources

Process multiple video sources in batches with scaling.

```python
from pyservicemaker.utils import MediaExtractor, MediaChunk
import torch  # pip install torch torchvision (not in base DS container)

def batch_process_videos(video_paths, batch_size=4, target_resolution=(1280, 720)):
    """
    Process multiple videos in batches with scaling
    
    Args:
        video_paths: List of video file paths
        batch_size: Number of videos to process in parallel
        target_resolution: Target (width, height) for scaling
    """
    # Create chunks
    chunks = [MediaChunk(source=path) for path in video_paths]
    
    # Create extractor with batching
    with MediaExtractor(
        chunks=chunks,
        batch_size=batch_size,
        scaling=target_resolution,
        n_thread=1,
        q_size=10
    ) as extractor:
        queues = extractor()
        
        # Process each batch queue
        for batch_idx, q in enumerate(queues):
            print(f"Processing batch {batch_idx}")
            frame_count = 0
            
            while True:
                frame = q.get()
                if frame is None:
                    break
                
                # Frame is already scaled to target resolution
                torch_tensor = torch.utils.dlpack.from_dlpack(frame.tensor)
                print(f"  Batch {batch_idx}, Frame {frame_count}: shape={torch_tensor.shape}")
                
                # Process batched frame
                # ... your processing logic ...
                
                frame_count += 1
            
            print(f"  Batch {batch_idx} complete: {frame_count} frames")

# Process 12 videos in batches of 4
videos = [f"video_{i}.mp4" for i in range(12)]
batch_process_videos(videos, batch_size=4, target_resolution=(1280, 720))
```

### Pattern 5: Dynamic Source Addition

Add video sources dynamically during runtime.

```python
from pyservicemaker.utils import MediaExtractor, MediaChunk
import threading
import time

def dynamic_extraction_system(n_threads=2):
    """
    System that accepts video processing requests dynamically
    """
    # Create extractor without initial chunks (for dynamic addition)
    with MediaExtractor(chunks=None, n_thread=n_threads, q_size=5) as extractor:
        # Start extractor threads
        extractor()
        
        def process_chunk(chunk, queue):
            """Process frames from a chunk"""
            print(f"Processing: {chunk.source}")
            frame_count = 0
            
            while True:
                frame = queue.get()
                if frame is None:
                    break
                
                # Process frame
                frame_count += 1
            
            print(f"Completed: {chunk.source} ({frame_count} frames)")
        
        # Simulate dynamic requests
        video_requests = [
            ("video1.mp4", 0, 10),    # (path, start_sec, duration_sec)
            ("video2.mp4", 5, 15),
            ("video3.mp4", 0, 20),
            ("video4.mp4", 10, 10),
        ]
        
        threads = []
        for path, start_sec, duration_sec in video_requests:
            # Create chunk
            chunk = MediaChunk(
                source=path,
                start_pts=int(start_sec * 1e9),
                duration=int(duration_sec * 1e9)
            )
            
            # Add to extractor (returns queue for this chunk)
            q = extractor.append(chunk)
            
            # Process in separate thread
            t = threading.Thread(target=process_chunk, args=(chunk, q))
            t.start()
            threads.append(t)
            
            # Simulate delay between requests
            time.sleep(0.5)
        
        # Wait for all processing to complete
        for t in threads:
            t.join()
        
        print("All requests processed")

# Run dynamic extraction system
dynamic_extraction_system(n_threads=2)
```

### Pattern 6: Frame Extraction with Seeking

Enable seeking for efficient frame retrieval with large intervals.

```python
from pyservicemaker.utils import MediaExtractor, MediaChunk

def extract_keyframes_with_seeking(video_path, keyframe_interval_sec=10.0):
    """
    Extract keyframes efficiently using seeking
    
    Args:
        video_path: Path to video file
        keyframe_interval_sec: Interval between keyframes in seconds
    """
    # Create chunk with large interval
    chunk = MediaChunk(
        source=video_path,
        interval=int(keyframe_interval_sec * 1e9)
    )
    
    # Enable seeking for efficient frame retrieval
    with MediaExtractor(
        chunks=[chunk],
        enable_seek=True,  # Enable seeking
        q_size=5
    ) as extractor:
        queues = extractor()
        q = queues[0]
        
        keyframes = []
        while True:
            frame = q.get()
            if frame is None:
                break
            
            keyframes.append(frame)
            print(f"Keyframe {len(keyframes)}: timestamp={frame.timestamp/1e9:.2f}s")
        
        print(f"Extracted {len(keyframes)} keyframes")
        return keyframes

# Extract keyframes every 10 seconds
keyframes = extract_keyframes_with_seeking("long_video.mp4", keyframe_interval_sec=10.0)
```

### Pattern 7: Blocking Mode for Controlled Processing

Use blocking mode to control frame processing rate.

```python
from pyservicemaker.utils import MediaExtractor, MediaChunk
import time

def controlled_frame_processing(video_path, processing_delay=0.1):
    """
    Process frames with controlled rate using blocking mode
    
    Args:
        video_path: Path to video file
        processing_delay: Simulated processing delay per frame
    """
    chunk = MediaChunk(source=video_path)
    
    # Use blocking mode with small queue
    with MediaExtractor(
        chunks=[chunk],
        q_size=2,          # Small queue
        blocking=True      # Block when queue is full
    ) as extractor:
        queues = extractor()
        q = queues[0]
        
        frame_count = 0
        while True:
            frame = q.get()
            if frame is None:
                break
            
            # Simulate slow processing
            print(f"Processing frame {frame_count}...")
            time.sleep(processing_delay)
            
            frame_count += 1
        
        print(f"Processed {frame_count} frames")

# Process with controlled rate
controlled_frame_processing("video.mp4", processing_delay=0.1)
```

## Advanced Usage

### Multi-Threaded Parallel Extraction

Process multiple videos in parallel using multiple threads.

```python
from pyservicemaker.utils import MediaExtractor, MediaChunk
from concurrent.futures import ThreadPoolExecutor
import torch  # pip install torch torchvision (not in base DS container)

def parallel_video_analysis(video_paths, n_workers=4):
    """
    Analyze multiple videos in parallel
    
    Args:
        video_paths: List of video file paths
        n_workers: Number of parallel workers
    """
    # Create chunks
    chunks = [MediaChunk(source=path) for path in video_paths]
    
    # Create extractor with multiple threads
    with MediaExtractor(
        chunks=chunks,
        n_thread=n_workers,
        q_size=10
    ) as extractor:
        queues = extractor()
        
        def analyze_video(video_idx, queue, video_path):
            """Analyze a single video"""
            print(f"Analyzing: {video_path}")
            
            frame_stats = {
                'count': 0,
                'total_intensity': 0.0,
                'timestamps': []
            }
            
            while True:
                frame = queue.get()
                if frame is None:
                    break
                
                # Analyze frame
                torch_tensor = torch.utils.dlpack.from_dlpack(frame.tensor)
                mean_intensity = torch_tensor.float().mean().item()
                
                frame_stats['count'] += 1
                frame_stats['total_intensity'] += mean_intensity
                frame_stats['timestamps'].append(frame.timestamp)
            
            # Compute statistics
            avg_intensity = frame_stats['total_intensity'] / frame_stats['count']
            duration_sec = (frame_stats['timestamps'][-1] - frame_stats['timestamps'][0]) / 1e9
            
            return {
                'video': video_path,
                'frames': frame_stats['count'],
                'avg_intensity': avg_intensity,
                'duration': duration_sec
            }
        
        # Process all videos in parallel
        with ThreadPoolExecutor(max_workers=n_workers) as executor:
            futures = [
                executor.submit(analyze_video, i, q, path)
                for i, (q, path) in enumerate(zip(queues, video_paths))
            ]
            
            results = [f.result() for f in futures]
        
        # Print results
        for result in results:
            print(f"\nVideo: {result['video']}")
            print(f"  Frames: {result['frames']}")
            print(f"  Duration: {result['duration']:.2f}s")
            print(f"  Avg Intensity: {result['avg_intensity']:.2f}")

# Analyze 8 videos with 4 workers
videos = [f"video_{i}.mp4" for i in range(8)]
parallel_video_analysis(videos, n_workers=4)
```

### Combining with Inference Pipeline

Extract frames and run inference on them.

```python
from pyservicemaker.utils import MediaExtractor, MediaChunk
from pyservicemaker import Pipeline, Flow
import torch  # pip install torch torchvision (not in base DS container)

def extract_and_infer(video_path, model_config, segment_duration=30):
    """
    Extract video segments and run inference on each
    
    Args:
        video_path: Path to video file
        model_config: Path to inference model config
        segment_duration: Duration of each segment in seconds
    """
    import cv2  # pip install opencv-python-headless (not in base DS container)
    
    # Get video duration (simplified - use actual video metadata in production)
    cap = cv2.VideoCapture(video_path)
    fps = cap.get(cv2.CAP_PROP_FPS)
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    total_duration = total_frames / fps
    cap.release()
    
    # Create chunks for each segment
    n_segments = int(total_duration / segment_duration) + 1
    chunks = [
        MediaChunk(
            source=video_path,
            start_pts=int(i * segment_duration * 1e9),
            duration=int(segment_duration * 1e9)
        )
        for i in range(n_segments)
    ]
    
    # Extract frames
    with MediaExtractor(chunks=chunks, n_thread=2, q_size=10) as extractor:
        queues = extractor()
        
        for seg_idx, q in enumerate(queues):
            print(f"Processing segment {seg_idx}...")
            
            # Collect frames from segment
            frames = []
            while True:
                frame = q.get()
                if frame is None:
                    break
                frames.append(frame)
            
            print(f"  Segment {seg_idx}: {len(frames)} frames")
            
            # Run inference on frames (simplified example)
            for frame in frames:
                torch_tensor = torch.utils.dlpack.from_dlpack(frame.tensor)
                # Run your inference model here
                # results = model(torch_tensor)
                pass

# Extract and infer on 30-second segments
extract_and_infer("long_video.mp4", "model_config.yml", segment_duration=30)
```

## Best Practices

### 1. Timestamp Conversion
Always use nanoseconds for timestamps:
```python
# Convert seconds to nanoseconds
seconds = 10.5
nanoseconds = int(seconds * 1e9)

# Convert nanoseconds to seconds
nanoseconds = 10_500_000_000
seconds = nanoseconds / 1e9
```

### 2. Queue Size Management
Choose appropriate queue size based on memory and processing speed:
```python
# Small queue for memory-constrained systems
extractor = MediaExtractor(chunks=[...], q_size=2)

# Larger queue for smooth processing
extractor = MediaExtractor(chunks=[...], q_size=20)

# Use blocking mode if processing is slow
extractor = MediaExtractor(chunks=[...], q_size=5, blocking=True)
```

### 3. Thread Count Selection
```python
# Single thread for sequential processing
extractor = MediaExtractor(chunks=[...], n_thread=1)

# Multiple threads for parallel processing
extractor = MediaExtractor(chunks=[...], n_thread=4)

# Match thread count to CPU cores
import os
n_cores = os.cpu_count()
extractor = MediaExtractor(chunks=[...], n_thread=n_cores)
```

### 4. Seeking Optimization
Enable seeking for large sampling intervals:
```python
# Enable seeking when interval > 1 second
if interval_sec > 1.0:
    extractor = MediaExtractor(chunks=[...], enable_seek=True)
else:
    extractor = MediaExtractor(chunks=[...], enable_seek=False)
```

### 5. Context Manager Usage
Always use context manager for automatic cleanup:
```python
# Good: Automatic cleanup
with MediaExtractor(chunks=[...]) as extractor:
    queues = extractor()
    # Process frames...
# Cleanup happens automatically

# Avoid: Manual cleanup required
extractor = MediaExtractor(chunks=[...])
queues = extractor()
# Must manually clean up
```

### 6. Error Handling
```python
from pyservicemaker.utils import MediaExtractor, MediaChunk

def safe_extraction(video_paths):
    """Extract frames with error handling"""
    chunks = [MediaChunk(source=path) for path in video_paths]
    
    try:
        with MediaExtractor(chunks=chunks, q_size=10) as extractor:
            queues = extractor()
            
            for i, q in enumerate(queues):
                try:
                    while True:
                        frame = q.get(timeout=30)  # Timeout to detect stalls
                        if frame is None:
                            break
                        
                        # Process frame
                        # ...
                        
                except Exception as e:
                    print(f"Error processing video {i}: {e}")
                    continue
    
    except Exception as e:
        print(f"Extraction error: {e}")
```

## Performance Tips

### 1. Batch Processing
Use batching for multiple sources:
```python
# Process 12 videos in batches of 4
extractor = MediaExtractor(
    chunks=chunks,
    batch_size=4,  # Process 4 at a time
    scaling=(1280, 720)
)
```

### 2. Memory Management
Control memory usage with queue size:
```python
# Low memory: small queue
extractor = MediaExtractor(chunks=[...], q_size=2)

# High throughput: larger queue
extractor = MediaExtractor(chunks=[...], q_size=20)
```

### 3. Parallel Processing
Use multiple threads for I/O-bound tasks:
```python
# Process 8 videos with 4 threads
extractor = MediaExtractor(
    chunks=chunks,
    n_thread=4
)
```

## Common Use Cases

### 1. Video Thumbnail Generation
Extract keyframes at regular intervals for thumbnails.

### 2. Video Segmentation
Split long videos into processable segments.

### 3. Frame Sampling for Training Data
Extract frames at intervals for ML training datasets.

### 4. Video Quality Analysis
Sample frames to analyze video quality metrics.

### 5. Event Detection
Extract frames around specific timestamps for event analysis.

### 6. Multi-Video Synchronization
Process multiple synchronized video sources in batches.

## Troubleshooting

### Issue 1: Frames Not Extracted
**Solution**: Check that source path is valid, verify timestamps are in nanoseconds

### Issue 2: Memory Issues
**Solution**: Reduce `q_size`, process frames immediately, use smaller batches

### Issue 3: Slow Extraction
**Solution**: Enable seeking for large intervals, increase thread count, use batching

### Issue 4: Queue Timeout
**Solution**: Increase queue size, enable blocking mode, check video file integrity

## Related APIs

- **BufferProvider/Feeder**: See `buffer_apis.md`
- **BufferRetriever/Receiver**: See `buffer_apis.md`
- **Pipeline API**: See `service_maker_api.md`

## Summary

The MediaExtractor, MediaChunk, and FrameSampler utilities provide powerful capabilities for advanced frame extraction:

1. **MediaChunk**: Define time segments and sampling parameters
2. **FrameSampler**: Intelligent frame sampling based on timestamps
3. **MediaExtractor**: High-level extraction with batching, threading, and seeking
4. **VideoFrame**: Container for extracted frames with timestamps

Key features:
- Precise timestamp-based extraction
- Frame sampling at intervals
- Batch processing multiple sources
- Dynamic source addition
- Seeking optimization
- Multi-threaded parallel processing
- Context manager support for cleanup

These utilities are ideal for video analysis, training data preparation, thumbnail generation, and any application requiring precise frame extraction from video sources.

# nvdsmetamux Configuration Reference

## Overview

The `nvdsmetamux` GStreamer plugin performs batch metadata multiplexing for the same source and the "same" frame. This plugin is essential for pipelines where multiple inference models process the same video stream parallelly and their metadata needs to be merged.

### Key Concepts

- **Same Frame Matching**: The "same" frame is determined based on the frame PTS (Presentation Timestamp). The plugin searches for the nearest frame PTS of the same source.
- **PTS Tolerance**: There is a configurable PTS difference tolerance for matching frames. If the PTS difference exceeds this tolerance, frames are not considered the same.
- **Active Pad Selection**: Applications can select which sink pad's video frame will be passed to the source pad.
- **Metadata Merging**: The plugin merges metadata from multiple inference models, allowing you to combine results from different GIEs.
- **Metadata Filtering**: Applications can configure to filter metadata based on source IDs from specific model.

---

## GStreamer Element Properties

The `nvdsmetamux` element exposes the following GStreamer properties:

### Core Properties

| Property | Type | Description | Default |
|----------|------|-------------|---------|
| `active-pad` | string | Active sink pad whose buffer will transfer to source pad | null |
| `config-file` | string | Path to the nvdsmetamux configuration file | null |
| `pts-tolerance` | int64 | Time difference tolerance when searching for the same frame of the same source ID (in microseconds) | 60000 |
| `name` | string | The name of the GStreamer object | "nvdsmetamux0" |
| `parent` | GstObject | The parent of the GStreamer object | - |

### Latency Properties

| Property | Type | Description | Default |
|----------|------|-------------|---------|
| `latency` | uint64 | Additional latency in live mode to allow upstream to take longer to produce buffers (in nanoseconds) | 0 |
| `min-upstream-latency` | uint64 | Override minimum latency for dynamically plugged sources with higher latency (in nanoseconds) | 0 |

### Start Time Properties

| Property | Type | Description | Default |
|----------|------|-------------|---------|
| `start-time` | uint64 | Start time to use if `start-time-selection=set` | 18446744073709551615 |
| `start-time-selection` | enum | Decides which start time is output | 0 (zero) |

**start-time-selection Values**:
| Value | Name | Description |
|-------|------|-------------|
| 0 | zero | Start at 0 running time (default) |
| 1 | first | Start at first observed input running time |
| 2 | set | Set start time with `start-time` property |

---

## Configuration File Reference

The `nvdsmetamux` plugin uses a configuration file (specified via `config-file` property) to define metadata muxing behavior.

### Configuration File Format

The configuration file uses INI-style format with the following structure:

```ini
[property]
enable=1
# sink pad name which data will be pass to src pad.
active-pad=sink_0
# default pts-tolerance is 60 ms.
pts-tolerance=60000

[user-configs]

[group-0]
# src-ids-model-<model unique ID>=<source ids>
# mux all source if don't set it.
src-ids-model-1=0;1;2
src-ids-model-2=1;2;3
```

### Property Section

The `[property]` section contains core configuration parameters.

| Config Key | Type | Description | Default |
|------------|------|-------------|---------|
| `enable` | int | Enable the functions of MetaMux (0=disabled, 1=enabled) | 1 |
| `active-pad` | string | Sink pad name whose data will be passed to source pad. Used to synchronize the sources from the branches. | - |
| `pts-tolerance` | int64 | When the difference between the branch source and the base source is larger than this tolerance value, metamux will not combine the metadata into current output (in microseconds) | 60000 |

### User-Configs Section

The `[user-configs]` section is a placeholder for user-defined configurations. This section can be empty or contain custom settings.

### Group Section

The `[group-0]` section (and additional `[group-N]` sections) configures source ID filtering for specific GIE models.

| Config Key Pattern | Type | Description |
|--------------------|------|-------------|
| `src-ids-model-<unique-id>` | string | The source IDs list to be output for the specified GIE. The GIE `unique-id` should be attached as the key postfix. Values are semicolon-separated. If not set, the metadata of all sources from the GIE will be muxed. |

**Example**:
```ini
[group-0]
src-ids-model-1=0;1;2
src-ids-model-2=1;2;3
```
This means:
- Output source 0, source 1, and source 2 inference results from the GIE with `unique-id=1`
- Output source 1, source 2, and source 3 inference results from the GIE with `unique-id=2`

**Note**: If `src-ids-model-<unique-id>` is not set for a particular GIE, the metadata of all sources from the GIE will be muxed by default.

---

## Complete Configuration Examples

### Example 1: Basic MetaMux Configuration

```ini
# config_metamux.txt
[property]
enable=1
# sink pad name which data will be pass to src pad.
active-pad=sink_0
# default pts-tolerance is 60 ms.
pts-tolerance=60000

[user-configs]

[group-0]
# src-ids-model-<model unique ID>=<source ids>
# mux all source if don't set it.
src-ids-model-1=0;1;2;3
src-ids-model-3=0;1;3
```

### Example 2: Configuration with Larger PTS Tolerance

```ini
# config_metamux_large_tolerance.txt
[property]
enable=1
active-pad=sink_0
# Increased tolerance for high-latency pipelines
pts-tolerance=100000

[user-configs]

[group-0]
src-ids-model-1=0;1;2;3
src-ids-model-2=0;1;2;3
```

### Example 3: Multiple GIE Source Filtering

```ini
# config_metamux_multi_gie.txt
[property]
enable=1
active-pad=sink_0
pts-tolerance=60000

[user-configs]

[group-0]
# Primary detector (unique-id=1): output sources 0, 1, 2
src-ids-model-1=0;1;2
# Primary detector (unique-id=2): output sources 1, 2, 3
src-ids-model-2=1;2;3
# Primary detector (unique-id=3): output all sources
src-ids-model-3=0;1;2;3
```

---

## Pipeline Examples

This example uses a `nvstreamdemux` element to split and select the stream, followed by muxing it for parallel inference with multiple models:
- Primary object detector (ResNet18 TrafficCamNet)
- YOLO26s detection model

Pipeline Architecture:
```
4 video streams → nvstreammux → tee
  ├─ Path 0 (Video): queue → nvdsmetamux sink_0
  └─ Path 1 (Inference): queue → nvstreamdemux
       ├─ Stream 0: queue → tee_0
       ├─ Stream 1: queue → tee_1
       ├─ Stream 2: queue → tee_2
       └─ Stream 3: queue → tee_3
            │
            ├─ Branch 1: tee_0,1,2 → nvstreammux → nvinfer(ResNet18) → tracker → metamux sink_1
            └─ Branch 2: tee_1,2,3 → nvstreammux → nvinfer(YOLO26s) → tracker → metamux sink_2
                 │
                 └─ nvdsmetamux → nvmultistreamtiler → nvdsosd → display
```

**Key Implementation Notes**:

1. **Pad naming conventions**:
   - `nvstreamdemux`: Use `"src_%u"` for output pads (auto-assigned in order)
   - `nvdsmetamux`: Use `"sink_%u"` for input pads (auto-assigned in order)
   - `nvstreammux`: Use `"sink_%u"` for input pads
   - `tee`: Use `"src_%u"` for output pads

2. **Linking order matters for `nvdsmetamux`**:
   - First link → `sink_0` (should match `active-pad` in config)
   - Second link → `sink_1`
   - Third link → `sink_2`

3. **`nvstreammux`**: Set `batched-push-timeout` to `40000` (microseconds).

4. **Adaptive batching (process environment)**: Set the `NVSTREAMMUX_ADAPTIVE_BATCHING=yes` environment variable before the pipeline starts. Adaptive batching dynamically adjusts the batch size when a stream finishes early, avoiding empty slots in the batch.

5. **`nvstreamdemux`**: Set `per-stream-eos: True` on `nvstreamdemux` so that each stream sends EOS independently upon completion, rather than waiting for all streams to finish. This prevents the pipeline from hanging while other streams are still active.

**Code Pattern**:
```python
pipeline.add("nvstreammux", "mux", {
    "batch-size": NUM_SOURCES,
    "width": 1920,
    "height": 1080,
    "batched-push-timeout": 40000,
})

pipeline.add("nvstreamdemux", "demux", {"per-stream-eos": True})

# Add queue and tee after demux for each stream
for i in range(NUM_SOURCES):
    pipeline.add("queue", f"queue_demux_{i}", {"max-size-buffers": 100})
    pipeline.add("tee", f"tee_stream_{i}")

# Link demux outputs - uses src_%u template
for i in range(NUM_SOURCES):
    pipeline.link(("demux", f"queue_demux_{i}"), ("src_%u", ""))
    pipeline.link(f"queue_demux_{i}", f"tee_stream_{i}")

# Link to metamux - use sink_%u template, order determines pad assignment
pipeline.link(("queue_video_path", "metamux"), ("", "sink_%u"))  # → sink_0
pipeline.link(("queue_branch1_out", "metamux"), ("", "sink_%u"))  # → sink_1
pipeline.link(("queue_branch2_out", "metamux"), ("", "sink_%u"))  # → sink_2
```

**Configuration File** (`config_metamux.txt`):
```ini
[property]
enable=1
active-pad=sink_0
pts-tolerance=60000

[user-configs]

[group-0]
src-ids-model-1=0;1;2
src-ids-model-2=1;2;3
```

---

## Common Use Cases

### Use Case 1: Multi-Model Inference

Combine results from multiple inference models (e.g., object detection + YOLO26s) into a single output stream.

### Use Case 2: Selective Source Output

Filter which source streams should have their inference results included in the final output using `src-ids-model-<model unique ID>=<source ids>` configuration.

---

## Common Pitfalls

### Pitfall 1: PTS Tolerance Too Small

**Problem**: Frames are not being matched correctly, resulting in missing metadata.

**Wrong**:
```ini
[property]
pts-tolerance=1000  # Too small for variable latency
```

**Correct**:
```ini
[property]
pts-tolerance=60000  # 60ms tolerance
```

### Pitfall 2: Incorrect Active Pad

**Problem**: Wrong video frame is being output to the source pad.

**Solution**: Ensure `active-pad` matches one of your sink pad names (e.g., `sink_0`, `sink_1`).

```ini
[property]
active-pad=sink_0  # Must match an existing sink pad
```

### Pitfall 3: Missing GIE Unique ID in src-ids-model

**Problem**: Source ID filtering not working for a specific model.

**Wrong**:
```ini
[group-0]
src-ids-model=0;1;2;3  # Missing unique-id suffix
```

**Correct**:
```ini
[group-0]
src-ids-model-1=0;1;2;3  # Include the GIE unique-id (1)
```

### Pitfall 5: Missing Required Sections

**Problem**: Configuration file missing required sections.

**Wrong**:
```ini
[property]
enable=1
active-pad=sink_0

# Missing [user-configs] and [group-0] sections
```

**Correct**:
```ini
[property]
enable=1
active-pad=sink_0
pts-tolerance=60000

[user-configs]

[group-0]
src-ids-model-1=0;1;2;3
```

### Pitfall 4: PTS Synchronization Issues

**Problem**: When using separate nvstreammux instances, frames may have different PTS values.

**Solution**:
- Use the `tee` approach when possible to ensure consistent PTS across branches
- Increase `pts-tolerance` if using separate streammux instances
- Set `sync-inputs=0` on nvstreammux for live sources

---

## Best Practices

1. **Use tee for Single Source**: When processing the same streams through multiple models, use a `tee` element after the first nvstreammux to ensure consistent PTS values.

2. **Set Appropriate PTS Tolerance**: Start with the default (60000 microseconds = 60ms) and adjust based on your pipeline's latency characteristics.

3. **Configure Source IDs Explicitly**: Always specify which source IDs should output from each model using `src-ids-model-<model unique ID>=<source ids>` to avoid unexpected metadata merging.

4. **Use Queues**: Add `queue` elements before and after inference elements to prevent pipeline stalls.

5. **Match Batch Sizes**: Ensure batch sizes are consistent across all branches feeding into nvdsmetamux.

---

## Related Documentation

- **GStreamer Plugins Overview**: `gstreamer_plugins.md`
- **Use Cases and Pipelines**: `use_cases_pipelines.md`
- **nvinfer Configuration Reference**: `nvinfer_config.md`
- **Best Practices**: `best_practices.md`

# DeepStream Messaging API Adapter

## Table of Contents

1. [nvds_msgapi Interface](#nvds_msgapi-protocol-adapter-interface)
2. [Implementation Patterns & Skeleton](#building-a-custom-protocol-adapter)
3. [GStreamer Integration](#integrating-with-gst-nvmsgbroker)
4. [Configuration File Format](#configuration-file-format)
5. [nvds_logger](#nvds_logger)
6. [Common Pitfalls](#common-pitfalls-and-troubleshooting)
7. [Checklist](#checklist-for-custom-adapter-development)

---

## Overview

Build a custom messaging adapter for any protocol (NATS, ZeroMQ, Pulsar, custom TCP) by implementing `nvds_msgapi.h` and compiling as a shared library (`.so`).

---

## nvds_msgapi: Protocol Adapter Interface

### Types and Callbacks

Key types/enums from `nvds_msgapi.h` (excerpt):

```c
/** Defines the handle used by messaging API functions. */
typedef void *NvDsMsgApiHandle;
/**
 * Defines events associated with connections to remote entities.
 */
typedef enum {
  /** Specifies that a connection attempt was successful. */
  NVDS_MSGAPI_EVT_SUCCESS,
  /** Specifies disconnection of a connection handle. */
  NVDS_MSGAPI_EVT_DISCONNECT,
  /** Specifies that the remote service is down. */
  NVDS_MSGAPI_EVT_SERVICE_DOWN
} NvDsMsgApiEventType;

/**
 * Defines completion codes for operations in the messaging API.
 */
typedef enum {
  NVDS_MSGAPI_OK,
  NVDS_MSGAPI_ERR,
  NVDS_MSGAPI_UNKNOWN_TOPIC
} NvDsMsgApiErrorType;

typedef void (*nvds_msgapi_connect_cb_t)(NvDsMsgApiHandle h_ptr, NvDsMsgApiEventType ds_evt);

typedef void (*nvds_msgapi_send_cb_t)(void *user_ptr, NvDsMsgApiErrorType completion_flag);

typedef void (*nvds_msgapi_subscribe_request_cb_t)(NvDsMsgApiErrorType flag,
                                                    void *msg,
                                                    int msg_len,
                                                    char *topic,
                                                    void *user_ptr);
```

### Functions to Export

---

#### nvds_msgapi_connect() — Create a Connection

```c
NvDsMsgApiHandle nvds_msgapi_connect(char *connection_str,
                                     nvds_msgapi_connect_cb_t connect_cb,
                                     char *config_path);
```

| Parameter | Type | Description |
|---|---|---|
| `connection_str` | `char *` | Connection parameters in adapter-defined format. NVIDIA convention is `"url;port;topic"`, but the adapter may use any format or accept `NULL`. |
| `connect_cb` | `nvds_msgapi_connect_cb_t` | Callback for events associated with the connection. Invoke with `NVDS_MSGAPI_EVT_SUCCESS` on successful connect, `NVDS_MSGAPI_EVT_DISCONNECT` or `NVDS_MSGAPI_EVT_SERVICE_DOWN` on disconnect/error. |
| `config_path` | `char *` | Path to a configuration file passed to the protocol adapter. May be `NULL`. |

**Return**: `NvDsMsgApiHandle` on success, `NULL` on failure.

---

#### nvds_msgapi_send() — Synchronous Send

```c
NvDsMsgApiErrorType nvds_msgapi_send(NvDsMsgApiHandle h_ptr,
                                     char *topic,
                                     const uint8_t *payload,
                                     size_t nbuf);
```

| Parameter | Type | Description |
|---|---|---|
| `h_ptr` | `NvDsMsgApiHandle` | Connection handle. |
| `topic` | `char *` | Destination topic for the payload. May be `NULL`. |
| `payload` | `const uint8_t *` | Pointer to the message bytes. The message may but need not be a NULL-terminated string. |
| `nbuf` | `size_t` | Number of bytes to send, including the terminating NULL if the message is a string. |

**Return**: Completion code for the send operation.

---

#### nvds_msgapi_send_async() — Asynchronous Send

```c
NvDsMsgApiErrorType nvds_msgapi_send_async(NvDsMsgApiHandle h_ptr,
                                           char *topic,
                                           const uint8_t *payload,
                                           size_t nbuf,
                                           nvds_msgapi_send_cb_t send_callback,
                                           void *user_ptr);
```

| Parameter | Type | Description |
|---|---|---|
| `h_ptr` | `NvDsMsgApiHandle` | Connection handle. |
| `topic` | `char *` | Topic name. May be `NULL`. |
| `payload` | `const uint8_t *` | Pointer to the message bytes. The message may but need not be a NULL-terminated string. |
| `nbuf` | `size_t` | Number of bytes to send, including the terminating NULL if the message is a string. |
| `send_callback` | `nvds_msgapi_send_cb_t` | Callback invoked when the operation completes: `void (*)(void *user_ptr, NvDsMsgApiErrorType completion_flag)`. |
| `user_ptr` | `void *` | Context pointer forwarded verbatim to `send_callback`. |

**Return**: Completion code for the send operation.

---

#### nvds_msgapi_do_work() — Periodic Work

```c
void nvds_msgapi_do_work(NvDsMsgApiHandle h_ptr);
```

Allows the adapter to execute underlying protocol logic — service pending incoming and outgoing messages, perform periodic housekeeping tasks such as sending heartbeats. The client must call this periodically, according to the adapter's requirements. If the adapter uses its own I/O threads, this can be a no-op.

---

#### nvds_msgapi_disconnect() — Terminate a Connection

```c
NvDsMsgApiErrorType nvds_msgapi_disconnect(NvDsMsgApiHandle h_ptr);
```

Terminates a connection. The adapter must release all resources associated with `h_ptr` and must not use the handle again after this call returns.

**Return**: Completion code for the terminate operation.

---

#### nvds_msgapi_getversion() / nvds_msgapi_get_protocol_name()

```c
char *nvds_msgapi_getversion(void);
char *nvds_msgapi_get_protocol_name(void);
```

| Function | Description |
|---|---|
| `nvds_msgapi_getversion()` | Returns the messaging API version string supported by the adapter (e.g., `"2.0"` in `"major.minor"` format). |
| `nvds_msgapi_get_protocol_name()` | Returns the name of the protocol used in the adapter (e.g., `"KAFKA"`, `"MQTT"`). |

---

#### nvds_msgapi_subscribe() — Subscribe

```c
NvDsMsgApiErrorType nvds_msgapi_subscribe(NvDsMsgApiHandle h_ptr,
                                          char **topics,
                                          int num_topics,
                                          nvds_msgapi_subscribe_request_cb_t cb,
                                          void *user_ctx);
```

Subscribes to a remote entity for receiving messages on particular topic(s). The adapter must invoke `cb(flag, msg, msg_len, topic, user_ctx)` for each incoming message on subscribed topics.

The subscribes API **MUST** be implemented; it may be used in the `libnvds_msgbroker.so`.
Multiple topic subscriptions **MUST** be supported.

| Parameter | Type | Description |
|---|---|---|
| `h_ptr` | `NvDsMsgApiHandle` | Connection handle. |
| `topics` | `char **` | Array of topic strings to subscribe for messages. |
| `num_topics` | `int` | Number of topics in the `topics` array. |
| `cb` | `nvds_msgapi_subscribe_request_cb_t` | Callback invoked for each incoming message: `void (*)(NvDsMsgApiErrorType flag, void *msg, int msg_len, char *topic, void *user_ptr)`. Reports consumption status and the received message/payload. |
| `user_ctx` | `void *` | Opaque pointer forwarded verbatim to `cb`. |

**Return**: `NVDS_MSGAPI_OK` on success.

---

#### nvds_msgapi_connection_signature() — Connection Sharing

```c
NvDsMsgApiErrorType nvds_msgapi_connection_signature(char *broker_str,
                                                     char *cfg,
                                                     char *output_str,
                                                     int max_len);
```

Generates a unique connection signature by parsing `broker_str` and `cfg`. A connection signature is a unique string used to identify a connection. It can be retrieved only if the `share-connection` config option is set to `1`.

| Parameter | Type | Description |
|---|---|---|
| `broker_str` | `char *` | Broker connection string used to create the connection. |
| `cfg` | `char *` | Path to the adapter config file. |
| `output_str` | `char *` | **Output**: buffer to write the connection signature string into. |
| `max_len` | `int` | Maximum length of `output_str` buffer. |

**Return**: Valid connection signature in `output_str` on success. Empty string (`""`) in case of errors or if `share-connection` config option is not set to `1`. The signature should be a deterministic function of `broker_str` and `cfg` (e.g., a hash or concatenation of key fields).

---

## Building a Custom Protocol Adapter

### Choosing Your Implementation Pattern

| Pattern | Library characteristics | Built-in reference |
|---------|------------------------|-------------------|
| **A — Library owns threads** | SDK fires per-message callback from its own thread. `do_work()` is a no-op. | Azure IoT SDK |
| **B — Event loop + opaque pointer** | Library has a `poll()` call; per-message opaque `void *` returned verbatim in delivery callback. | Kafka (librdkafka) |
| **C — Event loop + message-ID map** | Library returns integer message-ID from publish; delivery callback receives same ID. | MQTT (libmosquitto) |
| **D — Blocking library** | Only synchronous blocking send. No async or event-loop API. | AMQP (rabbitmq-c), Redis (hiredis) |

`connect_cb` calling convention:
- **Pattern A/C**: call with `NVDS_MSGAPI_EVT_SUCCESS` from library's own connection callback.
- **Pattern B/D**: do NOT call in `nvds_msgapi_connect()`; call with `NVDS_MSGAPI_EVT_SERVICE_DOWN` from `do_work()` or consumer thread on runtime error.

---

### Skeleton Implementation (C)

> **CRITICAL**: `send_async()` completion callback **must fire from a different thread** than the caller. Both legacy and new-api paths hold an internal mutex across `send_async()`. Calling the callback synchronously — including in the error path — causes immediate deadlock. Always return `NVDS_MSGAPI_ERR` directly without invoking the callback on error.

Skeleton below implements **Pattern B**. Adaptation notes for A/C/D follow.

```c
#include <stdlib.h>
#include <string.h>
#include "nvds_msgapi.h"
#include "nvds_logger.h"

#define LOG_CAT "DSLOG:CUSTOM_PROTO"

typedef struct {
    nvds_msgapi_send_cb_t  cb;
    void                  *user_ptr;
} PendingSend;

typedef struct {
    char *server_url;
    int   port;
    int   connected;
    /* your_client_t *client; */
    nvds_msgapi_connect_cb_t connect_cb;
    /* pthread_t consumer_tid; int stop_consumer; */
} CustomAdapterCtx;

/* Fired from your_proto_poll() inside do_work() — satisfies "different thread" requirement */
static void on_send_complete(int success, void *msg_opaque)
{
    PendingSend *ps = (PendingSend *)msg_opaque;
    if (ps->cb) ps->cb(ps->user_ptr, success ? NVDS_MSGAPI_OK : NVDS_MSGAPI_ERR);
    free(ps);
}

static void on_disconnect(void *user_data)
{
    CustomAdapterCtx *ctx = (CustomAdapterCtx *)user_data;
    if (ctx->connect_cb)
        ctx->connect_cb((NvDsMsgApiHandle)ctx, NVDS_MSGAPI_EVT_SERVICE_DOWN);
}

NvDsMsgApiHandle nvds_msgapi_connect(char *connection_str,
                                     nvds_msgapi_connect_cb_t connect_cb,
                                     char *config_path)
{
    nvds_log_open();
    CustomAdapterCtx *ctx = calloc(1, sizeof(CustomAdapterCtx));
    if (!ctx) return NULL;

    ctx->connect_cb = connect_cb;

    if (connection_str) {
        char *tmp = strdup(connection_str);
        char *host = strtok(tmp, ";"), *port = strtok(NULL, ";");
        if (host) ctx->server_url = strdup(host);
        if (port) ctx->port = atoi(port);
        free(tmp);
    }

    /* Parse config_path with GKeyFile if needed */

    /* your_proto_connect(ctx->server_url, ctx->port, on_send_complete, on_disconnect, ctx);
     * On failure: free ctx and return NULL. */

    ctx->connected = 1;
    nvds_log(LOG_CAT, LOG_INFO, "Connected to %s:%d", ctx->server_url, ctx->port);
    return (NvDsMsgApiHandle)ctx;
}

NvDsMsgApiErrorType nvds_msgapi_send(NvDsMsgApiHandle h_ptr,
                                     char *topic, const uint8_t *payload, size_t nbuf)
{
    CustomAdapterCtx *ctx = (CustomAdapterCtx *)h_ptr;
    if (!ctx || !ctx->connected || !topic || !payload || nbuf <= 0) return NVDS_MSGAPI_ERR;

    /* your_proto_send_sync(ctx->client, topic, payload, nbuf); */

    return NVDS_MSGAPI_OK;
}

NvDsMsgApiErrorType nvds_msgapi_send_async(NvDsMsgApiHandle h_ptr,
                                           char *topic, const uint8_t *payload, size_t nbuf,
                                           nvds_msgapi_send_cb_t send_callback, void *user_ptr)
{
    /* Do NOT call send_callback here — deadlock. See CRITICAL note above. */
    CustomAdapterCtx *ctx = (CustomAdapterCtx *)h_ptr;
    if (!ctx || !ctx->connected || !topic || !payload || nbuf <= 0) return NVDS_MSGAPI_ERR;

    PendingSend *ps = malloc(sizeof(PendingSend));
    if (!ps) return NVDS_MSGAPI_ERR;
    ps->cb = send_callback;
    ps->user_ptr = user_ptr;

    /* your_proto_send_async(ctx->client, topic, payload, nbuf, ps);
     * Pass ps as opaque pointer — freed in on_send_complete().
     * If library doesn't copy payload internally, malloc+memcpy here and free in on_send_complete. */

    int rc = 0; /* replace with actual call */
    if (rc != 0) { free(ps); return NVDS_MSGAPI_ERR; }
    return NVDS_MSGAPI_OK;
}

void nvds_msgapi_do_work(NvDsMsgApiHandle h_ptr)
{
    CustomAdapterCtx *ctx = (CustomAdapterCtx *)h_ptr;
    if (!ctx || !ctx->connected) return;
    /* your_proto_poll(ctx->client, 0); */
}

NvDsMsgApiErrorType nvds_msgapi_subscribe(NvDsMsgApiHandle h_ptr,
                                          char **topics, int num_topics,
                                          nvds_msgapi_subscribe_request_cb_t cb, void *user_ctx)
{
    CustomAdapterCtx *ctx = (CustomAdapterCtx *)h_ptr;
    if (!ctx || !ctx->connected) return NVDS_MSGAPI_ERR;

    /* Start dedicated consumer thread (NOT the do_work thread):
     *   pthread_create(&ctx->consumer_tid, NULL, consumer_thread_fn, ctx);
     * consumer_thread_fn: blocking recv loop → cb(flag, msg, len, topic, user_ctx)
     *                     → check ctx->stop_consumer to exit */

    return NVDS_MSGAPI_OK;
}

NvDsMsgApiErrorType nvds_msgapi_disconnect(NvDsMsgApiHandle h_ptr)
{
    CustomAdapterCtx *ctx = (CustomAdapterCtx *)h_ptr;
    if (!ctx) return NVDS_MSGAPI_ERR;
    ctx->connected = 0;

    /* ctx->stop_consumer = 1; pthread_join(ctx->consumer_tid, NULL); */
    /* your_proto_flush(ctx->client); your_proto_disconnect(ctx->client); */

    free(ctx->server_url);
    free(ctx);
    nvds_log_close();
    return NVDS_MSGAPI_OK;
}

char *nvds_msgapi_getversion(void)        { return (char *)"2.0"; }
char *nvds_msgapi_get_protocol_name(void) { return (char *)"CUSTOM_PROTOCOL"; }

NvDsMsgApiErrorType nvds_msgapi_connection_signature(char *broker_str, char *cfg,
                                                     char *output_str, int max_len)
{
    if (!output_str || max_len <= 0) return NVDS_MSGAPI_ERR;
    output_str[0] = '\0';
    if (!broker_str) return NVDS_MSGAPI_ERR;

    /* Check share-connection in cfg; return OK (empty string) to disable sharing.
     *
     * Requirements:
     * - nv_msgbroker wrapper uses signature as map key for connection sharing,
     *   so use SHA-256 (fixed 64-char hex, output buffer ≥ 65 bytes)
     * - Hash PARSED connection values (host, port, credentials), not raw
     *   broker_str or config file path — identical settings from different
     *   config paths must produce the same signature
     */
    snprintf(output_str, max_len, "%s", broker_str); /* TODO: parse → SHA-256 */
    return NVDS_MSGAPI_OK;
}
```

---

### Adaptation Notes for Other Patterns

**Pattern A — Library owns threads (Azure IoT SDK)**
- `do_work()` is a no-op.
- Call `connect_cb(..., NVDS_MSGAPI_EVT_SUCCESS)` from the SDK's connection-status callback.
- In `disconnect()`, wait for in-flight sends to complete before closing the SDK handle.

```c
void nvds_msgapi_do_work(NvDsMsgApiHandle h_ptr) { /* no-op */ }
```

---

**Pattern C — Event loop + message-ID map (MQTT/libmosquitto)**
- Replace `PendingSend *` with `map[mid] = {cb, user_ptr}` protected by a mutex.
- Call `mosquitto_threaded_set(mosq, true)` before starting.
- Return your protocol name (e.g., `"MQTT"`) from `get_protocol_name()`.

```c
void nvds_msgapi_do_work(NvDsMsgApiHandle h_ptr) {
    CustomAdapterCtx *ctx = (CustomAdapterCtx *)h_ptr;
    if (!ctx || !ctx->connected) return;
    mosquitto_loop(ctx->mosq, 0, 1);
}
```

---

**Pattern D — Blocking library (AMQP/rabbitmq-c, Redis/hiredis)**
- Replace `PendingSend` with a mutex-protected queue. `send_async()` enqueues; `do_work()` atomically steals the queue and calls blocking send inline.
- Redis is the recommended Pattern D reference (uses mutex + double-buffered queue swap). AMQP uses a simplified scheme (no lock on its list) that has data-race risk under high concurrency.

```c
void nvds_msgapi_do_work(NvDsMsgApiHandle h_ptr) {
    CustomAdapterCtx *ctx = (CustomAdapterCtx *)h_ptr;
    if (!ctx || !ctx->connected) return;

    pthread_mutex_lock(&ctx->primary_mu);
    QueueNode *list = ctx->primary_head;
    ctx->primary_head = ctx->primary_tail = NULL;
    pthread_mutex_unlock(&ctx->primary_mu);

    while (list) {
        QueueNode *node = list; list = list->next;
        int rc = your_proto_send_sync(ctx->client, node->topic, node->payload, node->payload_len);
        if (node->cb) node->cb(node->user_ptr, rc == 0 ? NVDS_MSGAPI_OK : NVDS_MSGAPI_ERR);
        free(node->topic); free(node->payload); free(node);
    }
}
```

---

### Compilation

```makefile
DS_ROOT := /opt/nvidia/deepstream/deepstream

CC      := gcc
CFLAGS  := -fPIC -Wall -I$(DS_ROOT)/sources/includes $(shell pkg-config --cflags glib-2.0)
LDFLAGS := -shared $(shell pkg-config --libs glib-2.0) -lssl -lcrypto
# LDFLAGS += -lmyproto

TARGET := libnvds_custom_proto.so
SRCS   := custom_proto_adapter.c

$(TARGET): $(SRCS)
	$(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS)

install:
	cp $(TARGET) $(DS_ROOT)/lib/

clean:
	rm -f $(TARGET)

.PHONY: install clean
```

---

## Integrating with Gst-nvmsgbroker

### GStreamer Pipeline

Set the `proto-lib` property to the path of your custom adapter library:

```python
from pyservicemaker import Pipeline
from multiprocessing import Process
import sys

def run_pipeline():
    pipeline = Pipeline("custom-protocol-pipeline")

    # ... source, decoder, mux, inference, tracker, msgconv setup ...

    # Message converter (converts metadata to message format)
    # IMPORTANT: msg2p-newapi=True uses NvDsObjectMeta directly (no NvDsEventMsgMeta required)
    pipeline.add("nvmsgconv", "msgconv", {
        "config": "msgconv_config.txt",
        "payload-type": 0,  # 0=deepstream full schema, 1=minimal
        "msg2p-newapi": True,  # CRITICAL: Use new API to avoid NvDsEventMsgMeta requirement
    })

    pipeline.add("nvmsgbroker", "msgbroker", {
        "proto-lib": "/path/to/libnvds_custom_proto.so",
        "conn-str": "myserver.example.com;4222",
        "topic": "ds-broker",
        "config": "/path/to/custom_proto_config.txt",
        "sync": 0,
        "async": 0,   # Required when using tee or dynamic sources
        "sleep-time": 10,  # ms between do_work() calls
    })

    pipeline.link("msgconv", "msgbroker")

    try:
        pipeline.start().wait()
    except Exception as e:
        print(f"Pipeline error: {e}")
        sys.exit(1)

if __name__ == "__main__":
    process = Process(target=run_pipeline)
    try:
        process.start()
        process.join()
    except KeyboardInterrupt:
        print("\nInterrupted. Terminating...")
        process.terminate()
        process.join()
```

### Gst-nvmsgbroker Properties for Custom Adapters

| Property | Description |
|----------|-------------|
| `proto-lib` | Absolute path to the custom adapter `.so` file. |
| `conn-str` | Connection string passed to `nvds_msgapi_connect()`. Format defined by the adapter. |
| `config` | Path to the adapter's configuration file, passed to `nvds_msgapi_connect()`. |
| `topic` | Topic name passed to `nvds_msgapi_send()` / `nvds_msgapi_send_async()`. |
| `new-api` | Set to `1` to use the `nv_msgbroker` wrapper library (supports auto-reconnect and connection sharing) instead of calling adapter functions directly. |
| `sleep-time` | Milliseconds between consecutive `nvds_msgapi_do_work()` calls. Default 0. |

### Using new-api (nv_msgbroker Wrapper)

When `new-api=1`, the plugin uses the `nv_msgbroker` wrapper library instead of calling your adapter directly. The wrapper provides:

- **Auto-reconnect**: Periodically retries connection after failures. **IMPORTANT** If this `new-api` property is set to true, the reconnection functionality in the custom adapter library needs to be disabled, as the adapter will conflict with the autoreconnect feature built into `nvmsgbroker`.
- **Connection sharing**: Multiple pipeline components can share a single connection within the same process.
- **Work interval control**: Configurable interval for calling `nvds_msgapi_do_work()`.

The wrapper configuration file (`cfg_nvmsgbroker.txt`):

```ini
[nvmsgbroker]
# Enable auto-reconnection (0=disable, 1=enable)
auto-reconnect=1

# Reconnection retry interval in seconds
retry-interval=1

# Maximum retry limit in seconds
max-retry-limit=3600

# Interval for calling do_work(), in microseconds
work-interval=10000
```

The wrapper internally calls your adapter's `nvds_msgapi_*` functions. Your adapter does not need any changes to work with the wrapper.

---

## Configuration File Format

The config file path is passed as `config_path` to `nvds_msgapi_connect()`. Uses GLib key-file (INI) format. Parse with `GKeyFile` API.

```ini
[message-broker]
server=myserver.example.com
port=5672
username=myuser
share-connection=1
```

```c
/* In nvds_msgapi_connect(), after parsing connection_str: */
GKeyFile *kf = g_key_file_new();
if (config_path && g_key_file_load_from_file(kf, config_path, G_KEY_FILE_NONE, NULL)) {
    gchar *val = g_key_file_get_string(kf, "message-broker", "server", NULL);
    if (val) { ctx->server_url = strdup(val); g_free(val); }
    /* read other keys the same way */
}
g_key_file_free(kf);
```

---

## nvds_logger

Use `nvds_log()` instead of `printf` so logs integrate with DeepStream's log system.

```c
#include "nvds_logger.h"
#define LOG_CAT "DSLOG:CUSTOM_PROTO"  // DSLOG: prefix required for filtering

// In nvds_msgapi_connect(): nvds_log_open();
// In nvds_msgapi_disconnect(): nvds_log_close();

nvds_log(LOG_CAT, LOG_INFO, "Connected to %s:%d", host, port);
nvds_log(LOG_CAT, LOG_ERR,  "Send failed: %d", rc);
nvds_log(LOG_CAT, LOG_DEBUG, "Payload (%zu bytes)", nbuf);
// Link: -L${DS_ROOT}/lib -lnvds_logger
// Logs written to: /tmp/nvds/ds.log
```

---

## Common Pitfalls and Troubleshooting

### Thread Safety and Handle Management

Critical rules for adapter implementations:

1. **Handle lifecycle**: The client (Gst-nvmsgbroker) manages the connection handle lifecycle. Once `nvds_msgapi_disconnect()` is called, the handle is retired and must not be used for send or do_work calls.

2. **Thread safety**: If your underlying protocol library is thread-safe, multiple application threads can share connection handles. If not, you must implement locking or document that handles are single-threaded.

3. **do_work() contract**: If your adapter executes in the client thread (no internal worker thread), you must document how often `nvds_msgapi_do_work()` should be called. If the adapter uses its own threads, `do_work()` can be a no-op.

4. **Graceful failure**: The adapter should attempt graceful failure if called with retired handles, but need not guarantee thread-safe behavior in that case.


### CRITICAL: `nvds_msgapi_send_async()` Callback Must Be Invoked from a Different Thread

**Symptom**: Pipeline deadlocks after the very first message send (or immediately on the first error).

**Root cause** (verified in DeepStream source):

Both code paths lock a mutex **before** calling `nvds_msgapi_send_async()`, and the completion callback tries to re-lock the **same** mutex:

- **Legacy path** (`gstnvmsgbroker.cpp` `legacy_gst_nvmsgbroker_render`): locks `self->flowLock`, calls `nvds_msgapi_send_async()`. The callback `nvds_msgapi_send_callback()` does `g_mutex_lock(&self->flowLock)`.
- **New-api path** (`nvmsgbroker.cpp` `nv_msgbroker_send_async`): locks `h_ptr->do_work_thread.lock`, calls `nvds_msgapi_send_async_ptr()`. The callback `adapter_send_cb()` does `pthread_mutex_lock(&myinfo->h_ptr->do_work_thread.lock)`.

This affects **all** custom adapters, regardless of protocol. Two specific scenarios both deadlock:

- **Success path**: calling `send_callback` synchronously inside `send_async` before returning
- **Error path**: calling `send_callback` with an error code before returning `NVDS_MSGAPI_ERR` — the mutex is still held

**Fix**: Use the **`do_work` thread as the callback boundary** — the pattern used by all five built-in adapters. `send_async()` queues or submits the message and returns immediately without calling the callback. The callback fires later on the do_work thread, which is separate from the render thread that holds the mutex. In the error path, always return `NVDS_MSGAPI_ERR` directly without invoking the callback.

Three proven approaches, in order of applicability:

1. **Event loop + opaque pointer (Pattern B — Kafka)**: pass a heap-allocated `{cb, user_ptr}` as the library's per-message opaque pointer; `do_work()` calls `your_proto_poll()` which fires the delivery callback from within, calling the DeepStream callback there.
2. **Blocking library + atomic queue steal (Pattern D — AMQP, Redis)**: `send_async()` enqueues a `QueueNode` (with copied topic/payload); `do_work()` atomically steals the entire queue, calls the blocking library send for each node, and fires the callback inline. No separate worker thread needed.
3. **Library owns threads (Pattern A — Azure)**: `do_work()` is a no-op; the SDK fires callbacks from its own internal threads, which are already different from the render thread.

Copy topic and payload in all cases where the library does not copy them internally — the caller may free those buffers immediately after `send_async()` returns.

---

## Checklist for Custom Adapter Development

1. **`send_async` callback must run on a different thread** -- see [Common Pitfalls](#common-pitfalls-and-troubleshooting). Both legacy and new-api paths hold a mutex across the `send_async` call; a synchronous callback (including in the error path) deadlocks. Use one of the three proven approaches: event loop + opaque pointer (Pattern B), blocking library + atomic queue steal in `do_work()` (Pattern D), or library-owned threads (Pattern A). Never call the callback before returning from `send_async()`.
2. **Compile as a shared library** (`.so`) with `-shared -fPIC`.
3. **Include `nvds_msgapi.h`** from the DeepStream SDK includes directory.
4. **Handle NULL parameters gracefully** -- `connection_str`, `config_path`, and `topic` may be NULL.
5. **Document the connection string format** your adapter expects.
6. **Document the `do_work()` contract** -- whether the client must call it and how often.
7. **Use `nvds_logger`** for logging instead of raw printf for production adapters.
8. **`deepstream-test5-app`** can be used to test the adapter's deployment and multi-topic subscription functionality.
9. **Test with `gst-launch-1.0`** before integrating with pyservicemaker:

```bash
gst-launch-1.0 \
    filesrc location=test.mp4 ! h264parse ! nvv4l2decoder ! \
    nvstreammux name=mux batch-size=1 width=1920 height=1080 ! \
    nvinfer config-file-path=config.txt ! \
    nvmsgconv config=msgconv_config.txt payload-type=0 msg2p-newapi=1 ! \
    nvmsgbroker proto-lib=/path/to/libnvds_custom_proto.so \
        conn-str="host;port" topic=test config=cfg_custom.txt
```

---

## Related Documentation

- **Kafka Messaging Reference**: `kafka_messaging.md` -- Full Kafka integration patterns and protocol adapter configuration.
- **GStreamer Plugins Overview**: `gstreamer_plugins.md` -- All DeepStream GStreamer plugins including nvmsgbroker and nvmsgconv.
- **Service Maker Python API**: `service_maker_api.md` -- pyservicemaker Pipeline API reference.

# nvinfer Configuration File Reference

## Overview

The `nvinfer` GStreamer plugin uses a configuration file to define model parameters, preprocessing settings, and postprocessing options. This document provides a complete reference for all configuration parameters.

## Configuration File Formats

nvinfer supports **two configuration file formats**:

### Format 1: YAML Format (`.yml` or `.yaml`) - Recommended

```yaml
property:
  gpu-id: 0
  net-scale-factor: 0.00392156862745098
  onnx-file: /path/to/model.onnx
  batch-size: 1
  # ... more properties

class-attrs-all:
  topk: 20
  pre-cluster-threshold: 0.2
```

### Format 2: INI-style Text Format (`.txt`)

```ini
[property]
gpu-id=0
net-scale-factor=0.00392156862745098
onnx-file=/path/to/model.onnx
batch-size=1
# ... more properties

[class-attrs-all]
topk=20
pre-cluster-threshold=0.2
```

### Key Syntax Differences

| Aspect | YAML Format | INI Format |
|--------|-------------|------------|
| File extension | `.yml` or `.yaml` | `.txt` |
| Section headers | `property:` (no brackets) | `[property]` (with brackets) |
| Key-value separator | `: ` (colon + space) | `=` (equals) |
| Indentation | Required for nested values | Not used |
| Comments | `#` at start of line | `#` at start of line |

---

## Property Section Reference

The `property` section contains core inference configuration.

### Model Definition

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `onnx-file` | string | Path to ONNX model file | - |
| `model-engine-file` | string | Path to a pre-built TensorRT engine file. When set, nvinfer loads this engine directly instead of regenerating it from the ONNX file on every run. The engine filename encodes the batch size, GPU index, and precision (see naming convention below). | - |
| `custom-network-config` | string | Path to custom network config file | - |
| `custom-lib-path` | string | Path to custom parsing library (.so) | - |
| `labelfile-path` | string | Path to class labels text file | - |
| `int8-calib-file` | string | Path to INT8 calibration file | - |
| `tlt-model-key` | string | Encryption key for TAO/TLT models | - |

**Usage Example (YAML)**:
```yaml
property:
  onnx-file: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx
  model-engine-file: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b1_gpu0_fp16.engine
  labelfile-path: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt
```

#### model-engine-file — Purpose and Naming Convention

**Purpose:** The first time nvinfer runs with an ONNX model, TensorRT builds an optimised engine file. This serialisation step can take **minutes**. By specifying `model-engine-file`, you tell nvinfer to load an already-built engine directly, **skipping the ONNX-to-engine conversion** on subsequent runs and dramatically reducing startup time.

> **Agent guidance:** When generating nvinfer config files, **always include `model-engine-file`** alongside `onnx-file`. This avoids expensive re-compilation every time the pipeline starts. The engine file is specific to the batch size, GPU, and precision — if any of these change, a new engine must be generated (i.e. the first run without a matching engine file will trigger generation automatically). The engine-cache location must be writable.

**Naming convention:** TensorRT engine files follow the pattern:

```
<onnx-filename>_b<batch-size>_gpu<gpu-id>_<precision>.engine
```

| Component | Meaning | Example |
|-----------|---------|---------|
| `<onnx-filename>` | Full ONNX filename including `.onnx` extension | `resnet18_trafficcamnet_pruned.onnx` |
| `b<batch-size>` | Batch size the engine was built for | `b1`, `b4`, `b16` |
| `gpu<gpu-id>` | GPU device index | `gpu0`, `gpu1` |
| `<precision>` | Network precision mode | `fp32`, `int8`, `fp16` |

**Examples by batch size:**

```yaml
# batch-size: 1
property:
  batch-size: 1
  model-engine-file: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b1_gpu0_fp16.engine

# batch-size: 4
property:
  batch-size: 4
  model-engine-file: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b4_gpu0_fp16.engine

# batch-size: 16 (e.g. secondary classifier)
property:
  batch-size: 16
  model-engine-file: /opt/nvidia/deepstream/deepstream/samples/models/Secondary_VehicleMake/resnet18_vehiclemakenet_pruned.onnx_b16_gpu0_fp16.engine
```

**INI-style equivalent:**
```ini
[property]
batch-size=4
model-engine-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b4_gpu0_fp16.engine
```

### Processing Configuration

| Parameter | Type | Values | Description | Default |
|-----------|------|--------|-------------|---------|
| `gpu-id` | int | 0, 1, 2... | GPU device ID | 0 |
| `batch-size` | int | 1-32 | Maximum batch size | 1 |
| `process-mode` | int | 1=Primary, 2=Secondary | Inference mode | 1 |
| `network-mode` | int | 0=FP32, 1=INT8, 2=FP16 | Precision mode | 0 |
| `network-type` | int | 0=Detector, 1=Classifier, 2=Segmentation, 3=Instance Segmentation | Network type. Use instead of the legacy `is-classifier` key. | 0 |
| `interval` | int | 0-N | Skip N consecutive batches | 0 |
| `gie-unique-id` | int | 1-N | Unique ID for this GIE | 1 |

**Usage Example (YAML)**:
```yaml
property:
  gpu-id: 0
  batch-size: 4
  process-mode: 1
  network-mode: 2  # FP16
  interval: 0
  gie-unique-id: 1
```

### Network Input Configuration

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `net-scale-factor` | float | Input normalization scale factor | 1.0 |
| `offsets` | string | Channel offsets (semicolon-separated) | - |
| `model-color-format` | int | 0=RGB, 1=BGR, 2=GRAY | 0 |
| `network-input-order` | int | 0=NCHW, 1=NHWC | 0 |
| `infer-dims` | string | Input tensor dimensions in C;H;W format (semicolon-separated). **Required** when the ONNX model has dynamic input shapes (e.g., exported with `dynamic=True`). Tells TensorRT the concrete dimensions to use for the optimization profile. | Inferred from ONNX (only works for static shapes) |
| `maintain-aspect-ratio` | int | 0=disabled, 1=enabled | 0 |
| `symmetric-padding` | int | 0=disabled, 1=enabled | 0 |
| `force-implicit-batch-dim` | int | 0=disabled, 1=enabled | 0 |

> **Agent guidance — `infer-dims` and dynamic ONNX models:** Many popular model frameworks (Ultralytics YOLO, HuggingFace, etc.) export ONNX models with dynamic axes by default. These models have symbolic dimension names (e.g., `batch`, `height`, `width`) instead of fixed integers, which TensorRT reads as `-1`. Without `infer-dims`, TensorRT's `setDimensions` call fails because all dimensions must be >= 0. **Always add `infer-dims` when the ONNX model has dynamic input shapes.**

**Usage Example (YAML)** — static-shape model (infer-dims optional):
```yaml
property:
  net-scale-factor: 0.00392156862745098  # 1/255
  offsets: 0;0;0
  model-color-format: 0  # RGB
  maintain-aspect-ratio: 1
```

**Usage Example (YAML)** — dynamic-shape ONNX model (infer-dims required):
```yaml
property:
  net-scale-factor: 0.00392156862745098  # 1/255
  model-color-format: 0  # RGB
  infer-dims: 3;640;640  # REQUIRED for dynamic ONNX models
  maintain-aspect-ratio: 1
```

**Usage Example (INI)** — dynamic-shape ONNX model:
```ini
[property]
net-scale-factor=0.00392156862745098
model-color-format=0
infer-dims=3;640;640
maintain-aspect-ratio=1
```

**Geometry contract for masks and boxes:** When postprocessing produces boxes or masks that will be used outside the model-input grid, declare the model-input, pipeline/mux, and final output coordinate spaces. If `maintain-aspect-ratio` or `symmetric-padding` is enabled, account for the resize scale and padding before projecting model output into the target space. Identify whether `nvinfer` or the custom parser owns each transform, and apply the resize/padding inversion exactly once. For a full-frame mask, project the box into the declared frame-mask grid before compositing its bbox-local mask; do not treat pipeline-space rectangles as source-frame rectangles.

### Detection Configuration

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `num-detected-classes` | int | Number of classes in model | - |
| `cluster-mode` | int | 1=DBSCAN, 2=NMS, 3=DBSCAN+NMS, 4=None | 2 |
| `parse-bbox-func-name` | string | Custom bbox parsing function name | - |
| `output-blob-names` | string | Model output layer names (semicolon-separated) | - |

**Usage Example (YAML)**:
```yaml
property:
  num-detected-classes: 4
  cluster-mode: 2  # NMS
```

> **Oriented bounding boxes (OBB) — `rotation_angle`:** `nvinfer` supports oriented bounding boxes via `NvDsInferObjectDetectionInfo.rotation_angle`. **If you are using an OBB model**, the angle output by the model can be **directly assigned** to `rotation_angle` in your custom bbox parser. **If you are not using an OBB model**, set `rotation_angle = 0`. In C++, `NvDsInferObjectDetectionInfo obj{};` value-initializes the struct and zero-initializes all fields, including `rotation_angle`; plain `NvDsInferObjectDetectionInfo obj;` does **not** and can leave rotated-box metadata uninitialized.
>
> Example (C++):
> ```cpp
> NvDsInferObjectDetectionInfo obj{};
> // ... fill classId, confidence, left/top/width/height ...
> obj.rotation_angle = is_obb_model ? angle_from_model : 0.0f;
> ```

### Custom Instance-Segmentation Parsers

Use this path only for a model that produces detections with one mask per detected object. It is distinct from semantic segmentation, which produces a full-frame class map.

```yaml
property:
  network-type: 3
  custom-lib-path: /path/to/libcustom_parser.so
  parse-bbox-instance-mask-func-name: NvDsInferParseCustomInstanceMask
  output-instance-mask: 1
```

- Use `parse-bbox-instance-mask-func-name`, not the ordinary `parse-bbox-func-name`, and implement the instance-mask parser ABI provided by the DeepStream SDK in the target image.
- After thresholding and NMS, emit each retained box, class ID, confidence, and matching bbox-local object mask together.
- When mask and final-box resolutions differ, resize or crop the mask into the final bbox-local geometry.
- `output-instance-mask=1` exposes parsed object masks to DeepStream metadata and OSD. It does not create a frame-level output mask.

### Secondary GIE Configuration (process-mode: 2)

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `operate-on-gie-id` | int | GIE ID to operate on | -1 (all) |
| `operate-on-class-ids` | string | Class IDs to process (semicolon-separated) | - |
| `classifier-async-mode` | int | 0=sync, 1=async | 0 |
| `classifier-threshold` | float | Classification confidence threshold | 0.0 |
| `classifier-type` | string | Classifier label type (e.g., `vehicletype`, `vehiclemake`, `color`). Used to label classification results in metadata. | - |
| `input-object-min-width` | int | Minimum object width to classify | 0 |
| `input-object-min-height` | int | Minimum object height to classify | 0 |
| `input-object-max-width` | int | Maximum object width to classify | INT_MAX |
| `input-object-max-height` | int | Maximum object height to classify | INT_MAX |

**Usage Example (YAML)** - Secondary classifier:
```yaml
property:
  gpu-id: 0
  onnx-file: /path/to/classifier.onnx
  batch-size: 16
  process-mode: 2
  network-mode: 2
  network-type: 1
  gie-unique-id: 2
  operate-on-gie-id: 1
  operate-on-class-ids: 0
  classifier-async-mode: 1
  classifier-threshold: 0.51
  classifier-type: vehicletype
```

### Tensor Output Configuration

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `output-tensor-meta` | int | 0=disabled, 1=enabled | 0 |
| `output-instance-mask` | int | 0=disabled, 1=enabled | 0 |
| `input-tensor-meta` | int | 0=disabled, 1=enabled | 0 |

**Usage Example (YAML)**:
```yaml
property:
  output-tensor-meta: 1  # Enable tensor output for custom postprocessing
```

### Scaling Configuration

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `scaling-filter` | int | Scaling filter type (0-5) | 0 |
| `scaling-compute-hw` | int | 0=default, 1=GPU, 2=VIC | 0 |

---

## Class Attributes Sections

Class attributes sections configure detection parameters per class or for all classes.

### class-attrs-all (All Classes)

Applies to all detected classes.

> **IMPORTANT — camelCase key**: The DBSCAN minimum cluster size parameter is `minBoxes` (camelCase). Do NOT use `min-boxes` (kebab-case) — it is not recognized and will produce an "unknown key" warning at runtime.

| Parameter | Type | Description | Default |
|-----------|------|-------------|---------|
| `topk` | int | Maximum detections to keep after NMS | 20 |
| `nms-iou-threshold` | float | NMS IoU threshold (0.0-1.0) | 0.3 |
| `pre-cluster-threshold` | float | Confidence threshold before clustering | 0.4 |
| `eps` | float | DBSCAN epsilon parameter | 0.0 |
| `dbscan-min-score` | float | DBSCAN minimum confidence | 0.0 |
| `minBoxes` | int | DBSCAN minimum cluster size (camelCase, NOT `min-boxes`) | 0 |
| `roi-top-offset` | int | ROI top offset in pixels | 0 |
| `roi-bottom-offset` | int | ROI bottom offset in pixels | 0 |
| `detected-min-w` | int | Minimum detection width | 0 |
| `detected-min-h` | int | Minimum detection height | 0 |
| `detected-max-w` | int | Maximum detection width | INT_MAX |
| `detected-max-h` | int | Maximum detection height | INT_MAX |

**Usage Example (YAML)** - NMS clustering:
```yaml
class-attrs-all:
  topk: 20
  nms-iou-threshold: 0.5
  pre-cluster-threshold: 0.2
```

**Usage Example (YAML)** - DBSCAN clustering:
```yaml
class-attrs-all:
  detected-min-w: 4
  detected-min-h: 4
  minBoxes: 3
  eps: 0.7
  dbscan-min-score: 0.5
```

### class-attrs-N (Per-Class)

Override attributes for specific class ID N.

```yaml
class-attrs-0:
  topk: 30
  nms-iou-threshold: 0.4
  pre-cluster-threshold: 0.3

class-attrs-1:
  topk: 10
  nms-iou-threshold: 0.6
  pre-cluster-threshold: 0.5
```

---

## Complete Configuration Examples

### Example 1: Primary Detector (YAML)

```yaml
# Primary detector using ResNet18 TrafficCamNet
property:
  gpu-id: 0
  net-scale-factor: 0.00392156862745098
  onnx-file: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx
  model-engine-file: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b1_gpu0_fp16.engine
  labelfile-path: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt
  batch-size: 1
  process-mode: 1
  model-color-format: 0
  network-mode: 2
  num-detected-classes: 4
  interval: 0
  gie-unique-id: 1
  cluster-mode: 2

class-attrs-all:
  topk: 20
  nms-iou-threshold: 0.5
  pre-cluster-threshold: 0.2

class-attrs-0:
  topk: 20
  nms-iou-threshold: 0.5
  pre-cluster-threshold: 0.4
```

### Example 2: Primary Detector (INI-style)

```ini
# Primary detector using ResNet18 TrafficCamNet
[property]
gpu-id=0
net-scale-factor=0.00392156862745098
onnx-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx
model-engine-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx_b1_gpu0_fp16.engine
labelfile-path=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt
batch-size=1
process-mode=1
model-color-format=0
network-mode=2
num-detected-classes=4
interval=0
gie-unique-id=1
cluster-mode=2

[class-attrs-all]
topk=20
nms-iou-threshold=0.5
pre-cluster-threshold=0.2

[class-attrs-0]
topk=20
nms-iou-threshold=0.5
pre-cluster-threshold=0.4
```

### Example 3: Secondary Classifier (YAML)

```yaml
# Secondary classifier for vehicle make
property:
  gpu-id: 0
  net-scale-factor: 1.0
  onnx-file: /opt/nvidia/deepstream/deepstream/samples/models/Secondary_VehicleMake/resnet18_vehiclemakenet_pruned.onnx
  model-engine-file: /opt/nvidia/deepstream/deepstream/samples/models/Secondary_VehicleMake/resnet18_vehiclemakenet_pruned.onnx_b16_gpu0_fp16.engine
  labelfile-path: /opt/nvidia/deepstream/deepstream/samples/models/Secondary_VehicleMake/labels.txt
  batch-size: 16
  process-mode: 2
  model-color-format: 1
  network-mode: 2
  network-type: 1
  gie-unique-id: 2
  operate-on-gie-id: 1
  operate-on-class-ids: 0
  classifier-async-mode: 1
  classifier-threshold: 0.51
  classifier-type: vehiclemake
```

### Example 4: Tensor Output for Custom Postprocessing (YAML)

```yaml
# Enable tensor output for custom postprocessing
property:
  gpu-id: 0
  net-scale-factor: 0.00392156862745098
  onnx-file: /path/to/custom_model.onnx
  batch-size: 1
  process-mode: 1
  model-color-format: 0
  network-mode: 2
  num-detected-classes: 4
  gie-unique-id: 1
  output-tensor-meta: 1
  cluster-mode: 4  # No clustering, use custom postprocessing

class-attrs-all:
  pre-cluster-threshold: 0.1
```

---

## Common Pitfalls

### Pitfall 1: Wrong Section Name

**Wrong (using `model:` instead of `property:`)**:
```yaml
model:
  onnx-file: /path/to/model.onnx
  batch-size: 1
```

**Correct**:
```yaml
property:
  onnx-file: /path/to/model.onnx
  batch-size: 1
```

### Pitfall 2: Missing Colons in YAML

**Wrong**:
```yaml
property
  gpu-id: 0
```

**Correct**:
```yaml
property:
  gpu-id: 0
```

### Pitfall 3: Wrong Indentation

**Wrong**:
```yaml
property:
gpu-id: 0
batch-size: 1
```

**Correct**:
```yaml
property:
  gpu-id: 0
  batch-size: 1
```

### Pitfall 4: Using YAML syntax in INI file

**Wrong (YAML in .txt file)**:
```ini
property:
  gpu-id: 0
```

**Correct (INI format in .txt file)**:
```ini
[property]
gpu-id=0
```

### Pitfall 5: Incorrect process-mode for Secondary GIE

**Wrong (using process-mode=1 for secondary)**:
```yaml
property:
  process-mode: 1
  operate-on-gie-id: 1  # Won't work with process-mode=1
```

**Correct**:
```yaml
property:
  process-mode: 2  # Must be 2 for secondary GIE
  operate-on-gie-id: 1
```

### Pitfall 6: Missing `infer-dims` for Dynamic ONNX Models

**Wrong (no `infer-dims` with a dynamic-shape ONNX model)**:
```yaml
# Model exported with dynamic=True (e.g., Ultralytics YOLO)
# ONNX input shape: [batch, 3, height, width] — all symbolic
property:
  onnx-file: yolo_model.onnx
  net-scale-factor: 0.00392156862745098
  # Missing infer-dims → TensorRT sees -1 for dynamic dims → engine build fails
```

**Error**: `IOptimizationProfile::setDimensions: Error Code 3: API Usage Error (Parameter check failed, condition: std::all_of(dims.d, dims.d + dims.nbDims, [](int32_t x) noexcept { return x >= 0; }))`

**Correct**:
```yaml
property:
  onnx-file: yolo_model.onnx
  net-scale-factor: 0.00392156862745098
  infer-dims: 3;640;640  # C;H;W — tells TensorRT the concrete input dimensions
```

**When to add `infer-dims`**: Whenever the ONNX model was exported with dynamic axes (e.g., `dynamic=True` in Ultralytics, dynamic batch in other frameworks). If unsure, inspect the model with `python -c "import onnx; m = onnx.load('model.onnx'); print(m.graph.input)"` and check for symbolic dimension names.

### Pitfall 7: Using Legacy `is-classifier` Instead of `network-type`

**Wrong (legacy key, produces deprecation warning)**:
```yaml
property:
  is-classifier: 1
```

**Correct (use `network-type` in YAML configs)**:
```yaml
property:
  network-type: 1  # 0=Detector, 1=Classifier, 2=Segmentation, 3=Instance Segmentation
```

For primary detectors, simply omit both keys — the default is detector (`network-type: 0`).

### Pitfall 8: Using `min-boxes` Instead of `minBoxes`

**Wrong (kebab-case — not recognized, produces "unknown key" warning)**:
```yaml
class-attrs-all:
  min-boxes: 3
```

**Correct (camelCase)**:
```yaml
class-attrs-all:
  minBoxes: 3
```

Unlike most nvinfer config keys which use kebab-case, `minBoxes` uses camelCase. This is a legacy naming exception in the parser.

---

## DeepStream Sample Model Paths

DeepStream includes sample models at:

```
/opt/nvidia/deepstream/deepstream/samples/models/
├── Primary_Detector/
│   ├── resnet18_trafficcamnet_pruned.onnx
│   ├── labels.txt
│   └── cal_trt.bin (INT8 calibration)
├── Secondary_VehicleMake/
│   ├── resnet18_vehiclemakenet_pruned.onnx
│   └── labels.txt
├── Secondary_VehicleTypes/
│   ├── resnet18_vehicletypenet_pruned.onnx
│   └── labels.txt
└── SONYC_Audio_Classifier/
    └── ...
```

**Primary Detector Labels** (4 classes):
- 0: Car
- 1: TwoWheeler
- 2: Person
- 3: RoadSign

---

## GObject Properties vs Config File Parameters

Some parameters can be set via GObject properties on the `nvinfer` element:

```python
pipeline.add("nvinfer", "infer", {
    "config-file-path": "/path/to/config.yml",  # Required
    "batch-size": 4,                             # Overrides config file
    "unique-id": 1,                              # Overrides config file
    "output-tensor-meta": 1,                     # Overrides config file
    "interval": 2                                # Overrides config file
})
```

**Properties settable via GObject** (override config file):
- `batch-size`
- `unique-id`
- `process-mode`
- `interval`
- `output-tensor-meta`
- `input-tensor-meta`
- `output-instance-mask`
- `model-engine-file`

**Properties only in config file**:
- `net-scale-factor`
- `onnx-file`
- `infer-dims`
- `labelfile-path`
- `num-detected-classes`
- `cluster-mode`
- All `class-attrs-*` parameters

---

## Validation Checklist

Before running your pipeline, verify:

- [ ] Config file extension matches format (`.yml` for YAML, `.txt` for INI)
- [ ] Section name is `property:` (YAML) or `[property]` (INI)
- [ ] Model file path exists and is accessible
- [ ] `model-engine-file` is set and its name matches the current `batch-size`, `gpu-id`, and `network-mode` (precision)
- [ ] `infer-dims` is set if the ONNX model has dynamic input shapes (e.g., exported with `dynamic=True`)
- [ ] `num-detected-classes` matches your model
- [ ] `batch-size` <= number of streams
- [ ] `process-mode` is correct (1=Primary, 2=Secondary)
- [ ] Secondary GIE has `operate-on-gie-id` set correctly
- [ ] `gie-unique-id` is unique across all nvinfer instances

---

## Related Documentation

- **GStreamer Plugins Overview**: `gstreamer_plugins.md`
- **Service Maker Python API**: `service_maker_api.md`
- **Use Cases & Pipelines**: `use_cases_pipelines.md`
- **Best Practices**: `best_practices.md`

# REST API and Dynamic Source Management

## Overview

DeepStream supports dynamic addition and removal of video sources at runtime through REST APIs. This capability is built into `nvmultiurisrcbin`, which integrates an HTTP REST server, multiple `nvurisrcbin` instances, and `nvstreammux` into a single GStreamer bin.

**CRITICAL: Always use the built-in REST server in nvmultiurisrcbin. Do NOT implement a separate Flask/FastAPI server for stream management.**

---

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    nvmultiurisrcbin                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │ nvds_rest_   │  │ nvurisrcbin  │  │   nvstreammux    │  │
│  │ server       │  │ (multiple)   │  │                  │  │
│  │ Port: 9000   │  │              │  │                  │  │
│  └──────────────┘  └──────────────┘  └──────────────────┘  │
└─────────────────────────────────────────────────────────────┘
```

---

## Critical Configuration for Dynamic Sources

### Sink Element Configuration

**CRITICAL: When using dynamic sources, the sink element MUST have `async=0`**

```python
# CORRECT - Required for dynamic source state transitions
pipeline.add("nveglglessink", "sink", {
    "sync": 0,   # Don't sync to clock (required for live sources)
    "qos": 0,    # Disable QoS events
    "async": 0   # CRITICAL: Synchronous state changes for dynamic streams
})

# WRONG - Will cause state transition deadlock
pipeline.add("nveglglessink", "sink", {"sync": 0})  # Missing async=0
```

**Why `async=0` is required:**
- Without it, the sink waits for preroll (first buffer) before allowing state transitions
- With dynamic streams, this creates a deadlock: source waits for sink, sink waits for data
- Setting `async=0` makes state changes synchronous, allowing proper transitions

### nvmultiurisrcbin Configuration

```python
source_props = {
    # REST API Server
    "ip-address": "0.0.0.0",        # Listen on all interfaces
    "port": 9000,                    # REST API port (0 to disable)
    
    # Batching
    "max-batch-size": 16,            # Maximum number of sources
    "batched-push-timeout": 33333,   # Push batch after 33ms even if not full
    "width": 1920,
    "height": 1080,
    
    # Dynamic source handling
    "live-source": 1,                # REQUIRED for dynamic streams
    "drop-pipeline-eos": 1,          # Keep pipeline alive when sources removed
    "async-handling": 1,             # Handle async state changes
    
    # RTSP settings
    "select-rtp-protocol": 0,        # 0=UDP+TCP auto, 4=TCP only
    "latency": 100,                  # Jitterbuffer size in ms
}

pipeline.add("nvmultiurisrcbin", "src", source_props)
```

---

## REST API Endpoints

The built-in REST server provides these endpoints:

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/stream/add` | POST | Add a new stream |
| `/api/v1/stream/remove` | POST | Remove a stream |
| `/api/v1/stream/get-stream-info` | GET | Get current stream info |
| `/api/v1/health/get-dsready-state` | GET | Check pipeline readiness |

### Add Stream Payload

```json
{
    "key": "sensor",
    "value": {
        "camera_id": "unique_sensor_id",
        "camera_name": "human_readable_name",
        "camera_url": "rtsp://camera-ip/stream",
        "change": "camera_add"
    }
}
```

**Mandatory fields:**
- `value/camera_id` - Unique identifier
- `value/camera_url` - Stream URI
- `value/change` - Must contain "add" substring

### Remove Stream Payload

```json
{
    "key": "sensor",
    "value": {
        "camera_id": "unique_sensor_id",
        "camera_url": "rtsp://camera-ip/stream",
        "change": "camera_remove"
    }
}
```

**Note:** The `change` field must contain "remove" substring.

### Example curl Commands

```bash
# Add a stream
curl -X POST 'http://localhost:9000/api/v1/stream/add' -d '{
    "key": "sensor",
    "value": {
        "camera_id": "cam_001",
        "camera_name": "Front Door",
        "camera_url": "rtsp://192.168.1.100/stream",
        "change": "camera_add"
    }
}'

# Remove a stream
curl -X POST 'http://localhost:9000/api/v1/stream/remove' -d '{
    "key": "sensor",
    "value": {
        "camera_id": "cam_001",
        "camera_url": "rtsp://192.168.1.100/stream",
        "change": "camera_remove"
    }
}'

# Get stream info
curl -X GET 'http://localhost:9000/api/v1/stream/get-stream-info'

# Check pipeline readiness
curl -X GET 'http://localhost:9000/api/v1/health/get-dsready-state'
```

---

## Complete Pipeline Example

```python
from pyservicemaker import (
    Pipeline, Probe, BatchMetadataOperator,
    StateTransitionMessage, DynamicSourceMessage
)
import platform

def run_dynamic_source_pipeline():
    """Pipeline with dynamic source management via REST API."""
    
    def on_message(message):
        """Handle pipeline messages for dynamic sources."""
        if isinstance(message, DynamicSourceMessage):
            if message.source_added:
                print(f"Camera ADDED: {message.sensor_name} "
                      f"(id={message.sensor_id}, source_id={message.source_id})")
            else:
                print(f"Camera REMOVED: source_id={message.source_id}")
        
        elif isinstance(message, StateTransitionMessage):
            state_name = str(message.new_state).split('.')[-1]
            print(f"{message.origin} -> {state_name}")
    
    pipeline = Pipeline("dynamic-source-pipeline")
    
    # Source with built-in REST server
    pipeline.add("nvmultiurisrcbin", "src", {
        "ip-address": "0.0.0.0",
        "port": 9000,                    # REST API on port 9000
        "max-batch-size": 16,
        "batched-push-timeout": 33333,
        "width": 1920,
        "height": 1080,
        "live-source": 1,                # Required for dynamic sources
        "drop-pipeline-eos": 1,
        "async-handling": 1,
        "select-rtp-protocol": 0,
        "latency": 100,
    })
    
    # Inference
    pipeline.add("nvinfer", "pgie", {
        "config-file-path": "/path/to/pgie_config.yml",
        "batch-size": 16
    })
    
    # Tiler for multi-stream display
    pipeline.add("nvmultistreamtiler", "tiler", {
        "width": 1920,
        "height": 1080,
        "rows": 4,
        "columns": 4
    })
    
    # OSD
    pipeline.add("nvosdbin", "osd")
    
    # Sink - CRITICAL: async=0 for dynamic sources
    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    pipeline.add(sink_type, "sink", {
        "sync": 0,
        "qos": 0,
        "async": 0  # CRITICAL for dynamic source state transitions
    })
    
    # Link pipeline
    pipeline.link("src", "pgie", "tiler", "osd", "sink")
    
    # Prepare and activate
    pipeline.prepare(on_message)
    pipeline.activate()
    
    print("Pipeline started. REST API available at http://localhost:9000")
    print("Add streams with: POST /api/v1/stream/add")
    
    pipeline.wait()

if __name__ == "__main__":
    from multiprocessing import Process
    process = Process(target=run_dynamic_source_pipeline)
    process.start()
    process.join()
```

---

## Handling DynamicSourceMessage

When streams are added or removed, the pipeline emits `DynamicSourceMessage`:

```python
from pyservicemaker import DynamicSourceMessage

def on_message(message):
    if isinstance(message, DynamicSourceMessage):
        source_id = message.source_id      # Internal source ID (int)
        sensor_id = message.sensor_id      # Your camera_id from REST API
        sensor_name = message.sensor_name  # Your camera_name from REST API
        
        if message.source_added:
            # Stream successfully added
            # Map source_id to your camera tracking
            print(f"Added: {sensor_name} (sensor_id={sensor_id})")
        else:
            # Stream removed
            print(f"Removed: source_id={source_id}")
```

---

## Common Errors and Solutions

### Error: Stream added but no video displayed

**Symptom:** REST API returns success, `DynamicSourceMessage` received, but elements stuck in PAUSED state.

**Cause:** Missing `async=0` on sink element.

**Solution:**
```python
# Add async=0 to sink
pipeline.add("nveglglessink", "sink", {
    "sync": 0,
    "qos": 0,
    "async": 0  # This is the fix
})
```

### Error: No data from source, reconnection attempts

**Symptom:**
```
WARNING from dsnvurisrcbin0: No data from source since last 10 sec. Trying reconnection
Could not send message. (Received end-of-file)
```

**Cause:** RTSP source issue - invalid URL, authentication required, or network problem.

**Solution:**
1. Test RTSP URL with ffplay: `ffplay rtsp://camera-ip/stream`
2. Include credentials: `rtsp://user:password@camera-ip/stream`
3. Try different RTP protocol: `select-rtp-protocol: 4` (TCP only)

### Error: Pipeline EOS when stream removed

**Symptom:** Pipeline stops when the last stream is removed.

**Solution:** Set `drop-pipeline-eos: 1` on nvmultiurisrcbin.

### Anti-Pattern: Implementing Custom REST Server

**WRONG - Do not implement a separate Flask/FastAPI server:**
```python
# DON'T DO THIS
from flask import Flask
app = Flask(__name__)

@app.route('/add-camera', methods=['POST'])
def add_camera():
    # Custom REST server adds complexity and potential bugs
    pass
```

**CORRECT - Use the built-in REST server:**
```python
# Just configure the port on nvmultiurisrcbin
pipeline.add("nvmultiurisrcbin", "src", {
    "port": 9000,  # Built-in REST server on port 9000
    # ... other properties
})
# REST API is automatically available at http://localhost:9000/api/v1/
```

If you need a proxy API for simplified requests, make HTTP calls to the built-in server instead of reimplementing stream management.

---

## Headless Operation

For headless (no display) operation, use `fakesink`:

```python
import os

if "DISPLAY" not in os.environ:
    # Headless mode
    pipeline.add("fakesink", "sink", {
        "sync": 0,
        "async": 0
    })
else:
    # Display mode
    pipeline.add("nveglglessink", "sink", {
        "sync": 0,
        "qos": 0,
        "async": 0
    })
```

---

## RTSP URL Formats

Common RTSP URL formats by manufacturer:

| Manufacturer | URL Format |
|--------------|------------|
| Hikvision | `rtsp://user:pass@ip:554/Streaming/Channels/101` |
| Dahua | `rtsp://user:pass@ip:554/cam/realmonitor?channel=1&subtype=0` |
| Axis | `rtsp://user:pass@ip/axis-media/media.amp` |
| Generic | `rtsp://user:pass@ip:554/stream1` |
| NVIDIA Demo | `rtsp://<your-camera-ip>:554/stream` |

---

## Quick Reference

| Requirement | Property | Value |
|-------------|----------|-------|
| Enable REST API | `port` | 9000 (or any port, 0 to disable) |
| Dynamic sources | `live-source` | 1 |
| Keep pipeline alive | `drop-pipeline-eos` | 1 |
| Async state changes | `async-handling` | 1 |
| **Sink async** | `async` | **0 (CRITICAL)** |
| Sink sync | `sync` | 0 |

---

## Related Documentation

- **GStreamer Plugins**: `gstreamer_plugins.md`
- **Service Maker API**: `service_maker_api.md`
- **Troubleshooting**: `troubleshooting.md`
- **Configuration Classes**: `utilities_config.md`

# DeepStream Service Maker for Python (pyservicemaker) API Reference

## Introduction

The DeepStream Service Maker provides a high-level Python API (`pyservicemaker`) for building DeepStream applications. It abstracts away the complexity of GStreamer C API and provides a more intuitive, Pythonic interface for constructing video analytics pipelines.

## Installation

The pyservicemaker package is installed as part of DeepStream SDK:
```bash
pip install /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl pyyaml
```

**Inside a virtual environment**: `pyservicemaker` is installed system-wide but is NOT accessible from a standard venv. If the application uses a virtual environment, you must install it inside the venv:
```bash
python3 -m venv venv
source venv/bin/activate
pip install /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl pyyaml
```

## Two API Approaches

Service Maker provides two APIs for building pipelines:

1. **Pipeline API**: Low-level, element-by-element pipeline construction
2. **Flow API**: High-level, declarative pipeline construction

---

## Pipeline API

The Pipeline API provides fine-grained control over pipeline construction, similar to GStreamer C API but with Python syntax.

### Core Classes

#### Pipeline
Main class for creating and managing DeepStream pipelines.

**Constructor**:
```python
from pyservicemaker import Pipeline

# Create empty pipeline
pipeline = Pipeline("pipeline-name")

# Create pipeline from YAML config
pipeline = Pipeline("pipeline-name", "/path/to/config.yml")
```

**Methods**:

##### `add(element_type, name, properties=None)`
Add a GStreamer element to the pipeline.

**Parameters**:
- `element_type` (str): GStreamer element factory name (e.g., "nvinfer", "nvstreammux")
- `name` (str): Unique name for the element
- `properties` (dict, optional): Element properties as key-value pairs

**Returns**: Pipeline instance (for method chaining)

**Example**:
```python
pipeline.add("filesrc", "src", {"location": "/path/to/video.h264"})
pipeline.add("h264parse", "parser")
pipeline.add("nvv4l2decoder", "decoder")
pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})
pipeline.add("nvinfer", "infer", {"config-file-path": "/path/to/config.yml"})
```

##### `link(*element_names)`
Link elements in sequence. Elements are connected in the order specified.

**Parameters**:
- `*element_names`: Variable number of element names or tuples for request pads

**Returns**: Pipeline instance (for method chaining)

**Example**:
```python
# Simple linear linking
pipeline.link("src", "parser", "decoder", "mux", "infer", "sink")

# Linking with request pads (for nvstreammux)
pipeline.link(("decoder", "mux"), ("", "sink_%u"))
# This connects decoder src pad to mux sink_0 pad
```

**Request Pad Linking**:
For elements with dynamic pads (like nvstreammux), use tuple syntax:
```python
# Format: (source_element, sink_element), (source_pad, sink_pad_template)
pipeline.link(("decoder1", "mux"), ("", "sink_%u"))  # Connects to sink_0
pipeline.link(("decoder2", "mux"), ("", "sink_%u"))  # Connects to sink_1
```

**CRITICAL: Always use "sink_%u" pad template, NOT "sink_0", "sink_1", or f"sink_{i}"**
- `"sink_%u"` is a GStreamer pad template that automatically assigns sink pads (sink_0, sink_1, sink_2, etc.)
- Using literal pad names like `"sink_0"` or `f"sink_{i}"` will FAIL because these pads don't exist until requested
- The `%u` format specifier tells GStreamer to automatically assign the next available sink pad index

**Examples with different source types**:
```python
# With nvv4l2decoder (decoded video source)
pipeline.link((f"decoder{i}", "mux"), ("", "sink_%u"))  # CORRECT

# With nvurisrcbin (RTSP/file source with dynamic pads)
pipeline.link((f"src{i}", "mux"), ("", "sink_%u"))  # CORRECT - nvurisrcbin has dynamic src pad

# WRONG - DO NOT USE:
pipeline.link((f"src{i}", "mux"), ("", f"sink_{i}"))  # INCORRECT - will fail!
pipeline.link((f"src{i}", "mux"), ("", "sink_0"))     # INCORRECT - pad doesn't exist!
```

##### `attach(target, what, name='', tips='', properties=None)`
Attach a probe (or other custom object) to a named element in the pipeline.

**Parameters**:
- `target` (str): Name of the pipeline element to attach to
- `what`: Probe instance or name of a built-in probe module (e.g. `"measure_fps_probe"`)
- `name` (str, optional): Name for the probe. Not needed when `what` is an explicitly created Probe object.
- `tips` (str, optional): Extra information for the custom object
- `properties` (dict, optional): Properties to set on the object. Not applicable for explicitly created Probe objects.

**CRITICAL**: The parameter is **`name`**, NOT `probe_name`. Using `probe_name` will raise `TypeError`.

**Returns**: Pipeline instance (for method chaining)

**Example**:
```python
from pyservicemaker import Probe, BatchMetadataOperator

class MyProbe(BatchMetadataOperator):
    def handle_metadata(self, batch_meta):
        # Process metadata
        pass

pipeline.attach("infer", Probe("my-probe", MyProbe()))
# Or attach built-in probe by module name, giving it a name
pipeline.attach("infer", "measure_fps_probe", name="fps-probe")
```

##### `start()`
Start the pipeline (set to PLAYING state).

**Returns**: Pipeline instance (for method chaining)

**Example**:
```python
pipeline.start()
```

##### `wait()`
Wait for pipeline to finish (blocking call until EOS or error).

**Returns**: None

**Example**:
```python
pipeline.start().wait()
```

##### `set(properties)`
Set properties on an element (when element is accessed via indexing).

**Parameters**:
- `properties` (dict): Properties to set

**Example**:
```python
pipeline["infer"].set({"batch-size": 4})
```

##### Element Access via Indexing
Access elements by name to get/set properties:

```python
# Get element
infer_element = pipeline["infer"]

# Set properties
pipeline["infer"].set({"batch-size": 4})

# Get properties
batch_size = pipeline["infer"].get("batch-size")
```

### Complete Pipeline API Example

```python
from pyservicemaker import Pipeline, Probe, BatchMetadataOperator
import platform

PIPELINE_NAME = "my-pipeline"
CONFIG_FILE = "/path/to/inference_config.txt"  # Must be INI-style text format, NOT YAML
VIDEO_FILE = "/path/to/video.h264"

class ObjectCounter(BatchMetadataOperator):
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            # IMPORTANT: object_items returns an ITERATOR, not a list
            # You cannot use len() directly - iterate and count instead
            obj_count = 0
            for obj in frame_meta.object_items:
                obj_count += 1
            print(f"Frame {frame_meta.frame_number}: {obj_count} objects")

# Create pipeline
pipeline = (Pipeline(PIPELINE_NAME)
    .add("filesrc", "src", {"location": VIDEO_FILE})
    .add("h264parse", "parser")
    .add("nvv4l2decoder", "decoder")
    .add("nvstreammux", "mux", {
        "batch-size": 1,
        "width": 1920,
        "height": 1080
    })
    .add("nvinfer", "infer", {"config-file-path": CONFIG_FILE})
    .add("nvosdbin", "osd")
    .add("nv3dsink" if platform.processor() == "aarch64" else "nveglglessink", "sink")
    .link("src", "parser", "decoder")
    .link(("decoder", "mux"), ("", "sink_%u"))
    .link("mux", "infer", "osd", "sink")
    .attach("infer", Probe("counter", ObjectCounter()))
    .start()
    .wait())
```

---

## Flow API

The Flow API provides a high-level, declarative interface for common pipeline patterns.

### Core Classes

#### Flow
High-level API for building pipelines using method chaining.

**Constructor**:
```python
from pyservicemaker import Flow, Pipeline

pipeline = Pipeline("pipeline-name")
flow = Flow(pipeline)
```

**Methods**:

##### `batch_capture(sources, record_config=None, **kwargs)`
Configure batch capture from multiple sources.

**Parameters**:
- `sources` (list): List of source file paths or URIs
- `record_config` (class RecordConfig): Optional smart recording (see full table in **`record_config` details** section below). If **`None`**, no smart recording is configured on sources. 
- `kwargs` (dict): Optional overrides merged into mux and/or source properties (see **`kwargs` dict details** section below). 

**`record_config` details**:
RecordConfig instance should be constructed as description in **`record_config` Construction examples** section. The following RecordConfig fields can be used to configure smart recording.
| Field | Type | Default | Used when | Meaning |
|-------|------|---------|-----------|---------|
| **`recording_type`** | **str** | **`"local"`** | Always | **`"local"`** or **`"cloud"`** (case-insensitive check in validation). |
| **`proto_lib`** | **Optional[str]** | **`None`** | **`recording_type == "cloud"`** (required) | Path to the protocol library (e.g. Kafka proto **`libnvds_kafka_proto.so`**). Set on the smart-recording controller as **`proto-lib`**. |
| **`conn_str`** | **Optional[str]** | **`None`** | Cloud (required) | Broker connection string (e.g. **`"localhost;9092"`**). Property **`conn-str`**. |
| **`msgconv_config_file`** | **Optional[str]** | **`None`** | Cloud (required) | Message converter config file path. Property **`msgconv-config-file`**. |
| **`proto_config_file`** | **Optional[str]** | **`None`** | Cloud (required) | Protocol adaptor config file path. Property **`proto-config-file`**. |
| **`topic_list`** | **Optional[str]** | **`None`** | Cloud (required) | Comma-separated topic list. Property **`topic-list`**. |
| **`rec_cache`** | **int** | **20** | **`record_config` is set** | Maps to **`smart-rec-cache`** on each source (cache size in seconds). |
| **`rec_container`** | **int** | **0** | **`record_config` is set** | Maps to **`smart-rec-container`** (**0**: MP4, **1**: MKV). |
| **`rec_dir_path`** | **str** | **`"."`** | **`record_config` is set** | Maps to **`smart-rec-dir-path`** (output directory for recordings). |
| **`rec_mode`** | **int** | **0** | **`record_config` is set** | Maps to **`smart-rec-mode`**. Docstring: **0** both, **1** video-only, **2** audio-only. |

**`record_config` Construction examples**:
```python
from pyservicemaker import RecordConfig

# Local smart recording (minimal)
rec_local = RecordConfig()  # recording_type defaults to "local"

# Local with explicit paths and cache
rec_local = RecordConfig(
    recording_type="local",
    rec_cache=20,
    rec_container=0,
    rec_dir_path="/data/recordings",
    rec_mode=0,
)

# Cloud smart recording (all cloud fields required)
rec_cloud = RecordConfig(
    recording_type="cloud",
    proto_lib="/path/to/broker_library.so",
    conn_str="localhost;9092",
    msgconv_config_file="/path/to/dstest5_msgconv_sample_config.txt",
    proto_config_file="/path/to/cfg_kafka.txt",
    topic_list="sr-test",
    rec_cache=20,
    rec_dir_path=".",
    rec_mode=0,
)
```

**`kwargs` dict details**:
Any matching **hyphenated** name in the merged **`kwargs`** dict overrides the default value of the corresponding property, the following keys are supported:
- `gpu_id` (int): Used as the `gpu-id` property of **`nvstreammux`** and as `gpu-id` on each **`nvurisrcbin`**.
- `width` (int): Used as the `width` property of **`nvstreammux`**, default value is 1920.
- `height` (int): Used as the `height` property of **`nvstreammux`**, default value is 1080.
- `batch_size` (int): Used as the `batch-size` property of **`nvstreammux`**, default value is the number of URIs (if non-empty).
- `batched_push_timeout` (int): Used as the `batched-push-timeout` property of **`nvstreammux`**, default value is 33000.
- `buffer_pool_size` (int): Used as the `buffer-pool-size` property of **`nvstreammux`**, default value is 4.
- `drop_pipeline_eos` (bool): Used as the `drop-pipeline-eos` property of **`nvstreammux`**, default value is False.
- `live_source` (bool): Used as the `live-source` property of **`nvstreammux`**, default value is False.
- `file_loop`(bool): Used as the `file-loop` property of **`nvstreammux`**, default value is False.

**Returns**: Flow instance (for method chaining)

**Example**:
```python
flow.batch_capture([
    "/path/to/video1.h264",
    "/path/to/video2.h264",
    "rtsp://camera-ip/stream"
])

# Mux resolution and batching setting
flow.batch_capture(uris, width=1280, height=720, batch_size=4)

# GPU and file loop for file sources
flow.batch_capture(uris, gpu_id=0, file_loop=True)

# Combine with YAML: kwargs override missing keys from source-config.properties
flow.batch_capture("/path/to/sources.yaml", width=1920, height=1080, live_source=True)
```
**Important**:
`batch_capture` function sets the nvstreammux batch-size according to the input stream number by default, it is not necessary to set 'batch-size' with `batch_capture` unless you want to support dynamic source adding/removing.


##### `infer(config_file_path, with_triton, **kwargs)`
Add inference stage to the pipeline.

**Parameters**:
- `config_file_path` (str): Path to inference configuration file
- `with_triton` (bool): If **`False`** (default), adds **`nvinfer`**. If **`True`**, adds **`nvinferserver`** for Triton-based inference.
- `kwargs` (dict): Optional properties passed to gst-nvinfer or gst-nvinferserver plugin of DeepStream. Underscores in keyword names are converted to hyphens for GStreamer properties (e.g. **`batch_size`** → **`batch-size`**). Common overrides include **`batch_size`**, **`unique_id`**, **`model_engine_file`**, **`gpu_id`**, and other keys supported by **nvinfer** / **nvinferserver** for your install.

**Returns**: Flow instance (for method chaining)

**Notes**: For multiple streams inferencing case, `batch_size` property should be set as the same value as the stream number.

**Examples**:
```python
flow.infer("/path/to/pgie_config.yml")

#set nvinfer/nvinferserver properties with Flow.infer function
flow.infer("/path/to/pgie_config.yml",unique_id=5, batch_size=4)
```

##### `track(**kwargs)`
Add tracker for object tracking. Must be used after primary inference.

**Parameters**:
The following keyword arguments(kwargs) are passed to **nvtrack** as properties.
| Property            | Type | Description |
|---------------------|------|-------------|
| **`ll_config_file`** | str  | Path to the low-level tracker config file (e.g. NvDCF, NvSORT, IOU). |
| **`ll_lib_file`**    | str  | Path to the tracker library (e.g. `libnvds_nvmultiobjecttracker.so`). |
| **`gpu_id`**         | int  | GPU device id (default 0). |

**Notes**:
Example tracker configs (paths may vary by installation):
- NvDCF (performance): `config_tracker_NvDCF_perf.yml`
- NvDCF (accuracy): `config_tracker_NvDCF_accuracy.yml`
- NvSORT: `config_tracker_NvSORT.yml`
- IOU: `config_tracker_IOU.yml`
- NvDeepSORT: `config_tracker_NvDeepSORT.yml`

**Example**:
```python
flow = flow.track(ll_config_file=config_tracker_NvDCF_perf.yml, ll_lib_file=libnvds_nvmultiobjecttracker.so)
```

##### `analyze(config_file_path,**kwargs)`
Add analytics for region-of-interest (ROI), line-crossing, overcrowding and direction analytics. The result will be output as AnalyticsFrameMeta in frame meta and AnalyticsObjInfo in object meta.

**Parameters**:
- `config_file_path` (str): Path to analytics configuration file
- `kwargs` (dict): Optional properties passed to gst-nvdsanalytics plugin of DeepStream

**Notes**:
analytics MUST follow tracker to work properly.

**Example**:
```python
from pyservicemaker import Pipeline, Flow, BatchMetadataOperator, Probe, RenderMode

PGIE_CONFIG = "/path/to/config_infer_primary.yml"
TRACKER_LL_CONFIG = "/path/to/config_tracker_NvDCF_perf.yml"
TRACKER_LL_LIB = "/path/to/libnvds_nvmultiobjecttracker.so"
ANALYTICS_CONFIG = "/path/to/config_analytics.txt"  # nvdsanalytics config
SOURCE = "/path/to/source_list.yaml"

class AnalyticsProbe(BatchMetadataOperator):
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            # Frame-level analytics (ROI counts, line-cross counts)
            for user_meta in frame_meta.nvdsanalytics_frame_items:
                afm = user_meta.as_nvdsanalytics_frame()
                if afm:
                    print(f"Frame {frame_meta.frame_number}: unique_id={afm.unique_id} "
                          f"obj_in_roi_cnt={afm.obj_in_roi_cnt} obj_lc_curr_cnt={afm.obj_lc_curr_cnt} "
                          f"obj_cnt={afm.obj_cnt} oc_status={afm.oc_status}")

            # Object-level analytics (which ROI/line each object is in)
            for obj_meta in frame_meta.object_items:
                for user_meta in obj_meta.nvdsanalytics_obj_items:
                    aoi = user_meta.as_nvdsanalytics_obj()
                    if aoi:
                        print(f"  object_id={obj_meta.object_id} roi_status={aoi.roi_status} "
                              f"lc_status={aoi.lc_status} dir_status={aoi.dir_status} obj_status={aoi.obj_status}")

pipeline = Pipeline("analytics-demo")
flow = Flow(pipeline).batch_capture(SOURCE, width=1920, height=1080)
flow = flow.infer(PGIE_CONFIG)
flow = flow.track(ll_config_file=TRACKER_LL_CONFIG, ll_lib_file=TRACKER_LL_LIB)
flow = flow.analyze(ANALYTICS_CONFIG)
flow = flow.attach(what=Probe("analytics_probe", AnalyticsProbe()))
flow = flow.render(RenderMode.DISCARD, sync=False)
flow()
```

##### `attach(what, name='', tips='', properties=None)`
Attach a probe to the current flow.

**Parameters**:
- `what`: Probe instance or element name
- `name` (str, optional): Name for the probe. Not applicable when `what` is an explicitly created Probe object.
- `tips` (str, optional): Extra information for the custom object
- `properties` (dict, optional): Properties to set on the object.

**Returns**: Flow instance (for method chaining)

**Example**:
```python
from pyservicemaker import Probe
# Attach a custom probe (name is embedded in the Probe object)
flow.attach(Probe("my-probe", MyProbe()))

# Attach built-in probe by module name and name the probe by 'name'
flow = flow.attach(
            what="measure_fps_probe",
            name="fps_probe"
        )
```

##### `render()`
Add rendering stage to the pipeline.

**Returns**: Flow instance (for method chaining)

**Example**:
```python
flow.render()
```

##### `__call__()` (Invocation)
Execute the pipeline (start and wait).

**Example**:
```python
flow()  # Starts and waits for completion
```

### Complete Flow API Example

```python
from pyservicemaker import Pipeline, Flow, Probe, BatchMetadataOperator

class ObjectCounter(BatchMetadataOperator):
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            # IMPORTANT: object_items is an ITERATOR - cannot use len()
            obj_count = 0
            for obj in frame_meta.object_items:
                obj_count += 1
            print(f"Frame {frame_meta.frame_number}: {obj_count} objects")

def main():
    pipeline = Pipeline("my-pipeline")
    flow = Flow(pipeline)
    
    flow.batch_capture(["/path/to/video.h264"]) \
        .infer("/path/to/inference_config.txt") \  # Must be INI-style text format
        .attach(Probe("counter", ObjectCounter())) \
        .render()()
    
if __name__ == "__main__":
    main()
```

---

## Metadata API

### CRITICAL: Iterator Handling

**WARNING**: Properties like `frame_meta.object_items`, `frame_meta.tensor_items`, and `frame_meta.user_items` return **ITERATORS**, not lists!

**Common Mistakes to Avoid**:
```python
# WRONG - Will crash with "TypeError: object of type 'iterator' has no len()"
count = len(frame_meta.object_items)

# WRONG - Iterator can only be consumed once
for obj in frame_meta.object_items:
    process(obj)
for obj in frame_meta.object_items:  # This loop will be empty!
    do_something(obj)
```

**Correct Patterns**:
```python
# CORRECT - Count by iterating
obj_count = 0
for obj in frame_meta.object_items:
    obj_count += 1
    process(obj)

# CORRECT - If you need to iterate multiple times, convert to list first
# (only if you actually need multiple iterations)
object_list = list(frame_meta.object_items)
count = len(object_list)
for obj in object_list:
    process(obj)
```

---

### BatchMetadataOperator
Base class for implementing custom metadata processing.

**Methods**:

##### `handle_metadata(batch_meta)`
Override this method to process batch metadata.

**Parameters**:
- `batch_meta`: BatchMetadata object containing frame and object metadata

**Example**:
```python
class MyOperator(BatchMetadataOperator):
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            # Process each frame
            # NOTE: object_items is an ITERATOR, not a list!
            for object_meta in frame_meta.object_items:
                # Process each object
                pass
```

### BatchMetadata Object

**Properties**:
- `frame_items`: List of FrameMetadata objects
- Methods for acquiring metadata objects

**Methods**:
- `acquire_object_meta()`: Create new object metadata
- `acquire_display_meta()`: Create new display metadata
- `acquire_user_meta()`: Create new user metadata
- `acquire_event_message_meta()`: Create new `EventMessageUserMetadata` for nvmsgconv (see EventMessageUserMetadata section below)

### FrameMetadata Object

**Properties**:
- `frame_number`: Frame number (int)
- `pad_index`: Source pad index (int)
- `batch_id`: Location of frame in the batch (int)
- `source_id`: Source ID of the frame, e.g., camera ID (int)
- `source_width`: Width of the frame at input to streammux (int)
- `source_height`: Height of the frame at input to streammux (int)
- `pipeline_width`: Width of the frame at output of streammux (int)
- `pipeline_height`: Height of the frame at output of streammux (int)
- `buffer_pts`: Presentation timestamp (PTS) of the frame in nanoseconds (int)
- `ntp_timestamp`: NTP timestamp of the frame (int)
- `object_items`: **ITERATOR** of ObjectMetadata objects (NOT a list - cannot use `len()`)
- `tensor_items`: **ITERATOR** of TensorOutputUserMetadata objects (NOT a list - cannot use `len()`)
- `segmentation_items`: **ITERATOR** of SegmentationUserMetadata objects (NOT a list - cannot use `len()`)
- `nvdsanalytics_frame_items`: **ITERATOR** of AnalyticsFrameMeta objects (NOT a list - cannot use `len()`)
**IMPORTANT**: The `*_items` properties return iterators that can only be consumed once. See "CRITICAL: Iterator Handling" section above.

**NOTE**: There is no `timestamp` property. Use `buffer_pts` for PTS timestamp or `ntp_timestamp` for NTP timestamp.

**Methods**:
- `append(meta)`: Add metadata to frame

### ObjectMetadata Object

**Properties**:
- `class_id`: Class ID (int)
- `confidence`: Confidence score (float)
- `object_id`: Unique tracking ID assigned by tracker (int). Value is `0xFFFFFFFFFFFFFFFF` (UNTRACKED_OBJECT_ID) if object has not been tracked.
- `tracker_confidence`: Confidence value from tracker (float). Set to -0.1 for KLT and IOU trackers.
- `rect_params`: Rectangle parameters object
  - `left`: Left coordinate (float)
  - `top`: Top coordinate (float)
  - `width`: Width (float)
  - `height`: Height (float)
  - `border_width`: Border width (int)
  - `border_color`: Border color (Color object)
- `label`: String describing the object class
- `text_params`: Text parameters for OSD display (NvOSD_TextParams)
- `mask_params`: Bbox-local mask parameters for a per-object OSD overlay (`NvOSD_MaskParams`); it does not represent a full-frame segmentation map.
- `classifier_items`: **ITERATOR** of ClassifierMetadata objects. (NOT a list - cannot use `len()`)
- `tensor_items`: **ITERATOR** of TensorOutputUserMetadata objects. (NOT a list - cannot use `len()`)
- `nvdsanalytics_obj_items`: **ITERATOR** of AnalyticsObjInfo objects. (NOT a list - cannot use `len()`)

**Note**: The attribute is `object_id`, NOT `tracking_id`. This is the unique ID assigned by the tracker to track objects across frames.

### RectParams Object

**Properties**:
- `left`, `top`, `width`, `height`: Coordinates and dimensions
- `border_width`: Border width
- `border_color`: Border color (Color object)

### TensorOutputUserMetadata Object

**Methods**:
- `as_tensor_output()`: Get tensor output object
  - `get_layers()`: Get output layers dictionary

**Example**:
```python
for user_meta in frame_meta.tensor_items:
    tensor_output = user_meta.as_tensor_output()
    layers = tensor_output.get_layers()
    # layers is a dict: {"layer_name": tensor, ...}
```

### SegmentationUserMetadata Object

**Properties**:
- `unique_id`: Unique id of the component that generates the segmentation output.
- `classes`: Number of classes in the segmentation output. |
- `width`, `height`: Width and height of the segmentation mask array.
- `class_map`: Class map array of the segmentation output; shape `(height, width)`, dtype int. Each pixel holds the class index.
- `class_probabilities_map`: Class probabilities map array; shape `(height, width, classes)`, dtype float. Optional; may be empty if not produced by the model.

**Example**:
```python
from pyservicemaker import Pipeline, Flow, BatchMetadataOperator

class MyOperator(BatchMetadataOperator):
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            # frame_meta is FrameMetadata
            for user_meta in frame_meta.segmentation_items:
                # user_meta is UserMetadata (segmentation type)
                seg_meta = user_meta.as_segmentation()
                if seg_meta:  # cast is valid when meta type matches
                    # Use SegmentationUserMetadata attributes
                    print("unique_id:", seg_meta.unique_id)
                    print("classes:", seg_meta.classes)
                    print("width:", seg_meta.width, "height:", seg_meta.height)
                    # class_map: (height, width) int array
                    print("class_map shape:", seg_meta.class_map.shape)
                    # class_probabilities_map: (height, width, classes) float array, if present
                    if seg_meta.class_probabilities_map.size > 0:
                        print("class_probabilities_map shape:", seg_meta.class_probabilities_map.shape)
```

### AnalyticsFrameMeta object

**Properties**:
- `oc_status`: Map of overcrowding status per ROI (key = ROI label). Type: dict[str, bool]
- `obj_in_roi_cnt`: Map of count of valid objects in each ROI (key = ROI label). Type: dict[str, int] 
- `obj_lc_curr_cnt`: Map of line-crossing count in the current frame per line (key = line/ROI label). Type: dict[str, int]              |  |
- `obj_lc_cum_cnt`: Map of cumulative line-crossing count per line (key = line/ROI label). Type: dict[str, int]
- `unique_id`: Unique identifier for the nvdsanalytics instance.
- `obj_cnt`: Map of object count per class ID (key = class ID). Type: dict[int, int]

**Example**:
```python
from pyservicemaker import Pipeline, Flow, BatchMetadataOperator

class MyOperator(BatchMetadataOperator):
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            # frame_meta is FrameMetadata
            for user_meta in frame_meta.nvdsanalytics_frame_items:
                # user_meta is UserMetadata (nvdsanalytics frame type)
                analytics_frame_meta = user_meta.as_nvdsanalytics_frame()
                if analytics_frame_meta:  # cast is valid when meta type matches
                    # Use AnalyticsFrameMeta attributes
                    print("Frame {0} component id: {1}".format(analytics_frame_meta.unique_id))
                    print("Frame {0} overcrowding status: {1}".format(frame_meta.frame_number, analytics_frame_meta.oc_status))
                    print("Frame {0} object in ROI count: {1}".format(frame_meta.frame_number, analytics_frame_meta.obj_in_roi_cnt))
                    print("Frame {0} object line crossing current count: {1}".format(frame_meta.frame_number, analytics_frame_meta.obj_lc_curr_cnt))
                    print("Frame {0} object line crossing cumulative count: {1}".format(frame_meta.frame_number, analytics_frame_meta.obj_lc_cum_cnt))
                    print("Frame {0} object count: {1}".format(frame_meta.frame_number,, analytics_frame_meta.obj_cnt))
```

### AnalyticsObjInfo object

**Properties**:
- `roi_status`: Array of ROI labels in which this object is present. Type: list[str].
- `oc_status`: Array of OverCrowding labels in which this object is present. Type: list[str].
- `lc_status`: Array of line-crossing labels which this object has crossed. Type: list[str].
- `dir_status`: Direction string for the tracked object.
- `unique_id`: Unique identifier for the nvdsanalytics instance.
- `obj_status`: Status string for the tracked object.

**Note**: AnalyticsObjInfo is stored as **user metadata** on the object. **ObjectMetadata** exposes an iterator **`nvdsanalytics_obj_items`** over user metadata of type **NVDS_USER_OBJ_META_NVDSANALYTICS**; each element is a **UserMetadata** instance, which you cast to **AnalyticsObjInfo** using **`as_nvdsanalytics_obj()`**.

**Example**:
```python
from pyservicemaker import Pipeline, Flow, BatchMetadataOperator

class MyOperator(BatchMetadataOperator):
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            for obj_meta in frame_meta.object_items:
                # obj_meta is ObjectMetadata
                for user_meta in obj_meta.nvdsanalytics_obj_items:
                    # user_meta is UserMetadata (nvdsanalytics object type)
                    analytics_obj = user_meta.as_nvdsanalytics_obj()
                    if analytics_obj:  # cast is valid when meta type matches
                        # Use AnalyticsObjInfo attributes
                        print("Object {0} ROI status: {1}".format(object_meta.object_id, analytics_obj.roi_status))
                        print("Object {0} overcrowding status: {1}".format(object_meta.object_id, analytics_obj.oc_status))
                        print("Object {0} line crossing status: {1}".format(obj_meta.object_id, analytics_obj.lc_status))
                        print("Object {0} moving in direction: {1}".format(obj_meta.object_id, analytics_obj.dir_status))
                        print("Object {0} unique ID: {1}".format(object_meta.object_id, analytics_obj.unique_id))
                        print("Object {0} status: {1}".format(object_meta.object_id, analytics_obj.obj_status))
```

### ClassifierMetadata object

**Properties**:
- `n_labels`: Number of output labels of the classifier.
- `unique_component_id`: Unique id of the component that generates the classifier metadata.

**Methods**:
- `get_n_label(n)`: Returns the nth label of the classifier (0-based index `n`).

**Example**:
```python
from pyservicemaker import Pipeline, Flow, BatchMetadataOperator

class MyOperator(BatchMetadataOperator):
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            for obj_meta in frame_meta.object_items:
                for classifier_meta in obj_meta.classifier_items:
                    # classifier_meta is ClassifierMetadata
                    print("n_labels:", classifier_meta.n_labels)
                    print("unique_component_id:", classifier_meta.unique_component_id)
                    for i in range(classifier_meta.n_labels):
                        label = classifier_meta.get_n_label(i)
                        print(f"  label[{i}]:", label)
```

---

## OSD (On-Screen Display) API

### osd Module

Provides classes for creating OSD elements.

#### Text
Text display element.

**Properties**:
- `display_text`: Text content (bytes)
- `x_offset`: X position (int)
- `y_offset`: Y position (int)
- `font`: Font object
- `set_bg_color`: Enable background color (bool)
- `bg_color`: Background color (Color object)

#### Font
Font specification.

**Properties**:
- `name`: Font family (FontFamily enum)
- `size`: Font size (int)
- `color`: Font color (Color object)

#### FontFamily Enum
- `Serif`
- `Sans`
- `Mono`

#### Color
Color specification (RGBA).

**Properties**:
- Red, Green, Blue, Alpha values (0.0 to 1.0)

**Constructor**:
```python
color = osd.Color(1.0, 0.0, 0.0, 1.0)  # Red, fully opaque
```

### DisplayMeta Object

**Methods**:
- `add_text(text)`: Add text element
- `add_rect(rect)`: Add rectangle element
- `add_line(line)`: Add line element
- `add_circle(circle)`: Add circle element

### Example: Adding Text Overlay

```python
from pyservicemaker import osd

display_meta = batch_meta.acquire_display_meta()
text = osd.Text()
text.display_text = b"Object Count: 5"
text.x_offset = 10
text.y_offset = 12
text.font.name = osd.FontFamily.Serif
text.font.size = 12
text.font.color = osd.Color(1.0, 1.0, 1.0, 1.0)
text.set_bg_color = True
text.bg_color = osd.Color(0.0, 0.0, 0.0, 1.0)
display_meta.add_text(text)
frame_meta.append(display_meta)
```

---

## Postprocessing API

### postprocessing Module

Provides classes for custom postprocessing.

#### ObjectDetectorOutputConverter
Base class for converting tensor outputs to object detections.

**Methods**:

##### `__call__(output_layers)`
Convert tensor outputs to list of bounding boxes.

**Parameters**:
- `output_layers` (dict): Dictionary of layer names to tensors

**Returns**: List of bounding boxes `[class_id, confidence, x1, y1, x2, y2]`

**Example**:
```python
from pyservicemaker import postprocessing
import torch

class MyConverter(postprocessing.ObjectDetectorOutputConverter):
    def __call__(self, output_layers):
        outputs = []
        bbox_tensor = output_layers.get('bbox_layer')
        conf_tensor = output_layers.get('conf_layer')
        
        if bbox_tensor and conf_tensor:
            # Convert DLPack tensors to PyTorch
            bbox = torch.utils.dlpack.from_dlpack(bbox_tensor)
            conf = torch.utils.dlpack.from_dlpack(conf_tensor)
            
            # Process and convert to format: [class_id, confidence, x1, y1, x2, y2]
            # ... processing logic ...
            
        return outputs
```

**Usage**:
```python
converter = MyConverter()
objects = converter(output_layers)
# objects is list of [class_id, confidence, x1, y1, x2, y2]
```

---

## Probe API

### Probe Class

Wrapper for attaching callback functions to pipeline elements.

**Constructor** (two overloads):
```python
from pyservicemaker import Probe

# Overload 1: Metadata-level probe (most common)
probe = Probe("probe-name", BatchMetadataOperator())

# Overload 2: Buffer-level probe (for raw buffer access)
probe = Probe("probe-name", BufferOperator())
```

**Parameters**:
- `name` (str): Name of the probe
- `operator`: `BatchMetadataOperator` instance **or** `BufferOperator` instance

**Built-in Probes**:
- `"measure_fps_probe"`: Measures FPS
- `"measure_latency_probe"`: Measures latency
- `"add_message_meta_probe"`: Automatically generates `EventMessageUserMetadata` (NvDsEventMsgMeta) from object metadata for downstream `nvmsgconv` consumption. Use this when `msg2p-newapi=0` and you don't need custom control over sensor mappings.

**Example**:
```python
# Custom probe
probe = Probe("my-probe", MyOperator())

# Built-in probe
pipeline.attach("infer", "measure_fps_probe", "fps-probe")

# Built-in message meta probe (for Kafka with msg2p-newapi=0)
pipeline.attach("osd", "add_message_meta_probe", "metadata generator")
```

### BufferOperator Class

Low-level probe interface for accessing raw `Buffer` objects flowing through a pad. Use `BufferOperator` instead of `BatchMetadataOperator` when you need to inspect or count raw buffers that do NOT carry batch metadata — e.g., on the `src` pad of `nvdsdynamicsrcbin` (before any `nvstreammux`).

**Methods to Override**:

##### `handle_buffer(buffer)`
Called for every buffer that passes through the probed pad.

**Parameters**:
- `buffer` (Buffer): The buffer flowing through the pad

**Returns**: `bool` — `True` to pass the buffer downstream (keep), `False` to drop it.

**Buffer Object Properties/Methods** (available inside `handle_buffer`):
- `buffer.timestamp` (int): PTS timestamp of the buffer
- `buffer.get_chunk_id(batch_id)` (int): Chunk/source ID assigned by `nvdsdynamicsrcbin`. Always 0 for `uridecodebin`.
- `buffer.extract(batch_id)` → `Tensor`: Extract frame data as a tensor

**Example**:
```python
from pyservicemaker import Pipeline, Probe, BufferOperator

class MyBufferProbe(BufferOperator):
    def __init__(self):
        super().__init__()
        self.count = 0

    def handle_buffer(self, buffer):
        self.count += 1
        print(f"Buffer #{self.count}  ts={buffer.timestamp}")
        return True

probe = MyBufferProbe()
pipeline.attach("dynamicsrcbin", Probe("buf-probe", probe), tips="src")
```

---

## EventMessageUserMetadata

`EventMessageUserMetadata` wraps `NvDsEventMsgMeta` and is **required** by `nvmsgconv` when `msg2p-newapi` is `0` (the default / legacy API). Without it, nvmsgconv silently produces zero messages.

It is acquired from the `BatchMetadata` pool and must be populated and appended to the corresponding `FrameMetadata`.

### Acquiring and Generating Event Message Metadata

```python
event_msg = batch_meta.acquire_event_message_meta()  # Acquire from pool
event_msg.generate(object_meta, frame_meta, sensor_id, uri, labels)  # Populate
frame_meta.append(event_msg)  # Attach to frame
```

**Parameters for `generate()`**:
- `object_meta` (ObjectMetadata): The detected object to create a message for
- `frame_meta` (FrameMetadata): The frame containing the object
- `sensor_id` (str): Camera/sensor identifier string (e.g., `"Camera1"`)
- `uri` (str): Source URI of the stream (e.g., `"file:///path/to/video.mp4"`)
- `labels` (list[str]): List of class label strings matching class IDs (e.g., `["person", "bag", "face"]`)

### Two Approaches

#### Approach 1: Built-in Probe (Simple)

Use the built-in `"add_message_meta_probe"` -- no custom Python class needed:

```python
# Attach AFTER inference/tracker, BEFORE nvmsgconv
pipeline.attach("osd", "add_message_meta_probe", "metadata generator")
```

Reference: `deepstream_test4_app` sample
(`/opt/nvidia/deepstream/deepstream/service-maker/sources/apps/python/pipeline_api/deepstream_test4_app/deepstream_test4.py`)

#### Approach 2: Custom EventMessageGenerator (Full Control)

For multi-camera pipelines where you need control over sensor mappings:

```python
from pyservicemaker import Pipeline, Probe, BatchMetadataOperator, SensorInfo

class EventMessageGenerator(BatchMetadataOperator):
    """Generate EventMessageUserMetadata for downstream nvmsgconv."""

    def __init__(self, sensor_map, labels):
        super().__init__()
        self._sensor_map = sensor_map  # dict: source_id -> SensorInfo or str
        self._labels = labels          # list of class label strings

    def handle_metadata(self, batch_meta, frame_interval=1):
        for frame_meta in batch_meta.frame_items:
            frame_num = frame_meta.frame_number
            for object_meta in frame_meta.object_items:
                if not (frame_num % frame_interval):
                    event_msg = batch_meta.acquire_event_message_meta()
                    if event_msg:
                        source_id = frame_meta.source_id
                        sensor_info = self._sensor_map.get(source_id)
                        sensor_id = sensor_info.sensor_id if sensor_info else "N/A"
                        uri = sensor_info.uri if sensor_info else "N/A"
                        event_msg.generate(
                            object_meta, frame_meta, sensor_id, uri, self._labels
                        )
                        frame_meta.append(event_msg)

# Attach probe upstream of nvmsgconv
labels = ["car", "bicycle", "person", "roadsign"]
sensor_map = {0: SensorInfo(sensor_id="Camera1", sensor_name="cam1", uri="file:///video1.mp4")}
pipeline.attach("tracker", Probe("event_msg_gen", EventMessageGenerator(sensor_map, labels)))
```

Reference: `deepstream_test5_app` sample
(`/opt/nvidia/deepstream/deepstream/service-maker/sources/apps/python/pipeline_api/deepstream_test5_app/deepstream_test5.py`)

### SensorInfo Class

Used to map source IDs to sensor metadata for `EventMessageGenerator`:

```python
from pyservicemaker import SensorInfo

sensor_info = SensorInfo(
    sensor_id="Camera1",       # Unique sensor identifier string
    sensor_name="front_cam",   # Human-readable name
    uri="rtsp://host/stream1"  # Source URI
)
```

---

## YAML Configuration Support

Pipelines can be created from YAML configuration files (for pipeline structure definition):

```python
pipeline = Pipeline("pipeline-name", "/path/to/pipeline_config.yml")
```

**Note**: This YAML config is for **pipeline structure** (elements, links, probes). The nvinfer `config-file-path` can point to either a YAML file (`.yml`) or INI-style text file (`.txt`) - both formats are supported.

### YAML Structure Example (Pipeline Definition)

```yaml
pipeline:
  name: my-pipeline
  elements:
    - name: src
      type: filesrc
      properties:
        location: /path/to/video.h264
    
    - name: parser
      type: h264parse
    
    - name: decoder
      type: nvv4l2decoder
    
    - name: mux
      type: nvstreammux
      properties:
        batch-size: 1
        width: 1920
        height: 1080
    
    - name: infer
      type: nvinfer
      properties:
        # nvinfer supports both YAML (.yml) and INI-style (.txt) config formats
        config-file-path: /path/to/pgie_config.yml
    
    - name: osd
      type: nvosdbin
    
    - name: sink
      type: nveglglessink
  
  links:
    - [src, parser, decoder]
    - [decoder, mux]
    - [mux, infer, osd, sink]
  
  probes:
    - element: infer
      probe-name: my-probe
      probe-type: custom
      operator: MyOperator
```

### nvinfer Configuration (Both Formats Supported)

The `config-file-path` for nvinfer supports **both YAML and INI-style text formats**:

**YAML Format** (`.yml`) - Recommended:
```yaml
# pgie_config.yml - YAML format for nvinfer
property:
  gpu-id: 0
  net-scale-factor: 0.00392156862745098
  onnx-file: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx
  labelfile-path: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt
  batch-size: 1
  process-mode: 1
  model-color-format: 0
  network-mode: 2
  num-detected-classes: 4
  cluster-mode: 2

class-attrs-all:
  topk: 20
  pre-cluster-threshold: 0.2
```

**INI-style Format** (`.txt`):
```ini
# pgie_config.txt - INI-style format for nvinfer
[property]
gpu-id=0
net-scale-factor=0.00392156862745098
onnx-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx
labelfile-path=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt
batch-size=1
process-mode=1
model-color-format=0
network-mode=2
num-detected-classes=4
cluster-mode=2

[class-attrs-all]
topk=20
pre-cluster-threshold=0.2
```

---

## Common Patterns and Examples

### Pattern 1: Single Stream with Detection

```python
from pyservicemaker import Pipeline, Probe, BatchMetadataOperator
import platform

def single_stream_detection(video_path, config_path):
    pipeline = (Pipeline("single-stream")
        .add("filesrc", "src", {"location": video_path})
        .add("h264parse", "parser")
        .add("nvv4l2decoder", "decoder")
        .add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})
        .add("nvinfer", "infer", {"config-file-path": config_path})
        .add("nvosdbin", "osd")
        .add("nv3dsink" if platform.processor() == "aarch64" else "nveglglessink", "sink")
        .link("src", "parser", "decoder")
        .link(("decoder", "mux"), ("", "sink_%u"))
        .link("mux", "infer", "osd", "sink")
        .start()
        .wait())
```

### Pattern 2: Multi-Stream with Detection

**Pattern 2a: Multi-Stream from Files**
```python
def multi_stream_detection(video_paths, config_path):
    pipeline = Pipeline("multi-stream")
    
    # Add sources
    for i, path in enumerate(video_paths):
        pipeline.add("filesrc", f"src{i}", {"location": path})
        pipeline.add("h264parse", f"parser{i}")
        pipeline.add("nvv4l2decoder", f"decoder{i}")
    
    # Add muxer
    pipeline.add("nvstreammux", "mux", {
        "batch-size": len(video_paths),
        "width": 1920,
        "height": 1080
    })
    
    # Add processing elements
    pipeline.add("nvinfer", "infer", {"config-file-path": config_path})
    pipeline.add("nvosdbin", "osd")
    pipeline.add("nveglglessink", "sink")
    
    # Link sources to muxer
    for i in range(len(video_paths)):
        pipeline.link(f"src{i}", f"parser{i}", f"decoder{i}")
        pipeline.link((f"decoder{i}", "mux"), ("", "sink_%u"))  # CRITICAL: Use "sink_%u", NOT f"sink_{i}"
    
    # Link processing chain
    pipeline.link("mux", "infer", "osd", "sink")
    pipeline.start().wait()
```

**Pattern 2b: Multi-Stream RTSP with nvurisrcbin**
```python
def multi_rtsp_stream_detection(rtsp_urls, config_path):
    """
    Process multiple RTSP streams using nvurisrcbin.
    
    Args:
        rtsp_urls: List of RTSP stream URLs (e.g., ["rtsp://...", "rtsp://..."])
        config_path: Path to inference config file
    """
    pipeline = Pipeline("multi-rtsp-stream")
    
    # Add RTSP sources with nvurisrcbin (auto-detects codec and creates dynamic pads)
    for i, url in enumerate(rtsp_urls):
        pipeline.add("nvurisrcbin", f"src{i}", {"uri": url})
    
    # Add muxer for batching
    pipeline.add("nvstreammux", "mux", {
        "batch-size": len(rtsp_urls),
        "width": 1920,
        "height": 1080,
        "batched-push-timeout": 40000,
        "live-source": 1  # Important for RTSP streams
    })
    
    # Add processing elements
    pipeline.add("nvinfer", "infer", {"config-file-path": config_path, "batch-size": len(rtsp_urls)})
    pipeline.add("nvmultistreamtiler", "tiler", {"rows": 2, "columns": 2})
    pipeline.add("nvosdbin", "osd")
    pipeline.add("nveglglessink", "sink")
    
    # Link sources to muxer - CRITICAL: Use "sink_%u" pad template, NOT f"sink_{i}"
    for i in range(len(rtsp_urls)):
        # nvurisrcbin has dynamic src pad, so link directly to mux sink pad template
        pipeline.link((f"src{i}", "mux"), ("", "sink_%u"))  # CORRECT - pad template auto-assigns sink_0, sink_1, etc.
        # WRONG: pipeline.link((f"src{i}", "mux"), ("", f"sink_{i}"))  # This will FAIL!
    
    # Link processing chain
    pipeline.link("mux", "infer", "tiler", "osd", "sink")
    pipeline.start().wait()
```

### Pattern 3: Custom Metadata Processing

```python
class CustomProcessor(BatchMetadataOperator):
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            # Count objects by class
            class_counts = {}
            for obj in frame_meta.object_items:
                class_id = obj.class_id
                class_counts[class_id] = class_counts.get(class_id, 0) + 1
            
            # Add text overlay
            display_meta = batch_meta.acquire_display_meta()
            text = osd.Text()
            text.display_text = f"Objects: {sum(class_counts.values())}".encode('ascii')
            text.x_offset = 10
            text.y_offset = 10
            text.font.name = osd.FontFamily.Serif
            text.font.size = 12
            text.font.color = osd.Color(1.0, 1.0, 1.0, 1.0)
            display_meta.add_text(text)
            frame_meta.append(display_meta)

# Attach probe
pipeline.attach("infer", Probe("processor", CustomProcessor()))
```

### Pattern 4: Tensor-Based Custom Postprocessing

```python
class TensorConverter(postprocessing.ObjectDetectorOutputConverter):
    def __call__(self, output_layers):
        outputs = []
        # Extract tensors
        bbox_layer = output_layers.get('bbox')
        conf_layer = output_layers.get('conf')
        
        if bbox_layer and conf_layer:
            import torch
            bbox = torch.utils.dlpack.from_dlpack(bbox_layer)
            conf = torch.utils.dlpack.from_dlpack(conf_layer)
            
            # Process tensors and convert to [class_id, conf, x1, y1, x2, y2]
            # ... processing logic ...
            
        return outputs

class TensorProcessor(BatchMetadataOperator):
    def __init__(self):
        super().__init__()
        self._converter = TensorConverter()
    
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            for tensor_meta in frame_meta.tensor_items:
                output_layers = tensor_meta.as_tensor_output().get_layers()
                objects = self._converter(output_layers)
                
                # Create object metadata
                for obj in objects:
                    obj_meta = batch_meta.acquire_object_meta()
                    obj_meta.class_id = obj[0]
                    obj_meta.confidence = obj[1]
                    obj_meta.rect_params.left = obj[2]
                    obj_meta.rect_params.top = obj[3]
                    obj_meta.rect_params.width = obj[4] - obj[2]
                    obj_meta.rect_params.height = obj[5] - obj[3]
                    frame_meta.append(obj_meta)

# Enable tensor output in nvinfer
pipeline["infer"].set({"output-tensor-meta": 1})
pipeline.attach("infer", Probe("tensor-processor", TensorProcessor()))
```

### Pattern 5: Cloud Integration (Kafka)

```python
from kafka import KafkaProducer
import json

class KafkaSender(BatchMetadataOperator):
    def __init__(self, kafka_config):
        super().__init__()
        self.producer = KafkaProducer(
            bootstrap_servers=kafka_config['servers'],
            value_serializer=lambda v: json.dumps(v).encode('utf-8')
        )
        self.topic = kafka_config['topic']
    
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            objects = [
                {
                    "class_id": obj.class_id,
                    "confidence": obj.confidence,
                    "bbox": {
                        "left": obj.rect_params.left,
                        "top": obj.rect_params.top,
                        "width": obj.rect_params.width,
                        "height": obj.rect_params.height
                    },
                    "object_id": obj.object_id  # Tracking ID assigned by tracker
                }
                for obj in frame_meta.object_items
            ]
            
            message = {
                "frame_number": frame_meta.frame_number,
                "source_id": frame_meta.source_id,
                "buffer_pts": frame_meta.buffer_pts,  # PTS timestamp in nanoseconds
                "objects": objects
            }
            
            self.producer.send(topic=self.topic, value=message)
    
    def __del__(self):
        if hasattr(self, 'producer'):
            self.producer.flush()
            self.producer.close()

# Usage
kafka_config = {
    "servers": "localhost:9092",
    "topic": "analytics"
}
pipeline.attach("infer", Probe("kafka-sender", KafkaSender(kafka_config)))
```

---

## Best Practices

1. **Use Pipeline API for fine-grained control**, Flow API for rapid prototyping
2. **Always use hardware-accelerated decoders** (nvv4l2decoder)
3. **Configure appropriate batch sizes** for your use case
4. **Use probes for custom processing** instead of modifying plugins
5. **Handle KeyboardInterrupt** properly (use multiprocessing.Process)
6. **Flush and close Kafka producers** in cleanup methods
7. **Use tensor metadata** for custom postprocessing when needed
8. **Match tracker dimensions** to inference input dimensions
9. **Use YAML configs** for complex pipelines to improve maintainability
10. **Monitor GPU memory** when processing multiple streams
11. **Use correct Queue types for inter-process/thread communication**:
    - `queue.Queue` → Use with `threading.Thread` (same process)
    - `multiprocessing.Queue` → Use with `multiprocessing.Process` (cross-process)
    - Using `queue.Queue` with `multiprocessing.Process` will silently lose data!

---

## Error Handling

```python
from multiprocessing import Process
import sys

def run_pipeline():
    try:
        pipeline.start().wait()
    except Exception as e:
        print(f"Pipeline error: {e}")
        sys.exit(1)

if __name__ == "__main__":
    process = Process(target=run_pipeline)
    try:
        process.start()
        process.join()
    except KeyboardInterrupt:
        print("\nInterrupted. Terminating...")
        process.terminate()
        process.join()
```

---

## Pipeline State and Message Handling API

### Pipeline States

DeepStream pipelines follow GStreamer state transitions:

| State | Description |
|-------|-------------|
| `PipelineState.NULL` | Initial state, no resources allocated |
| `PipelineState.READY` | Resources allocated, not processing |
| `PipelineState.PAUSED` | Paused, ready to play |
| `PipelineState.PLAYING` | Processing data |

### Pipeline Methods for State Management

#### `prepare(message_handler)`
Prepare the pipeline for activation with a message handler.

**Parameters**:
- `message_handler` (callable): Function to receive pipeline messages

**Returns**: Pipeline instance (for method chaining)

**Example**:
```python
def on_message(message):
    if isinstance(message, StateTransitionMessage):
        print(f"State changed to: {message.new_state}")
    elif isinstance(message, DynamicSourceMessage):
        print(f"Source event: {message.source_id}")

pipeline.prepare(on_message)
```

#### `activate()`
Activate the pipeline (set to PLAYING state).

**Returns**: Pipeline instance (for method chaining)

#### `deactivate()`
Deactivate the pipeline (set to NULL state).

**Returns**: Pipeline instance (for method chaining)

#### `wait()`
Wait for the pipeline to complete (blocking).

**Returns**: None

### Message Types

#### StateTransitionMessage
Indicates a pipeline state change.

**Properties**:
- `origin` (str): Element name that changed state
- `old_state` (PipelineState): Previous state
- `new_state` (PipelineState): New state

**Example**:
```python
from pyservicemaker import StateTransitionMessage, PipelineState

def on_message(message):
    if isinstance(message, StateTransitionMessage):
        if message.new_state == PipelineState.PLAYING:
            print(f"Element {message.origin} is now playing")
        elif message.new_state == PipelineState.NULL:
            print(f"Element {message.origin} stopped")
```

#### DynamicSourceMessage
Indicates a dynamic source change (add/remove).

**Properties**:
- `source_id` (int): Unique source identifier
- `source_added` (bool): True if added, False if removed
- `sensor_id` (str): Sensor identifier
- `sensor_name` (str): Human-readable sensor name
- `uri` (str): Source URI (for added sources)

**Example**:
```python
from pyservicemaker import DynamicSourceMessage

sensor_map = {}

def on_message(message):
    if isinstance(message, DynamicSourceMessage):
        if message.source_added:
            sensor_map[message.source_id] = {
                "sensor_id": message.sensor_id,
                "sensor_name": message.sensor_name,
                "uri": message.uri
            }
            print(f"Added source: {message.sensor_name}")
        else:
            if message.source_id in sensor_map:
                del sensor_map[message.source_id]
            print(f"Removed source: {message.source_id}")
```

### Complete Message Handling Example

```python
from pyservicemaker import (
    Pipeline, PipelineState, StateTransitionMessage,
    DynamicSourceMessage, SensorInfo, utils
)

def run_pipeline_with_messages(config_file):
    """Pipeline with comprehensive message handling"""
    pipeline = Pipeline("message-aware-pipeline", config_file=config_file)
    
    # Track sources
    active_sources = {}
    
    # Performance monitor
    perf_monitor = utils.PerfMonitor(
        batch_size=4,
        interval=5,
        source_type="nvmultiurisrcbin"
    )
    perf_monitor.apply(pipeline["tiler"], "sink")
    
    def handle_message(message):
        """Handle pipeline messages"""
        if isinstance(message, StateTransitionMessage):
            # Handle state transitions
            if message.new_state == PipelineState.PLAYING:
                if message.origin == "sink":
                    print("Pipeline fully started")
            elif message.new_state == PipelineState.NULL:
                print(f"Element {message.origin} stopped")
        
        elif isinstance(message, DynamicSourceMessage):
            # Handle dynamic source changes
            source_id = message.source_id
            
            if message.source_added:
                # Track new source
                active_sources[source_id] = SensorInfo(
                    sensor_id=message.sensor_id,
                    sensor_name=message.sensor_name,
                    uri=message.uri
                )
                
                # Add to performance monitor
                perf_monitor.add_stream(
                    source_id=source_id,
                    uri=message.uri,
                    sensor_id=message.sensor_id,
                    sensor_name=message.sensor_name
                )
                
                print(f"Source added: {message.sensor_name} ({message.uri})")
            else:
                # Remove source
                if source_id in active_sources:
                    del active_sources[source_id]
                perf_monitor.remove_stream(source_id)
                print(f"Source removed: {source_id}")
    
    # Prepare with message handler
    pipeline.prepare(handle_message)
    
    # Activate and wait
    pipeline.activate()
    pipeline.wait()

# Run
run_pipeline_with_messages("pipeline_config.yaml")
```

---

## Signal Handling API

### Signal Module

The `signal` module provides classes for custom signal handling.

#### Emitter Class
Base class for signal emitters.

**Methods**:
- `attach(signal_name, element)`: Attach signal to element
- `set(properties)`: Set properties on the emitter

#### Handler Class
Base class for signal handlers.

### Smart Recording Signals

Smart recording uses signals for start/stop events.

**Signal Names**:
- `"start-sr"`: Start smart recording
- `"stop-sr"`: Stop smart recording
- `"sr-done"`: Recording complete

**Example**:
```python
from pyservicemaker import Pipeline, CommonFactory

pipeline = Pipeline("smart-recording")
# ... build pipeline ...

# Create smart recording controller
sr_controller = CommonFactory.create("smart_recording_action", "sr_controller")

if sr_controller:
    sr_controller.set({
        "proto-lib": "/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
        "conn-str": "localhost;9092",
        "topic-list": "sr-events"
    })
    
    # Attach signals to source element
    sr_controller.attach("start-sr", pipeline["src"])
    sr_controller.attach("stop-sr", pipeline["src"])
    
    # Attach signal handler for completion
    pipeline.attach("src", "smart_recording_signal", "sr", "sr-done")
```

---

## Dynamic Source Management

### nvmultiurisrcbin Properties

For dynamic source management, use `nvmultiurisrcbin`:

| Property | Type | Description |
|----------|------|-------------|
| `uri-list` | string | Comma-separated initial URIs |
| `sensor-id-list` | string | Comma-separated sensor IDs |
| `sensor-name-list` | string | Comma-separated sensor names |
| `max-batch-size` | int | Maximum number of sources |

### Adding/Removing Sources Dynamically

Sources are added/removed via REST API or programmatically through source management APIs.

```python
from pyservicemaker import Pipeline, SourceConfig, SensorInfo

# Load initial sources from config
source_config = SourceConfig()
source_config.load("sources.yaml")

# Create pipeline
pipeline = Pipeline("dynamic-sources", config_file="pipe

# Video Streaming Sources

## Overview

DeepStream pipelines ingest video from HTTP progressive download, HLS, MPEG-DASH, RTSP, and
local files through a single element: `nvurisrcbin`. This reference covers how to build source
pipelines for each protocol, how to scale to multiple simultaneous streams, and the codec and
format constraints that apply.

All pipeline examples use the `pyservicemaker` Python API. GStreamer
command-line equivalents are included where useful.

---

## Quick Reference

| Source type | URI format | Notes |
|---|---|---|
| Local file | `file:///path/to/video.mp4` | No server required |
| HTTP progressive (MP4) | `http://host/video.mp4` | Server must support Range requests |
| HLS (VoD or live) | `http://host/stream.m3u8` | Requires `gstreamer1.0-plugins-bad` |
| MPEG-DASH | `http://host/stream.mpd` | Requires DASH demux support; AdaptationSet selection is automatic |
| RTSP | `rtsp://camera/stream` | Live source; set `live-source=1` on `nvstreammux` |

---

## nvurisrcbin

**Purpose**: Universal source bin for URI-based video ingestion. It creates the appropriate
source, demux, parse, and decode path for URI protocols such as file, HTTP, HLS, DASH, and RTSP.
For HTTP/HLS/DASH inputs this commonly involves `souphttpsrc` plus the relevant demuxer
(`qtdemux`, `hlsdemux`, or `dashdemux`) before hardware decode.

**Key Properties**:

| Property | Type | Default | Description |
|---|---|---|---|
| `uri` | string | - | Source URI (`file://`, `http://`, `rtsp://`) |
| `gpu-id` | int | 0 | GPU device for NVDEC decoding |
| `num-buffers` | int | -1 | Limit decoded buffer count (-1 = unlimited) |
| `drop-on-latency` | bool | false | Drop frames when downstream is too slow |

**Output**: `video/x-raw(memory:NVMM)` frames in GPU memory, directly compatible
with `nvstreammux`.

**Usage (pyservicemaker)**:
```python
p.add("nvurisrcbin", "src", {"uri": "http://host/video.mp4", "gpu-id": 0})
p.link(("src", "mux"), ("", "sink_%u"))
```

**Usage (GStreamer CLI)**:
```bash
nvurisrcbin uri=http://host/video.mp4 gpu-id=0
```

For source selection, NVMM memory expectations, and `sink_%u` dynamic request-pad syntax,
follow the critical rules in [../SKILL.md](../SKILL.md#critical-rules). Audio pads from muxed
sources are not linked to `nvstreammux`; it accepts video NVMM pads only.

---

## Pipeline Topology

The downstream inference and output chain is the same for these source types. Live sources still
need live-specific muxer and sink settings; see the protocol notes below.

```
nvurisrcbin  <- URI (http://, rtsp://, file://)
     |
     | video/x-raw(memory:NVMM)
     v
nvstreammux      batch N streams; batch-size must equal source count
     v
nvinfer          TensorRT inference (object detection or classification)
     v
[nvtracker]      optional; use only when tracking or object IDs are requested
     v
nvosdbin         GPU bounding-box and label rendering
     v
nvvideoconvert
     v
capsfilter       video/x-raw(memory:NVMM), format=NV12; required before encoder
     v
nvv4l2h264enc    NVENC hardware H.264 encoder
     v
h264parse -> qtmux -> filesink      output.mp4
```

---

## Local Files

**When to use**: Video files already available inside the container or on a mounted host path.
No HTTP server is required.

Convert plain filesystem paths to `file://` URIs before passing them to `nvurisrcbin`:

```python
from pathlib import Path

uri = Path(video_path).resolve().as_uri()
p.add("nvurisrcbin", "src", {"uri": uri, "gpu-id": 0})
```

**Notes**:
- Containerized apps must mount the host directory containing the video file.
- For MP4/MOV/MKV files, `nvurisrcbin` handles demuxing internally.
- If the user explicitly needs parser-level control for raw elementary streams such as `.h264`
  or `.h265`, use the manual parser patterns in [use_cases_pipelines.md](use_cases_pipelines.md).

---

## HTTP Progressive Download

**When to use**: Local or CDN-hosted MP4 files served over plain HTTP. The simplest setup
for offline or batch processing.

**Server requirement**: `souphttpsrc` issues `Range: bytes=X-Y` requests to seek to the
MP4 `moov` atom before decoding. The server must respond with `206 Partial Content` and
include `Accept-Ranges: bytes`. Python's built-in `SimpleHTTPRequestHandler` does not
implement byte-range serving - see [Local Test Servers](#local-test-servers).

**Pipeline**:
```python
p.add("nvurisrcbin", "src", {"uri": "http://host:8080/video.mp4", "gpu-id": 0})
```

**GStreamer CLI**:
```bash
gst-launch-1.0 \
  nvurisrcbin uri=http://host:8080/video.mp4 gpu-id=0 ! mux.sink_0 \
  nvstreammux name=mux batch-size=1 width=1280 height=720 ! \
  nvinfer config-file-path=pgie.yml ! \
  nvosdbin ! nvvideoconvert ! \
  "video/x-raw(memory:NVMM),format=NV12" ! \
  nvv4l2h264enc ! h264parse ! qtmux ! filesink location=output.mp4
```

**Notes**:
- Muxed MP4 (video + audio) works without modification - audio pads are silently discarded.
- MP4 files with multiple video tracks: `qtdemux` selects the first video track.
- Verify Range support: `curl -I -H "Range: bytes=0-0" http://host:8080/video.mp4` should
  return `206 Partial Content` with `Accept-Ranges: bytes`.

---

## HLS

**When to use**: Live or VoD streams exposed as `.m3u8` playlists - CDN delivery, broadcast
workflows, or local live-stream simulation. No Range request support is required; HLS segments
are complete files.

**The pipeline code is identical to HTTP progressive** - only the URI changes:

```python
# HTTP MP4
p.add("nvurisrcbin", "src", {"uri": "http://host:8080/video.mp4", "gpu-id": 0})

# HLS VoD or live - only the URL changes
p.add("nvurisrcbin", "src", {"uri": "http://host:8080/stream.m3u8", "gpu-id": 0})
```

GStreamer's `hlsdemux` (from `gstreamer1.0-plugins-bad`) is selected automatically from the
URI. Output remains `video/x-raw(memory:NVMM)`; everything downstream is unchanged.

**Container package requirement**: `gstreamer1.0-plugins-bad` must be installed. See
[docker_containers.md](docker_containers.md).

**Known limitations**:

| Limitation | Detail |
|---|---|
| Master playlist variant selection | `hlsdemux` picks a quality level automatically; bitrate/resolution targeting is not configurable through `nvurisrcbin` properties |
| CMAF / fMP4 segments | Modern HLS increasingly uses `.mp4`/`.m4s` segments instead of MPEG-TS; behavior is not covered by this reference |
| Encrypted segments (`EXT-X-KEY`) | Encryption handling is not covered by this reference |
| Separate audio renditions (`EXT-X-MEDIA`) | Emitted as dynamic pads and silently discarded |

**Notes**:
- Direct CDN `.m3u8` URLs can be passed straight to `nvurisrcbin` - no local server needed.
- For local testing, generate HLS segments with ffmpeg as shown in
  [Local Test Servers](#local-test-servers).
- For live HLS, set `live-source=1` on `nvstreammux`; use `sync=0` on display sinks.

---

## MPEG-DASH

**When to use**: Sources that expose a `.mpd` manifest URL directly.

```python
p.add("nvurisrcbin", "src", {"uri": "http://host/stream.mpd", "gpu-id": 0})
```

GStreamer's `dashdemux` is selected automatically when the DASH plugin is available. Output is
`video/x-raw(memory:NVMM)`.

**Known limitations**:

| Limitation | Detail |
|---|---|
| AdaptationSet selection | `dashdemux` selects a video AdaptationSet automatically; codec and bitrate targeting is not configurable |
| Codec variants | If the manifest lists AV1 and H.264 AdaptationSets, `dashdemux` may select AV1, which NVDEC cannot decode on Turing/Ampere GPUs (see [Codec Support](#codec-support)) |
| Audio AdaptationSets | Emitted as dynamic pads and silently discarded |
| Subtitle / text AdaptationSets | Behavior is untested |

**Notes**:
- Standard DASH streams with a proper `.mpd` manifest URL pass directly to `nvurisrcbin` -
  no intermediate download step is required.
- DASH is not suitable for sources that only expose a CDN segment URL without a manifest.

---

## RTSP

**When to use**: Live camera streams or RTSP servers.

```python
p.add("nvurisrcbin", "src", {"uri": "rtsp://camera/stream", "gpu-id": 0})
p.add("nvstreammux", "mux", {
    "batch-size": 1,
    "width": 1280,
    "height": 720,
    "live-source": 1,
    "batched-push-timeout": 33000,
    "gpu-id": 0,
})
```

**Notes**:
- Set `live-source=1` on `nvstreammux` for RTSP inputs.
- Use `sync=0` on display sinks for live pipelines to avoid clock-related stalls.
- Put credentials in the URI only when appropriate for the deployment; otherwise use the
  application's credential handling path.
- Validate camera URLs with a simple player before debugging the DeepStream pipeline.
- For multi-RTSP inference examples, see [use_cases_pipelines.md](use_cases_pipelines.md).

---

## Multi-Stream (Batched Inference)

Set `batch-size` on `nvstreammux` to N and add N `nvurisrcbin` sources. Sources can mix
protocols - HTTP, HLS, RTSP, and local files can coexist in the same batch:

```python
urls = [
    "http://host/cam1.mp4",
    "http://host/stream.m3u8",
    "rtsp://camera/stream",
]
n = len(urls)

p.add("nvstreammux", "mux", {
    "batch-size": n,
    "width": 1280,
    "height": 720,
    "batched-push-timeout": 33000,   # microseconds; about 33 ms at 30 fps
    "gpu-id": 0,
})

for i, url in enumerate(urls):
    name = f"src{i}"
    p.add("nvurisrcbin", name, {"uri": url, "gpu-id": 0})
    p.link((name, "mux"), ("", "sink_%u"))

p.link("mux", "infer", ...)
```

**Notes**:
- `batch-size` on `nvinfer` must match `nvstreammux` `batch-size`.
- `batched-push-timeout` controls how long `nvstreammux` waits for lagging streams before
  pushing a partial batch.
- For live sources (RTSP, live HLS), set `live-source=1` on `nvstreammux` and `sync=0`
  on display sinks.
- Each source's detections carry a `source_id` field in `NvDsFrameMeta` matching its
  `sink_%u` slot index.

---

## Inference Configuration

`nvinfer` is model-agnostic: source handling does not change when the model changes. Pass the
model-specific YAML or INI file with `config-file-path`:

```python
p.add("nvinfer", "infer", {"config-file-path": pgie_cfg})
```

For nvinfer config keys, detector/classifier `network-type` rules, engine caching, dynamic
shape `infer-dims`, and custom parser requirements, use
[nvinfer_config.md](nvinfer_config.md). The bundled TrafficCamNet sample config is under the
unversioned DeepStream root:

```text
/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.yml
```

---

## Codec Support

Codec support depends on the target GPU or Jetson generation and on the decoder selected by
GStreamer. For x86 dGPU source selection, use this quick reference when choosing online stream
variants:

| Codec | Turing / Ampere dGPU | Ada Lovelace dGPU (RTX 40+) |
|---|---|---|
| H.264 | yes | yes |
| H.265 | yes | yes |
| VP9 | yes | yes |
| AV1 | no | yes |

When a platform or CDN offers multiple encodings, prefer H.264 or H.265 for the widest DeepStream
compatibility. Avoid adding decoder plugin-rank overrides in source examples; keep decoder
selection and troubleshooting guidance in [gstreamer_plugins.md](gstreamer_plugins.md) and
[docker_containers.md](docker_containers.md).

---

## Local Test Servers

### HTTP Server (Range-aware)

For local MP4 testing, `souphttpsrc` requires byte-range serving. The server must return
`206 Partial Content` and `Accept-Ranges: bytes` when the client sends a `Range` header.
If no Range-aware server is already available, create this local helper as `range_server.py`.
It implements the single byte-range requests needed for local MP4 testing; it is not a production
HTTP server.

```python
#!/usr/bin/env python3
import argparse
import os
from functools import partial
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer


class RangeRequestHandler(SimpleHTTPRequestHandler):
    def send_head(self):
        path = self.translate_path(self.path)
        if os.path.isdir(path):
            return super().send_head()

        try:
            f = open(path, "rb")
        except OSError:
            self.send_error(404, "File not found")
            return None

        size = os.fstat(f.fileno()).st_size
        start, end = 0, size - 1
        range_header = self.headers.get("Range")

        if range_header:
            try:
                unit, spec = range_header.split("=", 1)
                if unit.strip() != "bytes":
                    raise ValueError
                start_s, end_s = spec.split("-", 1)
                start = int(start_s) if start_s else 0
                end = int(end_s) if end_s else size - 1
                if start < 0 or end >= size or start > end:
                    raise ValueError
            except ValueError:
                f.close()
                self.send_error(416, "Invalid Range")
                return None
            self.send_response(206)
            self.send_header("Content-Range", f"bytes {start}-{end}/{size}")
        else:
            self.send_response(200)

        self.range = (start, end)
        self.send_header("Accept-Ranges", "bytes")
        self.send_header("Content-Type", self.guess_type(path))
        self.send_header("Content-Length", str(end - start + 1))
        self.end_headers()
        f.seek(start)
        return f

    def copyfile(self, source, outputfile):
        start, end = getattr(self, "range", (0, None))
        if end is None:
            return super().copyfile(source, outputfile)
        remaining = end - start + 1
        while remaining > 0:
            chunk = source.read(min(64 * 1024, remaining))
            if not chunk:
                break
            outputfile.write(chunk)
            remaining -= len(chunk)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--dir", default=".")
    parser.add_argument("--port", type=int, default=8080)
    args = parser.parse_args()
    handler = partial(RangeRequestHandler, directory=args.dir)
    ThreadingHTTPServer(("0.0.0.0", args.port), handler).serve_forever()
```

Run and verify before starting the pipeline:

```bash
python3 range_server.py --dir /path/to/videos --port 8080
curl -I -H "Range: bytes=0-0" http://localhost:8080/sample.mp4
```

The curl response must include `206 Partial Content` and `Accept-Ranges: bytes`.

### HLS Server

For local HLS testing, generate segments with ffmpeg and serve the generated directory. No
custom Python helper is required.

```bash
mkdir -p hls
ffmpeg -y -i sample.mp4 -c:v libx264 -profile:v high -pix_fmt yuv420p -an \
  -f hls -hls_time 2 -hls_list_size 0 hls/stream.m3u8
python3 -m http.server 8080 --directory hls
```

Pipeline URL: `http://localhost:8080/stream.m3u8`

For live-style testing, run ffmpeg and the HTTP server in separate terminals. Start the server
first; it can serve the directory while ffmpeg updates the playlist and segments.

Terminal 1:
```bash
mkdir -p hls
python3 -m http.server 8080 --directory hls
```

Terminal 2:
```bash
mkdir -p hls
ffmpeg -re -stream_loop -1 -i sample.mp4 -c:v libx264 -profile:v high \
  -pix_fmt yuv420p -an -f hls -hls_time 2 -hls_list_size 3 \
  -hls_flags delete_segments hls/stream.m3u8
```

### Generate a Synthetic Test Video

```bash
ffmpeg -f lavfi -i "testsrc2=duration=30:size=1280x720:rate=30" \
       -c:v libx264 -profile:v high -pix_fmt yuv420p -preset fast \
       -an sample.mp4
```

---

## Docker and Container Notes

For Docker image selection, pyservicemaker installation, GPU runtime flags, codec package
installation, environment variables, and common container failures, use
[docker_containers.md](docker_containers.md).

Source-specific container notes:

- HLS requires `gstreamer1.0-plugins-bad` for `hlsdemux`.
- DASH requires the GStreamer DASH demuxer; install `gstreamer1.0-plugins-bad` if it is missing.
- HTTP MP4 inputs may require the codec packages covered in `docker_containers.md` if the
  source includes audio or codecs stripped from the base image.
- Install pyservicemaker with the wildcard wheel path documented in `docker_containers.md`:
  `/opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl`.

---

## Related Pipeline Patterns

For a compact URI-source detection-to-MP4 pattern, see
[URI Source Inference Pattern](use_cases_pipelines.md#uri-source-inference-pattern).
Use this file for source URI selection, protocol constraints, and local stream setup details.

# nvtracker Configuration Reference

## Overview

The `nvtracker` GStreamer plugin provides multi-object tracking capabilities in DeepStream pipelines. It tracks objects detected by inference engines across video frames, assigning unique tracking IDs and maintaining object trajectories. The plugin works with a reference low-level tracker library (`NvMultiObjectTracker`) that implements multiple tracking algorithms in a unified, composable architecture.

## Prerequisites

### Required System Dependencies

The tracker library (`libnvds_nvmultiobjecttracker.so`) requires the **libmosquitto** library for MQTT-based communication features (used by multi-view tracking). This must be installed before using the tracker.

**Install on Ubuntu/Debian:**
```bash
sudo apt-get update
sudo apt-get install -y libmosquitto1
```

**Install on RHEL/CentOS:**
```bash
sudo yum install mosquitto
```

**Common Error if Missing:**
```
gstnvtracker: Failed to open low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
dlopen error: libmosquitto.so.1: cannot open shared object file: No such file or directory
gstnvtracker: Failed to initialize low level lib.
```

If you see this error, install libmosquitto1 as shown above.

---

## Unified Tracker Architecture

The NvMultiObjectTracker library employs a **modular, composable architecture**. Different tracker algorithms share common modules (data association, target management, state estimation) while differing in core functionalities (visual tracking, deep association metric, segmentation).

### Module Composition by Tracker Type

| Module | IOU | NvSORT | NvDCF | NvDeepSORT | MaskTracker |
|--------|-----|--------|-------|------------|-------------|
| **State Estimator** | - | Kalman (Regular) | Kalman (Simple) | Kalman (Regular) | Kalman (Simple) |
| **Data Association** | Yes | Yes (Cascaded) | Yes (Cascaded) | Yes (Cascaded) | Yes (Cascaded) |
| **Visual Tracker (DCF)** | - | - | Yes | - | - |
| **Re-ID Network** | - | - | Optional | Yes | - |
| **Segmenter (SAM2)** | - | - | - | - | Yes |
| **Object Model Projection** | - | - | Optional (SV3DT) | - | - |
| **Pose Estimator** | - | - | Optional (SV3DT) | - | - |
| **Target Management** | Yes | Yes | Yes | Yes | Yes |
| **Target Re-Association** | - | - | Optional | - | - |

### Tracker Algorithm Summary

| Algorithm | Library | Use Case | GPU Usage | Accuracy |
|-----------|---------|----------|-----------|----------|
| **IOU** | `libnvds_nvmultiobjecttracker.so` | Bare-minimum baseline, simple scenes | Very Low | Low |
| **NvSORT** | `libnvds_nvmultiobjecttracker.so` | Balanced performance with medium/high accuracy detectors | Very Low | Medium |
| **NvDCF** | `libnvds_nvmultiobjecttracker.so` | High accuracy, robust against occlusion, supports PGIE interval > 0 | Medium | High |
| **NvDeepSORT** | `libnvds_nvmultiobjecttracker.so` | Re-identification, objects with similar appearance | Low | High |
| **MaskTracker** | `libnvds_nvmultiobjecttracker.so` | Precise segmentation + tracking using SAM2 (Developer Preview) | High | Very High |

**Library Location**: `/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so`

---

## GObject Properties

### Required Properties

| Property | Type | Description |
|----------|------|-------------|
| `ll-lib-file` | string | Path to low-level tracker library |
| `ll-config-file` | string | Path to tracker configuration file. When sub-batches are used, specify multiple configs delimited by semicolon |

### Optional Properties

| Property | Type | Default | Description |
|----------|------|---------|-------------|
| `tracker-width` | int | 0 | Tracker input width in pixels (0=auto) |
| `tracker-height` | int | 0 | Tracker input height in pixels (0=auto) |
| `gpu-id` | int | 0 | GPU device ID |
| `display-tracking-id` | int | 1 | Show tracking ID in OSD (0/1) |
| `tracking-id-reset-mode` | int | 0 | ID reset behavior: 0=no reset, 1=reset on stream reset, 2=reset on EOS, 3=both |
| `tracking-surface-type` | int | 0 | Surface type for tracking |
| `compute-hw` | int | 0 | Compute engine for scaling: 0=Default, 1=GPU, 2=VIC (Jetson only) |
| `input-tensor-meta` | int | 0 | Use tensor metadata from upstream (nvdspreprocess) |
| `tensor-meta-gie-id` | int | -1 | GIE ID for tensor metadata (valid only if input-tensor-meta=1) |
| `user-meta-pool-size` | int | 16 | Tracker user metadata buffer pool size. Increase if you see "Unable to acquire a user meta buffer" warning |
| `sub-batches` | string | - | Sub-batch configuration (see Sub-batching section) |
| `sub-batch-err-recovery-trial-cnt` | int | 3 | Max reinit trials on sub-batch error. -1=infinite |

### Usage Example

```python
pipeline.add("nvtracker", "tracker", {
    "ll-lib-file": "/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so",
    "ll-config-file": "/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml",
    "tracker-width": 640,
    "tracker-height": 384,
    "gpu-id": 0,
    "display-tracking-id": 1
})
```

---

## Sub-batching

The sub-batching feature allows splitting the input frame batch into multiple sub-batches, each processed by a **separate instance** of the low-level tracker library on dedicated threads. This enables:

- **Parallel processing** to minimize GPU idling due to CPU compute blocks
- **Different configs per sub-batch** (different algorithms, backends, parameters)
- **Scaling beyond 128 streams** (VPI backend limit per instance)

### Configuration Options

**Option 1: Static source-to-sub-batch mapping**
```
# Semicolon-delimited arrays of source IDs
sub-batches=0,1;2,3
# Sources 0,1 -> sub-batch 0; Sources 2,3 -> sub-batch 1
```

**Option 2: Dynamic sub-batch sizing**
```
# Colon-delimited sub-batch sizes
sub-batches=2:2
# Two sub-batches, each accommodating up to 2 streams
```

### Multiple Config Files with Sub-batches

When sub-batches are configured, specify one config file per sub-batch using semicolons:
```
ll-config-file=config_tracker_NvDCF_accuracy.yml;config_tracker_NvSORT.yml;config_tracker_IOU.yml
sub-batches=0,1;2;3
```

### Use Case: Mixed Algorithms
```ini
[tracker]
enable=1
tracker-width=960
tracker-height=544
ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
ll-config-file=config_tracker_NvDCF_accuracy.yml;config_tracker_NvSORT.yml
sub-batches=0,1;2,3
```

### Use Case: PVA Backend on Jetson
```ini
[tracker]
ll-config-file=config_tracker_NvDCF_accuracy.yml;config_tracker_NvDCF_accuracy_PVA.yml
sub-batches=0,1;2,3
```

> **Note**: The optimal sub-batches configuration depends on pipeline elements, hardware config, etc. Start with a single batch and keep splitting until an optimal performance point is reached.

---

## Tracker Configuration File (YAML)

The low-level tracker configuration is a YAML file with the following sections.

### Configuration File Structure

```yaml
%YAML:1.0

BaseConfig:
  minDetectorConfidence: 0.0

TargetManagement:
  maxTargetsPerStream: 150
  probationAge: 4
  maxShadowTrackingAge: 38
  earlyTerminationAge: 1

TrajectoryManagement:
  useUniqueID: 0

DataAssociator:
  dataAssociatorType: 0
  associationMatcherType: 0  # GREEDY=0, CASCADED=1

StateEstimator:
  stateEstimatorType: 0  # DUMMY=0, SIMPLE=1, REGULAR=2, SIMPLE_LOC=3

# Algorithm-specific sections (only one active):
VisualTracker:    # For NvDCF
ReID:             # For NvDeepSORT or NvDCF with Re-Assoc
Segmenter:        # For MaskTracker

# SV3DT-specific sections (NvDCF with stateEstimatorType=3):
ObjectModelProjection:  # Camera model + 3D projection output
PoseEstimator:          # Body pose estimation for 3D height
```

---

## Configuration Sections Reference

### BaseConfig

| Parameter | Type | Default | Description | Dynamic |
|-----------|------|---------|-------------|---------|
| `minDetectorConfidence` | float | 0.0 | Detections below this confidence are discarded | Yes |

### TargetManagement

Controls the lifecycle of tracked targets through three states: **Tentative** -> **Active** -> **Inactive** (shadow tracking).

| Parameter | Type | Description | Dynamic |
|-----------|------|-------------|---------|
| `maxTargetsPerStream` | int | Max targets per stream (includes shadow-tracked). Pre-allocates GPU memory | No |
| `preserveStreamUpdateOrder` | bool | Deterministic ID order across runs (single-threaded update) | No |
| `enableBboxUnClipping` | bool | Restore bboxes clipped by image border | Yes |
| `minIouDiff4NewTarget` | float | New detection is discarded if IOU with any existing target exceeds this | Yes |
| `minTrackerConfidence` | float | Below this confidence, target enters shadow mode [0.0, 1.0] | Yes |
| `probationAge` | int | Frames in Tentative mode before target becomes Active (Late Activation) | Yes |
| `maxShadowTrackingAge` | int | Max frames of shadow tracking before termination | Yes |
| `earlyTerminationAge` | int | If shadowTrackingAge reaches this during Tentative period, target is terminated early | Yes |
| `searchRegionPaddingScale` | float | Search region size as multiple of bbox diagonal (NvDCF) | Yes |
| `outputTerminatedTracks` | bool | Export terminated track history to metadata | No |
| `outputShadowTracks` | bool | Export shadow track data to metadata | No |
| `terminatedTrackFilename` | string | File prefix for saving terminated tracks | No |

#### Target State Transitions

```
  New Detection -> [Tentative] ---- (survives probationAge) ---> [Active]
                      |                                            |
                      | (earlyTerminationAge)                      | (no detection match for a while,
                      v                                            |  or confidence < minTrackerConfidence)
                  [Terminated]                                     v
                                                              [Inactive / Shadow]
                                                                   |
                                                                   | (maxShadowTrackingAge exceeded)
                                                                   v
                                                              [Terminated]
```

### TrajectoryManagement

Controls unique ID generation and target re-association.

| Parameter | Type | Description |
|-----------|------|-------------|
| `useUniqueID` | bool | Use 64-bit unique ID (random upper 32-bit per stream + sequential lower 32-bit) |
| `enableReAssoc` | bool | Enable motion-based target re-association |
| `minMatchingScore4Overall` | float | Min total score for re-association |
| `minTrackletMatchingScore` | float | Min tracklet IOU similarity for re-association |
| `minMatchingScore4ReidSimilarity` | float | Min ReID score for re-association |
| `matchingScoreWeight4TrackletSimilarity` | float | Weight for tracklet similarity in re-association |
| `matchingScoreWeight4ReidSimilarity` | float | Weight for ReID similarity in re-association |
| `minTrajectoryLength4Projection` | int | Min tracklet length to create projected trajectory |
| `prepLength4TrajectoryProjection` | int | Trajectory length used for projection state estimation |
| `trajectoryProjectionLength` | int | Length of projected trajectory |
| `maxAngle4TrackletMatching` | float | Max angle difference for tracklet matching [degrees] |
| `minSpeedSimilarity4TrackletMatching` | float | Min speed similarity for tracklet matching |
| `minBboxSizeSimilarity4TrackletMatching` | float | Min bbox size similarity for tracklet matching |
| `maxTrackletMatchingTimeSearchRange` | int | Time search range for tracklet matching |
| `trajectoryProjectionProcessNoiseScale` | float | Process noise scale for trajectory projection |
| `trajectoryProjectionMeasurementNoiseScale` | float | Measurement noise scale for trajectory projection |
| `trackletSpacialSearchRegionScale` | float | Spatial search region for peer tracklet |
| `reidExtractionInterval` | int | Frame interval for ReID feature extraction per target. -1=first frame only |

### DataAssociator

| Parameter | Type | Default | Description | Dynamic |
|-----------|------|---------|-------------|---------|
| `dataAssociatorType` | int | 0 | Data associator type {DEFAULT=0} | No |
| `associationMatcherType` | int | 0 | Matching algorithm {GREEDY=0, CASCADED=1} | No |
| `checkClassMatch` | bool | true | Only associate same-class objects | No |
| `usePrediction4Assoc` | bool | false | Use predicted state for association instead of last known state | Yes |
| **Similarity Thresholds** |||||
| `minMatchingScore4Overall` | float | 0.0 | Min total matching score | Yes |
| `minMatchingScore4SizeSimilarity` | float | 0.0 | Min bbox size similarity | Yes |
| `minMatchingScore4Iou` | float | 0.0 | Min IOU score | Yes |
| `minMatchingScore4VisualSimilarity` | float | 0.0 | Min visual similarity (NvDCF only) | Yes |
| `minMatchingScore4ReidSimilarity` | float | 0.0 | Min ReID similarity (NvDeepSORT only) | Yes |
| **Similarity Weights** |||||
| `matchingScoreWeight4Iou` | float | 1.0 | Weight for IOU | Yes |
| `matchingScoreWeight4SizeSimilarity` | float | 0.0 | Weight for size similarity | Yes |
| `matchingScoreWeight4VisualSimilarity` | float | 0.0 | Weight for visual similarity (NvDCF) | Yes |
| `matchingScoreWeight4ReidSimilarity` | float | 0.0 | Weight for ReID similarity (NvDeepSORT) | Yes |
| **Tentative Detection** |||||
| `tentativeDetectorConfidence` | float | 0.5 | Below this but above minDetectorConfidence = tentative detection | Yes |
| `minMatchingScore4TentativeIou` | float | 0.0 | Min IOU for tentative detection matching | Yes |
| **Mahalanobis Distance (NvDeepSORT)** |||||
| `thresholdMahalanobis` | float | -1.0 | Max Mahalanobis distance. Negative = disabled | Yes |

#### Cascaded Data Association (associationMatcherType: 1)

The cascaded matcher performs multi-stage matching with different priorities:

1. **Stage 1**: Confirmed detections <-> validated targets (joint similarity metrics)
2. **Stage 2**: Tentative detections <-> remaining active targets (IOU only)
3. **Stage 3**: Remaining confirmed detections <-> tentative targets (IOU only)

Total matching score formula:

`totalScore = w_iou * IOU + w_size * sizeSimilarity + w_reid * reidSimilarity + w_visual * visualSimilarity`

### StateEstimator

| Parameter | Type | Description |
|-----------|------|-------------|
| `stateEstimatorType` | int | Estimator type: **DUMMY=0**, **SIMPLE_BBOX_KF=1**, **REGULAR_BBOX_KF=2**, **SIMPLE_LOCATION_KF=3** |

**SIMPLE_BBOX_KF (type=1)**: 6-state Kalman filter `{x, y, w, h, dx, dy}` with absolute noise values:

| Parameter | Description |
|-----------|-------------|
| `processNoiseVar4Loc` | Process noise for bbox center |
| `processNoiseVar4Size` | Process noise for bbox size |
| `processNoiseVar4Vel` | Process noise for velocity |
| `measurementNoiseVar4Detector` | Measurement noise from detector |
| `measurementNoiseVar4Tracker` | Measurement noise from visual tracker (NvDCF) |

**REGULAR_BBOX_KF (type=2)**: 8-state Kalman filter `{x, y, w, h, dx, dy, dw, dh}` with height-proportional noise:

| Parameter | Description |
|-----------|-------------|
| `noiseWeightVar4Loc` | Noise weight proportional to bbox height (location) |
| `noiseWeightVar4Vel` | Noise weight proportional to bbox height (velocity) |
| `useAspectRatio` | Use aspect ratio `a` instead of width `w` in state vector (used by NvDeepSORT) |

**SIMPLE_LOCATION_KF (type=3)**: 4-state Kalman filter `{x, y, dx, dy}` for 3D world coordinate tracking (SV3DT). Tracks the projected foot location in image space rather than bounding box. The bounding box is reconstructed by projecting a 3D cylinder model (from `ObjectModelProjection`) back onto the image. **Does NOT use `processNoiseVar4Size`** since bbox size is derived from the 3D model projection rather than estimated directly.

| Parameter | Description |
|-----------|-------------|
| `processNoiseVar4Loc` | Process noise for foot location in image space |
| `processNoiseVar4Vel` | Process noise for velocity |
| `measurementNoiseVar4Detector` | Measurement noise from detector |
| `measurementNoiseVar4Tracker` | Measurement noise from visual tracker (NvDCF) |

> **Note**: When using `stateEstimatorType: 3`, the `ObjectModelProjection` section is required. The `PoseEstimator` section is optional but recommended for more accurate height estimation.

### VisualTracker (NvDCF)

| Parameter | Type | Description | Dynamic |
|-----------|------|-------------|---------|
| `visualTrackerType` | int | **DUMMY=0**, **NvDCF_legacy=1**, **NvDCF_VPI=2** | No |
| `useColorNames` | bool | Use ColorNames feature (10 channels) | No |
| `useHog` | bool | Use HOG feature (18 channels) | No |
| `useHighPrecisionFeature` | bool | 16-bit precision (vs 8-bit) | No |
| `featureImgSizeLevel` | int | Feature image size {1=12x12, 2=18x18, 3=24x24, 4=30x30, 5=36x36} per channel | No |
| `featureFocusOffsetFactor_y` | float | Hanning window center Y offset [-0.5, 0.5]. Negative moves up (good for surveillance) | Yes |
| `filterLr` | float | DCF filter learning rate [0.0, 1.0] | Yes |
| `filterChannelWeightsLr` | float | Channel weights learning rate [0.0, 1.0] | Yes |
| `gaussianSigma` | float | Gaussian sigma for desired response [pixels] | Yes |
| `vpiBackend4DcfTracker` | int | VPI backend: **CUDA=1**, **PVA=2** (Jetson only). Valid when visualTrackerType=2 | No |

#### PVA Backend Limitations (VPI)
- Max 512 objects per tracker instance
- Max 33 streams per instance (use sub-batching for more)
- Only supports: `useColorNames: 1`, `useHog: 1`, `featureImgSizeLevel: 3`

### ReID (Re-Identification)

| Parameter | Type | Description |
|-----------|------|-------------|
| `reidType` | int | **DUMMY=0**, **NvDEEPSORT=1**, **REASSOC=2** (re-association only), **BOTH=3** |
| `batchSize` | int | ReID network batch size |
| `workspaceSize` | int | TensorRT workspace (MB) |
| `reidFeatureSize` | int | Output feature dimension |
| `reidHistorySize` | int | Max features kept per target (gallery size) |
| `inferDims` | [int] | Network input dims [C, H, W] |
| `networkMode` | int | Precision: FP32=0, FP16=1, INT8=2 |
| `inputOrder` | int | NCHW=0, NHWC=1 |
| `colorFormat` | int | RGB=0, BGR=1 |
| `offsets` | [float] | Per-channel subtraction values |
| `netScaleFactor` | float | Scale factor after offset: `y = netScaleFactor * (x - offsets)` |
| `keepAspc` | bool | Preserve aspect ratio when resizing |
| `useVPICropScaler` | bool | Use VPI for crop and scale |
| `addFeatureNormalization` | bool | L2 normalize output features |
| `minVisibility4GalleryUpdate` | float | Min visibility to add ReID embedding to gallery (SV3DT only, e.g. 0.6) |
| `outputReidTensor` | bool | Export ReID features to user meta |
| `tltEncodedModel` | string | TAO model path |
| `tltModelKey` | string | TAO model key |
| `onnxFile` | string | ONNX model path |
| `modelEngineFile` | string | Pre-built TensorRT engine path |
| `calibrationTableFile` | string | INT8 calibration table path |

### Segmenter (MaskTracker)

| Parameter | Type | Description |
|-----------|------|-------------|
| `segmenterType` | int | **DUMMY=0**, **SAM2=1** |
| `segmenterConfigPath` | string | Path to segmenter config (e.g., `config_tracker_module_Segmenter.yml`) |

The segmenter config file defines four TensorRT-accelerated sub-networks (ImageEncoder, MaskDecoder, MemoryAttention, MemoryEncoder) and memory management parameters. See MaskTracker section for details.

### ObjectModelProjection (SV3DT)

Used for Single-View 3D Tracking (SV3DT). Projects a 3D cylinder model onto the image plane using camera calibration to estimate per-object visibility, foot location, and convex hull. This enables the tracker to recover complete bounding boxes and foot positions even under partial occlusion.

| Parameter | Type | Description |
|-----------|------|-------------|
| `cameraModelFilepath` | list[string] | Camera calibration file path per stream (one entry per stream, ordered by stream index) |
| `outputVisibility` | bool | Output per-object visibility (0.0\~1.0) estimated from occlusion via 3D model |
| `outputFootLocation` | bool | Output foot location in image and world coordinates, estimated from 3D model projection |
| `outputConvexHull` | bool | Output convex hull vertices for each object estimated from 3D cylinder model |
| `minPoseConfidence` | float | Minimum pose keypoint confidence for adaptive height estimation (0.0\~1.0) |

**Camera Model File (`camInfo.yml`):**

The camera model file provides the 3x4 camera projection matrix and a cylinder model representing the tracked object (human). The projection matrix maps 3D world coordinates to 2D image coordinates.

```yaml
%YAML:1.0

# 3x4 camera projection matrix (row-major)
# Maps 3D world coordinates (X, Y, Z) to 2D image coordinates (u, v)
projectionMatrix_3x4:
  - 2582.5691623002185
  - -485.10283397043617
  - 650.27745033162591
  - -89466.605755471101
  - -423.46809686390498
  - 1044.6870098337931
  - 2461.1283636622838
  - -214284.36100320917
  - -0.25563255317172684
  - -0.90495941862094287
  - 0.34014768617197644
  - -1181.960782357068

# Cylinder model dimensions for human (cm)
modelInfo:
  height: 205    # Height of the cylinder model
  radius: 33     # Radius of the cylinder model
```

> **Note**: The camera must be **static** (fixed position and orientation). The projection matrix can be obtained through standard camera calibration procedures. For multi-stream setups, provide one `camInfo.yml` per camera in the `cameraModelFilepath` list.

### PoseEstimator (SV3DT)

Estimates 2D body pose to determine precise target height for the 3D cylinder model. Used in conjunction with `ObjectModelProjection` for SV3DT. When enabled, the BodyPose3DNet model infers key body joints to compute the actual individual height rather than using a fixed default height.

| Parameter | Type | Description |
|-----------|------|-------------|
| `poseEstimatorType` | int | **0**=Disabled (use fixed-height model, match head to bbox top edge), **1**=Enabled (use BodyPose3DNet for precise height estimation) |
| `useVPICropScaler` | bool | Use VPI backend for cropping and scaling |
| `batchSize` | int | Batch size for pose estimation inference |
| `workspaceSize` | int | TensorRT workspace size (MB) |
| `inferDims` | [int] | Network input dims [C, H, W], e.g. `[3, 256, 192]` |
| `networkMode` | int | Precision: FP32=0, FP16=1, INT8=2 |
| `inputOrder` | int | NCHW=0, NHWC=1 |
| `colorFormat` | int | RGB=0, BGR=1 |
| `offsets` | [float] | Per-channel subtraction values |
| `netScaleFactor` | float | Scale factor after offset subtraction |
| `onnxFile` | string | Path to BodyPose3DNet ONNX model |
| `modelEngineFile` | string | Pre-built TensorRT engine path |
| `poseInferenceInterval` | int | Frame interval for pose inference. **-1**=first frame only (determine height once per target, most efficient) |

> **Note**: When `poseEstimatorType: 0`, no pose model is needed. The tracker uses a fixed-height human model matching the head to the bbox top edge. This is less accurate but has zero additional compute cost. When `poseEstimatorType: 1`, the BodyPose3DNet model (`bodypose3dnet_accuracy.onnx`) is required.

---

## Tracker Algorithm Configurations

### IOU Tracker

**Best for**: Bare-minimum baseline, sparse objects, detector runs every frame.

```yaml
%YAML:1.0

BaseConfig:
  minDetectorConfidence: 0

TargetManagement:
  preserveStreamUpdateOrder: 0
  maxTargetsPerStream: 150
  minIouDiff4NewTarget: 0.5
  probationAge: 4
  maxShadowTrackingAge: 38
  earlyTerminationAge: 1

TrajectoryManagement:
  useUniqueID: 0

DataAssociator:
  dataAssociatorType: 0
  associationMatcherType: 0    # GREEDY
  checkClassMatch: 1
  minMatchingScore4Overall: 0.0
  minMatchingScore4SizeSimilarity: 0.0
  minMatchingScore4Iou: 0.0
  matchingScoreWeight4SizeSimilarity: 0.4
  matchingScoreWeight4Iou: 0.6
```

### NvSORT Tracker

**Best for**: Balanced performance with medium/high accuracy detectors. Uses Kalman filter + cascaded data association.

```yaml
%YAML:1.0

BaseConfig:
  minDetectorConfidence: 0.1345

TargetManagement:
  enableBboxUnClipping: 0
  maxTargetsPerStream: 300
  minIouDiff4NewTarget: 0.5780
  minTrackerConfidence: 0.8216
  probationAge: 5
  maxShadowTrackingAge: 26
  earlyTerminationAge: 1

TrajectoryManagement:
  useUniqueID: 0

DataAssociator:
  dataAssociatorType: 0
  associationMatcherType: 1    # CASCADED
  checkClassMatch: 1
  minMatchingScore4Overall: 0.2543
  minMatchingScore4SizeSimilarity: 0.4019
  minMatchingScore4Iou: 0.2159
  matchingScoreWeight4SizeSimilarity: 0.1365
  matchingScoreWeight4Iou: 0.3836
  tentativeDetectorConfidence: 0.2331
  minMatchingScore4TentativeIou: 0.2867
  usePrediction4Assoc: 1

StateEstimator:
  stateEstimatorType: 2    # REGULAR_BBOX_KF
  noiseWeightVar4Loc: 0.0301
  noiseWeightVar4Vel: 0.0017
  useAspectRatio: 1
```

### NvDCF Tracker (Performance)

**Best for**: High accuracy, robust against occlusions, supports PGIE interval > 0.

```yaml
%YAML:1.0

BaseConfig:
  minDetectorConfidence: 0.0430

TargetManagement:
  enableBboxUnClipping: 1
  preserveStreamUpdateOrder: 0
  maxTargetsPerStream: 150
  minIouDiff4NewTarget: 0.7418
  minTrackerConfidence: 0.4009
  probationAge: 2
  maxShadowTrackingAge: 51
  earlyTerminationAge: 1

TrajectoryManagement:
  useUniqueID: 0

DataAssociator:
  dataAssociatorType: 0
  associationMatcherType: 1    # CASCADED
  checkClassMatch: 1
  minMatchingScore4Overall: 0.4290
  minMatchingScore4SizeSimilarity: 0.3627
  minMatchingScore4Iou: 0.2575
  minMatchingScore4VisualSimilarity: 0.5356
  matchingScoreWeight4VisualSimilarity: 0.3370
  matchingScoreWeight4SizeSimilarity: 0.4354
  matchingScoreWeight4Iou: 0.3656
  tentativeDetectorConfidence: 0.2008
  minMatchingScore4TentativeIou: 0.5296

StateEstimator:
  stateEstimatorType: 1    # SIMPLE_BBOX_KF
  processNoiseVar4Loc: 1.5110
  processNoiseVar4Size: 1.3159
  processNoiseVar4Vel: 0.0300
  measurementNoiseVar4Detector: 3.0283
  measurementNoiseVar4Tracker: 8.1505

VisualTracker:
  visualTrackerType: 2    # NvDCF_VPI
  useColorNames: 1
  useHog: 0
  featureImgSizeLevel: 2
  featureFocusOffsetFactor_y: -0.2000
  filterLr: 0.0750
  filterChannelWeightsLr: 0.1000
  gaussianSigma: 0.7500
```

### NvDCF Tracker (Accuracy with Re-Association)

Enables Re-Association for long-term tracking with ReID.

```yaml
%YAML:1.0

BaseConfig:
  minDetectorConfidence: 0.1894

TargetManagement:
  enableBboxUnClipping: 1
  maxTargetsPerStream: 150
  minIouDiff4NewTarget: 0.3686
  minTrackerConfidence: 0.1513
  probationAge: 2
  maxShadowTrackingAge: 42
  earlyTerminationAge: 1

TrajectoryManagement:
  useUniqueID: 0
  enableReAssoc: 1
  minMatchingScore4Overall: 0.6622
  minTrackletMatchingScore: 0.2940
  minMatchingScore4ReidSimilarity: 0.0771
  matchingScoreWeight4TrackletSimilarity: 0.7981
  matchingScoreWeight4ReidSimilarity: 0.3848
  minTrajectoryLength4Projection: 34
  prepLength4TrajectoryProjection: 58
  trajectoryProjectionLength: 33
  maxAngle4TrackletMatching: 67
  minSpeedSimilarity4TrackletMatching: 0.0574
  minBboxSizeSimilarity4TrackletMatching: 0.1013
  maxTrackletMatchingTimeSearchRange: 27
  trajectoryProjectionProcessNoiseScale: 0.0100
  trajectoryProjectionMeasurementNoiseScale: 100
  trackletSpacialSearchRegionScale: 0.0100
  reidExtractionInterval: 8

DataAssociator:
  dataAssociatorType: 0
  associationMatcherType: 1    # CASCADED
  checkClassMatch: 1
  minMatchingScore4Overall: 0.0222
  minMatchingScore4SizeSimilarity: 0.3552
  minMatchingScore4Iou: 0.0548
  minMatchingScore4VisualSimilarity: 0.5043
  matchingScoreWeight4VisualSimilarity: 0.3951
  matchingScoreWeight4SizeSimilarity: 0.6003
  matchingScoreWeight4Iou: 0.4033
  tentativeDetectorConfidence: 0.1024
  minMatchingScore4TentativeIou: 0.2852

StateEstimator:
  stateEstimatorType: 1    # SIMPLE_BBOX_KF
  processNoiseVar4Loc: 6810.8668
  processNoiseVar4Size: 1541.8647
  processNoiseVar4Vel: 1348.4874
  measurementNoiseVar4Detector: 100.0000
  measurementNoiseVar4Tracker: 293.3238

VisualTracker:
  visualTrackerType: 2    # NvDCF_VPI
  useColorNames: 1
  useHog: 1
  featureImgSizeLevel: 3
  featureFocusOffsetFactor_y: -0.1054
  filterLr: 0.0767
  filterChannelWeightsLr: 0.0339
  gaussianSigma: 0.5687

ReID:
  reidType: 2    # REASSOC only
  batchSize: 100
  workspaceSize: 1000
  reidFeatureSize: 256
  reidHistorySize: 100
  inferDims: [3, 256, 128]
  networkMode: 1    # FP16
  inputOrder: 0
  colorFormat: 0
  offsets: [123.6750, 116.2800, 103.5300]
  netScaleFactor: 0.01735207
  keepAspc: 1
  useVPICropScaler: 1
  addFeatureNormalization: 1
  tltEncodedModel: "/opt/nvidia/deepstream/deepstream/samples/models/Tracker/resnet50_market1501.etlt"
  tltModelKey: "nvidia_tao"
```

### NvDeepSORT Tracker

**Best for**: Re-identification across views, objects with similar appearance. Requires a Re-ID model.

```yaml
%YAML:1.0

BaseConfig:
  minDetectorConfidence: 0.0762

TargetManagement:
  preserveStreamUpdateOrder: 0
  maxTargetsPerStream: 150
  minIouDiff4NewTarget: 0.9847
  minTrackerConfidence: 0.4314
  probationAge: 2
  maxShadowTrackingAge: 68
  earlyTerminationAge: 1

TrajectoryManagement:
  useUniqueID: 0

DataAssociator:
  dataAssociatorType: 0
  associationMatcherType: 1    # CASCADED
  checkClassMatch: 1
  thresholdMahalanobis: 12.1875
  minMatchingScore4Overall: 0.1794
  minMatchingScore4SizeSimilarity: 0.3291
  minMatchingScore4Iou: 0.2364
  minMatchingScore4ReidSimilarity: 0.7505
  matchingScoreWeight4SizeSimilarity: 0.7178
  matchingScoreWeight4Iou: 0.4551
  matchingScoreWeight4ReidSimilarity: 0.3197
  tentativeDetectorConfidence: 0.2479
  minMatchingScore4TentativeIou: 0.2376

StateEstimator:
  stateEstimatorType: 2    # REGULAR_BBOX_KF
  noiseWeightVar4Loc: 0.0503
  noiseWeightVar4Vel: 0.0037
  useAspectRatio: 1

ReID:
  reidType: 1    # NvDEEPSORT
  batchSize: 100
  workspaceSize: 1000
  reidFeatureSize: 256
  reidHistorySize: 100
  inferDims: [3, 256, 128]
  networkMode: 1    # FP16
  inputOrder: 0
  colorFormat: 0
  offsets: [123.6750, 116.2800, 103.5300]
  netScaleFactor: 0.01735207
  keepAspc: 1
  useVPICropScaler: 1
  addFeatureNormalization: 1
  tltEncodedModel: "/opt/nvidia/deepstream/deepstream/samples/models/Tracker/resnet50_market1501.etlt"
  tltModelKey: "nvidia_tao"
  modelEngineFile: "/opt/nvidia/deepstream/deepstream/samples/models/Tracker/resnet50_market1501.etlt_b100_gpu0_fp16.engine"
```

**Setup ReID model:**
```bash
mkdir -p /opt/nvidia/deepstream/deepstream/samples/models/Tracker/
wget 'https://api.ngc.nvidia.com/v2/models/nvidia/tao/reidentificationnet/versions/deployable_v1.0/files/resnet50_market1501.etlt' \
  -P /opt/nvidia/deepstream/deepstream/samples/models/Tracker/
```

### MaskTracker (Developer Preview)

**Best for**: Precise object segmentation + tracking using SAM2. Works with diverse object classes.

```yaml
%YAML:1.0

BaseConfig:
  minDetectorConfidence: 0.3529

TargetManagement:
  enableBboxUnClipping: 1
  preserveStreamUpdateOrder: 0
  maxTargetsPerStream: 150
  minIouDiff4NewTarget: 0.7608
  minTrackerConfidence: 0.6223
  probationAge: 4
  maxShadowTrackingAge: 84
  earlyTerminationAge: 1

DataAssociator:
  dataAssociatorType: 0
  associationMatcherType: 1    # CASCADED
  checkClassMatch: 1
  minMatchingScore4Overall: 0.0293
  minMatchingScore4SizeSimilarity: 0.1047
  minMatchingScore4Iou: 0.0437
  matchingScoreWeight4SizeSimilarity: 0.2410
  matchingScoreWeight4Iou: 0.8590
  tentativeDetectorConfidence: 0.1866
  minMatchingScore4TentativeIou: 0.3660

TrajectoryManagement:
  useUniqueID: 0

StateEstimator:
  stateEstimatorType: 1    # SIMPLE_BBOX_KF
  processNoiseVar4Loc: 2856.7104
  processNoiseVar4Size: 8157.1946
  processNoiseVar4Vel: 2602.8703
  measurementNoiseVar4Detector: 0.1000
  measurementNoiseVar4Tracker: 8.6695

Segmenter:
  segmenterType: 1    # SAM2
  segmenterConfigPath: "/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_module_Segmenter.yml"
```

**Setup SAM2 model:**
```bash
git clone https://github.com/NVIDIA-AI-IOT/deepstream_tools.git
cd deepstream_tools/sam2-onnx-tensorrt
bash run.sh
```

The segmentation mask is stored in `mask_params` field of `NvDsObjectMeta`. Set `display-mask=1` in OSD config to visualize.

### NvDCF 3D Tracker (SV3DT)

**Best for**: Tracking people in 3D physical world coordinates from a static camera. Estimates foot location, body visibility, and convex hull using camera calibration and a 3D cylinder human model. Recovers complete bounding boxes even under partial occlusion.

**Overview**: Single-View 3D Tracking (SV3DT) extends NvDCF with 3D state estimation. Instead of tracking bounding box coordinates directly, it tracks object positions in 3D world coordinates by projecting a cylinder model using the camera projection matrix. Key capabilities:

- **3D world coordinate tracking**: Estimates object foot position in real-world coordinates
- **Occlusion-aware bounding box recovery**: Reconstructs complete bounding boxes from partially occluded objects
- **Visibility estimation**: Computes per-object visibility ratio (0.0\~1.0) based on mutual occlusion
- **Convex hull output**: Provides projected 3D model convex hull vertices for each tracked object
- **Pose-based height estimation**: Optionally uses BodyPose3DNet to determine individual person height

**Prerequisites**:
- Static camera with known camera projection matrix (`camInfo.yml`)
- PeopleNet or similar person detector as PGIE
- ReID model (e.g., `resnet50_market1501.etlt`) for re-association
- BodyPose3DNet ONNX model (optional, for `poseEstimatorType: 1`)

**Setup models:**
```bash
# peoplenet model
mkdir -p PeopleNet
cd PeopleNet; wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplenet/versions/deployable_quantized_onnx_v2.6.3/zip -O peoplenet_deployable_quantized_onnx_v2.6.3.zip; unzip peoplenet_deployable_quantized_onnx_v2.6.3.zip
```

The model files are now stored in PeopleNet directory as

```
PeopleNet
  ├── labels.txt
  ├── resnet34_peoplenet.onnx
  └── ...
```

```bash
mkdir -p /opt/nvidia/deepstream/deepstream/samples/models/Tracker/

# ReID model
wget 'https://api.ngc.nvidia.com/v2/models/nvidia/tao/reidentificationnet/versions/deployable_v1.0/files/resnet50_market1501.etlt' \
  -P /opt/nvidia/deepstream/deepstream/samples/models/Tracker/

# BodyPose3DNet model (for poseEstimatorType: 1)
wget 'https://api.ngc.nvidia.com/v2/models/nvidia/tao/bodypose3dnet/versions/deployable_accuracy_onnx_1.0/files/bodypose3dnet_accuracy.onnx' \
  -P /opt/nvidia/deepstream/deepstream/samples/models/Tracker/
```

**Full Configuration (`config_tracker_NvDCF_accuracy_3D.yml`):**

```yaml
%YAML:1.0

BaseConfig:
  minDetectorConfidence: 0.1894

TargetManagement:
  enableBboxUnClipping: 1
  preserveStreamUpdateOrder: 0
  maxTargetsPerStream: 150
  minIouDiff4NewTarget: 0.3686
  minTrackerConfidence: 0.1513
  probationAge: 2
  maxShadowTrackingAge: 42
  earlyTerminationAge: 1
  # Export terminated tracklets
  outputTerminatedTracks: 1
  terminatedTrackFilename: track_dump_

TrajectoryManagement:
  useUniqueID: 0
  enableReAssoc: 1
  minMatchingScore4Overall: 0.6622
  minTrackletMatchingScore: 0.2940
  minMatchingScore4ReidSimilarity: 0.0771
  matchingScoreWeight4TrackletSimilarity: 0.7981
  matchingScoreWeight4ReidSimilarity: 0.3848
  minTrajectoryLength4Projection: 34
  prepLength4TrajectoryProjection: 58
  trajectoryProjectionLength: 33
  maxAngle4TrackletMatching: 67
  minSpeedSimilarity4TrackletMatching: 0.0574
  minBboxSizeSimilarity4TrackletMatching: 0.1013
  maxTrackletMatchingTimeSearchRange: 27
  trajectoryProjectionProcessNoiseScale: 0.0100
  trajectoryProjectionMeasurementNoiseScale: 100
  trackletSpacialSearchRegionScale: 0.0100
  reidExtractionInterval: 8

DataAssociator:
  dataAssociatorType: 0
  associationMatcherType: 1    # CASCADED
  checkClassMatch: 1
  minMatchingScore4Overall: 0.0222
  minMatchingScore4SizeSimilarity: 0.3552
  minMatchingScore4Iou: 0.0548
  minMatchingScore4VisualSimilarity: 0.5043
  matchingScoreWeight4VisualSimilarity: 0.3951
  matchingScoreWeight4SizeSimilarity: 0.6003
  matchingScoreWeight4Iou: 0.4033
  tentativeDetectorConfidence: 0.1024
  minMatchingScore4TentativeIou: 0.2852

StateEstimator:
  stateEstimatorType: 3    # SIMPLE_LOCATION_KF (3D)
  # Note: NO processNoiseVar4Size (bbox size derived from 3D model projection)
  processNoiseVar4Loc: 6810.8668
  processNoiseVar4Vel: 1348.4874
  measurementNoiseVar4Detector: 100.0000
  measurementNoiseVar4Tracker: 293.3238

ObjectModelProjection:
  cameraModelFilepath:    # one camInfo.yml per stream
    - configs/camInfo.yml
  outputVisibility: 1
  outputFootLocation: 1
  outputConvexHull: 1
  minPoseConfidence: 0.5

VisualTracker:
  visualTrackerType: 2    # NvDCF_VPI
  vpiBackend4DcfTracker: 1    # CUDA
  useColorNames: 1
  useHog: 1
  featureImgSizeLevel: 3
  featureFocusOffsetFactor_y: -0.1054
  filterLr: 0.0767
  filterChannelWeightsLr: 0.0339
  gaussianSigma: 0.5687

ReID:
  reidType: 2    # REASSOC only
  batchSize: 100
  workspaceSize: 1000
  reidFeatureSize: 256
  reidHistorySize: 100
  inferDims: [3, 256, 128]
  networkMode: 1    # FP16
  inputOrder: 0
  colorFormat: 0
  offsets: [123.6750, 116.2800, 103.5300]
  netScaleFactor: 0.01735207
  keepAspc: 1
  useVPICropScaler: 1
  addFeatureNormalization: 1
  minVisibility4GalleryUpdate: 0.6    # Only update ReID gallery when visibility >= 0.6
  tltEncodedModel: "/opt/nvidia/deepstream/deepstream/samples/models/Tracker/resnet50_market1501.etlt"
  tltModelKey: "nvidia_tao"
  modelEngineFile: "/opt/nvidia/deepstream/deepstream/samples/models/Tracker/resnet50_market1501.etlt_b100_gpu0_fp16.engine"

PoseEstimator:
  poseEstimatorType: 1    # 1=BodyPose3DNet, 0=disabled (fixed height)
  useVPICropScaler: 1
  batchSize: 1
  workspaceSize: 1000
  inferDims: [3, 256, 192]
  networkMode: 1    # FP16
  inputOrder: 0
  colorFormat: 0
  offsets: [123.6750, 116.2800, 103.5300]
  netScaleFactor: 0.00392156
  onnxFile: "/opt/nvidia/deepstream/deepstream/samples/models/Tracker/bodypose3dnet_accuracy.onnx"
  modelEngineFile: "/opt/nvidia/deepstream/deepstream/samples/models/Tracker/bodypose3dnet_accuracy.onnx_b1_gpu0_fp16.engine"
  poseInferenceInterval: -1    # -1 = first frame only (determine height once per target)
```

> **Key Differences from Standard NvDCF Accuracy Config:**
> - `stateEstimatorType: 3` instead of `1` — uses 3D location KF instead of bbox KF
> - `StateEstimator` has NO `processNoiseVar4Size` — bbox size is derived from the 3D model projection, not estimated
> - `ObjectModelProjection` section — camera calibration and 3D output controls
> - `PoseEstimator` section — optional body pose for height estimation
> - `minVisibility4GalleryUpdate: 0.6` in `ReID` — prevents occluded appearances from corrupting the gallery
> - `outputTerminatedTracks: 1` + `terminatedTrackFilename` — exports track history for evaluation

#### Multi-Stream Camera Configuration

For multi-stream setups, provide one camera calibration file per stream in the `cameraModelFilepath` list:

```yaml
ObjectModelProjection:
  cameraModelFilepath:
    - configs/camInfo_stream0.yml    # stream 0
    - configs/camInfo_stream1.yml    # stream 1
    - configs/camInfo_stream2.yml    # stream 2
  outputVisibility: 1
  outputFootLocation: 1
  outputConvexHull: 1
  minPoseConfidence: 0.5
```

Each camera must have its own calibrated projection matrix since cameras have different positions and orientations.

#### SV3DT Output Formats

**MOT Format** (`track_dump_<stream_id>.txt`):

When `outputTerminatedTracks: 1` and `terminatedTrackFilename` are set, terminated tracklets are saved in extended MOT format:

```
<frame>, <id>, <bb_left>, <bb_top>, <bb_width>, <bb_height>, <conf>, <foot_world_x>, <foot_world_y>, <class_id>, -1, <visibility>, <foot_image_x>, <foot_image_y>, <convex_hull_points...>
```

| Field | Description |
|-------|-------------|
| `frame` | Frame number |
| `id` | Target tracking ID |
| `bb_left, bb_top, bb_width, bb_height` | Recovered bounding box (complete, not clipped by occlusion) |
| `conf` | Detection confidence |
| `foot_world_x, foot_world_y` | Foot location in 3D world coordinates |
| `class_id` | Object class ID |
| `visibility` | Visibility ratio (0.0\~1.0), where 1.0 = fully visible |
| `foot_image_x, foot_image_y` | Foot location in image coordinates |
| `convex_hull_points` | Convex hull vertex coordinates from 3D cylinder projection |

**KITTI Format** (`track_results/` directory):

Track results can also be exported in KITTI tracking format for evaluation with standard benchmarks.

---

## Tracker Comparisons and Tradeoffs

| Tracker | GPU Usage | Accuracy | Visual Features | Key Advantage | Best Use Case |
|---------|-----------|----------|-----------------|---------------|---------------|
| **IOU** | Very Low | Low | No | Lightest weight | Sparse objects, detector every frame |
| **NvSORT** | Very Low | Medium | No | Kalman + cascaded matching | Medium/high accuracy detectors |
| **NvDCF** | Medium | High | DCF correlation filter | Robust to occlusion, supports PGIE interval > 0, tracker confidence output | Complex scenes, partial occlusion |
| **NvDeepSORT** | Low | High | Re-ID network | Discriminative appearance matching | Similar-looking objects, multi-camera |
| **MaskTracker** | High | Very High | SAM2 segmentation | Precise segmentation masks, works across object classes | Segmentation + tracking, diverse objects |
| **NvDCF 3D (SV3DT)** | Medium-High | High | DCF + 3D model + optional pose | 3D world tracking, occlusion-aware bbox, foot location | Static camera surveillance, people tracking in physical space |

> **Note**: IOU and NvSORT do not require video frame data (only bounding boxes). NvDCF and NvDeepSORT require NV12 or RGBA frames. MaskTracker requires frames for SAM2 inference.

> **tracker_confidence**: Only NvDCF generates per-object tracker confidence values. For IOU, NvSORT, NvDeepSORT, and MaskTracker, `tracker_confidence` is set to `1.0` by default.

---

## Dynamic Runtime Configuration

The tracker supports parameter updates at runtime without restarting the pipeline. Only parameters marked as **Dynamic=Yes** in the tables above are supported.

### REST API

```bash
curl -XPOST 'http://localhost:9000/api/v1/nvtracker/config-path' -d '{
  "stream": {
    "stream_id": "0",
    "config_path": "trackerUpdate.yaml"
  }
}'
```

### GStreamer Event

Use `gst_nvevent_nvtracker_config_update` to trigger a config update from within the application.

### C++ API

`NvMOT_UpdateParams(contextHandle, configStr)` accepts a YAML config string directly (no file on disk required).

### Control Section (Dynamic Only)

```yaml
Control:
  tracker-reset: 1  # Soft reset: removes all tracks and track history
```

> **Note**: Reconfiguring any stream in a batch re-configures all streams in that batch/sub-batch.

---

## Pipeline Integration

### Basic Usage

```python
from pyservicemaker import Pipeline
import platform

def tracking_pipeline(video_path, infer_config):
    pipeline = Pipeline("tracking-pipeline")

    # Source and decoding
    pipeline.add("filesrc", "src", {"location": video_path})
    pipeline.add("h264parse", "parser")
    pipeline.add("nvv4l2decoder", "decoder")
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})

    # Inference
    pipeline.add("nvinfer", "pgie", {"config-file-path": infer_config})

    # Tracker
    pipeline.add("nvtracker", "tracker", {
        "ll-lib-file": "/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so",
        "ll-config-file": "/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml",
        "tracker-width": 640,
        "tracker-height": 384
    })

    # Display
    pipeline.add("nvosdbin", "osd")
    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    pipeline.add(sink_type, "sink")

    # Link
    pipeline.link("src", "parser", "decoder")
    pipeline.link(("decoder", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "pgie", "tracker", "osd", "sink")

    pipeline.start().wait()
```

### SV3DT (Single-View 3D Tracking) with PeopleNet

SV3DT reuses the `tracking_pipeline` structure above -- only the PGIE config and the `nvtracker` properties change. Splice these settings into that pipeline (do **not** call this snippet on its own; it assumes `pipeline`, `MUX_WIDTH`, and `MUX_HEIGHT` from the surrounding `tracking_pipeline` definition):

```python
# Call as: tracking_pipeline(video_path, "config_pgie_peoplenet.yml")
# Then override the tracker block with the 3D config below.

# --- nvtracker overrides for SV3DT ---
# Replace the "tracker" element added in tracking_pipeline with:
pipeline.add("nvtracker", "tracker", {
    # 3D tracker library + config (from deepstream_reference_apps/deepstream-tracker-3d)
    "ll-lib-file": "/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so",
    "ll-config-file": "config_tracker_NvDCF_accuracy_3D.yml",  # references camInfo.yml

    # SV3DT requires tracker dimensions to match the muxer / camera calibration,
    # not the inference input -- otherwise the 3D cylinder projection is wrong.
    "tracker-width": MUX_WIDTH,    # e.g. 1920
    "tracker-height": MUX_HEIGHT,  # e.g. 1080

    "gpu-id": 0,
    "display-tracking-id": 1,
})
```

**Key deltas vs. the basic `tracking_pipeline`:**

| Property | Basic NvDCF | SV3DT |
|----------|-------------|-------|
| `ll-config-file` | `config_tracker_NvDCF_perf.yml` | `config_tracker_NvDCF_accuracy_3D.yml` (+ `camInfo.yml`) |
| `tracker-width` / `tracker-height` | Match inference (e.g. 640x384) | **Must match muxer/calibration** (e.g. 1920x1080) |
| PGIE | Any detector | PeopleNet (SV3DT models humans) |

### Accessing Tracking Data

```python
from pyservicemaker import BatchMetadataOperator

class TrackingAnalyzer(BatchMetadataOperator):
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            print(f"Frame {frame_meta.frame_number}:")

            for obj_meta in frame_meta.object_items:
                print(f"  Object: class={obj_meta.class_id}, "
                      f"object_id={obj_meta.object_id}, "
                      f"confidence={obj_meta.confidence:.2f}, "
                      f"tracker_confidence={obj_meta.tracker_confidence:.2f}")
```

---

## Performance Tuning

### Tracker Dimensions

Match tracker dimensions to inference input for best performance:

```python
# If inference uses 960x544, match tracker
pipeline.add("nvtracker", "tracker", {
    "tracker-width": 960,
    "tracker-height": 544,
    # ...
})
```

### Track Lifecycle Parameters

| Scene Type | maxShadowTrackingAge | probationAge | earlyTerminationAge |
|------------|---------------------|--------------|---------------------|
| Simple | 15 | 2 | 1 |
| Moderate | 30 | 3 | 1 |
| Complex/Occlusion | 60 | 5 | 2 |

### Memory Pre-allocation

Total GPU memory is proportional to: `(number of streams) x maxTargetsPerStream`. The library pre-allocates all memory during init -- no growth during runtime.

### Accuracy Tuning

DeepStream 7.0+ includes **PipeTuner** for automatic accuracy tuning. It explores the parameter space and finds optimal parameters for metrics like HOTA, MOTA, and IDF1.

---

## Miscellaneous Data Output

The tracker can output additional data via `NvDsTargetMiscDataBatch` (controlled by `user-meta-pool-size`):

| Data Type | Enable Config | Description |
|-----------|---------------|-------------|
| **Past-frame data** | `enablePastFrame: 1` | Tracked data from Tentative period, reported after activation |
| **Terminated tracks** | `outputTerminatedTracks: 1` | Full trajectory history for terminated targets |
| **Shadow tracks** | `outputShadowTracks: 1` | Shadow tracking target data (not otherwise visible) |

---

## Sample Configuration Files

```
/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/
|-- config_tracker_IOU.yml                # Fast IOU tracker (GREEDY)
|-- config_tracker_NvSORT.yml             # NvSORT (CASCADED + Regular KF)
|-- config_tracker_NvDCF_max_perf.yml     # NvDCF maximum performance
|-- config_tracker_NvDCF_perf.yml         # NvDCF balanced performance
|-- config_tracker_NvDCF_accuracy.yml     # NvDCF highest accuracy (Re-Assoc + ReID)
|-- config_tracker_NvDeepSORT.yml         # NvDeepSORT with ReID
|-- config_tracker_MaskTracker.yml        # MaskTracker with SAM2
|-- config_tracker_module_Segmenter.yml   # Segmenter module config for MaskTracker

# SV3DT 3D Tracker config (from deepstream_reference_apps):
# https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps/tree/master/deepstream-tracker-3d
|-- config_tracker_NvDCF_accuracy_3D.yml   # NvDCF 3D tracking (SV3DT)
|-- camInfo.yml                            # Camera calibration for SV3DT
```

---

## Common Issues

### Issue 1: Tracking IDs Not Appearing

**Cause**: OSD not configured to display tracking IDs.

**Solution**:
```python
pipeline.add("nvtracker", "tracker", {
    "display-tracking-id": 1,
})
```

### Issue 2: Frequent ID Switches

**Cause**: Low matching thresholds or short shadow tracking age.

**Solutions**:
- Increase `maxShadowTrackingAge` in tracker config
- Increase `minMatchingScore4Iou` and similarity weights
- Switch from GREEDY to CASCADED matching (`associationMatcherType: 1`)
- Consider using NvDCF or NvDeepSORT for visual/ReID-based matching

### Issue 3: Too Many Simultaneous Tracks

**Solution**: Reduce `maxTargetsPerStream` and/or increase `minDetectorConfidence` in BaseConfig.

### Issue 4: "Unable to acquire a user meta buffer"

**Cause**: Buffer pool exhausted when downstream is slow to release.

**Solution**: Increase `user-meta-pool-size` from default 16 to 64 or higher.

### Issue 5: Failed to Open Low-Level Lib

**Cause**: Missing `libmosquitto1` dependency.

**Solution**: `sudo apt-get install -y libmosquitto1`

### Issue 6: NvDCF Performance Bottleneck on Jetson

**Solution**: Use PVA backend to offload DCF operations from GPU:
```yaml
VisualTracker:
  visualTrackerType: 2
  vpiBackend4DcfTracker: 2  # PVA backend
```

---

## Related Documentation

- **GStreamer Plugins Overview**: `gstreamer_plugins.md`
- **Service Maker Python API**: `service_maker_api.md`
- **nvinfer Configuration**: `nvinfer_config.md`
- **Use Cases & Pipelines**: `use_cases_pipelines.md`
- **Official Docs**: https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvtracker.html

# DeepStream Common Errors and Troubleshooting Guide

## Overview

This document provides a quick reference for common errors encountered when developing DeepStream applications, along with their causes and solutions.

---

## Python API Errors

### Error: `RuntimeError: Probe failure` when attaching `measure_fps_probe`

**Symptom**: Pipeline crashes with `RuntimeError: Probe failure` and message `unable to add probe fps-probe`.

**Cause**: The built-in `measure_fps_probe` cannot be attached to sink elements (`nveglglessink`, `nv3dsink`, `filesink`). It can only be attached to processing elements that have both sink and src pads.

**Wrong Code**:
```python
pipeline.attach("sink", "measure_fps_probe", "fps-probe")  # CRASH - sink has no src pad
```

**Solution**:
```python
# Attach to a processing element instead
pipeline.attach("pgie", "measure_fps_probe", "fps-probe")   # Works
pipeline.attach("osd", "measure_fps_probe", "fps-probe")     # Works
```

---

### Error: `TypeError: object of type 'iterator' has no len()`

**Symptom**: Crash when trying to get length of metadata items.

**Cause**: `frame_meta.object_items`, `frame_meta.tensor_items`, and `frame_meta.user_items` return **iterators**, not lists.

**Wrong Code**:
```python
count = len(frame_meta.object_items)  # CRASH
```

**Solution**:
```python
# Count by iterating
obj_count = 0
for obj in frame_meta.object_items:
    obj_count += 1
    process(obj)

# Or convert to list first (if needed)
objects = list(frame_meta.object_items)
count = len(objects)
```

---

### Error: `pad template "sink_X" not found`

**Symptom**: Pipeline fails to link elements with error about missing pad.

**Cause**: Using literal pad names like `"sink_0"` instead of pad template `"sink_%u"`.

**Wrong Code**:
```python
pipeline.link((f"decoder{i}", "mux"), ("", f"sink_{i}"))  # FAILS
pipeline.link((f"decoder{i}", "mux"), ("", "sink_0"))     # FAILS
```

**Solution**:
```python
# Use pad template - GStreamer auto-assigns sink_0, sink_1, etc.
pipeline.link((f"decoder{i}", "mux"), ("", "sink_%u"))  # CORRECT
```

---

### Error: Data not reaching downstream (Queue appears empty)

**Symptom**: 
- Pipeline runs without errors
- No data reaches Kafka, VLM, or other downstream processing
- Statistics show 0 batches/messages processed

**Cause**: Using `queue.Queue` with `multiprocessing.Process`.

**Wrong Code**:
```python
from multiprocessing import Process
from queue import Queue  # Wrong queue type

class Processor:
    def __init__(self):
        self.batch_queue = Queue()  # Won't work across processes!
    
    def start(self):
        process = Process(target=self._run, args=(self.batch_queue,))
        process.start()  # Data put in child process never reaches parent
```

**Solution**:
```python
# Option 1: Use multiprocessing.Queue for processes
from multiprocessing import Process, Queue as MPQueue

class Processor:
    def __init__(self):
        self.batch_queue = MPQueue()  # Works across processes

# Option 2: Use threading instead
import threading
from queue import Queue

class Processor:
    def __init__(self):
        self.batch_queue = Queue()  # OK for threads
    
    def start(self):
        thread = threading.Thread(target=self._run, args=(self.batch_queue,))
        thread.start()  # Works because threads share memory
```

---

### Error: `ModuleNotFoundError: No module named 'pyservicemaker'` inside virtual environment

**Symptom**: Application crashes on import when run inside a Python virtual environment:
```
from pyservicemaker import Pipeline, Probe, BatchMetadataOperator
ModuleNotFoundError: No module named 'pyservicemaker'
```

**Cause**: `pyservicemaker` is installed system-wide but a standard `python3 -m venv` does **not** inherit system packages. Any DeepStream app run inside such a venv cannot find `pyservicemaker`.

**Solution**: Install `pyservicemaker` (and its `pyyaml` dependency) inside the virtual environment:
```bash
source venv/bin/activate
pip install /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl pyyaml
```

> **Note for generated READMEs**: When generating setup instructions that create a virtual environment, always include the `pyservicemaker` install step in the venv setup so users don't hit this error.

---

## Configuration Errors

### Error: `Configuration file parsing failed`

**Symptom**: nvinfer fails to load configuration file.

**Common Causes**:

1. **Wrong section name in YAML**:
```yaml
# WRONG
model:
  onnx-file: /path/to/model.onnx

# CORRECT
property:
  onnx-file: /path/to/model.onnx
```

2. **Mixing YAML/INI syntax**:
```yaml
# WRONG (INI syntax in .yml file)
[property]
onnx-file=/path/to/model.onnx

# CORRECT (YAML syntax)
property:
  onnx-file: /path/to/model.onnx
```

3. **Missing indentation in YAML**:
```yaml
# WRONG
property:
gpu-id: 0

# CORRECT
property:
  gpu-id: 0
```

---

### Error: `Model file not found`

**Symptom**: nvinfer cannot find model file.

**Solution**: Verify paths exist and use absolute paths:
```python
import os

# Verify path exists
model_path = "/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx"
if not os.path.exists(model_path):
    print(f"Model not found: {model_path}")
```

**DeepStream Model Locations**:
```
/opt/nvidia/deepstream/deepstream/samples/models/
├── Primary_Detector/
│   └── resnet18_trafficcamnet_pruned.onnx
├── Secondary_VehicleMake/
│   └── resnet18_vehiclemakenet_pruned.onnx
└── Secondary_VehicleTypes/
    └── resnet18_vehicletypenet_pruned.onnx
```

---

### Error: `num-detected-classes mismatch`

**Symptom**: Incorrect detection results or crashes.

**Cause**: `num-detected-classes` doesn't match model output.

**Solution**: Check your model's output and set correctly:
```yaml
property:
  num-detected-classes: 4  # Must match model
  labelfile-path: /path/to/labels.txt  # Should have 4 lines
```

---

## Pipeline Errors

### Error: `Element could not be created`

**Symptom**: Pipeline fails to create GStreamer element.

**Common Causes**:

1. **Missing plugin**: Element not installed
```bash
# Check if element exists
gst-inspect-1.0 nvinfer
```

2. **Wrong element name**:
```python
# Wrong
pipeline.add("nvv4ldecoder", "decoder")  # Typo

# Correct
pipeline.add("nvv4l2decoder", "decoder")
```

3. **Missing DeepStream libraries**:
```bash
# Set library path
export LD_LIBRARY_PATH=/opt/nvidia/deepstream/deepstream/lib:$LD_LIBRARY_PATH
```

---

### Error: `Failed to open low-level lib` (Tracker)

**Symptom**: Tracker fails to initialize with error:
```
gstnvtracker: Failed to open low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
dlopen error: libmosquitto.so.1: cannot open shared object file: No such file or directory
gstnvtracker: Failed to initialize low level lib.
```

**Cause**: The tracker library requires `libmosquitto` (MQTT client library) as a dependency.

**Solution**: Install the mosquitto library:
```bash
# Ubuntu/Debian
sudo apt-get update
sudo apt-get install -y libmosquitto1

# RHEL/CentOS
sudo yum install mosquitto
```

> **Important**: `libmosquitto1` is the client *library* only. If you also need to run an MQTT broker locally (e.g., `mosquitto &`) or use CLI tools like `mosquitto_sub` / `mosquitto_pub` for testing, you must install **separate** packages:
> ```bash
> sudo apt-get install -y mosquitto           # broker daemon
> sudo apt-get install -y mosquitto-clients   # CLI tools (mosquitto_pub, mosquitto_sub)
> ```

---

### Error: `Command 'mosquitto' not found`

**Symptom**: Running `mosquitto &` to start a local MQTT broker fails:
```
Command 'mosquitto' not found, but can be installed with:
apt install mosquitto
```

**Cause**: The `mosquitto` broker package is separate from `libmosquitto1` (client library). Installing `libmosquitto1` does NOT install the broker.

**Solution**:
```bash
sudo apt-get install -y mosquitto mosquitto-clients
```

---

### Error: `Linking failed between elements`

**Symptom**: Elements cannot be linked.

**Common Causes**:

1. **Incompatible caps**: Format mismatch between elements
```python
# Add videoconvert if formats don't match
pipeline.add("nvvideoconvert", "convert")
pipeline.link("element1", "convert", "element2")
```

2. **Wrong pad names**:
```python
# Wrong
pipeline.link(("src", "mux"), ("video", "sink"))

# Correct - check actual pad names
pipeline.link(("src", "mux"), ("", "sink_%u"))
```

---

### Error: `Pipeline stalled` or `No frames received`

**Symptom**: Pipeline starts but no output appears.

**Common Causes**:

1. **Missing queue elements**:
```python
# Add queues after tee
pipeline.add("tee", "tee")
pipeline.add("queue", "queue1")
pipeline.add("queue", "queue2")
pipeline.link(("tee", "queue1"), ("src_%u", ""))
pipeline.link(("tee", "queue2"), ("src_%u", ""))
```

2. **Sync issues with live sources**:
```python
# Disable sync for live streams
pipeline.add("nveglglessink", "sink", {"sync": 0})

# Set live-source on muxer
pipeline.add("nvstreammux", "mux", {"live-source": 1})
```

3. **appsink not emitting signals**:
```python
# Enable signal emission
pipeline.add("appsink", "sink", {"emit-signals": True, "sync": False})
```

---

### Error: `Resource busy` or `Device not found`

**Symptom**: GPU or video device unavailable.

**Solutions**:

1. **Check GPU availability**:
```bash
nvidia-smi
```

2. **Verify correct GPU ID**:
```yaml
property:
  gpu-id: 0  # Use correct GPU ID
```

3. **Check decoder device**:
```bash
ls /dev/nvidia*
```

---

## Memory Errors

### Error: `CUDA out of memory`

**Symptom**: Application crashes with memory error.

**Solutions**:

1. **Reduce batch size**:
```python
pipeline.add("nvstreammux", "mux", {"batch-size": 2})  # Reduce from 8
```

2. **Reduce resolution**:
```python
pipeline.add("nvstreammux", "mux", {
    "batch-size": 4,
    "width": 1280,   # Reduce from 1920
    "height": 720    # Reduce from 1080
})
```

3. **Use FP16 instead of FP32**:
```yaml
property:
  network-mode: 2  # FP16
```

4. **Monitor GPU memory**:
```bash
watch -n 1 nvidia-smi
```

---

### Error: `Buffer corruption` or `Segmentation fault`

**Symptom**: Random crashes when processing buffers.

**Cause**: Not cloning buffer tensors before async processing.

**Wrong Code**:
```python
def consume(self, buffer):
    tensor = buffer.extract(0)  # Direct use
    # Tensor may be reused/freed by pipeline
```

**Solution**:
```python
def consume(self, buffer):
    tensor = buffer.extract(0).clone()  # Clone first
    # Now safe for async processing
```

---

## Inference Errors

### Error: `setDimensions` fails with dynamic ONNX model (negative dimensions)

**Symptom**: TensorRT engine build fails immediately with repeated `setDimensions` errors:
```
ERROR: [TRT]: IOptimizationProfile::setDimensions: Error Code 3: API Usage Error
  (Parameter check failed, condition: std::all_of(dims.d, dims.d + dims.nbDims,
  [](int32_t x) noexcept { return x >= 0; }))
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1263 Explicit config dims is invalid
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:906 Failed to configure builder options
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:595 failed to build trt engine.
```

**Cause**: The ONNX model has **dynamic input shapes** (e.g., exported with `dynamic=True` in Ultralytics, or with dynamic batch/height/width axes). Dynamic dimensions are stored as symbolic names in the ONNX file, which TensorRT reads as `-1`. Without `infer-dims`, nvinfer passes these `-1` values to TensorRT's `setDimensions`, which requires all dimensions to be >= 0.

This is extremely common with models from Ultralytics (YOLO), HuggingFace, and other frameworks that default to dynamic exports.

**Diagnosis** — check if your ONNX model has dynamic dimensions:
```bash
python -c "
import onnx
m = onnx.load('model.onnx')
for inp in m.graph.input:
    dims = []
    for d in inp.type.tensor_type.shape.dim:
        dims.append(d.dim_param if d.dim_param else d.dim_value)
    print(f'{inp.name}: {dims}')
"
# If output shows symbolic names like 'batch', 'height', 'width' → dynamic model
# If output shows integers like [1, 3, 640, 640] → static model (infer-dims not needed)
```

**Solution**: Add `infer-dims` to the nvinfer config with the concrete C;H;W dimensions:

```yaml
# YAML format
property:
  onnx-file: model.onnx
  infer-dims: 3;640;640  # C;H;W — concrete dimensions for the dynamic input
```

```ini
# INI format
[property]
onnx-file=model.onnx
infer-dims=3;640;640
```

> **Note**: The batch dimension is handled by `batch-size` — `infer-dims` only specifies C;H;W. Delete any stale `.engine` files after adding `infer-dims` so TensorRT rebuilds the engine with the correct optimization profile.

---

### Error: `TensorRT engine build failed` (general)

**Symptom**: First-time model loading takes long then fails.

**Solutions**:

1. **Check for dynamic ONNX dimensions first** (see `setDimensions` error above)

2. **Check ONNX model compatibility**:
```bash
# Verify ONNX model
python -c "import onnx; onnx.checker.check_model('model.onnx')"
```

3. **Provide pre-built engine file**:
```yaml
property:
  model-engine-file: /path/to/model.engine
```

4. **Check CUDA/TensorRT versions**:
```bash
# Engine must match installed TensorRT version
nvcc --version
dpkg -l | grep tensorrt
```

---

### Error: `Output layer not found`

**Symptom**: Custom postprocessing can't find expected output layers.

**Solution**: List actual output layers:
```python
def handle_metadata(self, batch_meta):
    for frame_meta in batch_meta.frame_items:
        for tensor_meta in frame_meta.tensor_items:
            layers = tensor_meta.as_tensor_output().get_layers()
            print(f"Available layers: {list(layers.keys())}")
            # Use actual layer names
```

---

### Error: `Secondary GIE not processing`

**Symptom**: Secondary inference not running on detected objects.

**Causes and Solutions**:

1. **Wrong process-mode**:
```yaml
property:
  process-mode: 2  # Must be 2 for secondary
```

2. **Wrong operate-on-gie-id**:
```yaml
property:
  process-mode: 2
  operate-on-gie-id: 1  # Must match primary GIE unique-id
```

3. **Wrong operate-on-class-ids**:
```yaml
property:
  process-mode: 2
  operate-on-gie-id: 1
  operate-on-class-ids: 0  # Must match class IDs from primary
```

---

## Display Errors

### Error: `Could not open display`

**Symptom**: Rendering fails on headless systems.

**Solution**: Use fakesink for headless operation:
```python
# Check if display is available
import os
if "DISPLAY" not in os.environ:
    pipeline.add("fakesink", "sink")
else:
    pipeline.add("nveglglessink", "sink")
```

Or use file output:
```python
pipeline.add("nvvideoconvert", "convert")
pipeline.add("nvv4l2h264enc", "encoder")
pipeline.add("h264parse", "parser")
pipeline.add("mp4mux", "mux")
pipeline.add("filesink", "sink", {"location": "output.mp4"})
```

---

### Error: `Platform not supported`

**Symptom**: Sink element fails on Jetson or x86.

**Solution**: Use platform-specific sink:
```python
import platform

if platform.processor() == "aarch64":
    # Jetson
    pipeline.add("nv3dsink", "sink")
else:
    # x86
    pipeline.add("nveglglessink", "sink")
```

---

## Kafka/Message Broker Errors

### Error: `unable to open shared library` / `Failed to start` (missing librdkafka)

**Symptom**: Any pipeline using `nvmsgbroker` with the Kafka protocol adapter fails at startup:
```
WARN nvmsgbroker gstnvmsgbroker.cpp:404:legacy_gst_nvmsgbroker_start:<msgbroker> error: unable to open shared library
WARN basesink gstbasesink.c:5906:gst_base_sink_change_state:<msgbroker> error: Failed to start
Unable to set the pipeline to the playing state.
```

**Cause**: DeepStream's Kafka protocol adapter (`libnvds_kafka_proto.so`) dynamically links against `librdkafka.so.1`, which is **NOT** bundled with the DeepStream SDK and not installed by default.

**Diagnosis**:
```bash
ldd /opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so | grep "not found"
# Output: librdkafka.so.1 => not found
```

**Solution**:
```bash
sudo apt-get install -y librdkafka-dev
```

> **Note**: This is different from the "unable to connect to broker library" error below, which is caused by wrong connection string format. This error is about a missing system library.

---

### Error: `unable to connect to broker library` / `Failed to start`

**Symptom**: Pipeline fails with error:
```
WARN nvmsgbroker: error: unable to connect to broker library
WARN basesink: error: Failed to start
Unable to set the pipeline to the playing state.
```

**Cause**: Wrong connection string format. DeepStream uses **semicolon (`;`)** separator, NOT colon (`:`).

**Wrong Code**:
```python
# WRONG - colon separator
pipeline.add("nvmsgbroker", "msgbroker", {
    "conn-str": "localhost:9092",  # Wrong!
    # ...
})
```

**Solution**:
```python
# CORRECT - semicolon separator
pipeline.add("nvmsgbroker", "msgbroker", {
    "conn-str": "localhost;9092",  # Correct: use semicolon
    # ...
})
```

---

### Error: No messages reaching Kafka (pipeline runs but no output)

**Symptom**: 
- Pipeline runs without errors
- Kafka consumer receives no messages
- No error in logs

**Cause**: `nvmsgconv` requires `NvDsEventMsgMeta` by default (`msg2p-newapi=0`), which is **NOT automatically generated** by inference or tracker plugins. Without either (a) setting `msg2p-newapi: True` or (b) attaching a probe that generates `EventMessageUserMetadata`, nvmsgconv silently produces zero messages.

**Wrong Code**:
```python
# Without msg2p-newapi AND without EventMessageUserMetadata probe,
# nvmsgconv has no input and produces no messages!
pipeline.add("nvmsgconv", "msgconv", {
    "config": msgconv_config,
    "payload-type": 0
})
```

**Solution A** (simple): Set `msg2p-newapi: True` to use the new API that reads directly from `NvDsObjectMeta`:
```python
# CORRECT - msg2p-newapi reads from NvDsObjectMeta directly
pipeline.add("nvmsgconv", "msgconv", {
    "config": msgconv_config,
    "payload-type": 0,
    "msg2p-newapi": True,  # CRITICAL: Enables direct object metadata reading
    "frame-interval": 30   # Send message every 30 frames
})
```

**Solution B** (legacy): Keep `msg2p-newapi: 0` and attach a probe to generate `EventMessageUserMetadata`:
```python
# Option B1: Use built-in probe (simplest)
pipeline.attach("osd", "add_message_meta_probe", "metadata generator")

# Option B2: Custom EventMessageGenerator (for multi-camera / custom sensor mappings)
from pyservicemaker import Probe, BatchMetadataOperator

class EventMessageGenerator(BatchMetadataOperator):
    def __init__(self, sensor_map, labels):
        super().__init__()
        self._sensor_map = sensor_map
        self._labels = labels

    def handle_metadata(self, batch_meta, frame_interval=1):
        for frame_meta in batch_meta.frame_items:
            for object_meta in frame_meta.object_items:
                event_msg = batch_meta.acquire_event_message_meta()
                if event_msg:
                    source_id = frame_meta.source_id
                    sensor_info = self._sensor_map.get(source_id)
                    sensor_id = sensor_info.sensor_id if sensor_info else "N/A"
                    uri = sensor_info.uri if sensor_info else "N/A"
                    event_msg.generate(object_meta, frame_meta, sensor_id, uri, self._labels)
                    frame_meta.append(event_msg)

# Attach UPSTREAM of nvmsgconv
pipeline.attach("tracker", Probe("event_msg_gen", EventMessageGenerator(sensor_map, labels)))
```

**Reference samples**:
- Built-in probe: `/opt/nvidia/deepstream/deepstream/service-maker/sources/apps/python/pipeline_api/deepstream_test4_app/deepstream_test4.py`
- Custom generator: `/opt/nvidia/deepstream/deepstream/service-maker/sources/apps/python/pipeline_api/deepstream_test5_app/deepstream_test5.py`

---

### Error: `nvmsgbroker: Failed to send message`

**Symptom**: Messages not reaching Kafka.

**Solutions**:

1. **Check connection string format** (semicolon, not colon):
```python
pipeline.add("nvmsgbroker", "msgbroker", {
    "conn-str": "localhost;9092",  # Use semicolon separator!
    # ...
})
```

2. **Verify Kafka is running**:
```bash
# Check Kafka
kafka-topics.sh --list --bootstrap-server localhost:9092
```

3. **Check protocol library path**:
```python
pipeline.add("nvmsgbroker", "msgbroker", {
    "proto-lib": "/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
    # ...
})
```

---

### Error: `nvmsgbroker cannot have downstream elements`

**Symptom**: Pipeline fails when linking elements after nvmsgbroker.

**Cause**: nvmsgbroker is a **sink** element.

**Wrong Code**:
```python
# Wrong - msgbroker is a sink
pipeline.link("tracker", "msgconv", "msgbroker", "osd", "sink")
```

**Solution**: Use tee to split pipeline:
```python
# Correct - use tee to split
pipeline.add("tee", "tee")
pipeline.add("queue", "queue_msg")
pipeline.add("queue", "queue_video")

pipeline.link("tracker", "tee")
pipeline.link(("tee", "queue_msg"), ("src_%u", ""))
pipeline.link("queue_msg", "msgconv", "msgbroker")
pipeline.link(("tee", "queue_video"), ("src_%u", ""))
pipeline.link("queue_video", "osd", "sink")
```

---

## Debugging Tips

### Enable GStreamer Debug Output

```bash
# Basic debugging
export GST_DEBUG=3

# Plugin-specific debugging
export GST_DEBUG=nvinfer:5,nvstreammux:4

# Write to file
export GST_DEBUG_FILE=debug.log
```

### Debug Levels

| Level | Name | Description |
|-------|------|-------------|
| 0 | NONE | No output |
| 1 | ERROR | Errors only |
| 2 | WARNING | Warnings and errors |
| 3 | INFO | Informational messages |
| 4 | DEBUG | Debug messages |
| 5 | LOG | All log messages |

### Check Plugin Availability

```bash
# List all DeepStream plugins
gst-inspect-1.0 | grep nv

# Check specific plugin
gst-inspect-1.0 nvinfer
gst-inspect-1.0 nvstreammux
gst-inspect-1.0 nvtracker
```

### Pipeline Visualization

```bash
# Generate pipeline graph
export GST_DEBUG_DUMP_DOT_DIR=/tmp/dots
# Run pipeline, then:
dot -Tpng /tmp/dots/*.dot > pipeline.png
```

---

## Quick Reference: Error → Solution

| Error | Quick Fix |
|-------|-----------|
| `iterator has no len()` | Iterate to count, don't use `len()` |
| `pad template not found` | Use `"sink_%u"` not `"sink_0"` |
| Queue data loss | Use `multiprocessing.Queue` with `Process` |
| Config parse failed | Use `property:` not `model:` in YAML |
| `is-classifier` deprecation warning | Use `network-type: 1` instead of `is-classifier: 1`; omit both for detectors |
| `min-boxes` unknown key warning | Use `minBoxes` (camelCase), not `min-boxes` |
| `setDimensions` negative dims / engine build failed | Add `infer-dims=C;H;W` for dynamic ONNX models (e.g., `infer-dims=3;640;640`) |
| Model not found | Use absolute paths, verify file exists |
| Element not created | Check plugin name, set `LD_LIBRARY_PATH` |
| Link failed | Add `nvvideoconvert` for format conversion |
| Pipeline stalled | Add queues, check sync settings |
| CUDA OOM | Reduce batch size, use FP16 |
| Buffer corruption | Clone tensors before async use |
| Secondary GIE inactive | Set `process-mode: 2`, check `operate-on-gie-id` |
| No display | Use `fakesink` for headless |
| Kafka connection failed | Use `localhost;9092` (semicolon, not colon) |
| Kafka no messages | Set `msg2p-newapi: True`, OR attach `EventMessageUserMetadata` probe (see Kafka section) |
| msgbroker downstream | Use `tee` to split pipeline |
| Dynamic source stuck in PAUSED | Set `async: 0` on sink element |
| No data from RTSP | Test URL with ffplay, check credentials |
| `No module named 'pyservicemaker'` in venv | `pip install /opt/nvidia/deepstream/deepstream/service-maker/python/pyservicemaker*.whl pyyaml` inside the venv |

---

## Dynamic Source Management Errors

### Error: Stream added but stuck in PAUSED state

**Symptom**: REST API returns success, `DynamicSourceMessage` received, but video doesn't display. Elements stay in PAUSED state.

```
[Pipeline] src -> READY
[Pipeline] src -> PAUSED
# Never transitions to PLAYING
```

**Cause**: Missing `async=0` on sink element. The sink waits for preroll (first buffer) before allowing state transitions, creating a deadlock.

**Solution**:
```python
# CORRECT - async=0 is CRITICAL for dynamic sources
pipeline.add("nveglglessink", "sink", {
    "sync": 0,
    "qos": 0,
    "async": 0  # This is the fix
})

# WRONG - Will cause state transition deadlock
pipeline.add("nveglglessink", "sink", {"sync": 0})
```

---

### Error: No data from source, reconnection attempts

**Symptom**:
```
WARNING from dsnvurisrcbin0: No data from source since last 10 sec. Trying reconnection
Could not send message. (Received end-of-file)
```

**Cause**: RTSP connection issue - invalid URL, authentication required, or network problem.

**Solutions**:
1. Test RTSP URL directly:
```bash
ffplay "rtsp://camera-ip/stream"
```

2. Include credentials in URL:
```
rtsp://username:password@camera-ip/stream
```

3. Try TCP-only mode:
```python
"select-rtp-protocol": 4  # TCP only instead of auto
```

---

### Anti-Pattern: Custom REST Server for Stream Management

**WRONG**: Implementing a separate Flask/FastAPI server for stream management.

```python
# Don't do this - adds complexity and potential bugs
from flask import Flask
app = Flask(__name__)

@app.route('/add-camera')
def add_camera():
    # Custom implementation
```

**CORRECT**: Use nvmultiurisrcbin's built-in REST server.

```python
pipeline.add("nvmultiurisrcbin", "src", {
    "port": 9000,  # Built-in REST API at http://localhost:9000/api/v1/
    # ...
})
```

See `rest_api_dynamic.md` for complete REST API documentation.

---

## Related Documentation

- **GStreamer Plugins Overview**: `gstreamer_plugins.md`
- **Service Maker Python API**: `service_maker_api.md`
- **Best Practices**: `best_practices.md`
- **nvinfer Configuration**: `nvinfer_config.md`
- **Tracker Configuration**: `tracker_config.md`

# Use Cases: Pipeline Construction Patterns

## Overview

This document covers two fundamental DeepStream pipeline construction patterns. **Part 1** explains how to build a simple video player -- reading video from a file or stream, decoding it with hardware acceleration, and displaying it on screen without any AI inference. **Part 2** builds on that foundation to construct multi-inference pipelines that chain primary and secondary inference engines for object detection, classification, and attribute extraction across one or more video streams.

---

## Part 1: Simple Video Player

### Use Case Requirements

- Read video from file (H.264/H.265) or network stream (RTSP)
- Hardware-accelerated video decoding
- Display video on screen
- Handle multiple video formats
- Support for different platforms (x86_64 and ARM64/Jetson)

### Pipeline Architecture

#### Minimal Pipeline
```
Source -> Parser -> Decoder -> Converter -> Renderer
```

#### Detailed Pipeline Elements

1. **Source**: `filesrc` (for files) or `nvurisrcbin` (for URIs)
2. **Parser**: `h264parse` or `h265parse`
3. **Decoder**: `nvv4l2decoder` (hardware-accelerated)
4. **Converter**: `nvvideoconvert` (format conversion if needed)
5. **Renderer**: `nveglglessink` (x86_64) or `nv3dsink` (Jetson)

### Implementation Approaches

#### Approach 1: Pipeline API (Python)

**Language: Python**
**Target Audience: Python developers**
**Recommended for: Python applications**

```python
from pyservicemaker import Pipeline
import platform
import sys

def simple_video_player(video_path):
    """
    Simple video player using DeepStream Pipeline API

    Args:
        video_path: Path to video file or URI (rtsp://, file://, etc.)
    """
    pipeline = Pipeline("simple-player")

    # Determine if it's a URI or file path
    if video_path.startswith(("rtsp://", "http://", "file://")):
        # Use nvurisrcbin for URI-based sources
        pipeline.add("nvurisrcbin", "src", {"uri": video_path})
    else:
        # Use filesrc for local files
        pipeline.add("filesrc", "src", {"location": video_path})
        # Add parser based on file extension or use qtdemux
        if video_path.endswith(('.h264', '.264')):
            pipeline.add("h264parse", "parser")
        elif video_path.endswith(('.h265', '.265', '.hevc')):
            pipeline.add("h265parse", "parser")
        else:
            # For MP4/MOV files, use qtdemux
            pipeline.add("qtdemux", "demux")
            pipeline.add("h264parse", "parser")

    # Hardware-accelerated decoder
    pipeline.add("nvv4l2decoder", "decoder")

    # Video converter (may be needed for format conversion)
    pipeline.add("nvvideoconvert", "converter", {"gpu-id": 0})

    # Renderer (platform-specific)
    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    pipeline.add(sink_type, "sink", {"sync": 1})

    # Link elements
    if "nvurisrcbin" in [elem.name for elem in pipeline.elements]:
        # nvurisrcbin handles parsing internally
        pipeline.link("src", "decoder", "converter", "sink")
    elif "qtdemux" in [elem.name for elem in pipeline.elements]:
        # Handle qtdemux video pad
        pipeline.link("src", "demux")
        pipeline.link(("demux", "parser"), ("video_%u", ""))
        pipeline.link("parser", "decoder", "converter", "sink")
    else:
        # Simple file with parser
        pipeline.link("src", "parser", "decoder", "converter", "sink")

    # Start and wait
    try:
        pipeline.start().wait()
    except KeyboardInterrupt:
        print("\nPlayback interrupted")
    except Exception as e:
        print(f"Error: {e}")

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python simple_player.py <video_file_or_uri>")
        sys.exit(1)

    simple_video_player(sys.argv[1])
```

#### Approach 2: Flow API (Python)

**Language: Python**
**Target Audience: Python developers**
**Recommended for: Python applications**

```python
from pyservicemaker import Pipeline, Flow
import platform
import sys

def simple_video_player_flow(video_path):
    """
    Simple video player using DeepStream Flow API
    """
    pipeline = Pipeline("simple-player-flow")
    flow = Flow(pipeline)

    # Flow API doesn't directly support simple playback
    # This is a simplified example - Flow API is better for inference pipelines
    # For simple playback, use Pipeline API instead

    # However, we can still use Flow API with custom pipeline construction
    # This requires manual pipeline building
    pass

if __name__ == "__main__":
    simple_video_player_flow(sys.argv[1])
```

#### Approach 3: GStreamer Command Line

```bash
# For H.264 file
gst-launch-1.0 filesrc location=/path/to/video.h264 ! \
    h264parse ! \
    nvv4l2decoder ! \
    nvvideoconvert ! \
    nveglglessink sync=1

# For MP4 file
gst-launch-1.0 filesrc location=/path/to/video.mp4 ! \
    qtdemux ! \
    h264parse ! \
    nvv4l2decoder ! \
    nvvideoconvert ! \
    nveglglessink sync=1

# For RTSP stream
gst-launch-1.0 nvurisrcbin uri=rtsp://camera-ip/stream ! \
    nvv4l2decoder ! \
    nvvideoconvert ! \
    nveglglessink sync=1

# For Jetson platform
gst-launch-1.0 filesrc location=/path/to/video.h264 ! \
    h264parse ! \
    nvv4l2decoder ! \
    nvvideoconvert ! \
    nv3dsink sync=1
```

#### Approach 4: C/C++ Application

**Note: This section is specifically for C/C++ applications only. For Python applications, use Approach 1 (Pipeline API) or Approach 2 (Flow API) instead.**

This approach demonstrates how to build a simple video player using the GStreamer C API directly. This is a native C/C++ implementation that provides low-level control over the GStreamer pipeline.

**Language: C/C++**
**Target Audience: C/C++ developers**
**Not applicable for: Python applications**

```c
#include <gst/gst.h>
#include <glib.h>

typedef struct {
    GstElement *pipeline;
    GstElement *source;
    GstElement *parser;
    GstElement *decoder;
    GstElement *converter;
    GstElement *sink;
} AppData;

int main(int argc, char *argv[]) {
    GstBus *bus;
    GstMessage *msg;
    AppData data;

    // Initialize GStreamer
    gst_init(&argc, &argv);

    // Create elements
    data.pipeline = gst_pipeline_new("simple-player");
    data.source = gst_element_factory_make("filesrc", "source");
    data.parser = gst_element_factory_make("h264parse", "parser");
    data.decoder = gst_element_factory_make("nvv4l2decoder", "decoder");
    data.converter = gst_element_factory_make("nvvideoconvert", "converter");

    // Platform-specific sink
    #ifdef __aarch64__
        data.sink = gst_element_factory_make("nv3dsink", "sink");
    #else
        data.sink = gst_element_factory_make("nveglglessink", "sink");
    #endif

    if (!data.pipeline || !data.source || !data.parser ||
        !data.decoder || !data.converter || !data.sink) {
        g_printerr("Not all elements could be created.\n");
        return -1;
    }

    // Set source location
    g_object_set(data.source, "location", argv[1], NULL);

    // Set sink sync
    g_object_set(data.sink, "sync", 1, NULL);

    // Add elements to pipeline
    gst_bin_add_many(GST_BIN(data.pipeline),
                      data.source, data.parser, data.decoder,
                      data.converter, data.sink, NULL);

    // Link elements
    if (gst_element_link_many(data.source, data.parser, data.decoder,
                              data.converter, data.sink, NULL) != TRUE) {
        g_printerr("Elements could not be linked.\n");
        gst_object_unref(data.pipeline);
        return -1;
    }

    // Set pipeline to PLAYING state
    gst_element_set_state(data.pipeline, GST_STATE_PLAYING);

    // Wait for EOS or error
    bus = gst_element_get_bus(data.pipeline);
    msg = gst_bus_timed_pop_filtered(bus, GST_CLOCK_TIME_NONE,
                                      GST_MESSAGE_ERROR | GST_MESSAGE_EOS);

    // Cleanup
    if (msg != NULL)
        gst_message_unref(msg);
    gst_object_unref(bus);
    gst_element_set_state(data.pipeline, GST_STATE_NULL);
    gst_object_unref(data.pipeline);

    return 0;
}
```

**End of C/C++ Implementation** - This section contains C/C++ code only. For Python implementations, refer to Approach 1 (Pipeline API) or Approach 2 (Flow API) above.

### Enhanced Video Player Features

#### Feature 1: Multi-Format Support

```python
from pyservicemaker import Pipeline
import platform
import os

def detect_video_format(video_path):
    """Detect video format from file extension"""
    ext = os.path.splitext(video_path)[1].lower()
    formats = {
        '.h264': 'h264',
        '.264': 'h264',
        '.h265': 'h265',
        '.265': 'h265',
        '.hevc': 'h265',
        '.mp4': 'mp4',
        '.mov': 'mp4',
        '.mkv': 'mkv'
    }
    return formats.get(ext, 'unknown')

def multi_format_player(video_path):
    """Video player supporting multiple formats"""
    pipeline = Pipeline("multi-format-player")
    format_type = detect_video_format(video_path)

    # Source
    if video_path.startswith(("rtsp://", "http://", "file://")):
        pipeline.add("nvurisrcbin", "src", {"uri": video_path})
        # nvurisrcbin handles format detection automatically
        pipeline.add("nvv4l2decoder", "decoder")
    else:
        pipeline.add("filesrc", "src", {"location": video_path})

        if format_type == 'h264':
            pipeline.add("h264parse", "parser")
            pipeline.add("nvv4l2decoder", "decoder")
        elif format_type == 'h265':
            pipeline.add("h265parse", "parser")
            pipeline.add("nvv4l2decoder", "decoder")
        elif format_type in ['mp4', 'mkv']:
            demux_type = "qtdemux" if format_type == 'mp4' else "matroskademux"
            pipeline.add(demux_type, "demux")
            pipeline.add("h264parse", "parser")
            pipeline.add("nvv4l2decoder", "decoder")
        else:
            print(f"Unsupported format: {format_type}")
            return

    # Converter and sink
    pipeline.add("nvvideoconvert", "converter")
    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    pipeline.add(sink_type, "sink", {"sync": 1})

    # Link based on format
    if "nvurisrcbin" in [e.name for e in pipeline.elements]:
        pipeline.link("src", "decoder", "converter", "sink")
    elif "demux" in [e.name for e in pipeline.elements]:
        pipeline.link("src", "demux")
        pipeline.link(("demux", "parser"), ("video_%u", ""))
        pipeline.link("parser", "decoder", "converter", "sink")
    else:
        pipeline.link("src", "parser", "decoder", "converter", "sink")

    pipeline.start().wait()
```

#### Feature 2: Window Controls

```python
def video_player_with_controls(video_path):
    """Video player with window positioning and sizing"""
    pipeline = Pipeline("controlled-player")

    pipeline.add("filesrc", "src", {"location": video_path})
    pipeline.add("h264parse", "parser")
    pipeline.add("nvv4l2decoder", "decoder")
    pipeline.add("nvvideoconvert", "converter")

    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    pipeline.add(sink_type, "sink", {
        "sync": 1,
        "window-x": 100,      # Window X position
        "window-y": 100,      # Window Y position
        "window-width": 1280, # Window width
        "window-height": 720  # Window height
    })

    pipeline.link("src", "parser", "decoder", "converter", "sink")
    pipeline.start().wait()
```

#### Feature 3: Frame Rate Control

```python
def video_player_with_framerate(video_path, fps=None):
    """Video player with frame rate control"""
    pipeline = Pipeline("framerate-player")

    pipeline.add("filesrc", "src", {"location": video_path})
    pipeline.add("h264parse", "parser")
    pipeline.add("nvv4l2decoder", "decoder")

    # Add videorate for frame rate control
    if fps:
        pipeline.add("videorate", "rate")
        pipeline.add("capsfilter", "caps", {
            "caps": f"video/x-raw(memory:NVMM),framerate={fps}/1"
        })

    pipeline.add("nvvideoconvert", "converter")
    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    pipeline.add(sink_type, "sink", {"sync": 1})

    if fps:
        pipeline.link("src", "parser", "decoder", "rate", "caps", "converter", "sink")
    else:
        pipeline.link("src", "parser", "decoder", "converter", "sink")

    pipeline.start().wait()
```

### Platform-Specific Considerations

#### x86_64 (Desktop/Server)
- Use `nveglglessink` for rendering
- Supports multiple displays
- Higher GPU memory bandwidth
- Better for high-resolution playback

#### ARM64 (Jetson)
- Use `nv3dsink` for rendering
- Optimized for power efficiency
- Integrated GPU with shared memory
- Better for embedded applications

### Performance Optimization Tips

1. **Always use hardware decoders**: `nvv4l2decoder` instead of software decoders
2. **Provide headroom**: Bump `num-extra-surfaces` to prevent surface starvation
3. **Use NVMM memory**: Keeps frames on GPU for nvvideoconvert/sinks
4. **Sync to display**: Set `sync=1` on sink for smooth playback
5. **Match resolutions**: Avoid unnecessary scaling

### Error Handling

```python
from multiprocessing import Process
import sys

def safe_video_player(video_path):
    """Video player with error handling"""
    try:
        pipeline = Pipeline("safe-player")
        # ... pipeline construction ...
        pipeline.start().wait()
    except KeyboardInterrupt:
        print("\nPlayback interrupted by user")
    except Exception as e:
        print(f"Error during playback: {e}")
        sys.exit(1)

if __name__ == "__main__":
    process = Process(target=safe_video_player, args=(sys.argv[1],))
    try:
        process.start()
        process.join()
    except KeyboardInterrupt:
        print("\nTerminating...")
        process.terminate()
        process.join()
```

### Common Issues and Solutions

#### Issue 1: Black Screen
**Solution**: Check if decoder is working, verify video format support

#### Issue 2: Stuttering Playback
**Solution**: check GPU utilization

#### Issue 3: Format Not Supported
**Solution**: Use `nvurisrcbin` for automatic format detection, or add appropriate parser

#### Issue 4: High CPU Usage
**Solution**: Ensure hardware decoder is used, not software decoder

---

## Part 2: Multi-Inference Pipelines

### Use Case Requirements

- Detect objects using primary inference engine
- Classify detected objects using secondary inference engines
- Extract multiple attributes (e.g., vehicle make, vehicle type, color)
- Process multiple video streams simultaneously
- Track objects across frames
- Visualize all inference results

### Pipeline Architecture

#### Cascaded Inference Pipeline
```
Source -> Decoder -> Muxer -> PGIE -> SGIE1 -> SGIE2 -> Tracker -> OSD -> Renderer
```

#### Parallel Inference Pipeline (Advanced)
```
Source -> Decoder -> Muxer -> PGIE -> [SGIE1, SGIE2] -> Merger -> Tracker -> OSD -> Renderer
```

#### URI Source Inference Pattern

Use this source pattern when the input has already been resolved to a URI, such as `file://`,
`http://`, `https://`, `rtsp://`, HLS, or MPEG-DASH, and the pipeline should write annotated
MP4 output. For protocol constraints and local stream setup, see
[streaming_sources.md](streaming_sources.md).

| Item | Pattern |
|---|---|
| Pipeline chain | `nvurisrcbin -> nvstreammux -> nvinfer -> nvosdbin -> nvvideoconvert -> capsfilter -> nvv4l2h264enc -> h264parse -> qtmux -> filesink` |
| Source link | `pipeline.link(("src", "mux"), ("", "sink_%u"))` |
| Encoder caps | `video/x-raw(memory:NVMM), format=NV12` before `nvv4l2h264enc` |
| Optional tracking | Add `nvtracker` only when tracking or persistent object IDs are requested |

```python
pipeline.add("nvurisrcbin", "src", {"uri": url, "gpu-id": 0})
pipeline.add("nvstreammux", "mux", {
    "batch-size": 1,
    "width": 1280,
    "height": 720,
    "batched-push-timeout": 33000,
    "gpu-id": 0,
})
pipeline.link(("src", "mux"), ("", "sink_%u"))
pipeline.add("capsfilter", "enc_caps", {
    "caps": "video/x-raw(memory:NVMM), format=NV12",
})
```

### Implementation Approaches

#### Approach 1: Cascaded Detection + Classification

This is the most common pattern: detect objects first, then classify each detected object.

##### Pipeline API Implementation

```python
from pyservicemaker import Pipeline, Probe, BatchMetadataOperator, osd
import platform
import sys

def cascaded_inference_pipeline(video_path, pgie_config, sgie1_config, sgie2_config=None):
    """
    Cascaded inference: Detection -> Classification -> Attribute Detection

    Args:
        video_path: Path to video file
        pgie_config: Primary GIE config (object detection)
        sgie1_config: Secondary GIE config (first classification)
        sgie2_config: Optional second secondary GIE config
    """
    pipeline = Pipeline("cascaded-inference")

    # Source and decoding
    pipeline.add("filesrc", "src", {"location": video_path})
    pipeline.add("h264parse", "parser")
    pipeline.add("nvv4l2decoder", "decoder")

    # Stream muxer (batch multiple streams if needed)
    pipeline.add("nvstreammux", "mux", {
        "batch-size": 1,
        "width": 1920,
        "height": 1080
    })

    # Primary Inference Engine (Object Detection)
    pipeline.add("nvinfer", "pgie", {
        "config-file-path": pgie_config,
        "unique-id": 1
    })

    # Secondary Inference Engine 1 (Classification)
    pipeline.add("nvinfer", "sgie1", {
        "config-file-path": sgie1_config,
        "unique-id": 2
    })

    # Secondary Inference Engine 2 (Optional - Additional Classification)
    if sgie2_config:
        pipeline.add("nvinfer", "sgie2", {
            "config-file-path": sgie2_config,
            "unique-id": 3
        })

    # Tracker
    pipeline.add("nvtracker", "tracker", {
        "ll-lib-file": "/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so",
        "ll-config-file": "/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml",
        "tracker-width": 640,
        "tracker-height": 384
    })

    # On-Screen Display
    pipeline.add("nvosdbin", "osd", {
        "gpu-id": 0
    })

    # Converter and Sink
    pipeline.add("nvvideoconvert", "nvvideoconvert", {"gpu-id": 0})
    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    pipeline.add(sink_type, "sink", {"sync": 1})

    # Link pipeline
    pipeline.link("src", "parser", "decoder")
    pipeline.link(("decoder", "mux"), ("", "sink_%u"))

    # Link inference chain
    if sgie2_config:
        pipeline.link("mux", "pgie", "sgie1", "sgie2", "tracker", "s", "nvvideoconvert", "sink")
    else:
        pipeline.link("mux", "pgie", "sgie1", "tracker", "s", "nvvideoconvert", "sink")

    # Start pipeline
    pipeline.start().wait()

if __name__ == "__main__":
    if len(sys.argv) < 4:
        print("Usage: python cascaded_inference.py <video> <pgie_config> <sgie1_config> [sgie2_config]")
        sys.exit(1)

    sgie2 = sys.argv[4] if len(sys.argv) > 4 else None
    cascaded_inference_pipeline(sys.argv[1], sys.argv[2], sys.argv[3], sgie2)
```

##### Configuration Files

**Primary GIE Config (pgie_config.yml)**:
```yaml
property:
  model-engine-file: /path/to/detector.engine
  labelfile-path: /path/to/detector_labels.txt
  batch-size: 1
  net-scale-factor: 0.0039215697906911373
  model-color-format: 0
  num-detected-classes: 4
  process-mode: 1
  gie-unique-id: 1
  network-mode: 0
  cluster-mode: 2

class-attrs-all:
  topk: 20
  nms-iou-threshold: 0.5
  pre-cluster-threshold: 0.2
```

**Secondary GIE Config (sgie1_config.yml)**:
```yaml
property:
  model-engine-file: /path/to/classifier.engine
  labelfile-path: /path/to/classifier_labels.txt
  batch-size: 16
  net-scale-factor: 0.0039215697906911373
  model-color-format: 0
  process-mode: 2
  network-mode: 0
  network-type: 1
  gie-unique-id: 2
  operate-on-gie-id: 1
  operate-on-class-ids: 0
  classifier-async-mode: 1
  classifier-threshold: 0.51
```

#### Approach 2: Multi-Stream with Cascaded Inference

Process multiple video streams with cascaded inference on each stream.

```python
def multi_stream_cascaded_inference(video_paths, pgie_config, sgie_configs):
    """
    Multi-stream cascaded inference

    Args:
        video_paths: List of video file paths
        pgie_config: Primary GIE config
        sgie_configs: List of secondary GIE configs
    """
    pipeline = Pipeline("multi-stream-cascaded")
    num_streams = len(video_paths)

    # Add sources
    for i, video_path in enumerate(video_paths):
        pipeline.add("filesrc", f"src{i}", {"location": video_path})
        pipeline.add("h264parse", f"parser{i}")
        pipeline.add("nvv4l2decoder", f"decoder{i}")

    # Stream muxer
    pipeline.add("nvstreammux", "mux", {
        "batch-size": num_streams,
        "width": 1920,
        "height": 1080
    })

    # Primary Inference
    pipeline.add("nvinfer", "pgie", {
        "config-file-path": pgie_config,
        "unique-id": 1
    })

    # Secondary Inferences
    for idx, sgie_config in enumerate(sgie_configs):
        pipeline.add("nvinfer", f"sgie{idx+1}", {
            "config-file-path": sgie_config,
            "unique-id": idx + 2
        })

    # Tracker
    pipeline.add("nvtracker", "tracker", {
        "ll-lib-file": "/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so",
        "ll-config-file": "/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml"
    })

    # Stream demuxer
    pipeline.add("nvstreamdemux", "demux")

    # OSD and sinks for each stream
    for i in range(num_streams):
        pipeline.add("nvosdbin", f"osd{i}")
        pipeline.add("nvvideoconvert", f"converter{i}")
        pipeline.add("nveglglessink", f"sink{i}", {"sync": 1})

    # Link sources to muxer
    # CRITICAL: Always use "sink_%u" pad template for nvstreammux, NOT f"sink_{i}" or "sink_0"
    for i in range(num_streams):
        pipeline.link(f"src{i}", f"parser{i}", f"decoder{i}")
        pipeline.link((f"decoder{i}", "mux"), ("", "sink_%u"))  # Pad template auto-assigns sink_0, sink_1, etc.

    # Link inference chain
    link_chain = ["mux", "pgie"]
    for idx in range(len(sgie_configs)):
        link_chain.append(f"sgie{idx+1}")
    link_chain.extend(["tracker", "demux"])
    pipeline.link(*link_chain)

    # Link demuxer outputs to sinks
    for i in range(num_streams):
        pipeline.link((f"demux", f"osd{i}"), (f"src_{i}", ""))
        pipeline.link(f"osd{i}", f"converter{i}", f"sink{i}")

    pipeline.start().wait()
```

#### Approach 2b: Multi-Stream RTSP with nvurisrcbin and Cascaded Inference

Process multiple RTSP streams using nvurisrcbin with cascaded inference.

```python
def multi_rtsp_cascaded_inference(rtsp_urls, pgie_config, sgie_configs):
    """
    Multi-stream RTSP cascaded inference using nvurisrcbin

    Args:
        rtsp_urls: List of RTSP stream URLs
        pgie_config: Primary GIE config
        sgie_configs: List of secondary GIE configs
    """
    pipeline = Pipeline("multi-rtsp-cascaded")
    num_streams = len(rtsp_urls)

    # Add RTSP sources with nvurisrcbin (handles codec detection and decoding automatically)
    for i, url in enumerate(rtsp_urls):
        pipeline.add("nvurisrcbin", f"src{i}", {"uri": url})

    # Stream muxer
    pipeline.add("nvstreammux", "mux", {
        "batch-size": num_streams,
        "width": 1920,
        "height": 1080,
        "batched-push-timeout": 40000,
        "live-source": 1  # Important for RTSP streams
    })

    # Primary Inference
    pipeline.add("nvinfer", "pgie", {
        "config-file-path": pgie_config,
        "unique-id": 1,
        "batch-size": num_streams
    })

    # Secondary Inferences
    for idx, sgie_config in enumerate(sgie_configs):
        pipeline.add("nvinfer", f"sgie{idx+1}", {
            "config-file-path": sgie_config,
            "unique-id": idx + 2,
            "batch-size": num_streams
        })

    # Tracker
    pipeline.add("nvtracker", "tracker", {
        "ll-lib-file": "/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so",
        "ll-config-file": "/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml"
    })

    # Tiler for multi-stream display
    pipeline.add("nvmultistreamtiler", "tiler", {
        "rows": 2,
        "columns": 2,
        "width": 1920,
        "height": 1080
    })

    # OSD and sink
    pipeline.add("nvosdbin", "osd")
    pipeline.add("nveglglessink", "sink", {"sync": 0})

    # Link sources to muxer - CRITICAL: Use "sink_%u" pad template
    # nvurisrcbin creates dynamic src pads, so link directly to mux sink pad template
    for i in range(num_streams):
        pipeline.link((f"src{i}", "mux"), ("", "sink_%u"))  # CORRECT - pad template auto-assigns
        # WRONG: pipeline.link((f"src{i}", "mux"), ("", f"sink_{i}"))  # This will FAIL!

    # Link inference chain
    link_chain = ["mux", "pgie"]
    for idx in range(len(sgie_configs)):
        link_chain.append(f"sgie{idx+1}")
    link_chain.extend(["tracker", "tiler", "osd", "sink"])
    pipeline.link(*link_chain)

    pipeline.start().wait()
```

#### Approach 3: Custom Postprocessing with Tensor Metadata

Use custom postprocessing when built-in parsers don't support your model format.

```python
from pyservicemaker import Pipeline, Probe, BatchMetadataOperator, postprocessing, osd
import torch  # pip install torch torchvision (not in base DS container)
import torchvision.ops as ops

class CustomDetectorConverter(postprocessing.ObjectDetectorOutputConverter):
    """Custom converter for detection model outputs"""
    NETWORK_WIDTH = 960
    NETWORK_HEIGHT = 544

    def __init__(self, threshold=0.5):
        self.threshold = threshold

    def __call__(self, output_layers):
        """Convert tensor outputs to detection format"""
        outputs = []

        # Extract output layers (adjust names based on your model)
        bbox_layer = output_layers.get('output_bbox/BiasAdd:0')
        conf_layer = output_layers.get('output_cov/Sigmoid:0')

        if bbox_layer is None or conf_layer is None:
            return outputs

        # Convert DLPack tensors to PyTorch
        bbox_tensor = torch.utils.dlpack.from_dlpack(bbox_layer).to('cpu')
        conf_tensor = torch.utils.dlpack.from_dlpack(conf_layer).to('cpu')

        # Process detections
        # ... custom processing logic ...

        return outputs

class CustomPostprocessor(BatchMetadataOperator):
    """Custom postprocessor for tensor outputs"""
    def __init__(self, converter):
        super().__init__()
        self.converter = converter
        self.stream_width = 1920
        self.stream_height = 1080

    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            # Process tensor metadata
            for tensor_meta in frame_meta.tensor_items:
                output_layers = tensor_meta.as_tensor_output().get_layers()
                detections = self.converter(output_layers)

                # Scale coordinates
                scale_x = self.stream_width / self.converter.NETWORK_WIDTH
                scale_y = self.stream_height / self.converter.NETWORK_HEIGHT

                # Create object metadata
                for det in detections:
                    class_id, conf, x1, y1, x2, y2 = det

                    obj_meta = batch_meta.acquire_object_meta()
                    obj_meta.class_id = int(class_id)
                    obj_meta.confidence = float(conf)
                    obj_meta.rect_params.left = x1 * scale_x
                    obj_meta.rect_params.top = y1 * scale_y
                    obj_meta.rect_params.width = (x2 - x1) * scale_x
                    obj_meta.rect_params.height = (y2 - y1) * scale_y
                    obj_meta.rect_params.border_width = 2
                    obj_meta.rect_params.border_color = osd.Color(1.0, 0.0, 0.0, 1.0)

                    frame_meta.append(obj_meta)

def custom_postprocessing_pipeline(video_path, infer_config):
    """Pipeline with custom postprocessing"""
    pipeline = Pipeline("custom-postprocess")

    # Source and decoding
    pipeline.add("filesrc", "src", {"location": video_path})
    pipeline.add("h264parse", "parser")
    pipeline.add("nvv4l2decoder", "decoder")
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})

    # Inference with tensor output
    pipeline.add("nvinfer", "infer", {
        "config-file-path": infer_config,
        "output-tensor-meta": 1  # Enable tensor metadata output
    })

    # Disable built-in object metadata generation
    pipeline["infer"].set({"filter-out-class-ids": "0;1;2;3"})

    # Custom postprocessing
    converter = CustomDetectorConverter(threshold=0.5)
    postprocessor = CustomPostprocessor(converter)

    # Tracker, OSD, Sink
    pipeline.add("nvtracker", "tracker", {
        "ll-lib-file": "/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so",
        "ll-config-file": "/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml"
    })
    pipeline.add("nvosdbin", "osd")
    pipeline.add("nvvideoconvert", "converter")
    pipeline.add("nveglglessink", "sink", {"sync": 1})

    # Link and attach probe
    pipeline.link("src", "parser", "decoder")
    pipeline.link(("decoder", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "infer", "tracker", "osd", "converter", "sink")
    pipeline.attach("infer", Probe("postprocess", postprocessor))

    pipeline.start().wait()
```

#### Approach 4: Preprocessing + Inference Pipeline

Use custom preprocessing before inference for ROI-based processing.

```python
def preprocessing_inference_pipeline(video_path, preprocess_config, infer_config):
    """Pipeline with custom preprocessing"""
    pipeline = Pipeline("preprocess-inference")

    # Source and decoding
    pipeline.add("filesrc", "src", {"location": video_path})
    pipeline.add("h264parse", "parser")
    pipeline.add("nvv4l2decoder", "decoder")
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})

    # Custom preprocessing
    pipeline.add("nvdspreprocess", "preprocess", {
        "config-file": preprocess_config,
        "gpu-id": 0
    })

    # Inference with tensor input
    pipeline.add("nvinfer", "infer", {
        "config-file-path": infer_config,
        "input-tensor-meta": 1,  # Use tensor metadata from preprocessing
        "batch-size": 1
    })

    # Postprocessing (if needed)
    pipeline.add("nvdspostprocess", "postprocess", {
        "postprocesslib-name": "/path/to/libpostprocess.so",
        "postprocesslib-config-file": "/path/to/postprocess_config.yml"
    })

    # Tracker, OSD, Sink
    pipeline.add("nvtracker", "tracker", {
        "ll-lib-file": "/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so",
        "ll-config-file": "/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml"
    })
    pipeline.add("nvosdbin", "osd")
    pipeline.add("nvvideoconvert", "converter")
    pipeline.add("nveglglessink", "sink", {"sync": 1})

    # Link
    pipeline.link("src", "parser", "decoder")
    pipeline.link(("decoder", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "preprocess", "infer", "postprocess", "tracker", "osd", "converter", "sink")

    pipeline.start().wait()
```

### Metadata Processing Examples

#### Example 1: Extract All Inference Results

```python
class InferenceResultExtractor(BatchMetadataOperator):
    """Extract and print all inference results"""
    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            print(f"\nFrame {frame_meta.frame_number}:")

            for obj_meta in frame_meta.object_items:
                print(f"  Object:")
                print(f"    Class ID: {obj_meta.class_id}")
                print(f"    Confidence: {obj_meta.confidence:.2f}")
                print(f"    BBox: ({obj_meta.rect_params.left:.1f}, "
                      f"{obj_meta.rect_params.top:.1f}, "
                      f"{obj_meta.rect_params.width:.1f}, "
                      f"{obj_meta.rect_params.height:.1f})")
                print(f"    Object ID (Tracking): {obj_meta.object_id}")

                # Check for secondary inference results
                # Secondary results are stored in object metadata
                # Access via obj_meta.obj_user_meta_list
```

#### Example 2: Filter Objects by Confidence

```python
class ConfidenceFilter(BatchMetadataOperator):
    """Filter objects by confidence threshold"""
    def __init__(self, threshold=0.5):
        super().__init__()
        self.threshold = threshold

    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            # Remove low-confidence objects
            objects_to_remove = []
            for obj_meta in frame_meta.object_items:
                if obj_meta.confidence < self.threshold:
                    objects_to_remove.append(obj_meta)

            # Note: Direct removal may not be supported
            # Instead, mark them or filter in downstream processing
```

#### Example 3: Aggregate Statistics

```python
class StatisticsAggregator(BatchMetadataOperator):
    """Aggregate statistics across frames"""
    def __init__(self):
        super().__init__()
        self.class_counts = {}
        self.total_frames = 0

    def handle_metadata(self, batch_meta):
        self.total_frames += len(batch_meta.frame_items)

        for frame_meta in batch_meta.frame_items:
            for obj_meta in frame_meta.object_items:
                class_id = obj_meta.class_id
                self.class_counts[class_id] = self.class_counts.get(class_id, 0) + 1

    def print_statistics(self):
        print(f"\nStatistics:")
        print(f"Total frames processed: {self.total_frames}")
        print(f"Class distribution:")
        for class_id, count in self.class_counts.items():
            print(f"  Class {class_id}: {count} objects")
```

### Performance Optimization

#### Batch Size Optimization

```python
def optimize_batch_size(num_streams, gpu_memory_gb):
    """Calculate optimal batch size"""
    # Rule of thumb: 1GB GPU memory per stream for 1080p
    max_batch = min(num_streams, gpu_memory_gb)
    # Use power of 2 for better GPU utilization
    batch_size = 1
    while batch_size * 2 <= max_batch:
        batch_size *= 2
    return batch_size
```

#### Inference Precision Selection

```python
# In inference config file:
# network-mode: 0 = FP32 (highest accuracy, slowest)
# network-mode: 1 = FP16 (good balance)
# network-mode: 2 = INT8 (fastest, may need calibration)

# For production, typically use FP16:
infer_config = {
    "network-mode": 1  # FP16
}
```

### Common Patterns

#### Pattern 1: Vehicle Detection + Make/Type Classification

```python
# PGIE: Vehicle detection (cars, trucks, buses)
# SGIE1: Vehicle make classification (Toyota, Honda, Ford, etc.)
# SGIE2: Vehicle type classification (sedan, SUV, truck, etc.)

pipeline.link("mux", "pgie", "sgie1", "sgie2", "tracker", "osd", "sink")
```

#### Pattern 2: Person Detection + Attribute Classification

```python
# PGIE: Person detection
# SGIE1: Gender classification
# SGIE2: Age estimation
# SGIE3: Clothing classification

pipeline.link("mux", "pgie", "sgie1", "sgie2", "sgie3", "tracker", "osd", "sink")
```

#### Pattern 3: Multi-Model Ensemble

```python
# Run multiple detection models and merge results
# Requires custom postprocessing to combine outputs
```

### Best Practices

1. **Use appropriate batch sizes**: Match number of streams
2. **Cascade inferences properly**: Ensure operate-on-gie-id is correct
3. **Filter classes appropriately**: Use operate-on-class-ids
4. **Optimize inference precision**: Use FP16 for production
5. **Monitor GPU memory**: Adjust batch sizes accordingly
6. **Use tracker after all inferences**: Ensures consistent tracking
7. **Test with representative data**: Use real-world video samples

# Utilities and Configuration Classes

## Overview

The `pyservicemaker` module and its `utils` submodule provide a collection of utility classes for monitoring, configuration management, and helper patterns used in DeepStream application development. This document covers:

- **Part 1 -- Performance Monitoring Utilities**: Real-time FPS measurement, stream-level performance tracking, dynamic source monitoring, and model engine file hot-swapping via `PerfMonitor` and `EngineFileMonitor`.
- **Part 2 -- Configuration and Helper Classes**: Source configuration management (`SourceConfig`, `SensorInfo`, `CameraInfo`), smart recording configuration (`SmartRecordConfig`), custom postprocessing interfaces (`PostProcessing`, `ObjectDetectorOutputConverter`), and factory-based plugin creation (`CommonFactory`).

---

# Part 1: Performance Monitoring Utilities

The `pyservicemaker.utils` module provides utilities for monitoring pipeline performance and managing model engine files. These utilities are essential for:
- Real-time FPS (Frames Per Second) measurement
- Stream-level performance tracking
- Dynamic source monitoring
- Model engine file hot-swapping (On-The-Fly updates)
- Production deployment monitoring

## Core Classes

### PerfMonitor

A performance monitoring utility that tracks FPS and throughput for DeepStream pipelines.

**Constructor**:
```python
from pyservicemaker import utils

perf_monitor = utils.PerfMonitor(
    batch_size=4,              # Number of streams in batch
    interval=1,                # Measurement interval in seconds
    source_type="nvurisrcbin", # Source element type name
    show_name=True             # Show stream names in output
)
```

**Parameters**:
- `batch_size` (int): Number of streams in the pipeline batch
- `interval` (int): Performance measurement interval in seconds
- `source_type` (str): Type name of the source bin (e.g., "nvurisrcbin", "nvmultiurisrcbin")
- `show_name` (bool): Whether to show stream names in performance logs (default: True)

**Methods**:

#### `apply(element, pad_name)`
Attach the performance monitor to a pipeline element.

**Parameters**:
- `element`: Pipeline element to monitor (typically tiler or sink)
- `pad_name` (str): Name of the pad to monitor (typically "sink")

**Example**:
```python
perf_monitor.apply(pipeline["tiler"], "sink")
```

#### `add_stream(source_id, uri, sensor_id, sensor_name)`
Add a new stream to monitor (for dynamic sources).

**Parameters**:
- `source_id` (int): Unique source ID
- `uri` (str): Stream URI
- `sensor_id` (str): Sensor identifier
- `sensor_name` (str): Sensor name

#### `remove_stream(source_id)`
Remove a stream from monitoring.

**Parameters**:
- `source_id` (int): Source ID to remove

#### `pause()`
Pause performance monitoring.

#### `resume()`
Resume performance monitoring.

### EngineFileMonitor

Monitors TensorRT engine files and triggers On-The-Fly (OTF) model updates when files change.

**Constructor**:
```python
from pyservicemaker import utils

engine_monitor = utils.EngineFileMonitor(
    infer_element,           # nvinfer element
    engine_file_path         # Path to engine file to monitor
)
```

**Parameters**:
- `infer_element`: The `nvinfer` element to update when engine file changes
- `engine_file_path` (str): Path to the TensorRT engine file to monitor

**Properties**:
- `started` (bool): Whether the monitor has been started

**Methods**:

#### `start()`
Start monitoring the engine file for changes.

**Returns**: bool (True if started successfully)

#### `stop()`
Stop monitoring the engine file.

**Returns**: bool (True if stopped successfully)

## Performance Monitoring Usage Patterns

### Pattern 1: Basic FPS Monitoring

Monitor FPS for a single-stream pipeline.

```python
from pyservicemaker import Pipeline, utils
import platform

def pipeline_with_fps_monitoring(video_uri, config_path):
    """Pipeline with FPS monitoring"""
    pipeline = Pipeline("fps-monitored-pipeline")

    # Build pipeline
    pipeline.add("nvurisrcbin", "src", {"uri": video_uri})
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})
    pipeline.add("nvinfer", "infer", {"config-file-path": config_path})
    pipeline.add("nvmultistreamtiler", "tiler", {"rows": 1, "columns": 1})
    pipeline.add("nvosdbin", "osd")

    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    pipeline.add(sink_type, "sink")

    # Link elements
    pipeline.link(("src", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "infer", "tiler", "osd", "sink")

    # Create and apply performance monitor
    perf_monitor = utils.PerfMonitor(
        batch_size=1,
        interval=1,  # Report every second
        source_type="nvurisrcbin",
        show_name=True
    )

    # Apply to tiler's sink pad
    perf_monitor.apply(pipeline["tiler"], "sink")

    # Start pipeline
    pipeline.start().wait()

# Run with FPS monitoring
pipeline_with_fps_monitoring(
    "file:///path/to/video.mp4",
    "/path/to/config.yml"
)
```

**Output Example**:
```
**PERF: FPS 0 (Avg) 29.87
**PERF: FPS 0 (Avg) 30.02
**PERF: FPS 0 (Avg) 29.95
```

### Pattern 2: Multi-Stream FPS Monitoring

Monitor FPS for multiple streams with names.

```python
from pyservicemaker import Pipeline, utils
import platform

def multi_stream_fps_monitoring(stream_uris, config_path):
    """Monitor FPS for multiple streams"""
    pipeline = Pipeline("multi-stream-fps")

    # Add sources
    for i, uri in enumerate(stream_uris):
        pipeline.add("nvurisrcbin", f"src{i}", {"uri": uri})

    # Add muxer
    pipeline.add("nvstreammux", "mux", {
        "batch-size": len(stream_uris),
        "width": 1920,
        "height": 1080
    })

    # Add processing
    pipeline.add("nvinfer", "infer", {"config-file-path": config_path})
    pipeline.add("nvmultistreamtiler", "tiler", {
        "rows": 2,
        "columns": 2,
        "width": 1920,
        "height": 1080
    })
    pipeline.add("nvosdbin", "osd")

    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    pipeline.add(sink_type, "sink")

    # Link sources
    for i in range(len(stream_uris)):
        pipeline.link((f"src{i}", "mux"), ("", "sink_%u"))

    pipeline.link("mux", "infer", "tiler", "osd", "sink")

    # Create performance monitor
    perf_monitor = utils.PerfMonitor(
        batch_size=len(stream_uris),
        interval=2,  # Report every 2 seconds
        source_type="nvurisrcbin",
        show_name=True  # Show stream names
    )

    # Apply monitor
    perf_monitor.apply(pipeline["tiler"], "sink")

    # Start pipeline
    pipeline.start().wait()

# Monitor 4 streams
streams = [
    "file:///path/to/video1.mp4",
    "file:///path/to/video2.mp4",
    "rtsp://camera1/stream",
    "rtsp://camera2/stream"
]
multi_stream_fps_monitoring(streams, "/path/to/config.yml")
```

**Output Example**:
```
**PERF: FPS 0 (Avg) 29.87
**PERF: FPS 1 (Avg) 29.92
**PERF: FPS 2 (Avg) 30.15
**PERF: FPS 3 (Avg) 29.78
```

### Pattern 3: Dynamic Source Monitoring

Monitor performance with dynamically added/removed sources.

```python
from pyservicemaker import (
    Pipeline, PipelineState, StateTransitionMessage,
    DynamicSourceMessage, utils, SensorInfo
)

def dynamic_source_fps_monitoring(initial_sources, config_path):
    """Monitor FPS with dynamic source addition/removal"""
    pipeline = Pipeline("dynamic-fps-monitoring", config_file=config_path)

    # Sensor map to track sources
    sensor_map = {}

    # Initialize with static sources
    for i, source in enumerate(initial_sources):
        sensor_map[i] = SensorInfo(
            sensor_id=f"sensor_{i}",
            sensor_name=f"Camera {i}",
            uri=source
        )

    # Create performance monitor
    perf_monitor = utils.PerfMonitor(
        batch_size=len(initial_sources),
        interval=1,
        source_type="nvmultiurisrcbin",
        show_name=True
    )

    # Apply to tiler
    perf_monitor.apply(pipeline["tiler"], "sink")

    # Message handler for dynamic sources
    def on_message(message):
        if isinstance(message, DynamicSourceMessage):
            source_id = message.source_id

            if message.source_added:
                # Add new stream to monitoring
                sensor_map[source_id] = SensorInfo(
                    sensor_id=message.sensor_id,
                    sensor_name=message.sensor_name,
                    uri=message.uri
                )

                perf_monitor.add_stream(
                    source_id=source_id,
                    sensor_id=message.sensor_id,
                    sensor_name=message.sensor_name,
                    uri=message.uri
                )

                print(f"Added stream {source_id}: {message.sensor_name}")
            else:
                # Remove stream from monitoring
                if source_id in sensor_map:
                    del sensor_map[source_id]

                perf_monitor.remove_stream(source_id)
                print(f"Removed stream {source_id}")

    # Prepare pipeline with message handler
    pipeline.prepare(on_message)

    # Start pipeline
    pipeline.activate()
    pipeline.wait()

# Start with 2 sources (more can be added dynamically via API)
initial = [
    "file:///path/to/video1.mp4",
    "file:///path/to/video2.mp4"
]
dynamic_source_fps_monitoring(initial, "/path/to/config.yml")
```

### Pattern 4: Performance Monitoring with Pause/Resume

Control monitoring based on pipeline state.

```python
from pyservicemaker import Pipeline, utils
import time
import threading

def controlled_fps_monitoring(video_uri, config_path):
    """FPS monitoring with pause/resume control"""
    pipeline = Pipeline("controlled-monitoring")

    # Build pipeline
    pipeline.add("nvurisrcbin", "src", {"uri": video_uri})
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})
    pipeline.add("nvinfer", "infer", {"config-file-path": config_path})
    pipeline.add("nvmultistreamtiler", "tiler", {"rows": 1, "columns": 1})
    pipeline.add("nvosdbin", "osd")
    pipeline.add("nveglglessink", "sink")

    pipeline.link(("src", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "infer", "tiler", "osd", "sink")

    # Create performance monitor
    perf_monitor = utils.PerfMonitor(
        batch_size=1,
        interval=1,
        source_type="nvurisrcbin"
    )
    perf_monitor.apply(pipeline["tiler"], "sink")

    # Control thread
    def control_monitoring():
        time.sleep(10)
        print("Pausing monitoring...")
        perf_monitor.pause()

        time.sleep(5)
        print("Resuming monitoring...")
        perf_monitor.resume()

    control_thread = threading.Thread(target=control_monitoring, daemon=True)
    control_thread.start()

    # Start pipeline
    pipeline.start().wait()

controlled_fps_monitoring("file:///path/to/video.mp4", "/path/to/config.yml")
```

### Pattern 5: Model Engine Hot-Swapping

Monitor and automatically reload updated model engine files.

```python
from pyservicemaker import Pipeline, PipelineState, StateTransitionMessage, utils
import platform

def pipeline_with_otf_model_update(video_uri, config_path):
    """Pipeline with On-The-Fly model engine updates"""
    pipeline = Pipeline("otf-model-update")

    # Build pipeline
    pipeline.add("nvurisrcbin", "src", {"uri": video_uri})
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})
    pipeline.add("nvinfer", "pgie", {"config-file-path": config_path})
    pipeline.add("nvosdbin", "osd")

    sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"
    pipeline.add(sink_type, "sink")

    pipeline.link(("src", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "pgie", "osd", "sink")

    # Get engine file path from nvinfer element
    engine_file = pipeline["pgie"].get("model-engine-file")

    # Create engine file monitor
    model_engine_monitor = utils.EngineFileMonitor(
        pipeline["pgie"],
        engine_file
    )

    # Message handler to start monitor when pipeline is ready
    def on_message(message):
        if isinstance(message, StateTransitionMessage):
            if message.new_state == PipelineState.PLAYING and message.origin == "sink":
                if not model_engine_monitor.started:
                    print("Starting model engine monitor...")
                    model_engine_monitor.start()

    pipeline.prepare(on_message)

    # Start pipeline
    pipeline.activate()
    pipeline.wait()

# Pipeline will automatically reload model when engine file changes
pipeline_with_otf_model_update(
    "file:///path/to/video.mp4",
    "/path/to/pgie_config.yml"
)
```

### Pattern 6: Combined Performance and Model Monitoring

Use both utilities together for production deployment. This pattern also uses `SourceConfig` and `SensorInfo` (see Part 2 below for details on those classes).

```python
from pyservicemaker import (
    Pipeline, PipelineState, StateTransitionMessage,
    DynamicSourceMessage, utils, SensorInfo, SourceConfig
)
import platform

def production_pipeline_monitoring(source_config_file, pipeline_config_file):
    """Production pipeline with full monitoring"""
    # Load configuration
    source_config = SourceConfig()
    source_config.load(source_config_file)

    # Create pipeline
    pipeline = Pipeline("production-pipeline", config_file=pipeline_config_file)

    # Sensor map
    sensor_map = {}
    for i, sensor in enumerate(source_config.sensor_list):
        sensor_map[i] = sensor

    # Create performance monitor
    perf_monitor = utils.PerfMonitor(
        batch_size=len(source_config.sensor_list),
        interval=5,  # Report every 5 seconds
        source_type=source_config.source_type,
        show_name=True
    )
    perf_monitor.apply(pipeline["tiler"], "sink")

    # Create model engine monitor
    engine_file = pipeline["pgie"].get("model-engine-file")
    model_engine_monitor = utils.EngineFileMonitor(
        pipeline["pgie"],
        engine_file
    )

    # Message handler
    def on_message(message):
        if isinstance(message, StateTransitionMessage):
            if message.new_state == PipelineState.PLAYING and message.origin == "sink":
                # Start monitors when pipeline is playing
                if not model_engine_monitor.started:
                    model_engine_monitor.start()
                    print("Model engine monitoring started")

        elif isinstance(message, DynamicSourceMessage):
            source_id = message.source_id

            if message.source_added:
                sensor_map[source_id] = SensorInfo(
                    sensor_id=message.sensor_id,
                    sensor_name=message.sensor_name,
                    uri=message.uri
                )
                perf_monitor.add_stream(
                    source_id=source_id,
                    sensor_id=message.sensor_id,
                    sensor_name=message.sensor_name,
                    uri=message.uri
                )
                print(f"Stream added: {message.sensor_name}")
            else:
                if source_id in sensor_map:
                    del sensor_map[source_id]
                perf_monitor.remove_stream(source_id)
                print(f"Stream removed: {source_id}")

    pipeline.prepare(on_message)

    # Start pipeline
    pipeline.activate()
    pipeline.wait()

# Run production pipeline
production_pipeline_monitoring(
    "source_config.yaml",
    "pipeline_config.yaml"
)
```

### Pattern 7: Custom FPS Logging

Capture FPS data for custom analysis.

```python
from pyservicemaker import Pipeline, Probe, BatchMetadataOperator, utils
import time
import json

class FPSLogger(BatchMetadataOperator):
    """Custom FPS logger"""
    def __init__(self, log_file="fps_log.json"):
        super().__init__()
        self.log_file = log_file
        self.frame_count = 0
        self.start_time = time.time()
        self.last_log_time = self.start_time
        self.fps_data = []

    def handle_metadata(self, batch_meta):
        self.frame_count += len(batch_meta.frame_items)

        current_time = time.time()
        elapsed = current_time - self.last_log_time

        if elapsed >= 1.0:  # Log every second
            fps = self.frame_count / elapsed

            log_entry = {
                "timestamp": current_time,
                "fps": fps,
                "total_frames": self.frame_count,
                "elapsed_total": current_time - self.start_time
            }

            self.fps_data.append(log_entry)
            print(f"FPS: {fps:.2f}")

            # Save to file
            with open(self.log_file, 'w') as f:
                json.dump(self.fps_data, f, indent=2)

            self.frame_count = 0
            self.last_log_time = current_time

def pipeline_with_custom_fps_logging(video_uri, config_path):
    """Pipeline with custom FPS logging"""
    pipeline = Pipeline("custom-fps-logging")

    # Build pipeline
    pipeline.add("nvurisrcbin", "src", {"uri": video_uri})
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})
    pipeline.add("nvinfer", "infer", {"config-file-path": config_path})
    pipeline.add("nvosdbin", "osd")
    pipeline.add("nveglglessink", "sink")

    pipeline.link(("src", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "infer", "osd", "sink")

    # Attach custom FPS logger
    from pyservicemaker import Probe
    fps_logger = FPSLogger("custom_fps_log.json")
    pipeline.attach("infer", Probe("fps_logger", fps_logger))

    # Also use built-in performance monitor
    perf_monitor = utils.PerfMonitor(
        batch_size=1,
        interval=1,
        source_type="nvurisrcbin"
    )
    perf_monitor.apply(pipeline["osd"], "sink")

    pipeline.start().wait()

pipeline_with_custom_fps_logging("file:///path/to/video.mp4", "/path/to/config.yml")
```

## Performance Monitoring Best Practices

### 1. Choose Appropriate Monitoring Interval
```python
# For real-time monitoring
perf_monitor = utils.PerfMonitor(batch_size=4, interval=1)

# For less frequent updates (production)
perf_monitor = utils.PerfMonitor(batch_size=4, interval=5)

# For detailed analysis
perf_monitor = utils.PerfMonitor(batch_size=4, interval=0.5)
```

### 2. Monitor at Appropriate Pipeline Point
```python
# Monitor after tiler (recommended for multi-stream)
perf_monitor.apply(pipeline["tiler"], "sink")

# Monitor at final sink
perf_monitor.apply(pipeline["sink"], "sink")

# Monitor after inference
perf_monitor.apply(pipeline["infer"], "src")
```

### 3. Start Engine Monitor After Pipeline is Ready
```python
def on_message(message):
    if isinstance(message, StateTransitionMessage):
        if message.new_state == PipelineState.PLAYING:
            if not model_engine_monitor.started:
                model_engine_monitor.start()
```

### 4. Keep References to Monitors
```python
# Store monitors to prevent garbage collection
reference_holders = []
reference_holders.append(perf_monitor)
reference_holders.append(model_engine_monitor)
```

### 5. Handle Dynamic Sources Properly
```python
# Add stream
perf_monitor.add_stream(
    source_id=source_id,
    sensor_id=sensor_id,
    sensor_name=sensor_name,
    uri=uri
)

# Remove stream
perf_monitor.remove_stream(source_id)
```

## Performance Tips

### 1. Monitoring Overhead
- Performance monitoring has minimal overhead (~0.1% CPU)
- Use longer intervals (5-10 seconds) for production
- Disable `show_name` if not needed to reduce string operations

### 2. Engine File Monitoring
- Engine monitor uses inotify (Linux) for efficient file watching
- Minimal overhead when file doesn't change
- Automatic reload triggers brief inference pause

### 3. Multi-Stream Monitoring
- Per-stream FPS tracking has negligible overhead
- Batch size should match actual number of streams
- Update batch size when adding/removing dynamic sources

## Performance Monitoring Common Use Cases

### 1. Production Deployment Monitoring
Monitor FPS and model updates in production systems.

### 2. Performance Benchmarking
Measure and log FPS for different configurations.

### 3. Dynamic Stream Management
Track performance as streams are added/removed.

### 4. Model A/B Testing
Monitor performance during model hot-swapping.

### 5. Quality of Service (QoS) Monitoring
Ensure FPS meets SLA requirements.

### 6. Resource Utilization Analysis
Correlate FPS with system resource usage.

## Performance Monitoring Troubleshooting

### Issue 1: No FPS Output
**Solution**: Ensure monitor is applied to correct element and pad, verify pipeline is running

### Issue 2: Incorrect FPS Values
**Solution**: Check batch_size matches actual number of streams, verify monitoring point

### Issue 3: Engine Monitor Not Triggering
**Solution**: Ensure monitor is started after pipeline is PLAYING, verify file path is correct

### Issue 4: Memory Leak with Dynamic Sources
**Solution**: Always call `remove_stream()` when removing sources, keep references to monitors

## Performance Monitoring Summary

The performance monitoring utilities provide essential capabilities for production DeepStream applications:

1. **PerfMonitor**: Real-time FPS tracking and throughput measurement
   - Per-stream FPS monitoring
   - Dynamic source support
   - Pause/resume capability
   - Minimal overhead

2. **EngineFileMonitor**: Automatic model engine hot-swapping
   - File change detection
   - Automatic inference engine reload
   - Zero-downtime model updates
   - Production-ready OTF updates

Key features:
- Real-time performance metrics
- Multi-stream support
- Dynamic source tracking
- Model hot-swapping
- Production deployment ready
- Minimal performance overhead

These utilities are essential for monitoring, debugging, and maintaining DeepStream applications in production environments.

---

# Part 2: Configuration and Helper Classes

The `pyservicemaker` module provides several configuration and helper classes that simplify DeepStream application development. These classes handle:
- Source configuration management (video streams, cameras)
- Smart recording configuration
- Custom postprocessing interfaces
- Common factory patterns
- Signal handling and events

## Core Classes

### SourceConfig

A configuration manager for video sources and cameras.

**Constructor**:
```python
from pyservicemaker import SourceConfig

source_config = SourceConfig()
```

**Properties**:
- `sensor_list`: List of `SensorInfo` objects (for URI-based sources)
- `camera_list`: List of `CameraInfo` objects (for physical cameras)
- `source_type`: Type of source bin (e.g., "nvurisrcbin", "nvmultiurisrcbin", "camerabin")
- `source_properties`: Dictionary of source properties

**Methods**:

#### `load(config_file)`
Load source configuration from a YAML file.

**Parameters**:
- `config_file` (str): Path to YAML configuration file

**Example**:
```python
from pyservicemaker import SourceConfig

config = SourceConfig()
config.load("source_config.yaml")

print(f"Source type: {config.source_type}")
print(f"Number of sensors: {len(config.sensor_list)}")

for sensor in config.sensor_list:
    print(f"  Sensor ID: {sensor.sensor_id}")
    print(f"  Name: {sensor.sensor_name}")
    print(f"  URI: {sensor.uri}")
```

**YAML Configuration Format**:

```yaml
# For URI-based sources (files, RTSP streams)
source-list:
  - uri: "file:///path/to/video1.mp4"
    sensor-id: "sensor-001"
    sensor-name: "Camera 1"

  - uri: "rtsp://192.168.1.100/stream"
    sensor-id: "sensor-002"
    sensor-name: "Camera 2"

source-config:
  source-bin: "nvurisrcbin"
  properties:
    gpu-id: 0
    cudadec-memtype: 0

# For physical cameras (CSI, V4L2)
camera-list:
  - camera-type: "CSI"
    camera-video-format: "NV12"
    camera-width: 1920
    camera-height: 1080
    camera-fps-n: 30
    camera-fps-d: 1
    camera-csi-sensor-id: 0
    gpu-id: 0
    nvbuf-mem-type: 0

  - camera-type: "V4L2"
    camera-video-format: "NV12"
    camera-width: 1280
    camera-height: 720
    camera-fps-n: 30
    camera-fps-d: 1
    camera-v4l2-dev-node: 0
    gpu-id: 0
    nvbuf-mem-type: 0
    nvvideoconvert-copy-hw: 0
```

### SensorInfo

Named tuple containing sensor information for URI-based sources.

**Fields**:
- `sensor_id` (str): Unique sensor identifier
- `sensor_name` (str): Human-readable sensor name
- `uri` (str): Video source URI

**Example**:
```python
from pyservicemaker import SensorInfo

sensor = SensorInfo(
    sensor_id="cam-001",
    sensor_name="Front Door Camera",
    uri="rtsp://192.168.1.100/stream"
)

print(f"ID: {sensor.sensor_id}")
print(f"Name: {sensor.sensor_name}")
print(f"URI: {sensor.uri}")
```

### CameraInfo

Named tuple containing camera configuration for physical cameras.

**Fields**:
- `camera_type` (str): Camera type ("CSI" or "V4L2")
- `camera_video_format` (str): Video format (e.g., "NV12", "RGB")
- `camera_width` (int): Frame width in pixels
- `camera_height` (int): Frame height in pixels
- `camera_fps_n` (int): Frame rate numerator
- `camera_fps_d` (int): Frame rate denominator
- `camera_csi_sensor_id` (int): CSI sensor ID (for CSI cameras)
- `camera_v4l2_dev_node` (int): V4L2 device node (for V4L2 cameras)
- `gpu_id` (int): GPU ID to use
- `nvbuf_mem_type` (int): Buffer memory type
- `nvvideoconvert_copy_hw` (int): Hardware copy mode

**Example**:
```python
from pyservicemaker import CameraInfo

# CSI camera configuration
csi_camera = CameraInfo(
    camera_type="CSI",
    camera_video_format="NV12",
    camera_width=1920,
    camera_height=1080,
    camera_fps_n=30,
    camera_fps_d=1,
    camera_csi_sensor_id=0,
    camera_v4l2_dev_node=None,
    gpu_id=0,
    nvbuf_mem_type=0,
    nvvideoconvert_copy_hw=0
)
```

### SmartRecordConfig

Configuration dataclass for smart recording functionality.

**Constructor**:
```python
from pyservicemaker import SmartRecordConfig

config = SmartRecordConfig(
    proto_lib="/path/to/libnvds_kafka_proto.so",
    conn_str="localhost;9092",
    msgconv_config_file="/path/to/msgconv_config.txt",
    proto_config_file="/path/to/proto_config.txt",
    topic_list="smart-recording-events",
    smart_rec_cache=30,
    smart_rec_container=0,
    smart_rec_dir_path="./recordings",
    smart_rec_mode=0
)
```

**Required Parameters**:
- `proto_lib` (str): Path to protocol library (e.g., Kafka protocol library)
- `conn_str` (str): Connection string for message broker (e.g., "localhost;9092")
- `msgconv_config_file` (str): Path to message converter configuration file
- `proto_config_file` (str): Path to protocol configuration file
- `topic_list` (str): Comma-separated list of topics for message publishing

**Optional Parameters**:
- `smart_rec_cache` (int): Cache size in seconds (default: 20, range: 0-4294967295)
- `smart_rec_container` (int): Container format (0=MP4, 1=MKV, default: 0)
- `smart_rec_dir_path` (str): Directory to save recordings (default: ".")
- `smart_rec_mode` (int): Recording mode (0=audio+video, 1=video only, 2=audio only, default: 0)

**Example**:
```python
from pyservicemaker import SmartRecordConfig

# Create smart recording configuration
sr_config = SmartRecordConfig(
    proto_lib="/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
    conn_str="localhost;9092",
    msgconv_config_file="/opt/nvidia/deepstream/deepstream/sources/libs/kafka_protocol_adaptor/cfg_kafka.txt",
    proto_config_file="/opt/nvidia/deepstream/deepstream/sources/libs/kafka_protocol_adaptor/cfg_kafka.txt",
    topic_list="sr-events",
    smart_rec_cache=30,      # 30 seconds cache
    smart_rec_container=0,   # MP4 format
    smart_rec_dir_path="./recordings",
    smart_rec_mode=0         # Record audio and video
)
```

### PostProcessing (Abstract Base Class)

Base class for custom tensor output postprocessing.

**Abstract Method**:

#### `__call__(output_layers)`
Convert output tensors to real-world representation.

**Parameters**:
- `output_layers` (Dict): Dictionary of (layer_name, tensor) pairs

**Returns**: Any (depends on implementation)

**Example**:
```python
from pyservicemaker import postprocessing
import torch

class CustomPostProcessing(postprocessing.PostProcessing):
    def __call__(self, output_layers):
        # Extract tensors
        output = output_layers.get('output_layer')

        if output:
            # Convert to PyTorch
            torch_tensor = torch.utils.dlpack.from_dlpack(output)

            # Custom processing
            result = self.process(torch_tensor)
            return result

        return None

    def process(self, tensor):
        # Your custom processing logic
        return tensor.cpu().numpy()
```

### ObjectDetectorOutputConverter (Abstract Base Class)

Specialized base class for object detection postprocessing.

**Abstract Method**:

#### `__call__(output_layers)`
Convert output tensors to object detection results.

**Parameters**:
- `output_layers` (Dict): Dictionary of (layer_name, tensor) pairs

**Returns**: List of bounding boxes in format `[class_id, confidence, x1, y1, x2, y2]`

**Example**:
```python
from pyservicemaker import postprocessing
import torch
import torchvision.ops as ops

class YOLOv5Converter(postprocessing.ObjectDetectorOutputConverter):
    def __init__(self, conf_threshold=0.5, nms_threshold=0.4):
        self.conf_threshold = conf_threshold
        self.nms_threshold = nms_threshold

    def __call__(self, output_layers):
        outputs = []

        # Extract output tensor
        predictions = output_layers.get('output')
        if predictions is None:
            return outputs

        # Convert to PyTorch
        pred_tensor = torch.utils.dlpack.from_dlpack(predictions).cpu()

        # Process predictions
        # pred_tensor shape: [batch, num_boxes, 85] (for COCO)
        # Format: [x, y, w, h, obj_conf, class_conf...]

        for detection in pred_tensor[0]:  # Assuming batch size 1
            obj_conf = detection[4]

            if obj_conf < self.conf_threshold:
                continue

            # Get class with highest confidence
            class_confs = detection[5:]
            class_id = torch.argmax(class_confs).item()
            class_conf = class_confs[class_id].item()

            confidence = obj_conf * class_conf

            if confidence < self.conf_threshold:
                continue

            # Convert center format to corner format
            x_center, y_center, width, height = detection[:4]
            x1 = (x_center - width / 2).item()
            y1 = (y_center - height / 2).item()
            x2 = (x_center + width / 2).item()
            y2 = (y_center + height / 2).item()

            outputs.append([class_id, confidence, x1, y1, x2, y2])

        # Apply NMS
        if outputs:
            boxes = torch.tensor([[o[2], o[3], o[4], o[5]] for o in outputs])
            scores = torch.tensor([o[1] for o in outputs])
            keep = ops.nms(boxes, scores, self.nms_threshold)
            outputs = [outputs[i] for i in keep]

        return outputs
```

### CommonFactory

Factory class for creating custom objects and plugins.

**Method**:

#### `create(factory_name, instance_name)`
Create an instance from a registered factory.

**Parameters**:
- `factory_name` (str): Name of the factory (e.g., "smart_recording_action")
- `instance_name` (str): Name for the created instance

**Returns**: Created object instance

**Example**:
```python
from pyservicemaker import CommonFactory

# Create smart recording controller
sr_controller = CommonFactory.create("smart_recording_action", "sr_controller")

# Configure the controller
if sr_controller:
    sr_controller.set({
        "proto-lib": "/path/to/libnvds_kafka_proto.so",
        "conn-str": "localhost;9092",
        "msgconv-config-file": "/path/to/msgconv_config.txt",
        "proto-config-file": "/path/to/proto_config.txt",
        "topic-list": "sr-events"
    })
```

## Configuration and Helper Usage Patterns

### Pattern 1: Load and Use Source Configuration

Load source configuration from YAML and build pipeline.

```python
from pyservicemaker import Pipeline, SourceConfig
import platform

def pipeline_from_source_config(source_config_file, pgie_config):
    """Build pipeline from source configuration file"""
    # Load source configuration
    source_config = SourceConfig()
    source_config.load(source_config_file)

    # Create pipeline
    pipeline = Pipeline("configured-pipeline")

    # Add sources based on configuration
    if source_config.source_type == "nvmultiurisrcbin":
        # Multi-URI source bin
        uri_list = ','.join([s.uri for s in source_config.sensor_list])
        sensor_id_list = ','.join([s.sensor_id for s in source_config.sensor_list])
        sensor_name_list = ','.join([s.sensor_name for s in source_config.sensor_list])

        properties = dict(source_config.source_properties)
        properties.update({
            "uri-list": uri_list,
            "sensor-id-list": sensor_id_list,
            "sensor-name-list": sensor_name_list
        })

        pipeline.add("nvmultiurisrcbin", "source", properties)
        pipeline.add("nvinfer", "pgie", {"config-file-path": pgie_config})
        pipeline.link("source", "pgie")

    elif source_config.source_type == "camerabin":
        # Physical cameras
        pipeline.add("nvstreammux", "mux", {
            "batch-size": len(source_config.camera_list),
            "width": 1920,
            "height": 1080,
            "live-source": 1
        })

        for i, camera in enumerate(source_config.camera_list):
            src_name = f"src_{i}"

            if camera.camera_type == "CSI":
                pipeline.add("nvarguscamerasrc" if platform.processor() == "aarch64" else "videotestsrc",
                           src_name, {"sensor-id": camera.camera_csi_sensor_id})
            elif camera.camera_type == "V4L2":
                device = f"/dev/video{camera.camera_v4l2_dev_node}"
                pipeline.add("v4l2src", src_name, {"device": device})

            pipeline.link((src_name, "mux"), ("", "sink_%u"))

        pipeline.add("nvinfer", "pgie", {"config-file-path": pgie_config})
        pipeline.link("mux", "pgie")

    else:
        # Individual URI sources
        pipeline.add("nvstreammux", "mux", {
            "batch-size": len(source_config.sensor_list),
            "width": 1920,
            "height": 1080
        })

        for i, sensor in enumerate(source_config.sensor_list):
            src_name = f"src_{i}"
            properties = dict(source_config.source_properties)
            properties["uri"] = sensor.uri

            pipeline.add(source_config.source_type, src_name, properties)
            pipeline.link((src_name, "mux"), ("", "sink_%u"))

        pipeline.add("nvinfer", "pgie", {"config-file-path": pgie_config})
        pipeline.link("mux", "pgie")

    # Add remaining elements
    pipeline.add("nvosdbin", "osd")
    pipeline.add("nveglglessink", "sink")
    pipeline.link("pgie", "osd", "sink")

    # Start pipeline
    pipeline.start().wait()

# Use configuration file
pipeline_from_source_config("sources.yaml", "pgie_config.yml")
```

### Pattern 2: Smart Recording with Configuration

Set up smart recording using SmartRecordConfig.

```python
from pyservicemaker import Pipeline, Flow, SmartRecordConfig

def pipeline_with_smart_recording(video_uris, pgie_config):
    """Pipeline with smart recording enabled"""
    # Create smart recording configuration
    sr_config = SmartRecordConfig(
        proto_lib="/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
        conn_str="localhost;9092",
        msgconv_config_file="/opt/nvidia/deepstream/deepstream/sources/libs/kafka_protocol_adaptor/cfg_kafka.txt",
        proto_config_file="/opt/nvidia/deepstream/deepstream/sources/libs/kafka_protocol_adaptor/cfg_kafka.txt",
        topic_list="sr-events",
        smart_rec_cache=30,
        smart_rec_container=0,  # MP4
        smart_rec_dir_path="./recordings",
        smart_rec_mode=0  # Audio + Video
    )

    # Create pipeline with Flow API
    pipeline = Pipeline("smart-recording-pipeline")
    flow = Flow(pipeline)

    # Build pipeline with smart recording
    flow.batch_capture(video_uris)
    flow.infer(pgie_config)
    flow.smart_record(sr_config)  # Enable smart recording
    flow.render()

    # Execute
    flow()

# Run with smart recording
video_sources = [
    "rtsp://192.168.1.100/stream",
    "rtsp://192.168.1.101/stream"
]
pipeline_with_smart_recording(video_sources, "pgie_config.yml")
```

### Pattern 3: Custom Postprocessing

Implement custom postprocessing for inference outputs.

```python
from pyservicemaker import Pipeline, Probe, BatchMetadataOperator, postprocessing
import torch

class CustomDetectorConverter(postprocessing.ObjectDetectorOutputConverter):
    """Custom object detector postprocessing"""
    def __init__(self, threshold=0.5):
        self.threshold = threshold

    def __call__(self, output_layers):
        outputs = []

        # Extract your model's output tensors
        bbox_layer = output_layers.get('bboxes')
        conf_layer = output_layers.get('confidences')
        class_layer = output_layers.get('classes')

        if not all([bbox_layer, conf_layer, class_layer]):
            return outputs

        # Convert to PyTorch
        bboxes = torch.utils.dlpack.from_dlpack(bbox_layer).cpu()
        confs = torch.utils.dlpack.from_dlpack(conf_layer).cpu()
        classes = torch.utils.dlpack.from_dlpack(class_layer).cpu()

        # Process detections
        for bbox, conf, cls in zip(bboxes, confs, classes):
            if conf > self.threshold:
                x1, y1, x2, y2 = bbox
                outputs.append([
                    int(cls),
                    float(conf),
                    float(x1), float(y1),
                    float(x2), float(y2)
                ])

        return outputs

class CustomPostprocessor(BatchMetadataOperator):
    """Apply custom postprocessing to inference results"""
    def __init__(self):
        super().__init__()
        self.converter = CustomDetectorConverter(threshold=0.6)

    def handle_metadata(self, batch_meta):
        for frame_meta in batch_meta.frame_items:
            # Process tensor outputs
            for tensor_meta in frame_meta.tensor_items:
                output_layers = tensor_meta.as_tensor_output().get_layers()
                detections = self.converter(output_layers)

                # Create object metadata from detections
                for det in detections:
                    obj_meta = batch_meta.acquire_object_meta()
                    obj_meta.class_id = det[0]
                    obj_meta.confidence = det[1]
                    obj_meta.rect_params.left = det[2]
                    obj_meta.rect_params.top = det[3]
                    obj_meta.rect_params.width = det[4] - det[2]
                    obj_meta.rect_params.height = det[5] - det[3]
                    frame_meta.append(obj_meta)

def pipeline_with_custom_postprocessing(video_uri, config_path):
    """Pipeline with custom postprocessing"""
    pipeline = Pipeline("custom-postproc")

    # Build pipeline
    pipeline.add("nvurisrcbin", "src", {"uri": video_uri})
    pipeline.add("nvstreammux", "mux", {"batch-size": 1, "width": 1920, "height": 1080})

    # Enable tensor output
    pipeline.add("nvinfer", "infer", {
        "config-file-path": config_path,
        "output-tensor-meta": 1  # Enable tensor output
    })

    pipeline.add("nvosdbin", "osd")
    pipeline.add("nveglglessink", "sink")

    pipeline.link(("src", "mux"), ("", "sink_%u"))
    pipeline.link("mux", "infer", "osd", "sink")

    # Attach custom postprocessor
    pipeline.attach("infer", Probe("custom-postproc", CustomPostprocessor()))

    pipeline.start().wait()

pipeline_with_custom_postprocessing("file:///path/to/video.mp4", "config.yml")
```

### Pattern 4: Dynamic Sensor Management

Manage sensors dynamically using SensorInfo. For combining this with performance monitoring, see Part 1 above (Pattern 3: Dynamic Source Monitoring).

```python
from pyservicemaker import Pipeline, SensorInfo, DynamicSourceMessage
import time
import threading

def dynamic_sensor_management():
    """Manage sensors dynamically"""
    pipeline = Pipeline("dynamic-sensors", config_file="pipeline_config.yml")

    # Sensor registry
    active_sensors = {}

    def on_message(message):
        if isinstance(message, DynamicSourceMessage):
            source_id = message.source_id

            if message.source_added:
                # Register new sensor
                sensor = SensorInfo(
                    sensor_id=message.sensor_id,
                    sensor_name=message.sensor_name,
                    uri=message.uri
                )
                active_sensors[source_id] = sensor
                print(f"Added sensor: {sensor.sensor_name} ({sensor.sensor_id})")
            else:
                # Unregister sensor
                if source_id in active_sensors:
                    sensor = active_sensors[source_id]
                    print(f"Removed sensor: {sensor.sensor_name}")
                    del active_sensors[source_id]

    pipeline.prepare(on_message)
    pipeline.activate()
    pipeline.wait()

dynamic_sensor_management()
```

### Pattern 5: Factory-Based Plugin Creation

Use CommonFactory to create custom plugins.

```python
from pyservicemaker import Pipeline, CommonFactory, signal

def pipeline_with_factory_plugins(video_uris, config_path):
    """Pipeline using factory-created plugins"""
    pipeline = Pipeline("factory-pipeline")

    # Build pipeline
    pipeline.add("nvstreammux", "mux", {
        "batch-size": len(video_uris),
        "width": 1920,
        "height": 1080
    })

    for i, uri in enumerate(video_uris):
        pipeline.add("nvurisrcbin", f"src{i}", {"uri": uri})
        pipeline.link((f"src{i}", "mux"), ("", "sink_%u"))

    pipeline.add("nvinfer", "pgie", {"config-file-path": config_path})
    pipeline.add("nvmsgbroker", "msgbroker", {
        "proto-lib": "/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
        "conn-str": "localhost;9092",
        "topic": "analytics"
    })

    pipeline.link("mux", "pgie", "msgbroker")

    # Create smart recording controller using factory
    sr_controller = CommonFactory.create("smart_recording_action", "sr_controller")

    if sr_controller and isinstance(sr_controller, signal.Emitter):
        # Configure smart recording
        sr_controller.set({
            "proto-lib": "/opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so",
            "conn-str": "localhost;9092",
            "msgconv-config-file": "/path/to/msgconv_config.txt",
            "proto-config-file": "/path/to/proto_config.txt",
            "topic-list": "sr-events"
        })

        # Attach to sources
        for i in range(len(video_uris)):
            sr_controller.attach("start-sr", pipeline[f"src{i}"])
            sr_controller.attach("stop-sr", pipeline[f"src{i}"])
            pipeline.attach(f"src{i}", "smart_recording_signal", "sr", "sr-done")

    pipeline.start().wait()

video_sources = ["rtsp://cam1/stream", "rtsp://cam2/stream"]
pipeline_with_factory_plugins(video_sources, "pgie_config.yml")
```

## Configuration and Helper Best Practices

### 1. Use Configuration Files
```python
# Good: Externalize configuration
source_config = SourceConfig()
source_config.load("sources.yaml")

# Avoid: Hardcoding configuration
sensors = [
    SensorInfo("001", "Camera 1", "rtsp://..."),
    SensorInfo("002", "Camera 2", "rtsp://...")
]
```

### 2. Validate Configuration
```python
source_config = SourceConfig()
source_config.load("sources.yaml")

if not source_config.sensor_list:
    raise ValueError("No sensors configured")

if source_config.source_type not in ["nvurisrcbin", "nvmultiurisrcbin"]:
    raise ValueError(f"Unsupported source type: {source_config.source_type}")
```

### 3. Use Dataclasses for Configuration
```python
# Good: Use SmartRecordConfig dataclass
sr_config = SmartRecordConfig(
    proto_lib="/path/to/lib.so",
    conn_str="localhost;9092",
    # ... other parameters
)

# Avoid: Manual dictionary management
sr_config = {
    "proto-lib": "/path/to/lib.so",
    "conn-str": "localhost;9092",
    # ... other parameters
}
```

### 4. Implement Proper Postprocessing
```python
class MyConverter(postprocessing.ObjectDetectorOutputConverter):
    def __call__(self, output_layers):
        # Always return list of [class_id, conf, x1, y1, x2, y2]
        outputs = []

        # Process tensors
        # ...

        return outputs  # Return empty list if no detections
```

### 5. Handle Factory Creation Errors
```python
plugin = CommonFactory.create("plugin_name", "instance_name")

if plugin is None:
    print("Warning: Failed to create plugin")
    # Handle gracefully
else:
    # Use plugin
    plugin.set(properties)
```

## Related APIs

- **Pipeline API**: See `service_maker_api.md`
- **Flow API**: See `service_maker_api.md`
- **Postprocessing**: See `service_maker_api.md`
- **Smart Recording**: See `service_maker_api.md` and `kafka_messaging.md`

## Configuration and Helper Summary

The configuration and helper classes provide essential utilities for DeepStream application development:

1. **SourceConfig**: Manage video sources and cameras from YAML
2. **SensorInfo/CameraInfo**: Structured sensor and camera information
3. **SmartRecordConfig**: Configure smart recording functionality
4. **PostProcessing**: Base class for custom tensor postprocessing
5. **ObjectDetectorOutputConverter**: Specialized postprocessing for object detection
6. **CommonFactory**: Create custom plugins and objects

Key features:
- YAML-based configuration management
- Structured data classes for type safety
- Abstract base classes for custom implementations
- Factory pattern for plugin creation
- Smart recording configuration
- Flexible postprocessing framework

These utilities simplify configuration management, enable code reuse, and provide clean interfaces for extending DeepStream functionality.