# CrewAI — DataRobot OTel Integration

## Overview

CrewAI works with the standard `configure_otel()` pattern — it does NOT override the global TracerProvider. Use the generic `dr_otel_config.py` without modification.

Auto-instrumentation is available via `opentelemetry-instrumentation-crewai`, which creates spans for crew executions, agent tasks, tool calls, and LLM interactions.

## OTel Strategy

| Signal    | Strategy                                    |
|-----------|---------------------------------------------|
| **Traces**  | Standard setup + auto-instrumentor         |
| **Metrics** | Standard setup (optional custom callbacks) |
| **Logs**    | Standard setup                             |

## Setup

### 1. Use the generic `dr_otel_config.py` as-is

No modifications needed. Call `configure_otel()` at startup.

### 2. Add auto-instrumentor after `configure_otel()`

In the agent's entrypoint:

```python
from dr_otel_config import configure_otel

configure_otel()

# Auto-instrument CrewAI — must be called AFTER configure_otel()
from opentelemetry.instrumentation.crewai import CrewAIInstrumentor

CrewAIInstrumentor().instrument()

# Your crew code below...
```

The auto-instrumentor captures:
- Crew kickoff and execution
- Individual agent task execution
- Tool calls within tasks
- LLM interactions per agent

## Extra Dependencies

```
opentelemetry-instrumentation-crewai
```

## Custom Metrics (Optional)

For custom metrics beyond what the auto-instrumentor provides, use CrewAI's callback system:

```python
from opentelemetry import metrics

meter = metrics.get_meter("my-crewai-agent")
task_counter = meter.create_counter("agent.tasks.completed", unit="1")
crew_duration = meter.create_histogram("agent.crew.duration_ms", unit="ms")
```

Wire these into CrewAI's task callbacks or measure around `crew.kickoff()`.

## Known Pitfalls

| Issue | Cause | Fix |
|-------|-------|-----|
| No spans from CrewAI | Instrumentor not called | Ensure `CrewAIInstrumentor().instrument()` is called AFTER `configure_otel()` |
| Import error on instrumentor | Wrong package version | Ensure `opentelemetry-instrumentation-crewai` is compatible with your CrewAI version |

# Generic Python — DataRobot OTel Integration

## Overview

For agents built without a specific framework (plain Python, custom frameworks, or frameworks not listed in the supported set), use the standard `configure_otel()` pattern with manual span instrumentation.

The coding agent should analyze the user's code and add spans around key operations: agent request handling, LLM calls, tool invocations, and data retrieval.

## OTel Strategy

| Signal    | Strategy                                |
|-----------|-----------------------------------------|
| **Traces**  | Standard setup + manual span instrumentation |
| **Metrics** | Standard setup + manual metric recording |
| **Logs**    | Standard setup                         |

## Setup

### 1. Use the generic `dr_otel_config.py` as-is

No modifications needed. Call `configure_otel()` at startup.

### 2. Add manual span instrumentation

After calling `configure_otel()`, create a tracer and wrap key operations:

```python
from dr_otel_config import configure_otel

configure_otel()

from opentelemetry import trace

tracer = trace.get_tracer("my-agent")


def handle_request(user_message: str) -> str:
    with tracer.start_as_current_span("agent-request") as span:
        span.set_attribute("gen_ai.prompt", user_message)

        # LLM call
        with tracer.start_as_current_span("llm-call") as llm_span:
            llm_span.set_attribute("gen_ai.request.model", "gpt-4o")
            response = call_llm(user_message)
            llm_span.set_attribute("gen_ai.completion", response)
            llm_span.set_attribute("gen_ai.usage.prompt_tokens", token_count)
            llm_span.set_attribute("gen_ai.usage.completion_tokens", completion_tokens)

        # Tool call
        with tracer.start_as_current_span("tool-call") as tool_span:
            tool_span.set_attribute("tool_name", "search_database")
            tool_span.set_attribute("tool.parameters", '{"query": "..."}')
            result = search_database(query)

        span.set_attribute("gen_ai.completion", final_response)
        return final_response
```

### 3. Add custom metrics

```python
from opentelemetry import metrics
import time

meter = metrics.get_meter("my-agent")

request_counter = meter.create_counter(
    "agent.requests",
    unit="1",
    description="Total requests processed",
)
request_duration = meter.create_histogram(
    "agent.request.duration_ms",
    unit="ms",
    description="End-to-end request duration",
)
llm_call_counter = meter.create_counter(
    "agent.llm.calls",
    unit="1",
    description="Number of LLM API calls",
)
llm_duration = meter.create_histogram(
    "agent.llm.duration_ms",
    unit="ms",
    description="Individual LLM call duration",
)
tool_call_counter = meter.create_counter(
    "agent.tool.calls",
    unit="1",
    description="Number of tool invocations",
)


def handle_request(user_message: str) -> str:
    start = time.time()
    try:
        # ... agent logic with spans as above ...
        elapsed_ms = (time.time() - start) * 1000
        request_counter.add(1, {"status": "success"})
        request_duration.record(elapsed_ms)
        return response
    except Exception:
        elapsed_ms = (time.time() - start) * 1000
        request_counter.add(1, {"status": "error"})
        request_duration.record(elapsed_ms)
        raise
```

## Span Attributes for DataRobot Tracing

Use these attributes for data to appear in DataRobot's tracing table:

| Attribute | Description | DataRobot Column | Rule |
|-----------|-------------|------------------|------|
| `gen_ai.prompt` | User input / prompt text | Prompt | First span wins |
| `gen_ai.completion` | Model output / response | Completion | Last span wins |
| `tool_name` | Tool/function name | Tools | All unique values listed |
| `datarobot.moderation.cost` | Cost of this operation | Cost | Summed across trace |
| `gen_ai.request.model` | Model used | — | Informational |
| `gen_ai.usage.prompt_tokens` | Input token count | — | Informational |
| `gen_ai.usage.completion_tokens` | Output token count | — | Informational |
| `tool.parameters` | Tool call parameters (JSON) | — | Informational |

**Important:** Use `tool_name` (underscore), not `tool.name` (dot). DataRobot's tracing UI specifically looks for `tool_name`.

## Auto-Instrumenting Common SDKs

Even without a framework, you can auto-instrument the underlying LLM SDKs:

```python
# OpenAI
from opentelemetry.instrumentation.openai import OpenAIInstrumentor

OpenAIInstrumentor().instrument()

# Anthropic
from opentelemetry.instrumentation.anthropic import AnthropicInstrumentor

AnthropicInstrumentor().instrument()

# HTTP clients (catches all outbound API calls)
from opentelemetry.instrumentation.requests import RequestsInstrumentor

RequestsInstrumentor().instrument()

from opentelemetry.instrumentation.httpx import HTTPXInstrumentor

HTTPXInstrumentor().instrument()
```

## Extra Dependencies

None beyond the generic OTel packages.

Optional (for SDK auto-instrumentation):
```
opentelemetry-instrumentation-openai       # If using OpenAI SDK
opentelemetry-instrumentation-anthropic     # If using Anthropic SDK
opentelemetry-instrumentation-requests      # If using requests library
opentelemetry-instrumentation-httpx         # If using httpx library
```

## Guidance for the Coding Agent

When instrumenting a generic Python agent:

1. **Identify the request handler** — the function that receives user input and returns output. Wrap it in a root span (`agent-request`).
2. **Identify LLM calls** — any call to an LLM API (OpenAI, Anthropic, Vertex AI, etc.). Wrap each in a child span (`llm-call`) with `gen_ai.*` attributes.
3. **Identify tool calls** — any external operation (database, API, search, etc.). Wrap each in a child span (`tool-call`) with `tool.*` attributes.
4. **Add metrics** — at minimum, add request count and duration. Add LLM call count/duration and tool call count if identifiable.
5. **Don't over-instrument** — focus on the key operations. Not every function needs a span.

# Google ADK — DataRobot OTel Integration

## Critical: ADK Overwrites the Global TracerProvider

Google ADK's web server calls `_setup_telemetry()` at startup, which **replaces any TracerProvider set earlier**. This means the standard `configure_otel()` trace setup will be overwritten. Logs and metrics are NOT affected.

| Signal    | ADK Overrides? | Strategy                                                        |
|-----------|----------------|-----------------------------------------------------------------|
| **Traces**  | YES            | Lazy injection — add span processor to ADK's provider on first request |
| **Metrics** | No*            | Standard setup at import time with direct exporter config       |
| **Logs**    | No             | Standard setup at import time                                   |

*ADK will override MeterProvider if `OTEL_EXPORTER_OTLP_*` env vars are set. Never set these.

## Modified `dr_otel_config.py` for ADK

For ADK, the generated `dr_otel_config.py` must be modified from the generic pattern:

1. **Do NOT configure traces in `configure_otel()`** — traces will be lost when ADK replaces the TracerProvider
2. **Build `dr_span_processor` at module level** — this is injected lazily later
3. **Export `dr_span_processor`** so the metrics callback module can access it
4. **Call `configure_otel()` at module level** (import time), not deferred

```python
# In dr_otel_config.py — ADK variant
# ... (same imports as generic, plus:)


def configure_otel():
    """Configure logs and metrics only. Traces use lazy injection."""
    headers = _build_dr_headers()
    endpoint = _get_endpoint()
    if not endpoint:
        return
    resource = Resource.create()

    # Logs — ADK does NOT override LoggerProvider
    # (same as generic pattern)

    # Logs — attach a custom Formatter so OTLP log bodies are never empty
    # (ADK and third-party code can emit records with empty getMessage())
    log_exporter = OTLPLogExporter(endpoint=f"{endpoint}/v1/logs", headers=headers)
    logger_provider = LoggerProvider(resource=resource)
    set_logger_provider(logger_provider)
    logger_provider.add_log_record_processor(SimpleLogRecordProcessor(log_exporter))
    handler = LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider)
    handler.setFormatter(logging.Formatter("%(levelname)s %(name)s: %(message)s"))
    logging.getLogger().addHandler(handler)

    # Metrics — direct exporter config (no env vars!)
    # Use a shorter export_interval_millis for serverless/short-lived workers.
    # Default 60s may miss metrics on Agent Engine playground or single-turn requests.
    metric_exporter = OTLPMetricExporter(
        endpoint=f"{endpoint}/v1/metrics",
        headers=headers,
        preferred_temporality=preferred_temporality,
    )
    meter_provider = MeterProvider(
        metric_readers=[
            PeriodicExportingMetricReader(
                metric_exporter,
                export_interval_millis=5000,
            )
        ],
        resource=resource,
    )
    metrics.set_meter_provider(meter_provider)

    # NOTE: Traces are NOT configured here. See dr_span_processor below.


# Build trace processor at import time — injected lazily on first request.
# May be None if endpoint is not configured (e.g. local dev without DataRobot).
_ep = _get_endpoint()
if _ep:
    dr_span_processor: SimpleSpanProcessor | None = SimpleSpanProcessor(
        OTLPSpanExporter(
            endpoint=f"{_ep}/v1/traces",
            headers=_build_dr_headers(),
        )
    )
else:
    dr_span_processor = None

configure_otel()
```

## Custom Metrics Callback Module

Generate `dr_agent_metrics.py` with callbacks for ADK agent lifecycle events.

```python
"""DataRobot metrics instrumentation for ADK agent.

Callbacks for LlmAgent lifecycle events that record OTel metrics
and handle lazy trace injection into ADK's TracerProvider.
"""

import logging
import threading
import time

from opentelemetry import metrics, trace

_meter = metrics.get_meter("agent-name")  # Replace with actual agent name

request_counter = _meter.create_counter(
    "agent.requests",
    unit="1",
    description="Total requests processed by the agent",
)
request_duration = _meter.create_histogram(
    "agent.request.duration_ms",
    unit="ms",
    description="End-to-end agent request duration",
)
llm_call_counter = _meter.create_counter(
    "agent.llm.calls",
    unit="1",
    description="Number of LLM API calls made",
)
llm_duration = _meter.create_histogram(
    "agent.llm.duration_ms",
    unit="ms",
    description="Individual LLM call duration",
)
tool_call_counter = _meter.create_counter(
    "agent.tool.calls",
    unit="1",
    description="Number of tool invocations",
)

_t = threading.local()
_trace_injected = False
_inject_warned = False


def _resolve_tracer_provider_for_processor():
    """Resolve SDK TracerProvider; skip API ``ProxyTracerProvider``.

    ``trace.get_tracer_provider()`` often returns ``ProxyTracerProvider``, which does
    **not** implement ``add_span_processor``. The real SDK provider is on
    ``opentelemetry.trace._TRACER_PROVIDER`` once something (e.g. ADK) has called
    ``set_tracer_provider``.
    """
    import opentelemetry.trace as trace_module

    candidates = []
    internal = getattr(trace_module, "_TRACER_PROVIDER", None)
    if internal is not None:
        candidates.append(internal)
    candidates.append(trace.get_tracer_provider())

    for entry in candidates:
        cur = entry
        for _ in range(5):
            if cur is None:
                break
            if type(cur).__name__ != "ProxyTracerProvider" and hasattr(
                cur, "add_span_processor"
            ):
                return cur
            cur = getattr(cur, "_real_tracer_provider", None)
    return None


def _ensure_trace_export():
    """Inject DataRobot span processor on first successful resolution of SDK provider."""
    global _trace_injected, _inject_warned
    if _trace_injected:
        return
    try:
        import dr_otel_config

        proc = dr_otel_config.dr_span_processor
        if proc is None:
            logging.debug(
                "DataRobot span processor not configured (missing OTEL endpoint)"
            )
            return

        target = _resolve_tracer_provider_for_processor()
        if target is not None:
            target.add_span_processor(proc)
            _trace_injected = True
            logging.info(
                "Injected DataRobot span processor into TracerProvider (%s)",
                type(target).__name__,
            )
        elif not _inject_warned:
            _inject_warned = True
            logging.warning(
                "DataRobot trace export: no SDK TracerProvider yet (will retry on later callbacks)"
            )
    except Exception as e:
        logging.error("Failed to inject DataRobot span processor: %s", e)


async def before_agent(callback_context):
    """Called before agent processes a request. Injects trace export on first call."""
    _ensure_trace_export()
    _t.agent_start = time.time()
    return None


async def after_agent(callback_context):
    """Called after agent completes. NOTE: only 1 argument, not 2."""
    elapsed_ms = (time.time() - getattr(_t, "agent_start", time.time())) * 1000
    request_counter.add(1, {"status": "success"})
    request_duration.record(elapsed_ms)
    # Flush DELTA metrics so they export before the worker ends.
    # Critical for serverless / short-lived workers where the periodic reader
    # may not tick before shutdown.
    try:
        mp = metrics.get_meter_provider()
        if hasattr(mp, "force_flush"):
            mp.force_flush(timeout_millis=5000)
    except Exception:
        pass
    return None


async def before_model(callback_context, llm_request):
    """Called before each LLM API call."""
    _ensure_trace_export()
    _t.llm_start = time.time()
    return None


async def after_model(callback_context, llm_response):
    """Called after each LLM API call."""
    elapsed_ms = (time.time() - getattr(_t, "llm_start", time.time())) * 1000
    model = getattr(llm_response, "model", "unknown") or "unknown"
    llm_call_counter.add(1, {"model": model})
    llm_duration.record(elapsed_ms, {"model": model})
    return None


async def after_tool(tool, args, tool_context, tool_response):
    """Called after each tool invocation."""
    name = getattr(tool, "name", "unknown") or "unknown"
    tool_call_counter.add(1, {"tool": str(name)})
    # Set tool_name on the current span so DataRobot's tracing table shows it
    # in the Tools column. DataRobot looks for "tool_name" (underscore), not "tool.name".
    span = trace.get_current_span()
    if span and span.is_recording():
        span.set_attribute("tool_name", str(name))
    return None
```

## Wiring Callbacks to ADK Agents

For multi-agent setups, use different callback sets for root vs sub-agents:
- **Root agent**: All callbacks (trace injection, request metrics, LLM/tool metrics)
- **Sub-agents**: Only LLM and tool metrics (avoids double-counting requests and thread-local corruption)

```python
import dr_otel_config  # noqa: F401 — configures logs/metrics at import
import dr_agent_metrics

# Root agent: trace injection + end-to-end request metrics + LLM/tool metrics
_ADK_DR_ROOT = {
    "before_agent_callback": dr_agent_metrics.before_agent,
    "after_agent_callback": dr_agent_metrics.after_agent,
    "before_model_callback": dr_agent_metrics.before_model,
    "after_model_callback": dr_agent_metrics.after_model,
    "after_tool_callback": dr_agent_metrics.after_tool,
}

# Sub-agents: LLM and tool metrics only (no request counting or trace injection)
_ADK_DR_SUB = {
    "before_model_callback": dr_agent_metrics.before_model,
    "after_model_callback": dr_agent_metrics.after_model,
    "after_tool_callback": dr_agent_metrics.after_tool,
}

search_agent = LlmAgent(
    name="search_agent",
    model="gemini-2.5-flash",
    # ...
    **_ADK_DR_SUB,
)

root_agent = LlmAgent(
    name="my_agent",
    model="gemini-2.5-flash",
    tools=[AgentTool(agent=search_agent)],
    # ...
    **_ADK_DR_ROOT,
)
```

**Important**: Import `dr_otel_config` before `dr_agent_metrics` to ensure logs and metrics providers are set before metrics instruments are created.

**Warning about thread-local state**: `before_agent`/`after_agent` use `threading.local()` for timing. Wiring these on sub-agents can corrupt timing if a sub-agent runs in the same thread as the root. Keep request-level callbacks on the root only.

## Deployment Prerequisites

### GCP / Vertex AI Agent Engine

Before deploying, verify:
- [ ] Google Application Default Credentials configured (`gcloud auth application-default login` or `GOOGLE_APPLICATION_CREDENTIALS`)
- [ ] Correct `GOOGLE_CLOUD_PROJECT` set
- [ ] Required APIs enabled (Vertex AI, Agent Engine)
- [ ] GCS staging bucket accessible (if using Agent Engine)
- [ ] `DATAROBOT_API_TOKEN`, `DATAROBOT_ENTITY_ID`, `DATAROBOT_OTEL_ENDPOINT` set in the **deployed** agent's environment (not just local). These must be injected into Agent Engine via deployment env vars, Secret Manager, or the deploy script.
- [ ] Optional: `OTEL_SERVICE_NAME` for consistent `service.name` across all OTel signals

**Common mistake:** Local `verify_otel_connection.py` passes because `DATAROBOT_*` env vars exist on your machine, but deployed Agent Engine has no telemetry because those vars weren't passed to the container.

### Cloud Run

- [ ] Same `DATAROBOT_*` env vars set via `gcloud run services update --set-env-vars`
- [ ] No `OTEL_EXPORTER_OTLP_*` env vars (conflicts with DataRobot-specific export)

### Docker / Kubernetes

- [ ] Inject `DATAROBOT_*` env vars at runtime (docker-compose, k8s ConfigMap/Secret)
- [ ] No `OTEL_EXPORTER_OTLP_*` env vars for DataRobot-specific exporters

## Extra Dependencies

Add to the project's dependency file **if deploying to GCP** (Cloud Run, Agent Engine, GKE):
```
opentelemetry-resourcedetector-gcp
```

This package adds GCP resource attributes (project ID, region, service name) to spans. **Omit it for non-GCP deployments** (Docker, k8s on AWS/Azure, local dev) — it adds unnecessary weight and may log warnings about missing GCP metadata.

Note: The package name is `opentelemetry-resourcedetector-gcp` (no hyphen between "resource" and "detector"). Getting this wrong causes build failures.

## Known Pitfalls

| Issue | Cause | Fix |
|-------|-------|-----|
| Traces not appearing | ADK replaced TracerProvider | Verify `_ensure_trace_export()` fires (check logs for "Injected DataRobot span processor") |
| `TracerProvider does not support add_span_processor` | `get_tracer_provider()` returned `ProxyTracerProvider` | Use `_resolve_tracer_provider_for_processor()` (reads `opentelemetry.trace._TRACER_PROVIDER`); call `_ensure_trace_export()` from `before_model` too so injection retries after ADK inits the SDK |
| `after_agent() missing 1 required positional argument` | Wrong callback signature | `after_agent(callback_context)` takes 1 arg, not 2 |
| Callbacks not called | Not `async` | All ADK callbacks must be `async def` (ADK v1.18+) |
| Metrics missing | `OTEL_EXPORTER_OTLP_*` env vars set | Remove all `OTEL_EXPORTER_OTLP_*` env vars |
| Build fails on `opentelemetry-resource-detector-gcp` | Wrong package name | Correct: `opentelemetry-resourcedetector-gcp` |
| ADK creates duplicate spans | ADK detects OTEL env vars | Never set `OTEL_EXPORTER_OTLP_ENDPOINT` or similar env vars |
| Metrics never reach DataRobot | `dr_agent_metrics` imported before `dr_otel_config` | `get_meter()` binds to the no-op provider. Import `dr_otel_config` first (it calls `configure_otel()` + `set_meter_provider` at import time) |
| Metrics missing on short requests | `PeriodicExportingMetricReader` default 60s interval | Set `export_interval_millis=5000` and call `force_flush` in `after_agent` |
| Empty OTLP log bodies | ADK/library log records with empty `getMessage()` | Attach `logging.Formatter("%(levelname)s %(name)s: %(message)s")` on the `LoggingHandler` |
| Local verify passes but Engine has no telemetry | `DATAROBOT_*` env vars not in deployed container | Copy env vars to Agent Engine deploy script / Secret Manager |

# LangChain / LangGraph — DataRobot OTel Integration

## Overview

LangChain and LangGraph work with the standard `configure_otel()` pattern — they do NOT override the global TracerProvider. Use the generic `dr_otel_config.py` without modification.

Auto-instrumentation is available via `opentelemetry-instrumentation-langchain`, which automatically creates spans for chains, LLM calls, tool invocations, and retriever operations.

## OTel Strategy

| Signal    | Strategy                                    |
|-----------|---------------------------------------------|
| **Traces**  | Standard setup + auto-instrumentor         |
| **Metrics** | Standard setup (optional custom callbacks) |
| **Logs**    | Standard setup                             |

## Setup

### 1. Use the generic `dr_otel_config.py` as-is

No modifications needed. Call `configure_otel()` at startup.

### 2. Add auto-instrumentor after `configure_otel()`

In the agent's entrypoint:

```python
from dr_otel_config import configure_otel

configure_otel()

# Auto-instrument LangChain — must be called AFTER configure_otel()
from opentelemetry.instrumentation.langchain import LangchainInstrumentor

LangchainInstrumentor().instrument()

# Your agent code below...
```

The auto-instrumentor captures:
- Chain executions (with input/output)
- LLM calls (model, prompt, completion, token usage)
- Tool/function calls
- Retriever operations (for RAG)
- Agent reasoning steps (for LangGraph)

**Important: `tool_name` attribute for DataRobot.** LangGraph does NOT set the `tool_name` span attribute by default. DataRobot's tracing table requires `tool_name` (underscore) to populate the Tools column. Add it manually inside your tools:

```python
from opentelemetry import trace


@tool
def search_database(query: str) -> str:
    """Search the database."""
    span = trace.get_current_span()
    span.set_attribute("tool_name", "search_database")
    # ... tool logic ...
```

Without this, tool spans will appear in the trace hierarchy but the Tools column in DataRobot will be empty.

### 3. Optional: Add OpenAI/Anthropic SDK instrumentors

If the agent uses OpenAI or Anthropic SDKs directly (in addition to LangChain), add their instrumentors too:

```python
# Optional — for direct SDK calls outside LangChain
from opentelemetry.instrumentation.openai import OpenAIInstrumentor

OpenAIInstrumentor().instrument()
```

## Extra Dependencies

```
opentelemetry-instrumentation-langchain
```

Optional:
```
opentelemetry-instrumentation-openai      # If using OpenAI SDK directly
opentelemetry-instrumentation-anthropic    # If using Anthropic SDK directly
```

## Custom Metrics (Optional)

LangChain/LangGraph agents don't require a custom metrics callback module — the auto-instrumentor handles trace spans. For custom metrics (request counts, latency histograms), you can optionally add a LangChain callback handler:

```python
from opentelemetry import metrics

meter = metrics.get_meter("my-langchain-agent")
request_counter = meter.create_counter("agent.requests", unit="1")
request_duration = meter.create_histogram("agent.request.duration_ms", unit="ms")

# Use LangChain's callback system to record metrics
from langchain_core.callbacks import BaseCallbackHandler


class DataRobotMetricsHandler(BaseCallbackHandler):
    def on_chain_start(self, serialized, inputs, **kwargs):
        import time

        kwargs.setdefault("metadata", {})["_dr_start"] = time.time()

    def on_chain_end(self, outputs, **kwargs):
        import time

        start = kwargs.get("metadata", {}).get("_dr_start", time.time())
        elapsed_ms = (time.time() - start) * 1000
        request_counter.add(1, {"status": "success"})
        request_duration.record(elapsed_ms)

    def on_chain_error(self, error, **kwargs):
        request_counter.add(1, {"status": "error"})
```

This is optional — the auto-instrumentor already provides comprehensive trace spans.

## Known Pitfalls

| Issue | Cause | Fix |
|-------|-------|-----|
| No spans from LangChain | Instrumentor not called | Ensure `LangchainInstrumentor().instrument()` is called AFTER `configure_otel()` |
| Duplicate spans | Multiple instrumentors active | Only instrument once; check for existing instrumentation |
| Missing retriever spans | Old instrumentor version | Update `opentelemetry-instrumentation-langchain` to latest |

# LlamaIndex — DataRobot OTel Integration

## Overview

LlamaIndex works with the standard `configure_otel()` pattern — it does NOT override the global TracerProvider. Use the generic `dr_otel_config.py` without modification.

Auto-instrumentation is available via `opentelemetry-instrumentation-llamaindex`, which creates spans for query engines, retrievers, LLM calls, and embedding operations.

## OTel Strategy

| Signal    | Strategy                                    |
|-----------|---------------------------------------------|
| **Traces**  | Standard setup + auto-instrumentor         |
| **Metrics** | Standard setup (optional custom callbacks) |
| **Logs**    | Standard setup                             |

## Setup

### 1. Use the generic `dr_otel_config.py` as-is

No modifications needed. Call `configure_otel()` at startup.

### 2. Add auto-instrumentor after `configure_otel()`

In the agent's entrypoint:

```python
from dr_otel_config import configure_otel

configure_otel()

# Auto-instrument LlamaIndex — must be called AFTER configure_otel()
from opentelemetry.instrumentation.llamaindex import LlamaIndexInstrumentor

LlamaIndexInstrumentor().instrument()

# Your LlamaIndex code below...
```

The auto-instrumentor captures:
- Query engine executions
- Retriever operations (vector search, keyword search)
- LLM calls (prompts, completions, token usage)
- Embedding generation
- Node postprocessing

## Extra Dependencies

```
opentelemetry-instrumentation-llamaindex
```

## Alternative: LlamaIndex Built-in Callback

LlamaIndex also has a built-in OpenTelemetry callback handler. If the auto-instrumentor package is unavailable or incompatible:

```python
from llama_index.core.callbacks import CallbackManager
from llama_index.core.callbacks.open_inference_callback import (
    OpenInferenceCallbackHandler,
)

callback_manager = CallbackManager([OpenInferenceCallbackHandler()])
# Pass callback_manager to your index/query engine
```

Prefer the auto-instrumentor when available — it's more comprehensive and doesn't require modifying query engine construction.

## Known Pitfalls

| Issue | Cause | Fix |
|-------|-------|-----|
| No spans from LlamaIndex | Instrumentor not called | Ensure `LlamaIndexInstrumentor().instrument()` is called AFTER `configure_otel()` |
| Import path changed | LlamaIndex v0.10+ restructured packages | Check if instrumentation package matches your LlamaIndex version |
| Missing embedding spans | Old instrumentor version | Update to latest `opentelemetry-instrumentation-llamaindex` |

# PydanticAI — DataRobot OTel Integration

## Overview

PydanticAI's OpenTelemetry instrumentation is **opt-in**: call `Agent.instrument_all()`
at startup to emit spans. `configure_otel()` sets up the provider, and PydanticAI reuses
the global `TracerProvider` it installs, so the generic `dr_otel_config.py` works
unchanged. Skip the `Agent.instrument_all()` call and PydanticAI emits **no spans** —
silently, with no error, so telemetry never appears in DataRobot.

PydanticAI also has built-in integration with Pydantic Logfire, which exports
OTel-compatible telemetry. The preferred approach for DataRobot is direct OTel setup
(no Logfire dependency needed) plus the required `instrument_all()` opt-in.

## OTel Strategy

| Signal    | Strategy                                                            |
|-----------|--------------------------------------------------------------------|
| **Traces**  | Standard `configure_otel()` **+ required `Agent.instrument_all()`** (opt-in) |
| **Metrics** | Standard setup (optional custom metrics)                           |
| **Logs**    | Standard setup                                                     |

## Required Setup

### 1. Use the generic `dr_otel_config.py` as-is

Call `configure_otel()` at startup, before any PydanticAI import or agent creation.

### 2. Opt in to instrumentation and wire the entrypoint

Call `Agent.instrument_all()` after `configure_otel()`. This is the step that makes
PydanticAI emit spans; skipping it is the most common cause of "no telemetry."

```python
import os

from dr_otel_config import configure_otel

configure_otel()  # sets the global TracerProvider (additive) BEFORE PydanticAI opt-in

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

# Opt in to PydanticAI's OTel instrumentation — without this, no spans are emitted.
Agent.instrument_all()

# Example: routing through the DataRobot LLM Gateway (any provider works).
provider = OpenAIProvider(
    base_url=f"{os.environ['DATAROBOT_ENDPOINT'].rstrip('/')}/genai/llmgw",
    api_key=os.environ["DATAROBOT_API_TOKEN"],
)
model = OpenAIChatModel("azure/gpt-5-mini-2025-08-07", provider=provider)

agent = Agent(model, system_prompt="You are a helpful assistant.")
```

`Agent.instrument_all()` uses the global `TracerProvider` that `configure_otel()` set,
so every agent in the process exports to DataRobot without editing each `Agent(...)`
constructor. PydanticAI creates spans for agent runs, LLM API calls, tool/function
calls, and retries/error handling.

## Version robustness

PydanticAI's instrumentation API has changed across versions. Prefer the global call;
fall back to per-agent only if you cannot call `instrument_all()` at startup.

- **Recommended — global, version-robust:**
  ```python
  Agent.instrument_all()
  ```

- **Per-agent alternative — PydanticAI v2.x:**
  ```python
  from pydantic_ai import Agent
  from pydantic_ai.capabilities import Instrumentation
  from pydantic_ai.models.instrumented import InstrumentationSettings

  agent = Agent(
      model,
      capabilities=[Instrumentation(settings=InstrumentationSettings())],
  )
  ```

- **Per-agent alternative — older PydanticAI v1.x:**
  ```python
  agent = Agent(model, instrument=True)
  ```

- **`InstrumentationSettings(version=…)`** controls the telemetry data format
  (`Literal[2, 3, 4, 5]`, default `5`, following OTel GenAI semantic conventions
  1.37.0). Versions 2–4 are deprecated and emit `PydanticAIDeprecationWarning`. Only
  pin a version if you must match a specific downstream schema (see Verify below).

- **Privacy — exclude prompt/completion content:**
  ```python
  from pydantic_ai import Agent
  from pydantic_ai.models.instrumented import InstrumentationSettings

  Agent.instrument_all(InstrumentationSettings(include_content=False))
  ```
  By default PydanticAI captures prompt and completion text in spans. Set
  `include_content=False` to suppress it (this also empties DataRobot's
  Prompt/Completion columns).

## Verify

After wiring, confirm telemetry lands in DataRobot: run the agent once, then check the
Use Case Tracing view (or the `dr xp` panel — see SKILL.md Step 5).

- **Spans present, Prompt/Completion columns populated** → done.
- **No spans at all** → `Agent.instrument_all()` was not called, or `configure_otel()`
  ran after agent creation. Fix ordering.
- **Spans present but Prompt/Completion columns empty** → attribute-name mismatch.
  PydanticAI's default `version=5` emits `gen_ai.input.messages` /
  `gen_ai.output.messages` (OTel GenAI semconv), while DataRobot's tracing table reads
  `gen_ai.prompt` / `gen_ai.completion`. To fix, add the `gen_ai.prompt` / `gen_ai.completion` span attributes yourself — note that **no** `InstrumentationSettings(version=…)` value produces them (every selectable format, 2–5, emits `gen_ai.input.messages` / `gen_ai.output.messages`), so pinning a version does not help. Also confirm `include_content` is not `False`, and check whether your DataRobot instance already normalizes the newer semconv attributes.

## Optional: Logfire instrumentation

If the user already uses Logfire or wants richer PydanticAI-specific spans:

```python
import logfire

logfire.configure(
    send_to_logfire=False
)  # don't send to Logfire cloud; keep global provider
logfire.instrument_pydantic_ai()
```

This layers detailed Pydantic validation spans on top of the standard OTel traces. Use
either direct OTel (`instrument_all()`) or Logfire, not both, to avoid duplicate traces.

## Extra Dependencies

None beyond the generic OTel packages.

Optional (if using Logfire):
```
logfire[pydantic-ai]
```

## Custom Metrics (Optional)

For custom metrics, wrap the agent run:

```python
import time
from opentelemetry import metrics

meter = metrics.get_meter("my-pydantic-agent")
request_counter = meter.create_counter("agent.requests", unit="1")
request_duration = meter.create_histogram("agent.request.duration_ms", unit="ms")


async def run_with_metrics(agent, prompt):
    start = time.time()
    try:
        result = await agent.run(prompt)
        request_counter.add(1, {"status": "success"})
        return result
    except Exception:
        request_counter.add(1, {"status": "error"})
        raise
    finally:
        request_duration.record((time.time() - start) * 1000)
```

## Known Pitfalls

| Issue | Cause | Fix |
|-------|-------|-----|
| No spans from PydanticAI | `Agent.instrument_all()` never called (instrumentation is opt-in) | Call `Agent.instrument_all()` after `configure_otel()`, before running agents |
| No spans from PydanticAI | `configure_otel()` called after agent creation | Ensure `configure_otel()` runs before any PydanticAI import or agent instantiation |
| Spans appear but Prompt/Completion columns empty | v5 semconv attributes (`gen_ai.input.messages`) differ from DataRobot's `gen_ai.prompt`/`gen_ai.completion`; or `include_content=False` | Add the `gen_ai.prompt`/`gen_ai.completion` span attributes yourself (no `version` value emits them); ensure `include_content` is not `False` (see Verify) |
| Logfire overrides TracerProvider | `logfire.configure()` called with default settings | Use `send_to_logfire=False` to keep the global provider |
| Duplicate traces | Both direct OTel and Logfire active | Choose one approach — prefer direct OTel |

# Generic `dr_otel_config.py` Template

This is the core `configure_otel()` function to generate for every project, regardless of framework. Framework-specific files in `frameworks/` layer additional setup (auto-instrumentors, callbacks) on top of this.

**Critical rules (also summarized in SKILL.md):**
1. Always pass `endpoint=` and `headers=` directly to exporters — NEVER use `OTEL_EXPORTER_OTLP_*` env vars (some frameworks detect these and create conflicting providers)
2. Be additive — add DataRobot as an additional span processor to any existing TracerProvider, don't replace it
3. Use `SimpleSpanProcessor` (not Batch) to avoid flush-before-shutdown issues
4. Use DELTA temporality for metrics (required by DataRobot)

The `DATAROBOT_ENTITY_ID` at runtime is the Use Case entity (`experiment_container-<use_case_id>`) by default, or a deployment entity (`deployment-<id>`) if a shell deployment was used instead.

## Template

```python
"""DataRobot OpenTelemetry configuration.

Configures traces, logs, and metrics export to DataRobot's OTel endpoint.
Call configure_otel() at application startup, before any agent code runs.

Required env vars at runtime:
    DATAROBOT_API_TOKEN      - DataRobot API key
    DATAROBOT_ENTITY_ID      - experiment_container-<use_case_id> (or deployment-<deployment_id>)
    DATAROBOT_OTEL_ENDPOINT  - https://<your-instance>.datarobot.com/otel
"""

import logging
import os

from opentelemetry import metrics, trace
from opentelemetry._logs import set_logger_provider
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import SimpleLogRecordProcessor
from opentelemetry.sdk.metrics import (
    Counter,
    Histogram,
    MeterProvider,
    ObservableCounter,
)
from opentelemetry.sdk.metrics.export import (
    AggregationTemporality,
    PeriodicExportingMetricReader,
)
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor


def _build_dr_headers():
    """Build DataRobot authentication headers for OTel exporters."""
    api_key = os.environ.get("DATAROBOT_API_TOKEN", "")
    entity_id = os.environ.get("DATAROBOT_ENTITY_ID", "")
    if not api_key:
        logging.warning(
            "DATAROBOT_API_TOKEN not set — OTel export to DataRobot will fail"
        )
    if not entity_id:
        logging.warning(
            "DATAROBOT_ENTITY_ID not set — OTel export to DataRobot will fail"
        )
    return {
        "X-DataRobot-Entity-Id": entity_id,
        "X-DataRobot-Api-Key": api_key,
    }


def _get_endpoint():
    """Get DataRobot OTel endpoint, auto-deriving from DATAROBOT_ENDPOINT if needed."""
    endpoint = os.environ.get("DATAROBOT_OTEL_ENDPOINT", "")
    if endpoint:
        return endpoint.rstrip("/")
    # Auto-derive from DATAROBOT_ENDPOINT (e.g. https://app.datarobot.com/api/v2 → .../otel)
    api_endpoint = os.environ.get("DATAROBOT_ENDPOINT", "")
    if api_endpoint:
        base = api_endpoint.rstrip("/")
        if base.endswith("/api/v2"):
            base = base[: -len("/api/v2")]
        return f"{base}/otel"
    return ""


def configure_otel():
    """Configure OpenTelemetry to export traces, logs, and metrics to DataRobot.

    This function is additive — it adds DataRobot as an additional exporter
    alongside any existing OTel setup. It does not replace existing providers.
    """
    headers = _build_dr_headers()
    endpoint = _get_endpoint()
    if not endpoint:
        logging.warning("DATAROBOT_OTEL_ENDPOINT not set — skipping OTel configuration")
        return
    resource = Resource.create()

    # --- Traces ---
    dr_span_processor = SimpleSpanProcessor(
        OTLPSpanExporter(endpoint=f"{endpoint}/v1/traces", headers=headers)
    )
    existing_provider = trace.get_tracer_provider()
    if hasattr(existing_provider, "add_span_processor"):
        existing_provider.add_span_processor(dr_span_processor)
    else:
        provider = TracerProvider(resource=resource)
        provider.add_span_processor(dr_span_processor)
        trace.set_tracer_provider(provider)

    # --- Logs ---
    log_exporter = OTLPLogExporter(endpoint=f"{endpoint}/v1/logs", headers=headers)
    logger_provider = LoggerProvider(resource=resource)
    set_logger_provider(logger_provider)
    logger_provider.add_log_record_processor(SimpleLogRecordProcessor(log_exporter))
    handler = LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider)
    # Custom formatter ensures OTLP log bodies are never empty
    # (some libraries emit records with empty getMessage())
    handler.setFormatter(logging.Formatter("%(levelname)s %(name)s: %(message)s"))
    logging.getLogger().addHandler(handler)

    # --- Metrics ---
    preferred_temporality = {
        Counter: AggregationTemporality.DELTA,
        Histogram: AggregationTemporality.DELTA,
        ObservableCounter: AggregationTemporality.DELTA,
    }
    metric_exporter = OTLPMetricExporter(
        endpoint=f"{endpoint}/v1/metrics",
        headers=headers,
        preferred_temporality=preferred_temporality,
    )
    meter_provider = MeterProvider(
        metric_readers=[PeriodicExportingMetricReader(metric_exporter)],
        resource=resource,
    )
    metrics.set_meter_provider(meter_provider)
```

## OTel provider initialization order warning

Some frameworks override the global TracerProvider at startup (notably Google ADK). When this happens, the standard trace setup above will lose the DataRobot exporter. The framework reference files document which frameworks have this issue and provide alternative patterns (e.g., lazy injection via callbacks). Always check the framework reference file.

Existing OTel setups (e.g., exporters to Jaeger, Datadog, Google Cloud Trace) are preserved when possible — DataRobot is added alongside, not replacing. However, note that OTel has a single global provider per signal. Whoever calls `set_tracer_provider()` last wins. The additive pattern above avoids calling `set_tracer_provider()` when a provider already exists, instead adding a processor to the existing one.

#!/usr/bin/env python3
# Copyright (c) 2026 DataRobot, Inc. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Create a shell deployment in DataRobot for receiving external agent OTel telemetry.

Uses RegisteredModelVersion.create_for_external (DataRobot Python SDK 3.x).

Usage:
    python create_shell_deployment.py --name "My Agent" --description "OTel sink"

Env vars:
    DATAROBOT_API_TOKEN  - DataRobot API token
    DATAROBOT_ENDPOINT   - DataRobot API endpoint (e.g. https://app.datarobot.com/api/v2)
"""

from __future__ import annotations

import argparse
import json
import os
import sys

import datarobot as dr

# Raised when a tenant does not support an optional target type or deployment
# setting: the API rejects the call (ClientError) or the async settings update
# never resolves. Each call site below degrades gracefully instead of failing.
_UNSUPPORTED_FEATURE_ERRORS = (
    dr.errors.ClientError,
    dr.errors.AsyncTimeoutError,
    dr.errors.AsyncFailureError,
    dr.errors.AsyncProcessUnsuccessfulError,
)


def _otel_endpoint(api_endpoint: str) -> str:
    """Derive OTel endpoint from the API endpoint.

    API endpoint:  https://app.datarobot.com/api/v2
    OTel endpoint: https://app.datarobot.com/otel
    """
    base = api_endpoint.rstrip("/")
    base = base.removesuffix("/api/v2")
    return f"{base}/otel"


def _find_or_create_prediction_environment() -> dr.PredictionEnvironment:
    """Find or create a prediction environment for external agent monitoring.

    Not all tenants support platform="external". Try common platforms in order
    of preference: gcp, other, aws, azure. Reuse an existing environment when
    possible to avoid clutter.
    """
    preferred_platforms = ("gcp", "other", "aws", "azure")
    envs = dr.PredictionEnvironment.list()

    for platform in preferred_platforms:
        for env in envs:
            if getattr(env, "platform", None) == platform:
                return env

    # No compatible environment found — create one
    return dr.PredictionEnvironment.create(
        name="External Agent OTel Environment",
        platform="other",
        description="Prediction environment for external agent OTel monitoring",
    )


def create_shell_deployment(name: str, description: str) -> dict:
    """Create a shell deployment in DataRobot for external agent monitoring.

    Creates an external registered model version (target type: AgenticWorkflow)
    and deploys it with prediction row storage and automatic association ID
    generation enabled. The deployment ID is used to route OTel telemetry to
    the correct DataRobot monitoring dashboard.

    Args:
        name: Display name for the deployment
        description: Description of what agent this monitors

    Returns:
        Dict with deployment_id, entity_id, and otel_endpoint
    """
    token = os.getenv("DATAROBOT_API_TOKEN")
    endpoint = os.getenv("DATAROBOT_ENDPOINT", "https://app.datarobot.com/api/v2")

    if not token:
        print("Error: DATAROBOT_API_TOKEN env var is required", file=sys.stderr)
        sys.exit(1)

    dr.Client(token=token, endpoint=endpoint)

    # --- Prediction environment ---
    pred_env = _find_or_create_prediction_environment()

    # --- Registered model version (external shell) ---
    # Prefer AgenticWorkflow (enables full monitoring dashboards);
    # fall back to TextGeneration if the tenant doesn't support it.
    try:
        model_version = dr.RegisteredModelVersion.create_for_external(
            name="external-agent-shell-v1",
            target={"name": "agent_output", "type": "AgenticWorkflow"},
            registered_model_name=name,
            registered_model_description=description,
        )
    except _UNSUPPORTED_FEATURE_ERRORS as e:
        print(
            f"Warning: AgenticWorkflow target type failed ({e}). "
            "Retrying with TextGeneration.",
            file=sys.stderr,
        )
        model_version = dr.RegisteredModelVersion.create_for_external(
            name="external-agent-shell-v1",
            target={"name": "prediction", "type": "TextGeneration"},
            registered_model_name=name,
            registered_model_description=description,
        )

    # --- Deploy ---
    deployment = dr.Deployment.create_from_registered_model_version(
        model_package_id=model_version.id,
        label=name,
        description=description,
        prediction_environment_id=pred_env.id,
    )

    # --- Enable monitoring settings ---
    # Prediction row storage: stores prediction inputs/outputs for monitoring
    try:
        deployment.update_predictions_data_collection_settings(enabled=True)
    except _UNSUPPORTED_FEATURE_ERRORS as e:
        print(
            f"Warning: Could not enable prediction row storage: {e}",
            file=sys.stderr,
        )

    # Automatic association ID generation: assigns unique IDs to prediction rows.
    # The API requires a column name alongside autoGenerateId.
    try:
        client = dr.client.get_client()
        resp = client.patch(
            f"deployments/{deployment.id}/settings/",
            json={
                "associationId": {
                    "columnNames": ["association_id"],
                    "autoGenerateId": True,
                    "requiredInPredictionRequests": False,
                },
            },
        )
        from datarobot.utils.waiters import wait_for_async_resolution

        wait_for_async_resolution(client, resp.headers["Location"])
    except (*_UNSUPPORTED_FEATURE_ERRORS, KeyError) as e:
        print(
            f"Warning: Could not enable automatic association ID: {e}",
            file=sys.stderr,
        )

    return {
        "deployment_id": deployment.id,
        "entity_id": f"deployment-{deployment.id}",
        "otel_endpoint": _otel_endpoint(endpoint),
    }


if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description="Create a DataRobot shell deployment for external agent monitoring"
    )
    parser.add_argument(
        "--name",
        required=True,
        help="Display name for the deployment",
    )
    parser.add_argument(
        "--description",
        default="External agent OTel telemetry sink",
        help="Description of what agent this monitors",
    )
    args = parser.parse_args()

    result = create_shell_deployment(args.name, args.description)
    print(json.dumps(result, indent=2))

#!/usr/bin/env python3
# Copyright (c) 2026 DataRobot, Inc. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Resolve a DataRobot Use Case to use as the OTel telemetry target.

A Use Case is the default (primary) telemetry entity for external agent
monitoring: telemetry is associated with it via the `experiment_container`
entity type, and traces appear under the Use Case's Tracing tab. This works
for both DataRobot-native agents and agents built elsewhere (brownfield).

Two modes:
    --use-case-id <id>   Validate an existing Use Case the user already has.
    --name "<name>"      Create a new Use Case (when the user has none yet).

Usage:
    python create_use_case.py --name "My Agent Monitoring"
    python create_use_case.py --use-case-id 6123abc...

Env vars:
    DATAROBOT_API_TOKEN  - DataRobot API token
    DATAROBOT_ENDPOINT   - DataRobot API endpoint (e.g. https://app.datarobot.com/api/v2)
"""

from __future__ import annotations

import argparse
import json
import os
import sys

import datarobot as dr


def _base_url(api_endpoint: str) -> str:
    """Strip a trailing /api/v2 to get the instance base URL."""
    base = api_endpoint.rstrip("/")
    base = base.removesuffix("/api/v2")
    return base


def _result(use_case: dr.UseCase, api_endpoint: str) -> dict:
    """Build the telemetry-target descriptor for a Use Case.

    Two id forms for the same Use Case, used in different places:
      * ``entity_id`` (``experiment_container-<id>``) — the OTel export header /
        ``DATAROBOT_ENTITY_ID`` at runtime.
      * ``view_command`` uses the *bare* ``use_case.id`` — ``dr xp`` expects the
        unprefixed id (``--entity-type`` defaults to ``experiment_container``).
    """
    return {
        "use_case_id": use_case.id,
        "entity_id": f"experiment_container-{use_case.id}",
        "otel_endpoint": f"{_base_url(api_endpoint)}/otel",
        "view_command": f"dr xp --entity-id {use_case.id} --enable-logs --enable-metrics",
    }


def resolve_use_case(
    name: str | None, description: str | None, use_case_id: str | None
) -> dict:
    """Validate an existing Use Case or create a new one for telemetry.

    Args:
        name: Display name for a Use Case to create (create mode).
        description: Optional description for the new Use Case (auto if omitted).
        use_case_id: Existing Use Case ID to validate and reuse (validate mode).

    Returns:
        Dict with use_case_id, entity_id, otel_endpoint, and view_command.
    """
    token = os.getenv("DATAROBOT_API_TOKEN")
    endpoint = os.getenv("DATAROBOT_ENDPOINT", "https://app.datarobot.com/api/v2")

    if not token:
        print("Error: DATAROBOT_API_TOKEN env var is required", file=sys.stderr)
        sys.exit(1)

    dr.Client(token=token, endpoint=endpoint)

    if use_case_id:
        # Validate mode: confirm the Use Case exists and is reachable.
        use_case = dr.UseCase.get(use_case_id)
    else:
        # Create mode: auto-describe when no description is provided.
        use_case = dr.UseCase.create(
            name=name,
            description=description
            or f"OTel telemetry target for external agent monitoring ({name})",
        )

    return _result(use_case, endpoint)


if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description="Resolve a DataRobot Use Case as an external agent OTel target"
    )
    # Validate an existing Use Case or create a new one — exactly one, never both.
    target = parser.add_mutually_exclusive_group(required=True)
    target.add_argument(
        "--use-case-id",
        dest="use_case_id",
        help="Existing Use Case ID to validate and reuse (validate mode)",
    )
    target.add_argument(
        "--name",
        help="Display name for a new Use Case (create mode)",
    )
    parser.add_argument(
        "--description",
        default="",
        help="Optional description for the new Use Case (auto-generated if omitted)",
    )
    args = parser.parse_args()

    result = resolve_use_case(args.name, args.description, args.use_case_id)
    print(json.dumps(result, indent=2))

#!/usr/bin/env python3
# Copyright (c) 2026 DataRobot, Inc. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""
Verify OTel connection to DataRobot by sending test telemetry.

Sends a test trace span, log record, and metric to DataRobot's OTel endpoint
to confirm the pipeline is working before deploying the instrumented agent.

Usage:
    python verify_otel_connection.py

Env vars:
    DATAROBOT_API_TOKEN      - DataRobot API token
    DATAROBOT_ENTITY_ID      - experiment_container-<use_case_id> (or deployment-<deployment_id>)
    DATAROBOT_OTEL_ENDPOINT  - https://<instance>.datarobot.com/otel
"""

import json
import logging
import os
import sys

from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import SimpleLogRecordProcessor
from opentelemetry.sdk.metrics import (
    Counter,
    Histogram,
    MeterProvider,
    ObservableCounter,
)
from opentelemetry.sdk.metrics.export import (
    AggregationTemporality,
    PeriodicExportingMetricReader,
)
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor


def verify_connection() -> dict:
    """Send test telemetry to DataRobot and report results.

    Returns:
        Dict with status and per-signal results
    """
    api_key = os.environ.get("DATAROBOT_API_TOKEN", "")
    entity_id = os.environ.get("DATAROBOT_ENTITY_ID", "")
    endpoint = os.environ.get("DATAROBOT_OTEL_ENDPOINT", "")

    errors = []
    if not api_key:
        errors.append("DATAROBOT_API_TOKEN env var is required")
    if not entity_id:
        errors.append("DATAROBOT_ENTITY_ID env var is required")
    if not endpoint:
        errors.append("DATAROBOT_OTEL_ENDPOINT env var is required")
    if entity_id and not entity_id.startswith(("experiment_container-", "deployment-")):
        errors.append(
            "DATAROBOT_ENTITY_ID must start with 'experiment_container-' (Use Case) "
            f"or 'deployment-' (deployment), got: {entity_id}"
        )

    if errors:
        return {"status": "error", "errors": errors}

    headers = {
        "X-DataRobot-Entity-Id": entity_id,
        "X-DataRobot-Api-Key": api_key,
    }
    resource = Resource.create()
    results = {
        "status": "success",
        "traces": "pending",
        "logs": "pending",
        "metrics": "pending",
    }

    # --- Traces ---
    try:
        trace_provider = TracerProvider(resource=resource)
        trace_provider.add_span_processor(
            SimpleSpanProcessor(
                OTLPSpanExporter(endpoint=f"{endpoint}/v1/traces", headers=headers)
            )
        )
        tracer = trace_provider.get_tracer("datarobot-otel-verification")
        with tracer.start_as_current_span("datarobot.otel.verification") as span:
            span.set_attribute("verification", True)
            span.set_attribute("source", "datarobot-external-agent-monitoring")
        trace_provider.shutdown()
        results["traces"] = "sent"
    # Broad catch is intentional: this is a connectivity diagnostic, and any
    # exporter/transport failure must be reported per-signal rather than abort
    # the remaining checks.
    except Exception as e:  # noqa: BLE001
        results["traces"] = f"error: {e}"
        results["status"] = "partial"

    # --- Logs ---
    try:
        logger_provider = LoggerProvider(resource=resource)
        logger_provider.add_log_record_processor(
            SimpleLogRecordProcessor(
                OTLPLogExporter(endpoint=f"{endpoint}/v1/logs", headers=headers)
            )
        )
        handler = LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider)
        test_logger = logging.getLogger("datarobot-otel-verification")
        test_logger.addHandler(handler)
        test_logger.setLevel(logging.INFO)
        test_logger.info("DataRobot OTel verification — test log record")
        test_logger.removeHandler(handler)
        logger_provider.shutdown()
        results["logs"] = "sent"
    # Broad catch is intentional: this is a connectivity diagnostic, and any
    # exporter/transport failure must be reported per-signal rather than abort
    # the remaining checks.
    except Exception as e:  # noqa: BLE001
        results["logs"] = f"error: {e}"
        results["status"] = "partial"

    # --- Metrics ---
    try:
        preferred_temporality = {
            Counter: AggregationTemporality.DELTA,
            Histogram: AggregationTemporality.DELTA,
            ObservableCounter: AggregationTemporality.DELTA,
        }
        meter_provider = MeterProvider(
            metric_readers=[
                PeriodicExportingMetricReader(
                    OTLPMetricExporter(
                        endpoint=f"{endpoint}/v1/metrics",
                        headers=headers,
                        preferred_temporality=preferred_temporality,
                    )
                )
            ],
            resource=resource,
        )
        meter = meter_provider.get_meter("datarobot-otel-verification")
        counter = meter.create_counter("datarobot.otel.verification", unit="1")
        counter.add(1, {"source": "verification"})
        meter_provider.shutdown()
        results["metrics"] = "sent"
    # Broad catch is intentional: this is a connectivity diagnostic, and any
    # exporter/transport failure must be reported per-signal rather than abort
    # the remaining checks.
    except Exception as e:  # noqa: BLE001
        results["metrics"] = f"error: {e}"
        results["status"] = "partial"

    return results


if __name__ == "__main__":
    result = verify_connection()
    print(json.dumps(result, indent=2))
    if result["status"] != "success":
        sys.exit(1)