# SHAP API Reference

Full parameter signatures for `datarobot.insights` classes.
SDK version: `datarobot>=3.6.0` for the full set below (`ShapDistributions` was added in 3.6;
`ShapMatrix`, `ShapImpact`, and `ShapPreview` are available in `datarobot>=3.4.0`)

Source: https://datarobot-public-api-client.readthedocs-hosted.com/en/latest-release/insights.html

---

## Import

```python
from datarobot.insights import ShapMatrix, ShapImpact, ShapPreview, ShapDistributions
```

---

## ShapMatrix

Raw SHAP values for each feature column and each row.

### `ShapMatrix.create(entity_id, source='validation', **kwargs)`

Blocking call - computes and returns a ShapMatrix.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `entity_id` | str | required | Model ID |
| `source` | str | `'validation'` | Partition: `'validation'`, `'crossValidation'`, `'holdout'`, `'externalTestSet'` |
| `data_slice_id` | str | None | Optional Data Slice ID to filter the selected partition |
| `external_dataset_id` | str | None | Dataset ID from AI Catalog; required when `source='externalTestSet'` |
| `quick_compute` | bool | None | If true/unspecified, compute on a 2500-row sample; if false, compute all rows |

### `ShapMatrix.compute(entity_id, source='validation', **kwargs)`

Non-blocking - returns a job reference.

Same parameters as `.create()`. Call `job.get_result_when_complete()` to wait.

### `ShapMatrix.get(entity_id, source)`

Retrieve an already-computed ShapMatrix by model + partition.

### `ShapMatrix.list(entity_id)`

List all computed ShapMatrix objects for a model (one per partition computed).

### Attributes

| Attribute | Type | Description |
|-----------|------|-------------|
| `matrix` | list[list[float]] | 2D array: rows x features |
| `columns` | list[str] | Feature names (same order as matrix columns) |
| `base_value` | float | Model's mean prediction (baseline) |
| `link_function` | str | Link function: `'identity'` for regression, `'logit'` for binary classification |
| `source` | str | Partition this matrix was computed on |

### Export to DataFrame or CSV

After `.create()` / `.compute()`, values are on the result (`matrix`, `columns`):

```python
import pandas as pd

df = pd.DataFrame(result.matrix, columns=result.columns)
```

`get_as_dataframe` and `get_as_csv` are **classmethods** on `ShapMatrix` (re-fetch from the API).
Pass the same `entity_id`, `source`, and optional kwargs used at compute time:

```python
df = ShapMatrix.get_as_dataframe(entity_id=model_id, source="validation")
csv = ShapMatrix.get_as_csv(
    entity_id=model_id,
    source="externalTestSet",
    external_dataset_id=dataset_id,
)
```

Do not call `result.get_as_dataframe()` — that API shape is for legacy `datarobot.models.ShapMatrix`.

### Notes

- Available for blenders
- The >1000-feature limitation applies to anomaly-detection models only
- When `link_function='logit'`, values are in log-odds space; add contributions to
  `base_value` in log-odds space, then use inverse-logit (`scipy.special.expit`) on the total.
  Do not apply `expit` to individual SHAP values as probability deltas

---

## ShapImpact

Aggregated feature importance based on SHAP matrix values.

### `ShapImpact.create(entity_id, source='training')`

Blocking call.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `entity_id` | str | required | Model ID |
| `source` | str | `'training'` | Source type to use when computing the insight |
| `data_slice_id` | str | None | Optional Data Slice ID |
| `quick_compute` | bool | None | If true/unspecified, compute on a 2500-row sample; if false, compute all rows |

### `ShapImpact.compute(entity_id, source='training')`

Non-blocking. Returns job; call `job.get_result_when_complete()`.

### `ShapImpact.get(entity_id, source=..., ...)` / `ShapImpact.list(entity_id)`

Retrieve existing ShapImpact results. Pass the same `source` used when computing the insight.

### Attributes

| Attribute | Type | Description |
|-----------|------|-------------|
| `shap_impacts` | list | Each entry is `[feature_name, normalized, unnormalized]` or a dict with `feature_name`, `impact_normalized`, `impact_unnormalized` |
| `base_value` | float or list[float] | Baseline prediction value(s) |
| `row_count` | int | Number of rows used for computation |
| `capping` | bool | Whether extreme SHAP values were capped |
| `link` | str | Link function |

### Notes

- `normalized_impact`: impact scaled so features sum to 1.0
- `unnormalized_impact`: raw mean absolute SHAP value for the feature
- Results are sorted descending by importance

---

## ShapPreview

Per-row top-feature SHAP explanations in a compact "preview" format.

### `ShapPreview.create(entity_id, source='validation', **kwargs)`

Blocking.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `entity_id` | str | required | Model ID |
| `source` | str | `'validation'` | Partition: same options as ShapMatrix |
| `data_slice_id` | str | None | Optional Data Slice ID |
| `external_dataset_id` | str | None | Required when `source='externalTestSet'` |

### `ShapPreview.compute(entity_id, source='validation', **kwargs)`

Non-blocking variant.

### Attributes

| Attribute | Type | Description |
|-----------|------|-------------|
| `previews` | list[dict] | Per-row preview data (see structure below) |
| `previews_count` | int | Total number of rows |

### `previews` row structure

```python
{
    "row_index": 0,
    "prediction_value": 0.72,
    "preview_values": [
        {
            "feature_rank": 1,
            "feature_name": "income",
            "feature_value": "85000",
            "shap_value": 0.18,
            "has_text_explanations": False,
            "text_explanations": [],
        },
        # ... top-N features
    ],
}
```

---

## ShapDistributions

Distribution of SHAP values across rows for each feature.

### `ShapDistributions.create(entity_id, source='validation', data_slice_id=None, **kwargs)`

Blocking.

### `ShapDistributions.compute(entity_id, source='validation', data_slice_id=None, **kwargs)`

Non-blocking.

### Attributes

| Attribute | Type | Description |
|-----------|------|-------------|
| `features` | list[dict] | Per-feature distribution data |
| `total_features_count` | int | Total number of features |

---

## Insights diagnostics (non-SHAP)

Same `BaseInsight` pattern as SHAP: `create`, `compute`, `get`, `list` with `entity_id=model_id`.

| Class | Import | Notable result attributes |
|-------|--------|---------------------------|
| `RocCurve` | `from datarobot.insights import RocCurve` | `auc`, `roc_points` |
| `LiftChart` | `from datarobot.insights import LiftChart` | `bins` |
| `ConfusionMatrix` | `from datarobot.insights import ConfusionMatrix` | `confusion_matrix_data`, `global_metrics` |

`FeatureEffects` / partial dependence is not in `datarobot.insights`; use
`model.get_feature_effect(source=...)` or `model.get_or_request_feature_effect(...)`.

---

## Constraints and limitations

| Constraint | Detail |
|-----------|--------|
| Blenders | SHAP insights are available for blenders |
| Feature count | The >1000-feature limitation applies to anomaly-detection models only |
| Holdout | `source='holdout'` requires holdout to be unlocked in the project |
| Data slices | Use `dr.DataSlice.create()`, `.list()`, or `.get()`, then pass `data_slice_id` |
| Custom models | SHAP for custom models requires additional setup (see SHAP insights user guide) |

# XEMP Prediction Explanations Reference

Parameter reference for `dr.PredictionExplanations` and related classes.
SDK version: `datarobot>=2.26` (XEMP API is stable; `datarobot.insights` SHAP API is preferred for new code).

Source: https://datarobot-public-api-client.readthedocs-hosted.com/en/latest-release/reference/modeling/insights/prediction_explanations.html

---

## When to use XEMP vs SHAP

| Situation | Use |
|-----------|-----|
| Default / new code | `ShapMatrix` / `ShapPreview` from `datarobot.insights` |
| Anomaly-detection models with >1000 features | XEMP PE if SHAP is unavailable |
| Regulatory requirement for XEMP | XEMP PE |
| Feature Impact methodology required | XEMP PE |

---

## Prerequisites

All must be satisfied before calling `PredictionExplanations.create()`:

1. **Feature Impact computed**:
   ```python
   job = model.request_feature_impact()
   job.wait_for_completion()
   ```

2. **PredictionExplanationsInitialization created** (one-time per model):
   ```python
   dr.PredictionExplanationsInitialization.create(project_id=project_id, model_id=model_id)
   ```

3. **Scoring dataset uploaded** to the AI Catalog (`dataset_id` passed to `.create()`).

---

## PredictionExplanationsInitialization

### `PredictionExplanationsInitialization.create(project_id, model_id)`

One-time initialization. Safe to call multiple times; check first with `.get()`.

### `PredictionExplanationsInitialization.get(project_id, model_id)`

Check whether initialization exists. Raises `ClientError` if not found.

### `PredictionExplanationsInitialization.delete(project_id, model_id)`

Delete initialization (forces re-initialization on next `.create()`).

---

## PredictionExplanations

### `PredictionExplanations.create(project_id, model_id, dataset_id, ...)`

Submit an async job to compute explanations on a dataset. Call
`pe_job.get_result_when_complete()` to retrieve the `PredictionExplanations` result.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `project_id` | str | required | DataRobot project ID |
| `model_id` | str | required | DataRobot model ID |
| `dataset_id` | str | required | AI Catalog dataset ID to explain |
| `max_explanations` | int | 3 | Top-N feature explanations per row; at most 50 can be returned |
| `threshold_high` | float | None | Only explain rows with prediction >= this |
| `threshold_low` | float | None | Only explain rows with prediction <= this |
| `mode` | `TopPredictionsMode` or `ClassListMode` | predicted class only | For multiclass/clustering: which classes to explain |

### `PredictionExplanations.create_on_training_data(project_id, model_id, ...)`

Same as `.create()` but runs on the model's training data instead of an uploaded dataset.

### `PredictionExplanations.get(project_id, prediction_explanations_id)`

Retrieve a computed PE object by ID.

### `PredictionExplanations.list(project_id, model_id=None)`

List all PE objects for a project (optionally filtered by model).

### Methods on PE result object

| Method | Returns | Description |
|--------|---------|-------------|
| `.get_rows(batch_size=None)` | iterator | Iterate explanation rows |
| `.get_all_as_dataframe(exclude_adjusted_predictions=True)` | `pd.DataFrame` | All rows as DataFrame |
| `.download_to_csv(filename, exclude_adjusted_predictions=True)` | None | Write to CSV |
| `.is_multiclass()` | bool | Whether this is a multiclass explanation |

---

## Explanation row structure

Each row from `.get_rows()` has:

| Field | Type | Description |
|-------|------|-------------|
| `row_index` | int | Position in dataset |
| `prediction` | float | Model's prediction for this row |
| `adjusted_prediction` | float | Exposure-adjusted prediction (if applicable) |
| `prediction_explanations` | list[dict] | Feature explanations |

Each entry in `prediction_explanations`:

```python
{
    "feature": "income",  # feature name
    "featureValue": "85000",  # actual value of the feature
    "strength": 0.18,  # XEMP contribution (positive = increases prediction)
    "label": "income",  # display label
    "qualitative_strength": "++",  # qualitative indicator
}
```

---

## Adjusted predictions (exposure projects)

For insurance or other exposure-normalized projects:

```python
df = pe_obj.get_all_as_dataframe(exclude_adjusted_predictions=False)
# DataFrame now includes 'adjusted_prediction' column
```

---

## Multiclass / clustering

```python
pe_job = dr.PredictionExplanations.create(
    project_id=project_id,
    model_id=model_id,
    dataset_id=dataset.id,
    mode=dr.models.ClassListMode(
        ["class_a", "class_b"]
    ),  # specify which classes to explain
)

# Check if multiclass
pe_obj = pe_job.get_result_when_complete()
print(pe_obj.is_multiclass())
```

---

## Notes

- `max_explanations` can return at most 50 explanations because XEMP computes explanations for
  the global top 50 features
- `threshold_high` and `threshold_low` can be combined to explain only extreme predictions
- XEMP explanations use the XEMP methodology (not SHAP); magnitudes are not comparable across models
- For SHAP-based explanations, prefer `datarobot.insights.ShapMatrix` / `ShapPreview` with
  `entity_id=model_id`. Export via `pd.DataFrame(result.matrix, columns=result.columns)` or
  `ShapMatrix.get_as_dataframe(entity_id=..., source=...)` — not legacy `datarobot.models.ShapMatrix`.

#!/usr/bin/env python3
# Copyright (c) 2026 DataRobot, Inc. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""
Compute a ShapMatrix for a DataRobot model and export to CSV or DataFrame.

Usage:
    python compute_shap_matrix.py --model-id <model_id> [--source validation] [--output out.csv]
    python compute_shap_matrix.py --model-id <model_id> --data-slice-id <slice_id>
    python compute_shap_matrix.py --model-id <model_id> --source externalTestSet \
        --dataset-path ./data/scoring.csv --output out.csv
    python compute_shap_matrix.py --model-id <model_id> --list-existing
"""

import argparse
from typing import Any

import datarobot as dr
import pandas as pd
from datarobot.insights import ShapMatrix


def compute_shap_matrix(
    model_id: str,
    source: str = "validation",
    dataset_path: str | None = None,
    output_path: str | None = None,
    data_slice_id: str | None = None,
    quick_compute: bool | None = None,
) -> Any:
    external_dataset_id: str | None = None
    if source == "externalTestSet":
        if not dataset_path:
            raise ValueError("--dataset-path required when source=externalTestSet")
        print(f"Uploading dataset: {dataset_path}")
        dataset = dr.Dataset.upload(dataset_path)
        external_dataset_id = dataset.id
        print(f"  Dataset ID: {external_dataset_id}")

    print(f"Computing ShapMatrix: model={model_id!r} source={source!r} ...")
    result = ShapMatrix.create(
        entity_id=model_id,
        source=source,
        data_slice_id=data_slice_id,
        external_dataset_id=external_dataset_id,
        quick_compute=quick_compute,
    )

    print(f"  Features:   {len(result.columns)}")
    print(f"  Rows:       {len(result.matrix)}")
    print(f"  Base value: {result.base_value:.6f}")
    print(f"  Link:       {result.link_function}")

    if output_path:
        df = pd.DataFrame(result.matrix, columns=result.columns)
        df.to_csv(output_path, index=False)
        print(f"  Exported to: {output_path}")

    return result


def main() -> None:
    parser = argparse.ArgumentParser(description="Compute DataRobot ShapMatrix")
    parser.add_argument("--model-id", required=True)
    parser.add_argument(
        "--source",
        default="validation",
        choices=["validation", "crossValidation", "holdout", "externalTestSet"],
    )
    parser.add_argument("--data-slice-id", default=None)
    parser.add_argument("--dataset-path", default=None)
    parser.add_argument("--output", default=None)
    parser.add_argument("--full-compute", action="store_true")
    parser.add_argument("--list-existing", action="store_true")
    args = parser.parse_args()

    dr.Client()

    if args.list_existing:
        matrices = ShapMatrix.list(entity_id=args.model_id)
        print(f"Found {len(matrices)} existing ShapMatrix computation(s):")
        for m in matrices:
            print(
                f"  source={m.source}  features={len(m.columns)}  rows={len(m.matrix)}"
            )
        return

    compute_shap_matrix(
        model_id=args.model_id,
        source=args.source,
        dataset_path=args.dataset_path,
        output_path=args.output,
        data_slice_id=args.data_slice_id,
        quick_compute=False if args.full_compute else None,
    )


if __name__ == "__main__":
    main()