# Remediation Patterns

Common data quality issue patterns with example remediation workflows. These are illustrative examples for reasoning — not rigid step-by-step procedures. Real incidents often combine multiple patterns or present unique variations. Use these as a starting point, then adapt based on the specific TSA findings and available tools.

---

## Pattern 1: Stale data (freshness alert)

### Root cause signals (from TSA)

- "Pipeline has not run since..."
- "DAG/job failed at [timestamp]"
- "No new rows since [timestamp]"
- "Upstream table also stale" (cascading staleness)

### Reasoning

Stale data usually means the pipeline that feeds this table has either failed or not run. The first step is identifying which pipeline is responsible, then determining why it stopped.

### Investigation steps

1. Check upstream lineage — is the source table also stale?
   ```
   getAssetLineage(mcons=["<table_mcon>"], direction="UPSTREAM")
   getTable(mcon="<upstream_mcon>")  # check freshness
   ```

2. Check recent write queries — what pipeline usually updates this table?
   ```
   getQueriesForTable(mcon="<table_mcon>", query_type="destination", limit=5)
   ```

3. If upstream is also stale, trace further upstream to find the true root cause.

### Remediation by available tools

**If pipeline orchestrator is available (Airflow, Dagster, Prefect):**
- Identify the DAG/pipeline responsible for updating the table
- Check if the last run failed or was delayed
- Trigger a new run or retry the failed task
- Example: "The DAG `etl_orders_daily` last ran 26 hours ago and failed on task `load_orders`. I'll retry that specific task."

**If dbt Cloud is available:**
- Identify the dbt job that builds this model
- Check the last run status
- Trigger a new run
- Example: "The dbt job 'Daily Transform' failed 18 hours ago with a compilation error. After reviewing the error, I'll trigger a rerun."

**If no execution tool is available:**
- Document which pipeline is responsible (from query analysis)
- Provide the manual commands: `airflow dags trigger <dag_id>` or `dbt run --select <model>`
- Present the diagnosis and recommended fix to the user and ask for next steps
- Comment on the alert with the diagnosis and recommended fix

### Verification

After triggering the pipeline:
- Wait for the job to complete (check status via the orchestrator MCP)
- Re-check table freshness: `getTable(mcon="<table_mcon>")` — has `last_updated_on` moved forward?
- Re-check the alert: `getAlerts(alert_ids=["<alert_id>"])` — has it resolved?

---

## Pattern 2: dbt model failure

### Root cause signals (from TSA)

- "dbt run failed with error..."
- "Compilation error in model..."
- "Database error during model execution"
- "Dependency failed — upstream model did not complete"

### Reasoning

dbt failures can be compilation errors (code issues), database errors (permissions, resource limits), or dependency failures (upstream model failed first). The fix depends on the error type.

### Investigation steps

1. Read the TSA `full_response` carefully — it often contains the actual dbt error message.

2. Check if the issue is the table itself or an upstream dependency:
   ```
   getAssetLineage(mcons=["<table_mcon>"], direction="UPSTREAM")
   ```

3. If upstream models also have alerts, the root cause is likely further upstream — remediate that first.

### Remediation by available tools

**If dbt Cloud is available:**
- For compilation errors: the error likely needs a code fix → create a GitHub PR if GitHub MCP is available, otherwise describe the fix for the user
- For transient database errors: rerun the job
- For dependency failures: find and fix the upstream failure first, then rerun

**If GitHub is available (for code fixes):**
- Create a branch with the fix
- Open a PR with clear description of what failed and why
- Example: "The model `stg_orders` fails because column `order_status` was renamed to `status` upstream. Creating a PR to update the column reference."

**If no execution tool is available:**
- Describe the error and the fix needed
- Provide the `dbt run --select <model>` command for manual execution
- If it's a code issue, describe exactly what file and line needs changing

### Verification

- Check dbt job status via dbt Cloud MCP (if available)
- Re-check table: `getTable(mcon="<table_mcon>")` — has the model been rebuilt?
- Verify downstream tables are also refreshing

---

## Pattern 3: Schema change

### Root cause signals (from TSA)

- "Column added/removed/renamed"
- "Column type changed"
- "Schema differs from expected"

### Reasoning

Schema changes can be intentional (upstream team made a planned change) or accidental (a deployment error). The remediation depends on whether the change is expected and whether downstream consumers can handle it.

### Investigation steps

1. Identify what changed:
   ```
   getTable(mcon="<table_mcon>", include_fields=true)
   ```

2. Check blast radius — who consumes this table?
   ```
   getAssetLineage(mcons=["<table_mcon>"], direction="DOWNSTREAM")
   ```

3. Check if downstream tables are also alerting:
   ```
   getAlerts(
     table_mcons=["<downstream_mcon_1>", "<downstream_mcon_2>"],
     created_after="<24 hours ago>"
   )
   ```

4. Check recent queries to identify who/what made the schema change:
   ```
   getQueriesForTable(mcon="<table_mcon>", query_type="destination", limit=10)
   ```

### Remediation by available tools

**If the change is intentional and downstream needs updating:**
- If GitHub is available: create a PR that updates downstream models to handle the new schema
- If dbt Cloud is available: after fixing the code, trigger a full rebuild of affected models
- Example: "Column `user_id` was renamed to `customer_id` in `raw_orders`. 4 downstream models reference this column. Creating a PR to update all references."

**If the change is accidental and should be reverted:**
- If GitHub is available: create a PR reverting the upstream change
- If the change was a direct DDL (not code-managed): provide the ALTER statement to revert
- Present findings to the user and recommend they contact the upstream table owner

**If no execution tool is available:**
- Document all affected downstream tables and the specific column changes
- List the files/models that need updating
- Escalate with a complete impact report

### Verification

- Re-check table schema: `getTable(mcon="<table_mcon>", include_fields=true)`
- Verify downstream models are rebuilding successfully
- Check that schema change alerts resolve

---

## Pattern 4: Volume anomaly

### Root cause signals (from TSA)

- "Row count dropped by X%"
- "Row count significantly higher than expected"
- "No new rows in expected time window"
- "Duplicate rows detected"

### Reasoning

Volume anomalies can indicate: data loss (rows missing), data duplication (rows doubled), source system issues (upstream stopped sending data), or filter/logic changes (a WHERE clause changed). The investigation must determine which case applies.

### Investigation steps

1. Quantify the anomaly:
   ```
   getTable(mcon="<table_mcon>")  # current row count
   ```

2. Check if the issue is in this table or upstream:
   ```
   getAssetLineage(mcons=["<table_mcon>"], direction="UPSTREAM")
   getTable(mcon="<upstream_mcon>")  # check upstream row counts
   ```

3. Analyze recent write queries for clues:
   ```
   getQueriesForTable(mcon="<table_mcon>", query_type="destination", limit=10)
   ```
   Look for: unusual DELETE statements, changed WHERE clauses, failed INSERT operations.

4. Check monitoring for more context:
   ```
   getMonitors(mcons=["<table_mcon>"])
   ```

### Remediation by available tools

**For missing data (row count drop):**
- If pipeline orchestrator is available: trigger a backfill for the affected time range
- If the drop is from a bad deployment: revert via GitHub, then rerun the pipeline
- If upstream source stopped: present findings to the user and recommend they contact the source system owner

**For duplicate data:**
- If warehouse access is available: run a deduplication query (with user confirmation!)
- Create a PR to fix the pipeline logic that caused duplication
- Trigger a rebuild after the fix

**For unexpected volume increase:**
- Investigate whether this is a genuine increase or a data quality issue
- Check if upstream sources are sending more data than expected
- If it's a filter/logic change: review recent code changes

**If no execution tool is available:**
- Quantify the anomaly (expected vs actual row counts, affected time range)
- Identify the likely cause from query analysis
- Provide specific remediation steps for the user to execute manually

### Verification

- Re-check row counts: `getTable(mcon="<table_mcon>")`
- Compare against expected values
- Monitor over the next few pipeline runs to confirm stability

---

## Pattern 5: Unknown or complex root cause

### Root cause signals (from TSA)

- TSA returned `failed` status
- TSA `tldr` is unclear or generic ("multiple issues detected")
- Root cause spans multiple systems or teams
- The issue is intermittent and hard to reproduce

### Reasoning

Not every issue has a clear, automatable fix. When the root cause is unclear or complex, the best approach is to present full context to the user and ask for direction — not guess at a fix.

### Context package

Compile a complete summary for the user:

1. **Alert details:** type, severity, when it fired, affected tables
2. **TSA findings:** whatever root cause analysis was available (even if partial)
3. **Blast radius:** downstream consumers, key assets affected
4. **Table state:** current freshness, row counts, schema
5. **Recent queries:** pipeline activity, any anomalous patterns
6. **Monitoring coverage:** what monitors exist, any gaps
7. **Your assessment:** what you think might be wrong and why, with confidence level

### What to do

Present the context package to the user and ask how they'd like to proceed. They may want to:
- Notify their team via Slack or PagerDuty
- Investigate further with specific queries
- Assign the alert to a specific person
- Take a manual remediation action you can help with

**Document on the alert regardless:**
```
createOrUpdateAlertComment(
  alert_id="<alert_uuid>",
  comment="## Investigation Summary\n\n[full context package]\n\n**Why automated remediation was not attempted:** [reason]\n**Recommended next steps:** [specific actions]"
)

updateAlert(
  alert_id="<alert_uuid>",
  status="WORK_IN_PROGRESS"
)
```

### When to use this pattern

- TSA failed or returned unclear results
- Root cause spans multiple systems (e.g., infrastructure + pipeline + data)
- The fix requires access or permissions you don't have
- You're not confident the proposed fix won't cause additional problems
- The issue is intermittent and the current state looks normal
- Multiple alerts are firing simultaneously on related tables (likely a systemic issue)

# Safety Rails

Detailed safety protocols for the remediation skill. These rules are non-negotiable — they apply in every remediation scenario, regardless of severity or urgency.

## Core principles

1. **Investigate before acting.** Never propose a fix without completing the investigation workflow.
2. **Explain before executing.** Never run a remediation action without telling the user what you're about to do and why.
3. **Confirm before destroying.** Any action that modifies data, restarts a pipeline, or changes configuration requires explicit user confirmation.
4. **One step at a time.** Execute one remediation action, verify it, then decide on the next step.
5. **Document everything.** Record all findings and actions on the alert.
6. **Escalate when uncertain.** A clear "I don't know" is safer than a confident wrong fix.

## Confirmation protocol

### Actions that ALWAYS require confirmation

These actions must not be executed without the user explicitly saying "yes", "go ahead", "proceed", or similar:

- **Pipeline triggers:** Starting DAG runs, triggering dbt jobs, launching pipeline executions
- **Data modifications:** Any SQL that includes INSERT, UPDATE, DELETE, DROP, ALTER, TRUNCATE
- **Configuration changes:** Modifying pipeline parameters, changing schedules, updating credentials
- **Code changes:** Creating PRs, committing code, merging branches
- **Incident escalation:** Paging on-call via PagerDuty, creating high-severity incidents
- **Alert status changes:** Marking alerts as FIXED, EXPECTED, or NO_ACTION_NEEDED

### Actions that do NOT require confirmation

These are safe to execute without asking:

- **Read-only investigation:** All Monte Carlo investigation tools (getAlerts, getTable, getAssetLineage, etc.)
- **Adding comments:** `createOrUpdateAlertComment` — documenting findings is always safe
- **Acknowledging alerts:** `updateAlert(status="ACKNOWLEDGED")` — this just signals awareness
- **Setting ownership:** `setAlertOwner` — assigning someone to look at it
- **Sending non-urgent notifications:** Posting informational messages to Slack channels (not paging)
- **Status updates:** `updateAlert(status="WORK_IN_PROGRESS")` — tracking progress

### How to ask for confirmation

Present the action clearly and wait for an explicit response:

> "I'd like to trigger a rerun of the Airflow DAG `etl_orders_daily`. This will:
> - Start a new run of all tasks in the DAG
> - Expected duration: ~45 minutes based on recent runs
> - Risk: minimal — this is a standard rerun, not a backfill
>
> Should I proceed?"

**Do NOT proceed on ambiguous responses.** "Maybe", "I guess", "hmm" are not confirmation. Ask again clearly: "Just to confirm — should I trigger the DAG rerun? (yes/no)"

## Destructive operation handling

### Definition

A "destructive operation" is any action that:
- Deletes or modifies existing data
- Cannot be easily undone
- Affects multiple systems or tables
- Changes infrastructure or configuration

### Required protocol for destructive operations

1. **State the action explicitly:** "I want to execute: `DELETE FROM orders WHERE created_at < '2024-01-01'`"
2. **Explain the impact:** "This will remove approximately 1.2M rows from the `orders` table"
3. **Describe the rollback plan:** "If this causes issues, the data can be restored from the daily backup at s3://backups/orders/2024-01-15/"
4. **Wait for explicit confirmation**
5. **Execute the action**
6. **Immediately verify the result**
7. **Document what was done**

### Actions that are NEVER automated

Even with user confirmation, suggest these be done manually rather than by the agent:

- Dropping tables or databases
- Modifying production credentials or secrets
- Changing IAM roles or permissions
- Directly modifying production infrastructure (scaling, networking)
- Running backfill operations that span more than 7 days of data

For these, provide the exact commands and let the user execute them.

## Escalation criteria

### When to stop and ask the user for direction

Present your findings and ask the user how to proceed when ANY of these conditions are true:

1. **No clear root cause:** TSA failed or returned ambiguous results, and your manual investigation didn't identify a clear cause.

2. **Multiple simultaneous alerts:** More than 3 alerts firing on related tables suggests a systemic issue that needs human judgment.

3. **High blast radius + uncertain fix:** The affected table has >10 downstream consumers AND you're not confident the fix will work.

4. **Data loss detected:** Any sign that data has been permanently deleted or corrupted. Do not attempt to fix data loss — stop and tell the user immediately.

5. **Permission or access issues:** The root cause involves permissions, credentials, or access controls. These require human intervention.

6. **Cross-system failure:** The issue spans multiple systems (e.g., ingestion + transformation + serving) and no single fix addresses it.

7. **Recurring incident:** The same alert has fired 3+ times in the past week. The underlying issue needs a permanent fix, not another band-aid.

8. **Production safety concern:** Any situation where the proposed fix could make things worse, even with a rollback plan.

### How to hand off to the user

1. **Present your findings clearly:** Summarize what you investigated, what you found, and why you're not confident in an automated fix.

2. **Document on the alert:**
   ```
   createOrUpdateAlertComment(
     alert_id="<alert_uuid>",
     comment="## Investigation Summary\n\n**Findings:** [full summary]\n**Why automated remediation was not attempted:** [reason]\n**Recommended next steps:**\n1. [specific step]\n2. [specific step]"
   )
   ```

3. **Set status to WORK_IN_PROGRESS:**
   ```
   updateAlert(alert_id="<alert_uuid>", status="WORK_IN_PROGRESS")
   ```

4. **Ask the user for next steps:** They may want to notify their team, page on-call, investigate further, or take a manual action you can assist with.

## What "uncertain" means in practice

You should consider yourself "uncertain" and ask the user for direction when:

- You can identify multiple plausible root causes and can't narrow it down
- The TSA summary says one thing but your manual investigation suggests something different
- The proposed fix addresses a symptom but not necessarily the root cause
- You've never seen this pattern before in the reference examples
- The fix requires making an assumption about the system that you can't verify
- The user seems uncertain or is asking "are you sure?" — respect their caution

**When in doubt, state your confidence level:**

**Example:**
> "I'm moderately confident (60-70%) that the root cause is [X], based on [evidence]. However, [alternative explanation] is also possible. I'd recommend [safer action] first. If that doesn't resolve it, the data platform team may need to investigate further. How would you like to proceed?"

## Rollback planning

Before executing any remediation action, have a rollback plan:

### For pipeline restarts
- **Rollback:** If the rerun produces bad data, the previous good state is usually available in the warehouse's time travel / versioning feature. Note the timestamp before triggering.

### For dbt reruns
- **Rollback:** dbt models can be rebuilt from source. If a rerun produces bad results, fix the model and rerun again. For incremental models, note the last successful run timestamp.

### For code changes (PRs)
- **Rollback:** Revert the PR. Always create changes as PRs (not direct commits) so they can be cleanly reverted.

### For data modifications
- **Rollback:** Before any data modification, recommend the user:
  1. Create a backup: `CREATE TABLE backup_<table>_<timestamp> AS SELECT * FROM <table>`
  2. Or verify that time travel / snapshots are available for recovery
  3. Document the rollback command alongside the modification

### For configuration changes
- **Rollback:** Document the previous configuration value before changing it. If available, use version-controlled configuration.

# Tool Discovery at Runtime

This reference explains how to discover what remediation tools are available to you at runtime. You have three categories of tools to check: MCP servers, CLI tools (via shell access), and APIs (via `curl` or language-specific clients). Check all three before deciding what's possible.

## Category 1: MCP servers

In Claude Code and other MCP-capable editors, MCP tools follow the naming convention:

```
mcp__<server_name>__<tool_name>
```

For example:
- `mcp__airflow__trigger_dag_run` — an Airflow MCP tool
- `mcp__dbt_cloud__trigger_run` — a dbt Cloud MCP tool
- `mcp__github__create_pull_request` — a GitHub MCP tool

Monte Carlo's own tools are bundled by this plugin and namespaced under the plugin server — see the **Monte Carlo tool routing** block at the top of the skill. Examples:
- `mcp__plugin_mc-agent-toolkit_monte-carlo-mcp__get_alerts`
- `mcp__plugin_mc-agent-toolkit_monte-carlo-mcp__search`

Scan your tool list for any `mcp__*__*` patterns and group by server name.

## Category 2: CLI tools

You have shell access. Many remediation actions can be done via CLI tools that may already be installed:

| CLI Tool | Capability | Example Commands |
| -------- | ---------- | ---------------- |
| `gh` | GitHub operations | `gh pr create`, `gh issue create`, `gh api` |
| `git` | Code changes | `git checkout -b fix/...`, `git commit`, `git push` |
| `dbt` | dbt operations | `dbt run --select <model>`, `dbt test`, `dbt retry` |
| `airflow` | Airflow operations | `airflow dags trigger <dag_id>`, `airflow tasks run` |
| `montecarlo` | Monte Carlo CLI | `montecarlo monitors apply`, `montecarlo collectors test-connection` |
| `curl` | Any HTTP API | Call REST APIs directly for any service with an API |
| `snowsql` / `bq` / `databricks` | Warehouse CLIs | Execute SQL queries, check table state |

**Check availability** by running `which <tool>` or `<tool> --version` before using a CLI tool.

## Category 3: APIs (via curl or HTTP)

If neither an MCP server nor a CLI tool is available for a service, you can often call its REST API directly using `curl`. This is the most flexible option — any service with an API is reachable.

Examples:
```
# Trigger an Airflow DAG via REST API
curl -X POST "https://airflow.example.com/api/v1/dags/<dag_id>/dagRuns" \
  -H "Authorization: Bearer $AIRFLOW_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"conf": {}}'

# Trigger a dbt Cloud job via REST API
curl -X POST "https://cloud.getdbt.com/api/v2/accounts/<account_id>/jobs/<job_id>/run/" \
  -H "Authorization: Token $DBT_CLOUD_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"cause": "Triggered by remediation skill"}'
```

**Note:** API calls require credentials. If you don't have the right tokens or environment variables, ask the user — don't guess at authentication.

## Discovery procedure

### Step 1: Check MCP servers

Scan your tool list for `mcp__*__*` patterns. Common MCP servers relevant to remediation:

| Server Name Pattern | Capability |
| ------------------- | ---------- |
| `*airflow*` | Pipeline orchestration — trigger DAG runs, retry failed tasks |
| `*dagster*` | Pipeline orchestration — launch runs, check status |
| `*prefect*` | Pipeline orchestration — create flow runs, check status |
| `*dbt*` | dbt operations — trigger job runs, get status, list jobs |
| `*github*` | Code changes — create PRs, issues, branches |
| `*gitlab*` | Code changes — create merge requests, issues |
| `*snowflake*` | Warehouse access — execute queries, get table info |
| `*bigquery*` | Warehouse access — execute queries, get table info |
| `*databricks*` | Warehouse access + orchestration — queries, trigger jobs |
| `*fivetran*` | Ingestion — trigger connector sync, check status |
| `*jira*` | Issue tracking — create issues, update status |

### Step 2: Check CLI tools

For any capability not covered by MCP, check if a relevant CLI tool is available. The most common:

- `gh` — covers GitHub operations (PRs, issues, API calls) without needing a GitHub MCP
- `dbt` — covers dbt operations (run, test, retry) without needing a dbt Cloud MCP
- `git` — always available for code changes
- `curl` — always available for calling any REST API

### Step 3: Assess coverage

For the current remediation task, determine:

1. **Can I investigate?** — Monte Carlo MCP is always needed. If it's not connected, you cannot proceed.
2. **Can I execute the fix?** — Check MCP servers first, then CLI tools, then API access.

### Step 4: Report capabilities

Present what you found to the user before proceeding:

> "For this remediation, I can:
> - ✅ Investigate the issue (Monte Carlo MCP)
> - ✅ Restart the Airflow DAG (Airflow MCP connected)
> - ✅ Create a code fix (`gh` CLI available — no GitHub MCP needed)
> - ❌ Rerun the dbt job (no dbt Cloud MCP or `dbt` CLI found)"

## Graceful degradation

When no tool (MCP, CLI, or API) is available for a needed action:

### Priority 1: Provide actionable instructions

Give the user the exact commands or steps to execute themselves:

```
# If no Airflow tool is available but you identified the DAG:
"The DAG `etl_orders_daily` needs to be triggered. Run:
  airflow dags trigger etl_orders_daily
Or via the Airflow UI: navigate to DAGs → etl_orders_daily → Trigger DAG"

# If no dbt tool is available:
"dbt model `stg_orders` needs to be rebuilt. Run:
  dbt run --select stg_orders+
Or via dbt Cloud UI: Jobs → Daily Transform → Run Now"

# If no GitHub tool is available:
"File `models/staging/stg_orders.sql` needs line 14 changed from
  `user_id` to `customer_id`. Create a branch and PR with this change."
```

### Priority 2: Present findings and ask for next steps

Tell the user what you found, what the fix is, and ask how they'd like to proceed. They may run the commands themselves, notify their team, or take a different approach.

### Priority 3: Document on the alert

Always, regardless of what other tools are available:

```
createOrUpdateAlertComment(
  alert_id="<alert_uuid>",
  comment="## Remediation Plan\n\n**Root cause:** [summary]\n**Required action:** [specific fix]\n**Manual steps:**\n1. [step]\n2. [step]\n\n**Investigated by:** AI agent via remediation skill"
)
```

## Tool-specific notes

### Airflow

**MCP tools:** `trigger_dag_run`, `get_dag_runs`, `get_task_instances`, `clear_task_instances`
**CLI:** `airflow dags trigger <dag_id>`, `airflow tasks run <dag_id> <task_id> <execution_date>`
**API:** `POST /api/v1/dags/<dag_id>/dagRuns`

**Caution:** `trigger_dag_run` starts a NEW run. If the issue was a failed task in an existing run, `clear_task_instances` (retry) may be more appropriate than starting fresh.

### dbt

**MCP tools:** `trigger_run` / `trigger_job`, `get_run`, `list_jobs`, `cancel_run`
**CLI:** `dbt run --select <model>`, `dbt retry`, `dbt test --select <model>`
**API:** `POST /api/v2/accounts/<id>/jobs/<id>/run/`

**Note:** dbt Cloud jobs often include multiple models. Triggering a job reruns ALL models in that job, not just the failed one. The `dbt` CLI with `--select` gives more granular control.

### GitHub

**MCP tools:** `create_pull_request`, `create_issue`, `create_or_update_file`, `create_branch`
**CLI:** `gh pr create`, `gh issue create`, `gh api`
**Git:** `git checkout -b`, `git commit`, `git push`

The `gh` CLI is often the most practical option — it doesn't require a GitHub MCP server and supports the full GitHub API via `gh api`.

**Best practice:** For code fixes, create a branch → make the change → open a PR. Don't push directly to main.