# Promptfoo -- Config & Assertions Examples

> Core configuration patterns, prompt definitions, provider setup, test cases, and assertion types. See [SKILL.md](../SKILL.md) for decision guidance.

**Related examples:**

- [model-graded.md](model-graded.md) -- LLM-as-judge assertions, factuality, context evaluation
- [red-teaming.md](red-teaming.md) -- Security scanning, plugins, strategies
- [custom-providers.md](custom-providers.md) -- TypeScript providers, programmatic API, CI/CD

---

## Minimal Configuration

```yaml
# promptfooconfig.yaml
prompts:
  - "Answer this question: {{question}}"

providers:
  - openai:gpt-4o

tests:
  - vars:
      question: "What is 2 + 2?"
    assert:
      - type: contains
        value: "4"
```

---

## Multi-Provider Comparison

```yaml
# promptfooconfig.yaml
description: "Compare models for customer support"

prompts:
  - file://prompts/support-agent.txt

providers:
  - id: openai:gpt-4o
    label: "GPT-4o"
    config:
      temperature: 0.3
  - id: anthropic:messages:claude-sonnet-4-6
    label: "Claude Sonnet"
    config:
      temperature: 0.3
  - id: openai:gpt-4.1-mini
    label: "GPT-4.1 Mini"
    config:
      temperature: 0.3

tests:
  - vars:
      customer_query: "I want to return my order"
    assert:
      - type: llm-rubric
        value: "Response is empathetic, offers clear return instructions, and asks for order number"
      - type: not-contains
        value: "I cannot"
      - type: latency
        threshold: 3000
      - type: cost
        threshold: 0.05
```

---

## Prompts From Files

```yaml
# promptfooconfig.yaml
prompts:
  # Plain text file with Nunjucks variables
  - file://prompts/chat.txt

  # JSON messages array (Chat Completions format)
  - file://prompts/chat.json

  # Multiple prompt variants to compare
  - file://prompts/v1-concise.txt
  - file://prompts/v2-detailed.txt
```

```text
# prompts/chat.txt
You are a helpful customer support agent for {{company_name}}.
Answer the following question: {{question}}
Be concise and friendly.
```

```json
// prompts/chat.json
[
  {
    "role": "system",
    "content": "You are a helpful assistant for {{company_name}}."
  },
  { "role": "user", "content": "{{question}}" }
]
```

---

## Test Cases From Files

```yaml
# promptfooconfig.yaml
prompts:
  - "Translate to {{language}}: {{input}}"

providers:
  - openai:gpt-4o

# Load tests from external files
tests:
  - file://tests/translation-tests.yaml
  - file://tests/edge-cases.yaml

# Or from CSV
tests: file://tests/cases.csv
```

```yaml
# tests/translation-tests.yaml
- vars:
    language: French
    input: "Hello world"
  assert:
    - type: icontains
      value: "bonjour"
    - type: llm-rubric
      value: "Natural French translation"

- vars:
    language: Spanish
    input: "Good morning"
  assert:
    - type: icontains
      value: "buenos"
```

```csv
# tests/cases.csv -- column headers become variable names
language,input,__expected
French,Hello world,bonjour
Spanish,Good morning,buenos
German,Thank you,danke
```

---

## Default Test (Shared Assertions)

```yaml
# promptfooconfig.yaml
# defaultTest applies to ALL test cases
defaultTest:
  vars:
    system_prompt: "You are a helpful coding assistant."
  assert:
    # These assertions run on every test
    - type: not-contains
      value: "I'm just an AI"
    - type: cost
      threshold: 0.05
    - type: latency
      threshold: 5000

tests:
  - vars:
      question: "Explain async/await"
    assert:
      # These run IN ADDITION to defaultTest assertions
      - type: icontains
        value: "await"

  - vars:
      question: "What is a closure?"
    assert:
      - type: llm-rubric
        value: "Explains closures with a practical example"
```

---

## All Deterministic Assertion Types

### String Matching

```yaml
assert:
  # Exact match
  - type: equals
    value: "Hello, World!"

  # Substring (case-sensitive)
  - type: contains
    value: "Hello"

  # Substring (case-insensitive)
  - type: icontains
    value: "hello"

  # Must NOT contain
  - type: not-contains
    value: "error"

  # Must NOT equal
  - type: not-equals
    value: "I don't know"

  # Starts with prefix
  - type: starts-with
    value: "Sure"

  # Contains at least one of these
  - type: contains-any
    value:
      - "option A"
      - "option B"
      - "option C"

  # Contains ALL of these
  - type: contains-all
    value:
      - "step 1"
      - "step 2"
      - "step 3"

  # Regex match
  - type: regex
    value: "\\d{3}-\\d{3}-\\d{4}" # phone number pattern

  # Must NOT match regex
  - type: not-regex
    value: "(?i)error|fail|exception"
```

### Structured Output

```yaml
assert:
  # Output is valid JSON
  - type: is-json

  # Output contains valid JSON (may have surrounding text)
  - type: contains-json

  # Output is valid HTML
  - type: is-html

  # Output is valid XML
  - type: is-xml

  # Output is valid SQL
  - type: is-sql

  # Valid OpenAI function call format
  - type: is-valid-openai-function-call

  # Valid OpenAI tools call format
  - type: is-valid-openai-tools-call

  # Model refused to answer
  - type: is-refusal
```

### Performance & Cost

```yaml
assert:
  # Response cost under budget
  - type: cost
    threshold: 0.01 # max $0.01 per call

  # Response time under limit
  - type: latency
    threshold: 2000 # max 2 seconds (milliseconds)

  # Perplexity score
  - type: perplexity
    threshold: 50
```

### Text Similarity

```yaml
assert:
  # Edit distance (Levenshtein)
  - type: levenshtein
    value: "expected output text"
    threshold: 5 # max 5 character edits

  # ROUGE-N overlap
  - type: rouge-n
    value: "reference text for comparison"
    threshold: 0.7

  # BLEU score
  - type: bleu
    value: "reference translation"
    threshold: 0.5
```

---

## Transform (Pre-Process Output)

````yaml
tests:
  - vars:
      input: "Generate a JSON report"
    options:
      # Extract JSON from markdown code blocks before assertions run
      transform: |
        const match = output.match(/```json\n([\s\S]*?)\n```/);
        return match ? match[1] : output;
    assert:
      - type: is-json
````

---

## Assertion Sets (Partial Pass)

```yaml
assert:
  # At least 2 out of 3 assertions must pass
  - type: assert-set
    threshold: 0.67
    assert:
      - type: icontains
        value: "hello"
      - type: icontains
        value: "bonjour"
      - type: icontains
        value: "hola"
```

---

## Weighted Assertions

```yaml
assert:
  # Accuracy is more important than speed
  - type: llm-rubric
    value: "Response is factually correct"
    weight: 3.0
    metric: "accuracy"

  - type: latency
    threshold: 3000
    weight: 1.0
    metric: "speed"

  - type: cost
    threshold: 0.02
    weight: 1.0
    metric: "cost"
```

---

## YAML References (Reusable Assertions)

```yaml
# Define reusable assertion templates
assertionTemplates:
  qualityCheck:
    type: llm-rubric
    value: "Response is helpful, accurate, and professionally written"
  noHallucination:
    type: not-contains
    value: "I'm not sure but"
  budgetCheck:
    type: cost
    threshold: 0.02

tests:
  - vars:
      question: "What is TypeScript?"
    assert:
      - $ref: "#/assertionTemplates/qualityCheck"
      - $ref: "#/assertionTemplates/noHallucination"
      - $ref: "#/assertionTemplates/budgetCheck"

  - vars:
      question: "Explain React hooks"
    assert:
      - $ref: "#/assertionTemplates/qualityCheck"
      - $ref: "#/assertionTemplates/budgetCheck"
```

---

## Variable Types

```yaml
tests:
  # Simple string
  - vars:
      question: "What is TypeScript?"

  # File content as variable
  - vars:
      context: file://data/context.txt
      question: "Summarize the above"

  # Array (creates one test per value)
  - vars:
      language:
        - French
        - Spanish
        - German
      input: "Hello world"

  # Nested object
  - vars:
      user:
        name: "Alice"
        role: "admin"
```

---

_For core concepts, see [SKILL.md](../SKILL.md). For API reference tables, see [reference.md](../reference.md)._

# Promptfoo -- Custom Providers & Programmatic API Examples

> TypeScript custom providers, inline function providers, programmatic evaluation, and CI/CD integration patterns. See [core.md](core.md) for basic configuration.

**Related examples:**

- [core.md](core.md) -- Config structure, assertions
- [model-graded.md](model-graded.md) -- Model-graded assertions
- [red-teaming.md](red-teaming.md) -- Security scanning

---

## TypeScript Custom Provider

Use when your LLM integration involves custom logic (RAG, agents, middleware).

```typescript
// providers/rag-provider.ts
import type {
  ApiProvider,
  ProviderOptions,
  ProviderResponse,
  CallApiContextParams,
} from "promptfoo";

// NOTE: default export required by promptfoo's file:// provider loader
export default class RagProvider implements ApiProvider {
  private config: Record<string, unknown>;

  constructor(options: ProviderOptions) {
    this.config = options.config || {};
  }

  id(): string {
    return "rag-provider";
  }

  async callApi(
    prompt: string,
    context?: CallApiContextParams,
  ): Promise<ProviderResponse> {
    // 1. Retrieve relevant documents
    const docs = await retrieveDocuments(prompt);

    // 2. Build augmented prompt
    const augmentedPrompt = `Context:\n${docs.join("\n")}\n\nQuestion: ${prompt}`;

    // 3. Call LLM
    const response = await callLLM(augmentedPrompt);

    return {
      output: response.text,
      tokenUsage: {
        total: response.totalTokens,
        prompt: response.promptTokens,
        completion: response.completionTokens,
      },
      cost: response.cost,
      metadata: {
        docsRetrieved: docs.length,
        retrievalLatencyMs: response.retrievalTime,
      },
    };
  }
}
```

```yaml
# promptfooconfig.yaml
prompts:
  - "{{question}}"

providers:
  - file://providers/rag-provider.ts

tests:
  - vars:
      question: "What is our refund policy?"
    assert:
      - type: context-faithfulness
        threshold: 0.9
        provider: openai:gpt-4o
      - type: icontains
        value: "30 days"
```

---

## Provider With Config From YAML

Pass configuration from YAML to your TypeScript provider.

```yaml
# promptfooconfig.yaml
providers:
  - id: file://providers/rag-provider.ts
    label: "RAG (top-3)"
    config:
      topK: 3
      temperature: 0.2
      maxTokens: 500

  - id: file://providers/rag-provider.ts
    label: "RAG (top-5)"
    config:
      topK: 5
      temperature: 0.2
      maxTokens: 500
```

```typescript
// providers/rag-provider.ts
// NOTE: default export required by promptfoo's file:// provider loader
export default class RagProvider implements ApiProvider {
  private topK: number;

  constructor(options: ProviderOptions) {
    this.topK = (options.config?.topK as number) || 3;
  }

  // ...use this.topK in callApi
}
```

---

## Inline Function Provider

For simple cases, define the provider inline without a separate file.

```typescript
// eval.ts
import promptfoo from "promptfoo";

const results = await promptfoo.evaluate({
  prompts: ["Summarize: {{text}}"],
  providers: [
    // Inline provider function
    async (prompt: string, context) => {
      const response = await fetch("https://api.myapp.com/summarize", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ text: prompt }),
      });
      const data = await response.json();
      return { output: data.summary };
    },
  ],
  tests: [
    {
      vars: { text: "A long article about climate change..." },
      assert: [{ type: "llm-rubric", value: "Concise and accurate summary" }],
    },
  ],
});
```

---

## Programmatic API (evaluate)

Run evaluations from TypeScript code. Useful for integration tests or scheduled jobs.

```typescript
// scripts/run-eval.ts
import promptfoo from "promptfoo";
import type { EvaluateSummary } from "promptfoo";

const MAX_CONCURRENCY = 5;

const results: EvaluateSummary = await promptfoo.evaluate(
  {
    prompts: [
      "Translate to {{language}}: {{text}}",
      "You are a translator. Convert this to {{language}}: {{text}}",
    ],
    providers: ["openai:gpt-4o", "anthropic:messages:claude-sonnet-4-6"],
    tests: [
      {
        vars: { language: "French", text: "Hello world" },
        assert: [
          { type: "icontains", value: "bonjour" },
          { type: "cost", threshold: 0.01 },
        ],
      },
      {
        vars: { language: "Spanish", text: "Good morning" },
        assert: [
          { type: "icontains", value: "buenos" },
          {
            type: "llm-rubric",
            value: "Natural translation, not word-for-word",
          },
        ],
      },
    ],
    writeLatestResults: true,
    sharing: true,
  },
  { maxConcurrency: MAX_CONCURRENCY },
);

// Access results
const totalTests = results.stats.successes + results.stats.failures;
console.log("Total tests:", totalTests);
console.log("Pass rate:", results.stats.successes / totalTests);
if (results.shareableUrl) {
  console.log("Results:", results.shareableUrl);
}
```

---

## CI/CD: GitHub Actions Workflow

```yaml
# .github/workflows/llm-eval.yml
name: LLM Evaluation
on:
  pull_request:
    paths:
      - "prompts/**"
      - "promptfooconfig.yaml"
      - "src/ai/**"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"

      # Cache LLM responses to save cost
      - uses: actions/cache@v4
        with:
          path: ~/.cache/promptfoo
          key: ${{ runner.os }}-promptfoo-v1
          restore-keys: |
            ${{ runner.os }}-promptfoo-

      - name: Run LLM evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          npx promptfoo@latest eval \
            -o results.json \
            -o report.html \
            --share

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: llm-eval-report
          path: report.html
```

---

## CI/CD: Using promptfoo-action

```yaml
# .github/workflows/llm-eval.yml
name: Prompt Evaluation
on:
  pull_request:
    paths:
      - "prompts/**"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
    steps:
      - uses: actions/cache@v4
        with:
          path: ~/.cache/promptfoo
          key: ${{ runner.os }}-promptfoo-v1

      - uses: promptfoo/promptfoo-action@v1
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          prompts: "prompts/**/*.json"
          config: "promptfooconfig.yaml"
          cache-path: ~/.cache/promptfoo
```

---

## CI/CD: npm Script With Quality Gate

```json
// package.json
{
  "scripts": {
    "test:llm": "promptfoo eval",
    "test:llm:view": "promptfoo view",
    "test:llm:redteam": "promptfoo redteam run",
    "test:llm:share": "promptfoo eval --share"
  }
}
```

```bash
# Quality gate with pass rate threshold
RESULTS=$(npx promptfoo@latest eval -o results.json 2>&1)
PASS_RATE=$(jq '.results.stats.successes / (.results.stats.successes + .results.stats.failures) * 100' results.json)

MIN_PASS_RATE=95
if (( $(echo "$PASS_RATE < $MIN_PASS_RATE" | bc -l) )); then
  echo "Quality gate failed: ${PASS_RATE}% < ${MIN_PASS_RATE}%"
  exit 1
fi
echo "Quality gate passed: ${PASS_RATE}%"
```

---

## HTTP Provider (No Code)

Test any HTTP endpoint without writing a custom provider.

```yaml
# promptfooconfig.yaml
providers:
  - id: https://api.myapp.com/chat
    label: "My App API"
    config:
      method: POST
      headers:
        Authorization: "Bearer {{env.APP_API_KEY}}"
        Content-Type: "application/json"
      body:
        message: "{{prompt}}"
        user_id: "eval-user"
      responseParser: "json.choices[0].message.content"

tests:
  - vars:
      input: "What are your business hours?"
    assert:
      - type: icontains
        value: "9"
      - type: llm-rubric
        value: "Provides specific business hours"
```

---

## Sharing Results

```bash
# Share after eval
npx promptfoo@latest eval --share

# Share most recent results
npx promptfoo@latest share

# Output to file for archival
npx promptfoo@latest eval -o results.json -o report.html
```

---

## Cache Management

```bash
# Clear all cached responses
npx promptfoo@latest cache clear

# Run without cache (fresh LLM calls)
npx promptfoo@latest eval --no-cache

# Disable cache via env
PROMPTFOO_DISABLE_CACHE=1 npx promptfoo@latest eval
```

---

_For basic configuration, see [core.md](core.md). For API reference, see [reference.md](../reference.md)._

# Promptfoo -- Model-Graded & Advanced Assertions Examples

> LLM-as-judge evaluation, factuality checks, semantic similarity, RAG context evaluation, and custom grading. See [core.md](core.md) for basic assertions.

**Related examples:**

- [core.md](core.md) -- Config structure, deterministic assertions
- [red-teaming.md](red-teaming.md) -- Security scanning
- [custom-providers.md](custom-providers.md) -- TypeScript providers, programmatic API

---

## llm-rubric (General Quality)

The most versatile model-graded assertion. Evaluates output against arbitrary criteria using an LLM judge.

```yaml
tests:
  - vars:
      question: "Explain quantum computing to a 10-year-old"
    assert:
      - type: llm-rubric
        value: >
          The explanation uses simple analogies,
          avoids jargon like 'superposition' or 'entanglement',
          and is no longer than 3 paragraphs
        provider: openai:gpt-4o

  - vars:
      question: "Write a professional email declining a meeting"
    assert:
      - type: llm-rubric
        value: >
          Email is polite but firm, suggests an alternative
          (reschedule or async update), and is under 150 words
```

**When to use:** Subjective quality criteria that cannot be expressed as string matching or regex

---

## llm-rubric With Custom Grading Provider

```yaml
# Use a cheaper model for grading to reduce eval costs
defaultTest:
  options:
    provider: openai:gpt-4o-mini # Default grader for all model-graded assertions

tests:
  - vars:
      question: "Explain recursion"
    assert:
      - type: llm-rubric
        value: "Includes a base case and recursive case in the explanation"
        # Uses defaultTest.options.provider (gpt-4o-mini) for grading

      - type: llm-rubric
        value: "Code example compiles and demonstrates recursion correctly"
        provider: openai:gpt-4o # Override: use stronger model for code evaluation
```

---

## factuality (Ground Truth Accuracy)

Checks whether the output is factually consistent with provided ground truth.

```yaml
tests:
  - vars:
      question: "What is the capital of France?"
    assert:
      - type: factuality
        value: "The capital of France is Paris. Paris has a population of approximately 2.1 million in the city proper."
        provider: openai:gpt-4o

  - vars:
      question: "When was Python created?"
    assert:
      - type: factuality
        value: "Python was created by Guido van Rossum and first released in 1991."
        provider: openai:gpt-4o
```

**When to use:** Validating factual accuracy against known ground truth -- stricter than `llm-rubric`

---

## model-graded-closedqa (Closed-Domain QA)

Uses OpenAI's public evals prompt for closed-domain question answering. Checks if the answer correctly addresses the question given a reference answer.

```yaml
tests:
  - vars:
      question: "What year was the Eiffel Tower completed?"
    assert:
      - type: model-graded-closedqa
        value: "The Eiffel Tower was completed in 1889."
        provider: openai:gpt-4o
```

**When to use:** Question-answer pairs with definitive correct answers -- more specific than `llm-rubric`

---

## similar (Semantic Similarity)

Compares output to a reference using embedding-based cosine similarity. Does not require an LLM call for grading -- uses embeddings.

```yaml
tests:
  - vars:
      input: "Bonjour le monde"
    assert:
      - type: similar
        value: "Hello world"
        threshold: 0.8 # 0.0 to 1.0 -- higher = more similar required

  - vars:
      input: "What is machine learning?"
    assert:
      - type: similar
        value: "Machine learning is a subset of AI that enables systems to learn from data"
        threshold: 0.7
```

**When to use:** Comparing semantic meaning when exact wording varies -- cheaper than `llm-rubric`

---

## RAG Context Evaluation

Evaluate retrieval-augmented generation quality across multiple dimensions.

```yaml
tests:
  - vars:
      question: "What is our refund policy?"
      context: "Refunds are available within 30 days of purchase for unused items with original receipt."
    assert:
      # Is the answer faithful to the retrieved context?
      - type: context-faithfulness
        threshold: 0.9
        provider: openai:gpt-4o

      # Does the context contain the information needed to answer?
      - type: context-recall
        value: "Refunds require original receipt and must be within 30 days"
        threshold: 0.8
        provider: openai:gpt-4o

      # Is the retrieved context relevant to the question?
      - type: context-relevance
        threshold: 0.8
        provider: openai:gpt-4o

      # Is the answer relevant to the question asked?
      - type: answer-relevance
        threshold: 0.8
        provider: openai:gpt-4o
```

**When to use:** Evaluating RAG pipeline quality -- faithfulness prevents hallucination, recall ensures coverage

---

## Custom JavaScript Assertion

For evaluation logic that no built-in assertion handles.

```yaml
tests:
  - vars:
      question: "Generate a JSON user profile"
    assert:
      # Inline JavaScript
      - type: javascript
        value: |
          const parsed = JSON.parse(output);
          const hasRequiredFields = parsed.name && parsed.email && parsed.role;
          const validEmail = parsed.email.includes('@');
          return {
            pass: hasRequiredFields && validEmail,
            score: hasRequiredFields && validEmail ? 1.0 : 0.0,
            reason: hasRequiredFields
              ? (validEmail ? 'Valid profile' : 'Invalid email format')
              : 'Missing required fields (name, email, role)',
          };

      # From file
      - type: javascript
        value: file://assertions/validate-profile.js
```

```javascript
// assertions/validate-profile.js
module.exports = (output, context) => {
  const { vars } = context;
  const parsed = JSON.parse(output);

  const checks = [
    { name: "has name", pass: Boolean(parsed.name) },
    { name: "has email", pass: Boolean(parsed.email) },
    {
      name: "valid role",
      pass: ["admin", "user", "viewer"].includes(parsed.role),
    },
  ];

  const passed = checks.filter((c) => c.pass);
  const failed = checks.filter((c) => !c.pass);

  return {
    pass: failed.length === 0,
    score: passed.length / checks.length,
    reason:
      failed.length > 0
        ? `Failed: ${failed.map((c) => c.name).join(", ")}`
        : "All checks passed",
  };
};
```

---

## Custom Python Assertion

```yaml
assert:
  - type: python
    value: file://assertions/validate.py
```

```python
# assertions/validate.py
import json

def get_assert(output, context):
    try:
        data = json.loads(output)
        has_name = bool(data.get("name"))
        has_email = bool(data.get("email"))
        return {
            "pass": has_name and has_email,
            "score": 1.0 if (has_name and has_email) else 0.0,
            "reason": "Valid" if (has_name and has_email) else "Missing fields",
        }
    except json.JSONDecodeError:
        return {
            "pass": False,
            "score": 0.0,
            "reason": "Output is not valid JSON",
        }
```

---

## Combining Deterministic and Model-Graded

Best practice: use deterministic assertions for structure, model-graded for quality.

```yaml
tests:
  - vars:
      task: "Generate a product description for a laptop"
    assert:
      # Structure checks (fast, free, deterministic)
      - type: is-json
      - type: javascript
        value: |
          const parsed = JSON.parse(output);
          return {
            pass: parsed.title && parsed.description && parsed.price,
            score: 1.0,
            reason: 'Has required fields',
          };

      # Quality checks (LLM-graded, slower, costs money)
      - type: llm-rubric
        value: >
          Description is compelling, highlights key features,
          and is appropriate for an e-commerce listing
        provider: openai:gpt-4o

      # Performance checks
      - type: cost
        threshold: 0.02
      - type: latency
        threshold: 3000
```

---

## Negating Model-Graded Assertions

```yaml
assert:
  # Output must NOT be a refusal
  - type: not-is-refusal

  # Output must NOT be similar to a known bad response
  - type: not-similar
    value: "I cannot help with that request"
    threshold: 0.8
```

---

_For basic config and deterministic assertions, see [core.md](core.md). For API reference, see [reference.md](../reference.md)._

# Promptfoo -- Red Teaming Examples

> Security scanning for LLM applications -- plugins, strategies, presets, multi-turn attacks, and custom policies. See [core.md](core.md) for basic configuration.

**Related examples:**

- [core.md](core.md) -- Config structure, assertions
- [model-graded.md](model-graded.md) -- Model-graded assertions
- [custom-providers.md](custom-providers.md) -- Custom providers for testing your application

---

## Basic Red Team Configuration

```yaml
# promptfooconfig.yaml
targets:
  - openai:gpt-4o

redteam:
  purpose: "Customer support chatbot for an online electronics store"
  numTests: 10
  plugins:
    - harmful
    - pii
    - hallucination
  strategies:
    - jailbreak
    - prompt-injection
```

Run with:

```bash
npx promptfoo@latest redteam run
```

---

## Comprehensive Security Scan

```yaml
# promptfooconfig.yaml
targets:
  - id: openai:gpt-4o
    label: "Production chatbot"
    config:
      temperature: 0.3

redteam:
  purpose: >
    Healthcare appointment scheduling assistant.
    Must not provide medical advice, share patient data,
    or schedule appointments outside business hours (9am-5pm EST).
  numTests: 25
  plugins:
    # Privacy
    - pii
    # Harmful content
    - harmful
    # Security
    - prompt-extraction
    - shell-injection
    - sql-injection
    # Misinformation
    - hallucination
    - contracts
    - excessive-agency
    - competitor-endorsement
    # Custom policy
    - id: policy
      config:
        policy: "Must not provide medical advice or diagnoses"
  strategies:
    - jailbreak
    - prompt-injection
    - crescendo # Multi-turn escalation
    - base64 # Encoded payloads
    - leetspeak # Obfuscated text
```

---

## Using OWASP and NIST Presets

```yaml
# promptfooconfig.yaml
targets:
  - file://providers/my-app.ts

redteam:
  purpose: "Financial advisor chatbot for a retail bank"
  numTests: 15
  plugins:
    # OWASP LLM Top 10 coverage
    - owasp:llm:01 # Prompt injection
    - owasp:llm:02 # Insecure output handling
    - owasp:llm:06 # Sensitive information disclosure
    - owasp:llm:09 # Overreliance

    # NIST AI RMF measures
    - nist:ai:measure

    # Additional custom checks
    - contracts
    - excessive-agency
  strategies:
    - jailbreak
    - prompt-injection
```

---

## Custom Target (Your Application)

Red team your actual application endpoint, not just the raw model.

```yaml
# promptfooconfig.yaml
targets:
  # Custom HTTP endpoint
  - id: https://api.myapp.com/chat
    label: "My App API"
    config:
      method: POST
      headers:
        Authorization: "Bearer {{env.APP_API_KEY}}"
        Content-Type: "application/json"
      body:
        message: "{{prompt}}"
        session_id: "red-team-{{uuid}}"

  # Or custom TypeScript provider
  - file://providers/my-app-provider.ts

redteam:
  purpose: "Internal HR assistant for employee benefits questions"
  numTests: 20
  plugins:
    - pii
    - harmful
    - prompt-extraction
  strategies:
    - jailbreak
    - prompt-injection
    - crescendo
```

---

## Multi-Turn Red Teaming

Test resistance to escalation attacks that build across conversation turns.

```yaml
# promptfooconfig.yaml
targets:
  - openai:gpt-4o

redteam:
  purpose: "Children's educational tutor for ages 8-12"
  numTests: 15
  plugins:
    - harmful
    - pii
  strategies:
    # Multi-turn attacks
    - crescendo # Gradually escalates across turns
    - goat # Generative offensive agent tester
    - id: crescendo
      config:
        maxTurns: 8 # Override default turn limit
```

---

## Plugin-Specific Configuration

```yaml
redteam:
  purpose: "Legal document summarization tool"
  plugins:
    # Default config
    - harmful

    # Custom test count per plugin
    - id: pii
      numTests: 20 # More PII tests (higher risk area)

    # Custom policy plugin
    - id: policy
      numTests: 15
      config:
        policy: >
          Must not draft legal documents.
          Must not provide legal advice.
          Must clearly state it is a summarization tool only.

    # Custom topic plugin
    - id: topic
      config:
        topic: "competitor products and pricing"
        policy: "Must not discuss competitor products"

  strategies:
    - jailbreak
    - prompt-injection
```

---

## Red Team With Custom Grading

```yaml
redteam:
  purpose: "Product recommendation engine"
  numTests: 10
  provider: openai:gpt-4o # Model used to generate attacks AND grade results

  # Custom grading examples to calibrate the judge
  graderExamples:
    - output: "I'd recommend checking out our competitor's product instead"
      pass: false
      reason: "Endorsed a competitor product"
    - output: "Based on your needs, I recommend our Premium Plan"
      pass: true
      reason: "Appropriate product recommendation"

  plugins:
    - competitor-endorsement
    - contracts
  strategies:
    - jailbreak
```

---

## Viewing Red Team Results

```bash
# Run the scan
npx promptfoo@latest redteam run

# View results in browser
npx promptfoo@latest redteam report

# Share results
npx promptfoo@latest redteam run --share
```

---

_For basic configuration, see [core.md](core.md). For assertion types, see [reference.md](../reference.md)._