Claude Skill

ai-observability-promptfoo

Testing and evaluation framework for LLM prompts and applications -- promptfooconfig.yaml, assertions, model-graded evals, red teaming, CI/CD integration, custom providers, and comparative evaluation

LLM Mart · 0 points · 3 views 0 listing impressions 0 install-command copies
Download agents-inc-skills-dist_plugins_ai-observability-promptfoo_skills_ai-observability-promptfoo-707467a.zip · 20 KB

Install

skills CLI npx skills add https://github.com/agents-inc/skills/tree/main/dist/plugins/ai-observability-promptfoo/skills/ai-observability-promptfoo
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install agents-inc-skills@llmmart
Git git clone https://github.com/agents-inc/skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole agents-inc/skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Promptfoo Patterns

Quick Guide: Use promptfoo for systematic LLM evaluation. Define prompts, providers, and test cases in promptfooconfig.yaml. Use assertion types (contains, is-json, llm-rubric, similar, cost, latency) to validate outputs. Use promptfoo eval to run (exits with code 100 on test failures), promptfoo view for results UI. Use model-graded assertions (llm-rubric, factuality) for subjective quality. Use promptfoo redteam run for security scanning. Use --share flag or promptfoo share to share results. All provider API keys come from environment variables -- never hardcode them.


<critical_requirements>

CRITICAL: Before Using This Skill

(You MUST define test cases with explicit assert arrays -- tests without assertions only capture output without validating it)

(You MUST use llm-rubric for subjective quality evaluation -- do NOT rely solely on deterministic assertions for natural language output)

(You MUST set threshold on similarity and model-graded assertions -- omitting thresholds uses defaults that may not match your quality bar)

(You MUST use environment variables for all API keys -- never hardcode keys in promptfooconfig.yaml or provider configs)

(You MUST verify promptfoo eval exit code in CI pipelines -- it returns exit code 100 on test failures, exit code 1 on other errors)

</critical_requirements>


Auto-detection: promptfoo, promptfooconfig, promptfooconfig.yaml, promptfoo eval, promptfoo view, promptfoo redteam, llm-rubric, model-graded-closedqa, promptfoo share, promptfoo cache, assertion type, LLM evaluation, prompt testing, red teaming, PROMPTFOO_CONFIG

When to use:

  • Writing or evaluating LLM prompts across one or more providers
  • Setting up automated test suites for LLM-powered features
  • Comparing model outputs side-by-side (GPT vs Claude vs Gemini)
  • Running model-graded evaluations (LLM-as-a-judge)
  • Red teaming LLM applications for security vulnerabilities
  • Integrating LLM quality gates into CI/CD pipelines
  • Validating structured output (JSON, function calls) from LLMs

Key patterns covered:

  • promptfooconfig.yaml structure (prompts, providers, tests, defaultTest)
  • Assertion types (deterministic, model-graded, performance)
  • Custom TypeScript providers
  • Red teaming configuration (plugins, strategies)
  • CI/CD integration with GitHub Actions
  • Programmatic API (evaluate() function)
  • Result sharing and caching

When NOT to use:

  • Unit testing application code (use your test runner)
  • Load testing / benchmarking API throughput (use a load testing tool)
  • Runtime monitoring of production LLM calls (use observability tooling)

Examples Index




<decision_framework>

Decision Framework

Which Assertion Type to Use

What are you validating?
+-- Exact or structural match?
|   +-- Exact text -> equals
|   +-- Contains substring -> contains / icontains
|   +-- Regex pattern -> regex
|   +-- Valid JSON -> is-json
|   +-- Valid function call -> is-valid-openai-tools-call
|   +-- Cost under budget -> cost (with threshold)
|   +-- Response time -> latency (with threshold)
+-- Subjective quality?
|   +-- General quality criteria -> llm-rubric
|   +-- Factual accuracy against ground truth -> factuality
|   +-- Semantic similarity -> similar (with threshold)
|   +-- Closed-domain QA accuracy -> model-graded-closedqa
|   +-- RAG context fidelity -> context-faithfulness
+-- Custom logic?
    +-- JavaScript function -> javascript
    +-- Python function -> python
    +-- External service -> webhook

When to Use Red Teaming vs Eval

What are you testing?
+-- Prompt quality and correctness?
|   +-- Use promptfoo eval with test cases and assertions
+-- Security vulnerabilities?
|   +-- Use promptfoo redteam run with plugins and strategies
+-- Both?
    +-- Run eval for quality, redteam for security -- separate configs or sections

Provider Selection

How does your LLM integration work?
+-- Direct API call to OpenAI/Anthropic/etc?
|   +-- Use built-in provider: openai:gpt-4o, anthropic:messages:claude-sonnet-4-6
+-- Custom pipeline (RAG, agents, middleware)?
|   +-- Use custom TypeScript provider: file://providers/my-app.ts
+-- HTTP endpoint?
|   +-- Use HTTP provider: id: https://api.example.com/chat
+-- Multiple providers to compare?
    +-- List all in providers array -- promptfoo runs tests against each

</decision_framework>


<red_flags>

RED FLAGS

High Priority Issues:

  • Tests without assert arrays (output is captured but never validated -- tests always "pass")
  • Not checking promptfoo eval exit code in CI (promptfoo eval exits 100 on test failures -- ensure your CI pipeline treats non-zero exit codes as failures)
  • Hardcoded API keys in promptfooconfig.yaml (use environment variables)
  • Using llm-rubric for checks that is-json or contains can do deterministically (wastes money and adds non-determinism)
  • Red teaming without purpose (generic attacks miss application-specific vulnerabilities)

Medium Priority Issues:

  • Missing threshold on similar assertions (default may not match your quality bar)
  • Not caching in CI (every run makes full API calls -- expensive and slow)
  • Using model-graded-closedqa when llm-rubric would be simpler (closedqa is for specific ground-truth QA)
  • Not setting provider on model-graded assertions (uses default which may not be the grader you want)
  • Running red team with default numTests: 5 in production scans (too few for comprehensive coverage)

Common Mistakes:

  • Confusing prompts (the LLM prompt templates) with tests (the evaluation cases) -- prompts define what to send, tests define what to check
  • Using equals for natural language output (LLM output is non-deterministic, use llm-rubric or similar)
  • Forgetting {{variable}} syntax in prompts (promptfoo uses Nunjucks templating, not ${variable})
  • Putting assertions in defaultTest that should only apply to specific tests (assertions in defaultTest apply to ALL tests)
  • Using file:// paths without the prefix (promptfoo treats bare paths as literal strings, not file references)

Gotchas & Edge Cases:

  • promptfoo eval caches LLM responses by default -- use promptfoo cache clear or --no-cache to force fresh calls
  • --share uploads results to promptfoo's servers -- do not use with sensitive data unless self-hosting
  • Red team strategies wrap plugins output -- a plugin generates the malicious content, a strategy delivers it (e.g., via jailbreak encoding)
  • defaultTest.assert merges with per-test assertions, it does not replace them -- both arrays run
  • CSV test files map column headers to variable names -- header input becomes {{input}} in prompts
  • transform in test options runs JavaScript on the output before assertions -- useful for extracting JSON from markdown-wrapped responses
  • Provider configs in YAML use config: key for model parameters (temperature, max_tokens), not top-level fields
  • The weight property on assertions affects scoring in the results UI but does not change pass/fail behavior

</red_flags>


<critical_reminders>

CRITICAL REMINDERS

(You MUST define test cases with explicit assert arrays -- tests without assertions only capture output without validating it)

(You MUST use llm-rubric for subjective quality evaluation -- do NOT rely solely on deterministic assertions for natural language output)

(You MUST set threshold on similarity and model-graded assertions -- omitting thresholds uses defaults that may not match your quality bar)

(You MUST use environment variables for all API keys -- never hardcode keys in promptfooconfig.yaml or provider configs)

(You MUST verify promptfoo eval exit code in CI pipelines -- it returns exit code 100 on test failures, exit code 1 on other errors)

Failure to follow these rules will produce untested, insecure, or falsely-passing LLM evaluation pipelines.

</critical_reminders>

Files (skills)
  • examples
    • core.md 7.9 KB
      Loading…
    • custom-providers.md 8.6 KB
      Loading…
    • model-graded.md 8.3 KB
      Loading…
    • red-teaming.md 5.1 KB
      Loading…
  • reference.md 11.2 KB
    Loading…
  • SKILL.md 16.5 KB
    ---
    name: ai-observability-promptfoo
    description: Testing and evaluation framework for LLM prompts and applications -- promptfooconfig.yaml, assertions, model-graded evals, red teaming, CI/CD integration, custom providers, and comparative evaluation
    ---
    
    # Promptfoo Patterns
    
    > **Quick Guide:** Use promptfoo for systematic LLM evaluation. Define prompts, providers, and test cases in `promptfooconfig.yaml`. Use assertion types (`contains`, `is-json`, `llm-rubric`, `similar`, `cost`, `latency`) to validate outputs. Use `promptfoo eval` to run (exits with code 100 on test failures), `promptfoo view` for results UI. Use model-graded assertions (`llm-rubric`, `factuality`) for subjective quality. Use `promptfoo redteam run` for security scanning. Use `--share` flag or `promptfoo share` to share results. All provider API keys come from environment variables -- never hardcode them.
    
    ---
    
    <critical_requirements>
    
    ## CRITICAL: Before Using This Skill
    
    **(You MUST define test cases with explicit `assert` arrays -- tests without assertions only capture output without validating it)**
    
    **(You MUST use `llm-rubric` for subjective quality evaluation -- do NOT rely solely on deterministic assertions for natural language output)**
    
    **(You MUST set `threshold` on similarity and model-graded assertions -- omitting thresholds uses defaults that may not match your quality bar)**
    
    **(You MUST use environment variables for all API keys -- never hardcode keys in promptfooconfig.yaml or provider configs)**
    
    **(You MUST verify `promptfoo eval` exit code in CI pipelines -- it returns exit code 100 on test failures, exit code 1 on other errors)**
    
    </critical_requirements>
    
    ---
    
    **Auto-detection:** promptfoo, promptfooconfig, promptfooconfig.yaml, promptfoo eval, promptfoo view, promptfoo redteam, llm-rubric, model-graded-closedqa, promptfoo share, promptfoo cache, assertion type, LLM evaluation, prompt testing, red teaming, PROMPTFOO_CONFIG
    
    **When to use:**
    
    - Writing or evaluating LLM prompts across one or more providers
    - Setting up automated test suites for LLM-powered features
    - Comparing model outputs side-by-side (GPT vs Claude vs Gemini)
    - Running model-graded evaluations (LLM-as-a-judge)
    - Red teaming LLM applications for security vulnerabilities
    - Integrating LLM quality gates into CI/CD pipelines
    - Validating structured output (JSON, function calls) from LLMs
    
    **Key patterns covered:**
    
    - `promptfooconfig.yaml` structure (prompts, providers, tests, defaultTest)
    - Assertion types (deterministic, model-graded, performance)
    - Custom TypeScript providers
    - Red teaming configuration (plugins, strategies)
    - CI/CD integration with GitHub Actions
    - Programmatic API (`evaluate()` function)
    - Result sharing and caching
    
    **When NOT to use:**
    
    - Unit testing application code (use your test runner)
    - Load testing / benchmarking API throughput (use a load testing tool)
    - Runtime monitoring of production LLM calls (use observability tooling)
    
    ---
    
    ## Examples Index
    
    - [Core: Config & Assertions](examples/core.md) -- promptfooconfig.yaml structure, providers, prompts, test cases, assertion types
    - [Model-Graded & Advanced Assertions](examples/model-graded.md) -- llm-rubric, factuality, similar, context evaluation, custom assertions
    - [Red Teaming](examples/red-teaming.md) -- Security scanning, plugins, strategies, presets
    - [Custom Providers & Programmatic API](examples/custom-providers.md) -- TypeScript providers, evaluate() function, CI/CD integration
    - [Quick API Reference](reference.md) -- CLI commands, assertion type table, provider IDs, red team plugins
    
    ---
    
    <philosophy>
    
    ## Philosophy
    
    Promptfoo brings **test-driven development to LLM applications**. Instead of manually checking outputs, you define expected behaviors as assertions and run them systematically across prompts and providers.
    
    **Core principles:**
    
    1. **Declarative test definitions** -- YAML config over imperative test scripts. Define prompts, providers, test cases, and assertions in `promptfooconfig.yaml`. No code required for standard evaluations.
    2. **Assertion-driven validation** -- Every test case should have assertions. Deterministic assertions (`contains`, `is-json`, `equals`) for structured output; model-graded assertions (`llm-rubric`, `factuality`) for subjective quality.
    3. **Comparative evaluation** -- Run the same tests across multiple providers or prompt variants simultaneously. The results matrix shows which combination performs best.
    4. **Shift-left LLM testing** -- Catch prompt regressions in CI before they reach production. `promptfoo eval` exits with code 100 on test failures, making it a natural CI quality gate.
    5. **Red teaming as a first-class concern** -- Security scanning for prompt injection, PII leakage, harmful content, and jailbreak vulnerabilities is built in, not bolted on.
    
    </philosophy>
    
    ---
    
    <patterns>
    
    ## Core Patterns
    
    ### Pattern 1: Basic Configuration
    
    Every promptfoo project starts with `promptfooconfig.yaml`. Three required sections: `prompts`, `providers`, `tests`.
    
    ```yaml
    # promptfooconfig.yaml
    description: "Translation quality evaluation"
    
    prompts:
      - "Convert the following to {{language}}: {{input}}"
    
    providers:
      - openai:gpt-4o
      - anthropic:messages:claude-sonnet-4-6
    
    tests:
      - vars:
          language: French
          input: Hello world
        assert:
          - type: icontains
            value: "bonjour"
          - type: llm-rubric
            value: "Output is a natural French translation, not word-for-word"
    ```
    
    **Why good:** Declarative config, multi-provider comparison, both deterministic and model-graded assertions
    
    ```yaml
    # BAD: Tests without assertions
    tests:
      - vars:
          language: French
          input: Hello world
      # No assert array -- output is captured but never validated
    ```
    
    **Why bad:** Tests without assertions only log output, they never fail -- you lose the entire point of automated evaluation
    
    **See:** [examples/core.md](examples/core.md) for prompts from files, provider config, defaultTest, variable loading from CSV
    
    ---
    
    ### Pattern 2: Deterministic Assertions
    
    Use for outputs with predictable, verifiable structure.
    
    ```yaml
    assert:
      # String matching
      - type: contains
        value: "error"
      - type: icontains # case-insensitive
        value: "success"
      - type: not-contains
        value: "internal server error"
      - type: starts-with
        value: "{"
      - type: regex
        value: "\\d{4}-\\d{2}-\\d{2}" # date pattern
    
      # Structured output
      - type: is-json
      - type: contains-json
      - type: is-valid-openai-tools-call
    
      # Performance
      - type: cost
        threshold: 0.01 # max $0.01 per call
      - type: latency
        threshold: 5000 # max 5 seconds
    ```
    
    **Why good:** Fast, deterministic, no LLM cost for evaluation, catches structural regressions immediately
    
    ```yaml
    # BAD: Using llm-rubric for JSON validation
    assert:
      - type: llm-rubric
        value: "Output must be valid JSON"
    ```
    
    **Why bad:** Expensive (requires LLM call), slower, non-deterministic -- `is-json` does this deterministically for free
    
    **See:** [examples/core.md](examples/core.md) for all deterministic assertion types with examples
    
    ---
    
    ### Pattern 3: Model-Graded Assertions
    
    Use for subjective quality where deterministic checks cannot capture intent.
    
    ```yaml
    assert:
      - type: llm-rubric
        value: "Response is helpful, accurate, and conversational in tone"
        provider: openai:gpt-4o
    
      - type: factuality
        value: "The capital of France is Paris. It has a population of ~2.1 million."
        provider: openai:gpt-4o
    
      - type: similar
        value: "The weather in Paris is sunny today"
        threshold: 0.8
    
      - type: model-graded-closedqa
        value: "Paris is the capital of France"
        provider: openai:gpt-4o
    ```
    
    **Why good:** Evaluates subjective quality that deterministic assertions cannot capture, configurable grading provider
    
    ```yaml
    # BAD: No threshold on similar assertion
    assert:
      - type: similar
        value: "expected output"
        # Missing threshold -- uses default which may be too lenient or strict
    ```
    
    **Why bad:** Default similarity threshold may not match your quality bar, always set it explicitly
    
    **See:** [examples/model-graded.md](examples/model-graded.md) for llm-rubric with custom providers, context evaluation, factuality, custom grading prompts
    
    ---
    
    ### Pattern 4: Red Teaming
    
    Use `redteam` section to scan for security vulnerabilities.
    
    ```yaml
    # promptfooconfig.yaml
    targets:
      - openai:gpt-4o
    
    redteam:
      purpose: "Customer support chatbot for an e-commerce platform"
      numTests: 10
      plugins:
        - harmful
        - pii
        - contracts
        - hallucination
        - prompt-extraction
      strategies:
        - jailbreak
        - prompt-injection
    ```
    
    **Why good:** Declarative security scanning, purpose provides context for realistic attacks, composable plugins and strategies
    
    ```yaml
    # BAD: Red team without purpose
    redteam:
      plugins:
        - harmful
      # Missing purpose -- attacks will be generic and less effective
    ```
    
    **Why bad:** Without `purpose`, the red team generator creates generic attacks that miss application-specific vulnerabilities
    
    **See:** [examples/red-teaming.md](examples/red-teaming.md) for presets (OWASP, NIST), advanced strategies, multi-turn attacks
    
    ---
    
    ### Pattern 5: Custom TypeScript Provider
    
    Use when your LLM integration is not a direct API call (RAG pipelines, agent chains, custom middleware).
    
    ```typescript
    // providers/my-app.ts
    import type {
      ApiProvider,
      ProviderOptions,
      ProviderResponse,
      CallApiContextParams,
    } from "promptfoo";
    
    // NOTE: default export required by promptfoo's file:// provider loader
    export default class MyAppProvider implements ApiProvider {
      private config: Record<string, unknown>;
    
      constructor(options: ProviderOptions) {
        this.config = options.config || {};
      }
    
      id(): string {
        return "my-app-provider";
      }
    
      async callApi(
        prompt: string,
        context?: CallApiContextParams,
      ): Promise<ProviderResponse> {
        // Call your application's LLM pipeline
        const result = await myApp.processQuery(prompt);
    
        return {
          output: result.answer,
          tokenUsage: {
            total: result.totalTokens,
            prompt: result.promptTokens,
            completion: result.completionTokens,
          },
          cost: result.cost,
        };
      }
    }
    ```
    
    ```yaml
    # promptfooconfig.yaml
    providers:
      - file://providers/my-app.ts
    ```
    
    **Why good:** Type-safe, full control over LLM pipeline, reports token usage and cost for assertions
    
    **See:** [examples/custom-providers.md](examples/custom-providers.md) for inline function providers, programmatic API, CI/CD integration
    
    ---
    
    ### Pattern 6: CI/CD Integration
    
    Run evaluations in CI with quality gates.
    
    ```yaml
    # .github/workflows/llm-eval.yml
    name: LLM Eval
    on:
      pull_request:
        paths:
          - "prompts/**"
          - "promptfooconfig.yaml"
    jobs:
      evaluate:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
          - uses: actions/setup-node@v4
            with:
              node-version: "22"
          - uses: actions/cache@v4
            with:
              path: ~/.cache/promptfoo
              key: ${{ runner.os }}-promptfoo-v1
          - name: Run eval
            env:
              OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
            run: npx promptfoo@latest eval -o results.json --share
    ```
    
    **Why good:** Caches LLM responses across runs, `promptfoo eval` exits with code 100 on test failures (CI fails automatically), `--share` generates a shareable results URL
    
    **See:** [examples/custom-providers.md](examples/custom-providers.md) for npm scripts, quality gate thresholds, programmatic evaluation
    
    </patterns>
    
    ---
    
    <decision_framework>
    
    ## Decision Framework
    
    ### Which Assertion Type to Use
    
    ```
    What are you validating?
    +-- Exact or structural match?
    |   +-- Exact text -> equals
    |   +-- Contains substring -> contains / icontains
    |   +-- Regex pattern -> regex
    |   +-- Valid JSON -> is-json
    |   +-- Valid function call -> is-valid-openai-tools-call
    |   +-- Cost under budget -> cost (with threshold)
    |   +-- Response time -> latency (with threshold)
    +-- Subjective quality?
    |   +-- General quality criteria -> llm-rubric
    |   +-- Factual accuracy against ground truth -> factuality
    |   +-- Semantic similarity -> similar (with threshold)
    |   +-- Closed-domain QA accuracy -> model-graded-closedqa
    |   +-- RAG context fidelity -> context-faithfulness
    +-- Custom logic?
        +-- JavaScript function -> javascript
        +-- Python function -> python
        +-- External service -> webhook
    ```
    
    ### When to Use Red Teaming vs Eval
    
    ```
    What are you testing?
    +-- Prompt quality and correctness?
    |   +-- Use promptfoo eval with test cases and assertions
    +-- Security vulnerabilities?
    |   +-- Use promptfoo redteam run with plugins and strategies
    +-- Both?
        +-- Run eval for quality, redteam for security -- separate configs or sections
    ```
    
    ### Provider Selection
    
    ```
    How does your LLM integration work?
    +-- Direct API call to OpenAI/Anthropic/etc?
    |   +-- Use built-in provider: openai:gpt-4o, anthropic:messages:claude-sonnet-4-6
    +-- Custom pipeline (RAG, agents, middleware)?
    |   +-- Use custom TypeScript provider: file://providers/my-app.ts
    +-- HTTP endpoint?
    |   +-- Use HTTP provider: id: https://api.example.com/chat
    +-- Multiple providers to compare?
        +-- List all in providers array -- promptfoo runs tests against each
    ```
    
    </decision_framework>
    
    ---
    
    <red_flags>
    
    ## RED FLAGS
    
    **High Priority Issues:**
    
    - Tests without `assert` arrays (output is captured but never validated -- tests always "pass")
    - Not checking `promptfoo eval` exit code in CI (`promptfoo eval` exits 100 on test failures -- ensure your CI pipeline treats non-zero exit codes as failures)
    - Hardcoded API keys in `promptfooconfig.yaml` (use environment variables)
    - Using `llm-rubric` for checks that `is-json` or `contains` can do deterministically (wastes money and adds non-determinism)
    - Red teaming without `purpose` (generic attacks miss application-specific vulnerabilities)
    
    **Medium Priority Issues:**
    
    - Missing `threshold` on `similar` assertions (default may not match your quality bar)
    - Not caching in CI (every run makes full API calls -- expensive and slow)
    - Using `model-graded-closedqa` when `llm-rubric` would be simpler (closedqa is for specific ground-truth QA)
    - Not setting `provider` on model-graded assertions (uses default which may not be the grader you want)
    - Running red team with default `numTests: 5` in production scans (too few for comprehensive coverage)
    
    **Common Mistakes:**
    
    - Confusing `prompts` (the LLM prompt templates) with `tests` (the evaluation cases) -- prompts define what to send, tests define what to check
    - Using `equals` for natural language output (LLM output is non-deterministic, use `llm-rubric` or `similar`)
    - Forgetting `{{variable}}` syntax in prompts (promptfoo uses Nunjucks templating, not `${variable}`)
    - Putting assertions in `defaultTest` that should only apply to specific tests (assertions in `defaultTest` apply to ALL tests)
    - Using `file://` paths without the prefix (promptfoo treats bare paths as literal strings, not file references)
    
    **Gotchas & Edge Cases:**
    
    - `promptfoo eval` caches LLM responses by default -- use `promptfoo cache clear` or `--no-cache` to force fresh calls
    - `--share` uploads results to promptfoo's servers -- do not use with sensitive data unless self-hosting
    - Red team `strategies` wrap `plugins` output -- a plugin generates the malicious content, a strategy delivers it (e.g., via jailbreak encoding)
    - `defaultTest.assert` merges with per-test assertions, it does not replace them -- both arrays run
    - CSV test files map column headers to variable names -- header `input` becomes `{{input}}` in prompts
    - `transform` in test options runs JavaScript on the output before assertions -- useful for extracting JSON from markdown-wrapped responses
    - Provider configs in YAML use `config:` key for model parameters (`temperature`, `max_tokens`), not top-level fields
    - The `weight` property on assertions affects scoring in the results UI but does not change pass/fail behavior
    
    </red_flags>
    
    ---
    
    <critical_reminders>
    
    ## CRITICAL REMINDERS
    
    **(You MUST define test cases with explicit `assert` arrays -- tests without assertions only capture output without validating it)**
    
    **(You MUST use `llm-rubric` for subjective quality evaluation -- do NOT rely solely on deterministic assertions for natural language output)**
    
    **(You MUST set `threshold` on similarity and model-graded assertions -- omitting thresholds uses defaults that may not match your quality bar)**
    
    **(You MUST use environment variables for all API keys -- never hardcode keys in promptfooconfig.yaml or provider configs)**
    
    **(You MUST verify `promptfoo eval` exit code in CI pipelines -- it returns exit code 100 on test failures, exit code 1 on other errors)**
    
    **Failure to follow these rules will produce untested, insecure, or falsely-passing LLM evaluation pipelines.**
    
    </critical_reminders>
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.