{
  "schema_version": 1,
  "skill_name": "ai-operating-economics",
  "evals": [
    {
      "id": "aggregate-speed-claim",
      "prompt": "Our support pilot reduced average handle time by 18% and the vendor says this proves a 20% productivity gain. Customer satisfaction was measured for only 10% of chats, and we have no baseline comparison. Should we roll it out to every team?",
      "expected_output": "A bounded recommendation that rejects the conclusion that lower handle time proves productivity or supports universal rollout. It identifies the missing baseline, incomplete quality measurement, and the need to measure a meaningful outcome such as resolved cases or customer outcomes alongside quality countermetrics. It proposes a scoped comparison and a constrained next step rather than a universal rollout.",
      "assertions": [
        "The response explicitly says reduced handle time alone does not establish business value or productivity",
        "The response identifies both the missing baseline and the 10% customer-satisfaction coverage as material evidence gaps",
        "The response distinguishes the vendor claim from observed or causal evidence",
        "The response proposes a meaningful primary outcome and at least one quality countermetric",
        "The response does not automatically convert time saved into salary savings and identifies the benefit-realization mechanism that would be needed",
        "The recommendation is bounded to a scoped pilot, hold, or constrained rollout rather than universal rollout"
      ],
      "case_set": "release"
    },
    {
      "id": "heterogeneous-effects",
      "prompt": "An AI assistant increased completed cases per hour by 14% overall. New employees improved by 34%, experienced employees showed no improvement, and quality audits suggest a small decline among the highest performers. What should the company do next?",
      "expected_output": "A segmented decision that treats heterogeneous effects as central evidence. It does not recommend universal scale from the mean. It proposes differentiated deployment, further quality investigation, and separate authority or workflow treatment for affected groups, with an owner and reversal trigger.",
      "assertions": [
        "The response treats worker heterogeneity as a decision-changing finding rather than noise",
        "The response does not recommend unconditional universal rollout based on the 14% average",
        "The response addresses the possible quality decline among highest performers",
        "The response proposes a differentiated or bounded next intervention",
        "The response includes a review or reversal trigger tied to quality and subgroup outcomes"
      ],
      "case_set": "release"
    },
    {
      "id": "incomplete-tco",
      "prompt": "Our agent costs $0.004 per request in model tokens. Finance wants to approve the project because that is cheaper than a $0.12 human-handled request. The agent also uses retrieval, three APIs, a vector database, observability, human exception review, and a dedicated GPU that is idle 40% of the time. Prepare the economic comparison.",
      "expected_output": "A corrected comparison that rejects token-only cost as total cost. It defines a meaningful denominator, separates marginal from fully loaded cost, includes retrieval, tools, storage, observability, human review, and committed or idle capacity, and identifies the evidence needed to calculate each component. It does not claim savings until the workflow outcomes and quality boundary are established.",
      "assertions": [
        "The response explicitly rejects model-token cost as the complete economic comparison",
        "The response separates marginal cost from fully loaded cost",
        "The response includes retrieval, APIs/tools, vector storage, observability, human exception review, and idle dedicated capacity",
        "The response defines or requests a meaningful denominator beyond raw requests",
        "The response states that cost savings cannot be claimed without workflow outcome and quality evidence"
      ],
      "case_set": "dev"
    },
    {
      "id": "vendor-survey-evidence",
      "prompt": "A consulting firm's survey says 66% of organizations report AI productivity gains and 40% report cost reductions. The board wants a memo saying our AI program will produce the same results. How should the evidence be used?",
      "expected_output": "A calibrated memo that labels the survey as vendor-reported self-report evidence, uses it only as context or hypothesis-generation, and refuses to project the percentages onto the organization without a local baseline, workflow evidence, cost boundary, and countermetrics.",
      "assertions": [
        "The response labels the survey as vendor-reported and self-reported rather than audited causal evidence",
        "The response refuses to treat the survey percentages as a forecast for this organization",
        "The response identifies a local baseline and workflow-specific evidence as needed",
        "The response requests quality or other countermetrics alongside productivity or cost claims",
        "The response offers a calibrated alternative for board language"
      ],
      "case_set": "regression"
    },
    {
      "id": "authority-after-pilot",
      "prompt": "Our read-only internal agent passed its pilot evaluation with a strong average score. The product team now wants to let it update customer records automatically. What does the economics review need to decide?",
      "expected_output": "A decision that separates value evidence from authority expansion. It requires a new intervention and risk boundary, side-effect evaluation, authorization and rollback controls, cost and quality monitoring, staged expansion, and a named human owner. A positive read-only pilot is not treated as evidence for write authority.",
      "assertions": [
        "The response says a positive read-only pilot does not establish safety or readiness for customer-record writes",
        "The response identifies authority expansion as a new intervention requiring separate evidence",
        "The response requires authorization, rollback or containment, and human escalation controls",
        "The response proposes staged or bounded authority rather than an immediate unrestricted change",
        "The response names cost, quality, and side-effect monitoring as part of the decision"
      ],
      "case_set": "release"
    },
    {
      "id": "retirement-decision",
      "prompt": "After six months, an AI workflow has high adoption but no measurable improvement in the intended customer outcome. Reviewers also spend significant time correcting its outputs, and the cost ledger is incomplete. The sponsor says adoption proves the system should continue. Produce the lifecycle recommendation.",
      "expected_output": "A recommendation that does not equate adoption with value. It makes the incomplete cost ledger and correction burden visible, assesses the intended outcome and quality countermetric, and recommends hold, redesign, or retirement with a bounded evidence plan and a transition or containment path.",
      "assertions": [
        "The response explicitly distinguishes adoption from realized value",
        "The response identifies correction burden and incomplete cost accounting as material",
        "The response centers the intended customer outcome rather than usage volume",
        "The response recommends a bounded hold, redesign, or retirement path rather than indefinite continuation",
        "The response includes a named owner, review trigger, and transition or containment consideration"
      ],
      "case_set": "release"
    }
  ]
}

# Evidence Method for AI Operating Economics

This reference turns the core skill into a repeatable investigation and decision method. Use it when the user needs more than a short recommendation or when a claim is consequential enough to preserve.

## 1. Scope the decision before gathering numbers

Write a one-sentence decision question:

> Should [accountable owner] [scale, constrain, redesign, hold, retire, or approve an exception for] [AI intervention] in [workflow/population] by [date or trigger], based on [required outcome and guardrails]?

Record the intervention mode:

- Assist: provides information or drafts while a person decides.
- Recommend: proposes a ranked or selected action.
- Route: classifies or directs work.
- Execute: takes an external action.
- Replace or remove: changes staffing, process, or service capacity.

The mode determines what evidence and authority are appropriate. A positive assist result does not automatically justify execution authority.

## 1a. Separate the business case from benefit realization

A forecasted benefit is not a realized benefit. Track four states separately:

1. **Expected benefit:** the hypothesis or business-case estimate.
2. **Enabled capacity:** time, throughput, or capability the intervention appears to make available.
3. **Operational benefit:** a verified change in the target workflow, such as more resolved cases, shorter cycle time without quality loss, or fewer avoidable escalations.
4. **Realized economic or mission benefit:** the operational change is converted into an attributable financial, service, capacity, or mission result under the organization's actual decision and accounting rules.

Record the owner, baseline, realization mechanism, timing, dependencies, and disbenefits for each expected benefit. Do not call time saved “savings” until the organization has a credible mechanism for converting it into reduced spend, additional output, avoided cost, improved service, or another explicitly valued result. If the benefit is capacity released for higher-value work, measure whether that work actually occurred.

Benefits-realization planning can borrow from public-sector benefits-management guidance, but the local owner and accounting treatment remain authoritative. Load the source index when a board or investment claim depends on the distinction.

## 2. Gather evidence by claim, not by headline

Build a claim table before writing the conclusion:

| Claim | Evidence class | Source and scope | What it supports | What it does not support | Open challenge | Permitted interpretation |
|---|---|---|---|---|---|---|
| [claim] | observed / causal / inferred / vendor-reported / asserted / normative | [citation, date, version] | [supported claim] | [boundary] | [test or missing evidence] | [calibrated language] |

Use the strongest source appropriate to the claim:

- Workflow outcomes: controlled studies, internal experiments, or verified operational data.
- Cost: billing records, usage telemetry, allocation rules, and explicit assumptions.
- Quality and safety: outcome audits, incident records, domain review, and representative samples.
- Worker or user impact: segmented operational data plus worker/user research where relevant.
- Controls: official standards, provider documentation, and observed enforcement behavior.

A research summary is not a substitute for source-level claims. Preserve rejected, inaccessible, redundant, or out-of-scope sources in the research log or evidence record so omission is distinguishable from oversight.

## 3. Build an outcome model

Separate the mechanism from the outcome:

| Layer | Question | Example |
|---|---|---|
| Intervention | What changed? | Agent receives suggested responses |
| Behavior | What did people or systems do differently? | More chats handled; recommendations accepted selectively |
| Immediate outcome | What changed in the workflow? | Resolved cases per hour |
| Quality outcome | Did the result remain correct and acceptable? | Resolution rate, customer sentiment, rework |
| Business or mission outcome | Did the organization get the intended value? | Cost per resolved case, retention, revenue, service access |
| Distributional outcome | Who gained, lost, or carried new burden? | Novices improve; experts see no gain; review work shifts |

Do not jump from intervention to business outcome without observing the intermediate mechanism and its failure modes.

## 4. Design the comparison

Choose the strongest feasible design and name the downgrade if it is weaker:

1. Randomized or staggered assignment.
2. Matched comparison or difference-in-differences with a defensible control and pre-period.
3. Controlled pilot with explicit inclusion, exclusion, baseline, and observation window.
4. Before/after descriptive comparison with seasonality, selection, and concurrent-change caveats.
5. Qualitative or vendor evidence used only for hypothesis formation.

Before interpreting a result, ask:

- Did people self-select into treatment or into using the tool?
- Did task mix, volume, staffing, incentives, or pay change?
- Did the intervention change which tasks were attempted?
- Did quality measurement cover the treated population equally?
- Did learning, novelty, or temporary support affect the result?
- Were concurrent tools, process changes, or policy changes present?
- Is the comparison at the same workflow boundary as the decision?

If these questions cannot be answered, narrow the causal language and classify the result as descriptive or inferred.

## 5. Model the economic boundary

Use three cost layers, then show marginal and fully loaded views within the economic layer:

1. **Billing truth:** provider or infrastructure charges as recorded in source billing data.
2. **Allocated cost:** shared charges assigned to a product, team, tenant, workflow, or business unit under documented allocation rules.
3. **Economic cost:** allocated cost plus material human, engineering, governance, risk, idle-capacity, transition, and opportunity costs needed to operate the intervention.

A common billing schema can improve reconciliation and allocation, but it does not decide the right allocation policy or establish business value.

### Marginal view

What changes when one more meaningful unit of work is served?

- Input and output tokens
- Model and routing charges
- Data acquisition, preparation, licensing, and retention
- Tool and API calls
- Retrieval and data transfer
- Human review, correction, escalation, and rework
- Incremental compute or capacity

### Fully loaded view

What must exist for the intervention to operate responsibly?

- Engineering and integration
- Evaluation and test data
- Red-teaming and monitoring
- Observability and retention
- Security, privacy, and governance
- Training, change management, and support
- Incident response and recovery
- Committed or idle capacity
- Model/provider migration, retraining, deprecation, and exit costs

Classify each cost as fixed, variable, step-function, avoided, transferred, or uncertain. State the allocation method for shared resources. Keep token attribution separate from realized value: tokens are a computation unit, not a value unit.

The denominator must represent meaningful work. Possible denominators include completed workflow, resolved case, accepted decision, active user-month, or customer outcome. Use separate denominators for materially different task classes rather than hiding them in an average.

## 6. Inspect worker, user, and task heterogeneity

At minimum, slice by the variables that could reverse the decision:

- Experience, skill, role, and training status
- Task complexity, risk, and exception rate
- Customer or user segment
- Language, geography, accessibility, or demographic group where appropriate and lawful
- Human-review burden and escalation frequency
- Quality, error severity, and rework
- Adoption intensity and non-user comparison

A heterogeneous result may require differentiated deployment: assistance for novices, review support for experts, a higher-quality model for high-risk tasks, or no deployment in a harmed segment. Do not “solve” heterogeneity by reporting only the mean.

Worker-impact evidence should include more than productivity. Consider autonomy, workload, learning, skill development, job quality, schedule, stress, discretion, and who absorbs monitoring or correction work. Route formal labor, legal, or collective-bargaining questions to qualified specialists.

## 7. Separate evidence from decision language

Use calibrated verbs:

- **Observed:** “The treated group resolved more cases per hour in this deployment.”
- **Causal:** “The staggered comparison estimates an increase within this population and period.”
- **Inferred:** “The pattern is consistent with knowledge transfer, but does not establish it.”
- **Vendor-reported:** “The provider reports that customers experienced…”
- **Asserted:** “The sponsor claims…” or “The proposal assumes…”; do not present it as observed.
- **Normative:** “The framework recommends monitoring and assigned responsibility.”
- **Unknown:** “The available evidence does not establish…”

Never use “proves,” “guarantees,” “will save,” or “safe” unless the evidence and scope truly support that strength.

## 7a. Require a governance evidence packet for authority expansion

Before moving from a bounded pilot to a materially broader population, higher-risk task, or side-effect authority, require a compact governance packet:

- Accountable owner, intended use, risk tier, and affected population
- System, model, prompt/policy, tool, provider, and version inventory
- Acceptable-use, refusal, escalation, and human-oversight rules
- Pre-deployment evaluation results and the release threshold used
- Third-party/provider assessment and relevant contractual evidence
- Incident, override, and near-miss record with after-action ownership
- Change and revalidation trigger for model, prompt, tool, data, or workflow changes
- Retention, dependency, leakage, user-impact, and decommissioning plan

These are governance inputs, not proof that the system is safe or valuable. Include the cost of producing and operating these controls in the economic boundary. Route detailed risk, privacy, security, and runtime control work to their owning skills.

## 8. Apply a decision rule without pretending to score everything

Use a structured disposition rather than a universal numeric score:

| Disposition | Minimum basis | Required next control |
|---|---|---|
| Scale | Outcome improvement, countermetrics within bounds, cost boundary understood, no disqualifying slice | Define next population and authority slice |
| Constrain | Plausible value with unresolved cost, quality, distributional, or authority risk | Set quota, population, task, or human-review boundary |
| Redesign | Mechanism or workflow creates avoidable failure or burden | Change workflow/model/control and rerun comparison |
| Hold | Required evidence is missing or conflicting | Name evidence owner, method, and review trigger |
| Retire | Value absent or countermetrics exceed bounds after review | Protect users/workers, migrate, and record learning |
| Exception | Material gap accepted temporarily | Named human approver, expiry, containment, and revisit |

A positive average cannot override a hard safety, privacy, authorization, or material quality failure. Conversely, a single weak metric should not automatically kill an intervention if the decision record explains the tradeoff and names the accountable owner.

## 9. Close the loop

At the review trigger, compare:

- Expected versus observed outcome
- Expected versus observed cost
- Countermetrics and subgroup effects
- Adoption, substitution, and new human work
- Incidents, near misses, overrides, and escalations
- Model, prompt, tool, policy, or population changes

Classify the hypothesis as supported, weakened, refuted, or unresolved. Preserve the result and link it to the next decision. A value-realization process that never changes authority, scope, or investment is measurement theater.

## Research thin spots to keep visible

The foundational evidence is strongest for workflow-level productivity, measurement caution, cost/usage controls, telemetry, and lifecycle governance. It is thinner for:

- Realized financial returns after implementation and change-management cost
- Long-term worker learning, job quality, and distributional effects
- Agentic workflows with multiple tools and autonomous side effects
- Comparable cross-vendor or cross-model economic benchmarks
- Enterprise counterfactuals where adoption is voluntary and task mix changes

Treat these as active evidence gaps. Do not fill them with vendor case studies or invented benchmarks.

# Source Index and Evidence Boundaries

This index records the research basis for the skill. Access dates and versions should be refreshed when a decision depends on a time-sensitive provider capability. These sources inform method and guardrails; they do not establish universal ROI.

## Workflow outcome and measurement

| Source | Evidence type | Supports | Does not support |
|---|---|---|---|
| [Brynjolfsson, Li, and Raymond, Generative AI at Work](https://www.nber.org/papers/w31161) | Independent working paper, revised 2023 | In one customer-support deployment, AI assistance increased resolved issues per hour by about 14% on average, with much larger gains for novice/lower-skilled workers and minimal gains for experienced/high-skilled workers; the paper also examines quality, sentiment, retention, adherence, and learning | General enterprise ROI, current agentic-system performance, or universal productivity claims |
| [NBER digest summary](https://www.nber.org/digest/20236/measuring-productivity-impact-generative-ai) | Independent study summary | Plain-language description of the measured deployment and heterogeneous effects | A substitute for the working paper when methodological detail matters |
| [METR, Uplift Update](https://metr.org/blog/2026-02-24-uplift-update/) | Independent measurement research | Selection effects, task-selection bias, concurrent-agent accounting, and why an attractive developer-speed estimate may not support a strong causal claim | A general estimate of enterprise AI productivity |
| [METR, AI Usage Survey](https://metr.org/blog/2026-05-11-ai-usage-survey/) | Independent survey research | The distinction between speed and value, plus caveats on self-reported uplift and counterfactual estimation | Audited financial ROI or causal productivity evidence |
| [Noy and Zhang, Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence](https://economics.mit.edu/sites/default/files/inline-files/Noy_Zhang_1.pdf) | Randomized experiment manuscript | Short writing-task effects on completion time, blinded quality, performer heterogeneity, task composition, and the limits of inferring durable workplace value | Firm-level ROI, long-term skill development, or generalization to organization-specific work |
| [OECD Employment Outlook 2023: AI, job quality and inclusiveness](https://www.oecd.org/en/publications/oecd-employment-outlook-2023_08785bba-en/full-report/artificial-intelligence-job-quality-and-inclusiveness_a713d0ad.html) | Independent policy research | Worker, management, working-condition, skill, productivity, wage, employment, and transition dimensions that should accompany a narrow productivity measure | A universal prediction of AI's labor-market impact or a substitute for local worker evidence |
| [OECD case studies of AI implementation](https://www.oecd.org/content/dam/oecd/en/publications/reports/2023/03/the-impact-of-ai-on-the-workplace-evidence-from-oecd-case-studies-of-ai-implementation_b4c2c6ee/2247ce58-en.pdf) | Independent case research | Heterogeneity across worker profiles, sectors, countries, task composition, skill requirements, and job quality | A causal benchmark for a specific organization |

## Cost and usage

| Source | Evidence type | Supports | Does not support |
|---|---|---|---|
| [FinOps for AI overview](https://www.finops.org/wg/finops-for-ai-overview/) | Primary foundation guidance | Extending FinOps practices to volatile model pricing, token meters, GPU scarcity, allocation, quotas, tagging, and outcome alignment | Proof that any organization has realized savings |
| [How to Build a Generative AI Cost and Usage Tracker](https://www.finops.org/wg/how-to-build-a-generative-ai-cost-and-usage-tracker/) | Primary foundation guidance | Token attribution levels, centralized or common interfaces, shared-throughput allocation, and the need to account for more than inference | A universal architecture or exact savings formula |
| [GenAI FinOps: How Token Pricing Really Works](https://www.finops.org/wg/genai-finops-how-token-pricing-really-works/) | Primary foundation guidance | The warning that advertised token prices do not describe complete application TCO | A measured cross-provider cost comparison |
| [Token Economics: The Atomic Unit of AI Value](https://www.finops.org/insights/token-economics-the-atomic-unit-of-ai-value/) | Primary foundation insight | Tokens are computation units and require contextual interpretation | Tokens as a direct measure of business value |

## Benefits realization and cost estimation

| Source | Evidence type | Supports | Does not support |
|---|---|---|---|
| [UK Digital and Data Benefits Framework](https://www.gov.uk/government/publications/digital-and-data-benefits-framework/digital-and-data-benefits-framework) | Government guidance | Distinguishing benefits, disbenefits, measures, owners, baselines, dependencies, and realization tracking in a digital business case | A universal accounting treatment or proof that a forecast benefit will be realized |
| [UK Magenta Book evaluation guidance](https://www.gov.uk/government/publications/the-magenta-book/magenta-book-central-government-guidance-on-evaluation-html) | Government evaluation guidance | Evaluation planning, theory of change, counterfactual reasoning, monitoring, and proportionate evidence design | A replacement for domain-specific statistical or financial expertise |
| [FinOps Open Cost and Usage Specification (FOCUS)](https://focus.finops.org/focus-specification/) | Primary specification | A common structure for normalizing billing data and supporting reconciliation, allocation, chargeback, budgeting, and forecasting | Complete TCO, causal ROI, labor cost, or the correct local allocation policy |
| [GAO Cost Estimating and Assessment Guide](https://www.gao.gov/products/gao-20-195g) | Government cost-estimation guidance | Lifecycle cost categories, documented assumptions, uncertainty, sensitivity, independent review, and updating estimates as evidence changes | An AI-specific cost model or a guarantee of estimate accuracy |

## Governance and worker impact

| Source | Evidence type | Supports | Does not support |
|---|---|---|---|
| [NIST AI RMF: Generative AI Profile](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf) | Primary voluntary framework | Generative-AI-specific risk considerations, testing and evaluation, monitoring, incident response, human oversight, and lifecycle controls | Certification, compliance, safety proof, or realized value |
| [ISO/IEC 42001](https://www.iso.org/standard/42001) | International standard description | The AI management-system idea: objectives, responsibilities, performance evaluation, corrective action, and continual improvement | Certification or conformance without an authorized audit and the applicable standard |
| [ILO, Generative AI and Jobs](https://webapps.ilo.org/static/english/intserv/working-papers/wp096/index.html) | Independent policy research | Job quantity and quality, autonomy, work organization, and worker voice as dimensions beyond productivity | A prediction of the impact on a particular employer or occupation |

## Telemetry and controls

| Source | Evidence type | Supports | Does not support |
|---|---|---|---|
| [OpenTelemetry GenAI observability](https://opentelemetry.io/blog/2026/genai-observability/) | Primary technical guidance | Agent, model-call, tool-execution, model/version, duration, and token telemetry; prompt and tool content should not be captured by default when sensitive | Complete production observability or a guarantee of privacy |
| [OpenTelemetry GenAI attributes](https://opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/) | Primary technical specification | Portable provider, model, operation, token, and agent attributes, with version/movement caveats | Stable interoperability across every implementation without version pinning |
| [NIST AI RMF Core](https://airc.nist.gov/airmf-resources/airmf/5-sec-core/) | Primary voluntary framework | Govern, map, measure, and manage lifecycle structure; inventory, roles, monitoring, incident response, recovery, and deactivation expectations | Certification, safety proof, or legal compliance |
| [NIST Generative AI Profile](https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence) | Primary voluntary framework | Generative-AI-specific risk-management considerations to supplement AI RMF 1.0 | A complete implementation design or outcome guarantee |
| [AWS Bedrock prompt routing](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html) | Vendor documentation | An example of provider-side quality/cost routing and its documented limitations | Independent savings or quality superiority |
| [Microsoft Foundry cost management](https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-models/how-to/manage-costs) | Vendor documentation | An example of estimation, representative test traffic, cost grouping, budgets, alerts, and dependent-resource TCO | Hard spend stops, universal controls, or independent ROI |

## Research-use rules

- Preserve the source URL, access date, evidence type, scope, and caveat with every retained claim.
- Treat working papers as research evidence, not peer-reviewed consensus unless the source says otherwise.
- Treat vendor documentation as capability evidence only.
- Treat vendor surveys and case studies as reported claims, not causal outcomes.
- Do not copy copyrighted source text into public skill content; paraphrase and link.
- Re-verify provider capabilities and moving OpenTelemetry conventions before using them as implementation requirements.

# AI Economics Review

Use this template for an executive, portfolio, experiment, launch, or lifecycle review that consumes one or more AI initiative evidence records.

## Review header

- Review purpose:
- Review date:
- Operating mode: lightweight / high-assurance
- Accountable decision-maker:
- Records reviewed:
- Decision deadline or trigger:

## Minimum decision record

- Intervention and population:
- Value hypothesis:
- Primary outcome:
- Countermetrics:
- Comparison and limitations:
- Cost boundary:
- Relevant slices:
- Evidence classes:
- Missing evidence, owner, and next trigger:
- Authority limit:
- Reversal path:

## Executive decision

- Disposition: scale / constrain / redesign / hold / retire / exception
- Decision in one sentence:
- Approved population and workflow:
- Approved authority:
- Budget or cost boundary:
- Review trigger:

## Evidence summary

| Dimension | What was observed | Confidence/class | Material caveat | Owner |
|---|---|---|---|---|
| Workflow outcome | | | | |
| Quality and safety | | | | |
| Worker/user/customer effect | | | | |
| Adoption and substitution | | | | |
| Marginal economics | | | | |
| Fully loaded economics | | | | |
| Reliability and operations | | | | |
| Governance and authority | | | | |

## Slices that change the decision

| Slice | Result | Why it matters | Required response |
|---|---|---|---|
| | | | |

## Evidence gaps and exceptions

| Gap or exception | Consequence if wrong | Owner | Control or containment | Revisit date/trigger |
|---|---|---|---|---|
| | | | | |

## Options and trade-offs

| Option | Value evidence | Cost | Quality/worker/user effects | Authority and reversibility | Recommendation |
|---|---|---|---|---|---|
| | | | | | |

## Decision record

- Why this disposition is justified:
- What would falsify or reverse it:
- What will not be claimed:
- Next action:
- Named owner:
- Verification evidence required before the next authority or investment step:

# AI Initiative Evidence Record

## Decision header

- Initiative:
- Workflow:
- Population and scope:
- Intervention mode: assist / recommend / route / execute / replace
- Decision sought: scale / constrain / redesign / hold / retire / exception
- Accountable decision owner:
- Review date or trigger:
- Record version:

## Value hypothesis

> For [population] doing [workflow], [intervention] will change [outcome] by [direction/range] without exceeding [countermetric boundary], at [full cost boundary], compared with [baseline], over [period].

- Hypothesis status: supported / weakened / refuted / unresolved
- Expected benefit:
- Enabled capacity:
- Operational benefit:
- Realized economic or mission benefit:
- Benefit realization mechanism:
- Human-control boundary:
- Authority proposed for next slice:

## Evidence comparison

- Baseline:
- Comparison design:
- Treatment period:
- Comparison period:
- Inclusion and exclusion rules:
- Known selection effects:
- Concurrent changes:
- Quality measurement limitations:
- Statistical or causal analysis owner:

| Claim | Evidence class | Source and scope | What it supports | What it does not support | Open challenge | Permitted interpretation |
|---|---|---|---|---|---|---|
| | observed / causal / inferred / vendor-reported / asserted / normative | | | | | |

## Outcome and countermetrics

| Metric | Type | Definition and denominator | Baseline | Observed | Target/boundary | Owner | Evidence source |
|---|---|---|---:|---:|---|---|---|
| | primary / leading / countermetric / adoption | | | | | | |

## Segment review

| Slice | Adoption or exposure | Outcome | Quality/countermetric | New burden or benefit | Decision implication |
|---|---:|---:|---:|---|---|
| | | | | | |

Slices not available and why:

## Cost boundary

### Billing truth

- Provider/infrastructure source:
- Billing period and version:
- Reconciliation status:

### Allocated cost

- Allocation target:
- Shared-cost rule:
- Allocation owner:

### Economic cost

- Meaningful unit:
- Model/inference:
- Tools and APIs:
- Retrieval/storage/networking:
- Human review and exception handling:
- Incremental capacity:
- Engineering, evaluation, observability, governance, training, support, and exit:
- Fixed, variable, step-function, avoided, transferred, and uncertain costs:
- Calculation and allocation method:
- Range or sensitivity:

## Findings and gaps

### Supported findings

1.

### Unresolved or conflicting findings

1.

### Missing evidence

| Gap | Why it matters | Owner | Next evidence | Due date or trigger |
|---|---|---|---|---|
| | | | | |

## Governance evidence packet

- Intended use and risk tier:
- System/model/prompt/policy/tool/provider/version inventory:
- Acceptable-use, refusal, escalation, and human-oversight rules:
- Pre-deployment evaluation and release threshold:
- Third-party/provider assessment and contractual evidence:
- Incident, override, and near-miss record:
- Change/revalidation trigger:
- Retention, dependency, leakage, user-impact, and decommissioning plan:

## Decision and controls

- Disposition:
- Scope of approval:
- Authority limit:
- Budget or quota limit:
- Human review or escalation rule:
- Stop trigger:
- Rollback, containment, or retirement path:
- Exception approver, if applicable:
- Revisit condition:

## Learning closure

- Expected versus observed outcome:
- Expected versus observed cost:
- Countermetric and subgroup result:
- Incidents, overrides, or near misses:
- Changes since prior record:
- Hypothesis update:
- Follow-up artifact or owner: