# Artifact Inventory

Track what exists at each layer. Update as artifacts are promoted or archived.

---

## Layer 1: Summary

| File | Key Findings | Key Implications | Status | SOURCES Complete? |
|------|-------------|-----------------|--------|-------------------|
| `01-summary/` | | | draft/review/published | yes/no |

## Layer 2: Analysis Collection

| File | Dimension | Sources Claim | Key Claim | Status |
|------|-----------|--------------|-----------|--------|
| `02-analysis/` | | | | draft/review/complete |
| `02-analysis/` | | | | |

## Layer 3: Detailed Dossiers

| Source ID | Title | Type | Captured | Evidence Count |
|-----------|-------|------|----------|---------------|
| | | paper/transcript/data | | |

## Quality Gate Log

| Gate | Date | Verdict | Notes |
|------|------|---------|-------|
| C (L3→L2) | | pass/fail | |
| B (L2→L1) | | pass/fail | |
| A (L1 publish) | | pass/fail | |

# Artifact Pyramid Project Scaffold

Copy this file into your research project as `00-index.md` and fill in the blanks.

---

# [Project Title]

## Overview

Research question or problem. One paragraph.

**Mission Brief:** [Brief from orchestrator]
**Started:** YYYY-MM-DD
**Status:** exploration / gathering / analysis / summary / complete

## Layer 1: Summary

| Status | File |
|--------|------|
| draft / review / published | `01-summary/` |

## Layer 2: Analysis Collection

| Dimension | File | Status | Key Claim |
|-----------|------|--------|-----------|
| | `02-analysis/` | draft / review / complete | |

## Layer 3: Detailed Dossiers

| Source ID | Title | Type | Captured | URL |
|-----------|-------|------|----------|-----|
| | | paper / transcript / data | | |

## Open Questions

1.
2.

## Quality Gate Status

- [ ] Gate C (L3 dossiers ready for analysis)
- [ ] Gate B (L2 analysis ready for summary)
- [ ] Gate A (L1 summary ready for distribution)

{
  "schema_version": 1,
  "skill_name": "artifact-pyramids",
  "evals": [
    {
      "id": "pyramid-scaffold",
      "prompt": "I am starting a research project on the competitive landscape of the observability market and need to produce durable, agent-consumable research artifacts. How do I scaffold the output as an artifact pyramid, and what goes in each layer?",
      "expected_output": "A scaffold following the three-layer pyramid: an L1 summary file that states the research question, the key findings, and the most important implications with links down to the L2 analysis files; L2 analysis files per dimension (market sizing, competitor profiles, technical feasibility) that are self-contained and each link to L3 dossiers; and L3 dossiers holding the raw evidence: source excerpts, data tables, interview notes, and methodology notes. The response explains that production is top-down with recursive gap analysis: write the summary, embed links to analysis files, write analysis files that link to dossiers, and re-check for gaps after each round. It also requires the SOURCES navigation section at the bottom of every file with absolute paths and a description of what each deeper file contains.",
      "assertions": [
        "The scaffold defines L1 summary, L2 analysis collection, and L3 dossiers with distinct content contracts",
        "L1 links down to L2 files and L2 files link down to L3 dossiers",
        "Production is top-down with recursive gap analysis between layers",
        "Every file carries the SOURCES navigation section with paths and descriptions",
        "The layers are applied to the observability-market research topic with concrete example files"
      ]
    },
    {
      "id": "l1-summary-authoring",
      "prompt": "I have completed the research for our market-entry question and need to write the top-layer summary file. What belongs in an L1 summary, what must be excluded, and how do I know when the summary is complete?",
      "expected_output": "An L1 summary written to the layer's contract: the research question and its scope, the key findings as conclusions with the most important implications for the decision, and nothing else — no raw data dumps, no methodology narration, and no findings that lack a home in an L2 file. The response explains the boundary rules: anything that is supporting evidence belongs in an L2 analysis or L3 dossier and is linked, not embedded; the summary must be independently consumable by an agent that reads only L1; and completion is checked by the quality gates for the layer: the question is answered, every finding is traceable to a linked L2 file, and no gap remains that the summary papered over.",
      "assertions": [
        "The L1 summary contains the research question, key findings, and implications only",
        "Raw evidence and methodology detail are excluded and routed to L2 or L3 via links",
        "The summary is independently consumable without reading the lower layers",
        "Every finding is traceable to a linked L2 file",
        "Completion is checked against the layer's quality gates, including uncovered gaps"
      ]
    },
    {
      "id": "l2-analysis-sources",
      "prompt": "I am writing the market-analysis layer of a pyramid about the developer-tooling market. Each analysis file must be self-contained and consumable. How do I structure one L2 file, and what is the SOURCES convention that keeps the pyramid navigable?",
      "expected_output": "An L2 analysis file structured as a self-contained dimension analysis: the dimension's scope and the question it answers, the analysis with its conclusions, the caveats and uncertainty, and links down to the L3 dossiers holding the underlying evidence. The response demonstrates the SOURCES convention: a section at the bottom of the file listing each referenced dossier with its absolute path and a one-line description of what the consumer will find there, phrased to answer what the deeper file contains rather than just naming it. It explains why the navigation section matters: it is the affordance that lets a consuming agent decide whether to pull the next layer, and it is required on every file at every layer.",
      "assertions": [
        "The L2 file is structured around the dimension's question, analysis, conclusions, and caveats",
        "It links down to the L3 dossiers that hold the evidence",
        "The SOURCES section lists absolute paths with descriptions of what each deeper file contains",
        "The response explains the navigation section as the agent's pull-decision affordance",
        "The convention is applied with a concrete market-analysis example"
      ]
    },
    {
      "id": "pyramid-audit",
      "prompt": "I inherited a research output directory with a 00-index file, several markdown files, and a dump of raw interview transcripts, but nothing links to anything. I suspect it is not a valid artifact pyramid. How do I audit it and what do I fix?",
      "expected_output": "An audit against the pyramid's structural contracts: the response checks layer presence (an L1 summary file, L2 analysis files, and L3 dossiers), checks the navigation mechanism (every file carries a SOURCES section with absolute paths), checks content placement (findings belong in L1, analysis in L2, raw evidence in L3 — not a 00-index stuffing findings meant for L1, and not raw transcripts masquerading as analysis), and checks link integrity between layers. The response produces a concrete remediation plan: promote or split the 00-index content into a proper L1 summary, reorganize the raw transcripts into dossiers, rewrite analysis files to be self-contained, and add the SOURCES sections and cross-links. It also notes the validation-script pitfall: structural checks pass on directory shape alone and do not prove content contracts, so the audit must read the files.",
      "assertions": [
        "The audit checks all three layers exist with the right content contracts",
        "It verifies the SOURCES navigation and link integrity between layers",
        "Misplaced content (findings in an index, raw transcripts as analysis) is specifically flagged",
        "A concrete remediation plan reorganizes the inherited files into a valid pyramid",
        "The response warns that directory-shape checks do not prove content compliance"
      ]
    },
    {
      "id": "composite-synthesis",
      "prompt": "I ran three subagent research teams in parallel — one on market, one on competitors, one on technical feasibility — and each returned its own pyramid. I need one root-level deliverable for the decision maker. How do I merge them into a composite pyramid?",
      "expected_output": "A composite synthesis procedure: the root pyramid's L1 summary is built from the three subagent pyramids' L1 findings, with a per-dimension L2 analysis layer that each maps to the corresponding subagent pyramid as its evidence source, using the SOURCES convention to reference the subagent pyramids rather than copying their content. The response explains the orchestrator flow: define the root question, map each subagent pyramid to a dimension, verify each subagent pyramid is complete before synthesis, reconcile conflicting findings across teams explicitly rather than silently averaging them, and produce the root SOURCES sections pointing at each sub-pyramid with a description of what it contains.",
      "assertions": [
        "The root L1 is synthesized from the sub-pyramids' L1 findings per dimension",
        "Each root L2 dimension references its subagent pyramid as the evidence source",
        "Conflicting findings across teams are reconciled explicitly, not averaged silently",
        "Sub-pyramids are verified complete before synthesis",
        "The root SOURCES sections point at each sub-pyramid with descriptions"
      ]
    }
  ]
}

# The Artifact Pyramid: Framework & Symmetry

## The Core Insight

Progressive disclosure governs how we feed AI agents context: start with metadata (~100 tokens), expand to instructions on activation (<5000 tokens), and load resources on demand. **The Artifact Pyramid applies the same principle to what agents produce.**

The same constraint — finite context windows and nonlinear quality degradation from overload — applies whether an agent is reading a skill description or reading a research artifact. The same solution — progressive disclosure via layered, linked, on-demand-loadable units — applies whether the agent is loading procedural knowledge or receiving declarative findings.

## The Asymmetry Problem

In current AI agent workflows:

| | Input Side | Output Side |
|---|---|---|
| Structure | Progressive disclosure (3 tiers) | Single flat document |
| Consumption | Load only what's needed | Everything bundled together |
| Cost | ~2500 tokens overhead for 50 skills | Full context window per consumer |

Every agent in a multi-agent pipeline inherits context from upstream agents. Without the pyramid, context accumulates across stages until every downstream agent operates in a window polluted by material irrelevant to its specific role.

## The Symmetry

| Input Tier (Skill Loading) | Output Tier (Artifact Pyramid) |
|---|---|
| **Metadata:** name + description (~100 tokens) | **L1 Summary:** key findings, implications (entry point) |
| **Instructions:** full skill body on activation | **L2 Analysis Collection:** per-dimension files |
| **Resources:** reference files loaded on demand | **L3 Detailed Dossiers:** source excerpts, transcripts |

The symmetry is the core architectural insight: the same three-tier model, the same context economy discipline, applied to what agents write rather than what they read.

## Relationship to Other Frameworks

### DIKW Pyramid (Ackoff 1989)

| DIKW | Artifact Pyramid |
|---|---|
| Wisdom | L1 Summary (actionable insights, decisions) |
| Knowledge | L2 Analysis Collection (connected understanding) |
| Information | L3 Detailed Dossiers (extracted data points) |
| Data | Raw primary sources (unprocessed captures) |

### Agent Skills (agentskills.io)

The Artifact Pyramid is the Agent Skills model reflected outward:

```
Skill File:     SKILL.md (metadata) → SKILL.md body (instructions) → references/ (resources)
     ↓ mirror
Artifact:       L1 Summary (metadata) → L2 Analysis (instructions) → L3 Dossiers (resources)
```

## Multi-Agent Routing

The pyramid gives an orchestrator the precision to deliver only what each downstream profile requires:

| Agent Profile | Receives |
|---|---|
| Product Manager Agent | L1 Summary only |
| Market Analyst Agent | Specific L2 analysis files |
| Data Architect Agent | L3 Dossiers |
| Validator Agent | L3 Dossiers + L2 files |

This prevents the context accumulation problem: each agent gets exactly what it needs, nothing more.

## When the Pyramid Breaks

1. **Ephemeral output**: A one-off calculation doesn't need three layers.
2. **Source IS the output**: Collecting data without synthesizing collapses the pyramid.
3. **Single consumer only**: If the same agent consumes and produces, the overhead of structured handoffs may not justify itself.
4. **Tiny output**: A three-sentence answer doesn't need three layers.

Use the pyramid when: research depth > 5 sources, multi-agent handoffs are involved, or outputs need to be auditable and reusable.

# Canonical Article: The Artifact Pyramid

**Published at Groktopus** — May 31, 2026

The Artifact Pyramid concept is published as a full article at groktop.us. This is the
canonical Layer 3 artifact for the methodology itself:

## Link

[https://www.groktop.us/artifact-pyramid-progressive-disclosure/](https://www.groktop.us/artifact-pyramid-progressive-disclosure/)

## What the article covers that the skill references don't

- **Full narrative argument** for why progressive disclosure must be symmetric
- **Quantitative motivation** — token budgets saved by layered output consumption
- **Worked examples** from real multi-agent pipelines
- **Related work** — how the pyramid relates to DIKW, Zettelkasten, and build pipelines
  (expanded beyond the framework reference)
- **Anti-patterns** beyond the skill's "When NOT to use" section

## How to use this in-context

When helping a user understand the Artifact Pyramid as a concept (rather than just
executing the methodology), reference this article. It serves as the "why" to the
skill's "how."

When building an Artifact Pyramid project, the article is itself an example of
what a Layer 3 published artifact should look like: self-contained, audience-ready,
with traceable claims back to the underlying framework.

# Composite Pyramid Synthesis

When multiple parallel subagents each produce their own artifact pyramid (one per research dimension), the orchestrator synthesizes them into a single **composite pyramid** organized by **opportunity/entity** rather than by **source agent**.

## When to Use This Pattern

Any parallel research pipeline where 2+ subagents investigate independent dimensions and their findings need to be merged into a unified view. Examples:
- Jobs Finder: 4 researchers (ATS, exec search, aggregators, company pages) → 1 consolidated opportunity pyramid
- Competitive analysis: market sizing + technical feasibility + competitive landscape → 1 merged pyramid
- Due diligence: financial + technical + legal + operational → 1 deal pyramid

## The Problem With Leaving Sub-Pyramids In Place

Each subagent pyramid is organized by what that agent discovered. When a downstream orchestrator or scoring agent needs to evaluate, say, "Komodo Health," it must:
1. Read R1's pyramid to see the Greenhouse listing
2. Read R3's pyramid to see if Built In also found it
3. Cross-reference mentally whether both sources refer to the same opportunity

This defeats the purpose of progressive disclosure. The composite pyramid reorganizes the data so one `02-analysis/komodo-health.md` file contains everything from all sources.

## Synthesis Protocol

### Step 1: Cross-Reference All Candidates

Read the L1 summary from each sub-pyramid. Build a master list of every unique candidate identified as `company|role`. Note which sub-pyramids found each one.

### Step 2: Build the Composite Pyramid

```
<project-root>/
├── 00-index.md              ← Entry point with cross-dedup summary
├── 01-summary/
│   └── findings.md          ← L1: merged findings with triage
├── 02-analysis/             ← L2: one file per OPPORTUNITY
│   ├── komodo-health.md
│   ├── picnichealth.md
│   ├── mie-cto.md
│   └── ...
└── 03-dossiers/
    ├── dedup-log.md          ← Cross-result dedup audit trail
    └── sub-pyramids-archive.md  ← What was absorbed and from where
```

The critical difference from a single-agent pyramid: L2 files are named after **opportunities** (companies + roles), not after **research dimensions**. A downstream agent reads `02-analysis/komodo-health.md` and gets everything — regardless of whether R1 or R3 found it.

### Step 3: Cross-Result Dedup

For each candidate, check whether it appeared in multiple sub-pyramids. If the same `company|role` was found by R1 (Greenhouse) and R4 (career page), collapse to a single entry. Document the collision in `03-dossiers/dedup-log.md`:

```markdown
| Company | Role | Found By | Also Found By | Verdict |
|---------|------|----------|---------------|---------|
| Veeva Systems | VP Engineering | R3 (Built In) | R4 (career page) | Already in dedup set — no collision |
| Komodo Health | VP AI Engineering | R1 (Greenhouse) | — | Unique to R1 |
```

### Step 4: Remove Sub-Pyramids

After the composite pyramid is verified complete:
1. Confirm all candidates from all sub-pyramids are represented in the composite
2. Delete the individual sub-pyramid directories
3. Update the `03-dossiers/sub-pyramids-archive.md` to document what was removed
4. Fix all SOURCES sections in the composite that referenced the old sub-pyramid paths

### Step 5: Fix SOURCES Navigation

After removal, update every SOURCES section that pointed to a deleted sub-pyramid. Replace researcher-specific `../researcher-1/00-index.md` references with a pointer to `03-dossiers/sub-pyramids-archive.md`.

# Delegation Context Template: Artifact Pyramid Mandate

When delegating research to a subagent via your agent's delegation mechanism, the subagent does NOT load any skills. The folder structure, SOURCES requirement, and output format MUST be explicitly included in the `context` string. This template provides the exact text to include.

## Minimal Template

Copy this block verbatim into every research delegation context:

```
OUTPUT FORMAT: You MUST produce a compliant Artifact Pyramid, not flat files.

Create this directory structure at <your-output-path>/:
├── 00-index.md              ← entry point (mandatory)
├── 01-summary/findings.md   ← L1: one findings file
├── 02-analysis/             ← L2: one file per dimension/opportunity
└── 03-dossiers/             ← L3: raw scrape outputs, search results

Every file MUST end with a ## SOURCES section linking to deeper layers:
- L1 files link to L2 analysis files
- L2 files link to L3 dossiers
- Each SOURCES entry includes a description answering: "what will I find if I go deeper?"

When the output directory doesn't exist yet, create it with the full pyramid structure before writing any files.

Respond with ONLY the absolute path to the completed pyramid's 00-index.md.
Do NOT respond with natural language summaries or prose.
```

## Expanded Template (with rationale)

For first-time delegations or when the subagent needs more context:

```
OUTPUT FORMAT: You MUST produce a compliant Artifact Pyramid (https://www.groktop.us/artifact-pyramid-progressive-disclosure/).

The pyramid has three layers consumed top-down:
- L1 (01-summary/): Key findings, most important implications. One file only.
- L2 (02-analysis/): Per-dimension analysis files. One file per discovery. Self-contained.
- L3 (03-dossiers/): Raw source excerpts, scrape outputs, methodology notes.

Directory structure at <your-output-path>/:
├── 00-index.md              ← entry point (mandatory)
├── 01-summary/findings.md   ← L1: one findings file
├── 02-analysis/             ← L2: one file per promising result
└── 03-dossiers/             ← L3: raw outputs

Every file MUST end with a ## SOURCES section. SOURCES are navigation affordances
for downstream agents — each entry answers "what will I find if I go deeper?"
L1 SOURCES links to 02-analysis/ files. L2 SOURCES links to 03-dossiers/ files.

If the output directory doesn't exist, create it with mkdir -p before writing.

Respond with ONLY the absolute path to 00-index.md. No summary, no natural language.
```

## Why This Is Necessary

your agent's delegation mechanism subagents have no access to the parent's loaded skills, memory, or conversation history. They receive only the `context` string. If the artifact-pyramid folder structure isn't in that string, they will produce flat files at the root of their output directory. The flat files will contain good content but will lack navigation affordances, making them unusable by downstream agents in the pipeline.

# Flat Output to Artifact-Pyramid Migration

When converting existing flat-file outputs to artifact-pyramid format, follow this pattern. It has been applied to baseline caches (SkillOpt PR #23) and validation results (PR #29), with the same structure reused across both.

## The Pattern

Given an existing flat output directory like:

```
outputs/
├── item-1.json            ← item-level result
├── item-2.json
└── item-3.json
```

Convert to:

```
output-pyramid/epoch-<N>/
├── 00-index.md                    ← navigation + provenance only (no findings)
├── 01-summary/findings.md         ← L1: YAML frontmatter (aggregate metrics) + human summary
├── 02-analysis/
│   ├── category-a.md              ← L2: one file per meaningful category of items
│   └── category-b.md              ← L2: each with SOURCES linking to L3
└── 03-dossiers/
    ├── item-1.json                ← L3: original JSON files, schema unchanged
    ├── item-2.json
    └── item-3.json
```

## Rules

### L1 YAML Frontmatter
Machine-parseable via `json.loads()`. Contains aggregate metrics that downstream consumers need at a glance (counts, rates, averages). No nested objects that require recursive parsing — flat key-value pairs and lists of IDs only.

### L2 Analysis Files
One file per meaningful category of items (accepted vs rejected, success vs failure, by type). Each file:
- Lists every item in its category with a one-line summary per item
- Ends with a `## SOURCES (LAYER 3 NAVIGATION)` section that maps to individual L3 dossiers
- Paths in SOURCES are **relative** (e.g., `03-dossiers/item-1.json`) for portability

### L3 Dossiers
The original flat JSON files, schema unchanged. Copied (not symlinked) so the pyramid is self-contained. Copy via `shutil.copy2` to preserve metadata.

### 00-index.md
Navigation + provenance only. No findings, no verdicts, no scores. Per the output-classification-framework reference: "00-index.md exists only to get the consumer to the right layer file."

## Downstream Consumer Changes

When a downstream process previously read the flat files, add a pyramid-aware read path:

```python
# Try pyramid first, fall back to flat glob
pyramid_index = os.path.join(state_dir, "output-pyramid", f"epoch-{epoch}", "00-index.md")
if os.path.exists(pyramid_index):
    summary_path = os.path.join(os.path.dirname(pyramid_index), "01-summary", "findings.md")
    meta = json.loads(open(summary_path).read().split("---", 2)[1])
    item_ids = [e["id"] for e in meta.get("items", [])]
    dossier_dir = os.path.join(os.path.dirname(pyramid_index), "03-dossiers")
    result_files = sorted(os.path.join(dossier_dir, f"{eid}.json") for eid in item_ids)
else:
    result_files = sorted(glob.glob(os.path.join(flat_dir, "*.json")))
```

## When to Use

- You have existing flat JSON outputs that are consumed by downstream processes
- The items naturally fall into categories (accepted/rejected, pass/fail, by type)
- A human or agent needs to scan aggregate results without loading every item
- You want backward compatibility during the transition (flat files still written, pyramid is additive)

# Intellectual Lineage: Progressive Disclosure in Software Architecture Documentation

The Artifact Pyramid applies progressive disclosure to what AI agents produce. But the principle was independently discovered (under different names) by software architecture documentation methodology over 30 years. This reference traces that lineage and clarifies what the Artifact Pyramid generalizes beyond what architecture already found.

## The Six Discoveries (Chronological)

| Year | Methodology | Their Word for Progressive Disclosure | What They Built |
|------|-------------|--------------------------------------|-----------------|
| 1995 | Kruchten's 4+1 Views | "concurrent views," "partial expression" | Role-based information partitioning by stakeholder |
| 2000 | IEEE 1471 / ISO 42010 | "viewpoints," "stakeholder concerns" | Formal standard: views are "partial expressions" of architecture |
| 2002 | Views and Beyond (SEI/Clements) | "view selection," "scoping" | Systematic process for choosing which views to produce |
| 2005 | Rozanski & Woods | "viewpoints framework" | Audience-targeted viewpoint selection |
| 2008 | arc42 | "tailoring," "lean/essential/thorough" | Three built-in depth modes + a Canvas "zip-version" |
| 2011 | ADRs (Nygard) | "small, modular documents" | Atomic decisions — read one at a time, chain as needed |
| ~2011 | C4 Model (Brown) | "zoom levels," "hierarchical abstractions" | Context → Container → Component → Code drill-down |

## The Key Distinctions

### Architecture's version: role-based, synchronous
Architecture view models partition information **by stakeholder role**. Different views exist simultaneously for different people. An executive sees the Context view; a developer sees the Component view. This is progressive disclosure as a *social coordination mechanism* — telling different stories to different audiences at the same time.

### Artifact Pyramid: role-based AND temporal
The Artifact Pyramid adds **temporal unfolding** — the same consumer (whether human or agent) gets more detail as they go deeper. This is progressive disclosure as a *consumption mechanism* — the same person or agent starts at the summary and drills in on demand.

Architecture discovered the first axis. The Artifact Pyramid adds the second.

### Domain scope
Architecture documentation covers structural visualization and decisions. The Artifact Pyramid covers the full delivery lifecycle: strategy → architecture → implementation → operations → learning. No architecture methodology has a broader scope.

### Consumer type
Architecture documentation assumes a *human* reader who can flip pages, open files, and decide what to read next. The Artifact Pyramid assumes an *agent* consumer — which changes the constraints entirely:
- Context windows and token budgets make progressive disclosure a **hard requirement**, not a convenience
- Agents are stateless between calls — each layer must be independently consumable
- Navigation must be explicit (SOURCES blocks with absolute paths), not implicit ("see chapter 4")

## What This Means

The architecture community validates that the mechanism works — 30 years of practice, thousands of teams, documented in IEEE standards. The Artifact Pyramid did not rediscover progressive disclosure. It:

1. **Named** a principle that architecture docs had stumbled toward without naming
2. **Generalized** it across the full lifecycle (not just structural views)
3. **Extended** it to temporal unfolding (not just role-based)
4. **Applied** it to a new domain (AI agent output) where it's a hard requirement, not an optimization

## Sources

### Vault (permanent knowledge base)

The full research with source excerpts, 9 supporting atoms, and methodology-by-methodology analysis lives in the Magnus v2 vault:

```
2 - Molecules/Intellectual Lineage of the Artifact Pyramid — Progressive Disclosure in Architecture Documentation.md
 -> Full synthesis: 9 atoms covering Kruchten (1995), IEEE 1471, C4 Model, arc42, ADRs, Views and Beyond, Rozanski & Woods. Source excerpts from each canonical source.
```

### Canonical URLs

### Vault (permanent knowledge base)

The full research with source excerpts, 9 supporting atoms, and methodology-by-methodology analysis lives in the Magnus v2 vault:

```
2 - Molecules/Intellectual Lineage of the Artifact Pyramid — Progressive Disclosure in Architecture Documentation.md
 -> Full synthesis with 9 atoms covering each methodology, source excerpts, and cross-cutting analysis
```

### Canonical URLs

- Kruchten (1995): https://www.cs.ubc.ca/~gregor/teaching/papers/4+1view-architecture.pdf
- IEEE 1471: https://ieeexplore.ieee.org/document/875998
- ISO 42010: https://www.iso.org/obp/ui/en/#!iso:std:74393:en
- Views and Beyond (Clements et al.): https://www.pearson.com/en-us/subject-catalog/p/documenting-software-architectures-views-and-beyond/P200000000186/9780132488594
- arc42 template: https://docs.arc42.org/
- ADRs (Nygard, 2011): https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions
- C4 Model: https://c4model.com
- Rozanski & Woods: https://www.viewpoints-and-perspectives.info/

# Methodology-to-Pyramid Mapping — Specialist Profile Convention

## The Pattern

Specialist profiles (researcher, technical-architect, data-architect, product-manager) each own a set of domain-specific methodologies. When consuming a task, they produce their domain's natural artifacts — but they structure those artifacts as an Artifact Pyramid at a known filesystem path and hand off ONLY the path to 00-index.md.

The pyramid structure is universal. The methodology mapping is domain-specific.

## Existing Mappings

### Researcher Profile

| Domain Artifact | Methodology | Pyramid Layer |
|----------------|-------------|--------------|
| Research question + key findings | Mission interpolation | L1 Summary |
| Per-dimension analysis (market, technical, competitive) | Systematic gathering + gap evaluation | L2 Analysis Collection |
| Source excerpts, transcripts, raw data | Source capture | L3 Dossiers |
| Path: `/tmp/researcher-workflow/<slug>/00-index.md` |

### Technical-Architect Profile

| Domain Artifact | Methodology | Pyramid Layer |
|----------------|-------------|--------------|
| C4 Level 1 (System Context) + arc42 Canvas + Decision dashboard | C4 Model, arc42 | L1 Summary |
| C4 Level 2 (Container), C4 Level 3 (Component), Active ADRs, arc42 sections 1-4, Quality attributes, Failure mode analysis | C4 + ADRs + arc42 | L2 Analysis Collection |
| C4 Level 4 (by exception), Superseded ADRs, Spike results, API contracts | C4 + ADRs | L3 Dossiers |
| Path: `/tmp/architect-workflow/<slug>/00-index.md` |

## Dimension Boundary Rules

When multiple methodologies overlap (e.g., C4 says "PostgreSQL 16" and arc42 says "must use RDS" and an ADR says "why PostgreSQL over MongoDB"), use these rules to decide where the fact lives:

| Claim Type | Goes In | Because |
|-----------|---------|---------|
| Structural choice (what) | C4-level file (L2) | Structural view shows the choice |
| Rationale (why) | ADR (L2) | ADRs capture reasoning |
| Constraint (must) | arc42 context file (L2) | Constraints are external forces |
| Quantified target (how good) | Quality attributes file (L2) | Scenarios, not decisions |
| Degradation (when it breaks) | Failure mode file (L2) | Different analytical dimension |
| Historical (was, but no longer) | L3 dossier | Reference only, not governing |

## Creating a New Profile Mapping

1. **Identify the profile's core methodologies.** What does a human in this role produce? (e.g., product manager produces PRDs, OKRs, competitive analyses)
2. **Map each artifact to a pyramid layer.** The summary document goes in L1. The per-dimension analysis goes in L2 (one file per dimension). The supporting reference materials go in L3.
3. **Define the output path.** Convention: `/tmp/<profile-role>-workflow/<slug>/00-index.md`
4. **Document the handoff.** The profile responds with ONLY the path to 00-index.md. Same convention as every other specialist profile.
5. **Define dimension boundaries.** If two methodologies would both claim the same fact, document which one owns it.

## Why This Matters

Without explicit methodology-to-pyramid mapping, each profile invents its own convention. With it, downstream orchestrators can route agents to any profile and know the handoff format is identical — only the content methodology changes. This makes specialist profiles plug-compatible.

# Nested Pyramid Pattern — Unified Single Hierarchy for Multi-Artifact Systems

When a single system produces multiple distinct artifact streams (rollout records, validation results, proposals, baseline metrics, meta-reflections), the natural instinct is to create a separate pyramid directory for each stream:

```
❌ Scatter — 6 independent pyramids
<state_dir>/
├── baseline/epoch-N/00-index.md
├── validation/epoch-N/00-index.md
├── rollout/epoch-N/00-index.md
├── reflection/epoch-N/00-index.md
├── proposal/epoch-N/00-index.md
├── slow-meta/epoch-N/00-index.md
└── run-summary.json
```

This forces a consumer to check 6 entry points to understand what's available. Each pyramid is disconnected from the others — no cross-referencing, no shared navigation.

## The Unified Alternative

Nest all artifact streams under a single root pyramid. Each epoch's data lives flat in `03-dossiers/` with epoch-prefixed filenames. Root-level files are **amended** as new epochs complete — they grow to reflect the current state.

```
✅ Unified — single navigable tree (flat 03-dossiers/)
<state_dir>/
├── 00-index.md                          ← AMENDED per epoch: navigation links, epoch list
├── 01-summary/findings.md              ← AMENDED per epoch: updated final_epoch, scores
├── 02-analysis/
│   ├── epoch-trajectory.md             ← AMENDED per epoch: appended trend row
│   └── epoch-1-overview.md             ← NEW per epoch: phase summary + SOURCES → dossiers
└── 03-dossiers/                        ← FLAT: no subdirectories
    ├── epoch-1-baseline.json
    ├── epoch-1-validation-edit-1.json
    ├── epoch-1-validation-edit-2.json
    ├── epoch-1-rollout-task-train-1.json
    ├── epoch-1-reflection.json
    ├── epoch-1-proposals.json
    ├── epoch-2-baseline.json
    ├── epoch-2-validation-edit-1.json
    └── epoch-2-slowmeta.json
```

### Critical constraint: 03-dossiers/ is flat

**03-dossiers/ MUST NOT contain subdirectories.** The artifact-pyramid spec defines the dossier layer as the broadest, flattest reference library. Subdirectories inside 03-dossiers/ violate the spec's flat-file contract.

Use epoch-prefixed filenames instead of nested directories:

| ❌ Wrong (subdirectories) | ✅ Correct (flat naming) |
|---|---|
| `03-dossiers/epoch-1/baseline.json` | `03-dossiers/epoch-1-baseline.json` |
| `03-dossiers/epoch-1/rollout/task-1.json` | `03-dossiers/epoch-1-rollout-task-1.json` |
| `03-dossiers/epoch-1/validation/edit-1.json` | `03-dossiers/epoch-1-validation-edit-1.json` |

Naming convention: `epoch-<N>-<phase>[-<item>].json`

### Amending pattern

Root-level files are never rewritten from scratch. They are amended as epochs complete:

| File | Action per epoch |
|---|---|
| `00-index.md` | Update epoch list + navigation links to new epoch overview |
| `01-summary/findings.md` | Update final_epoch, final_pass_rate, epoch count |
| `02-analysis/epoch-trajectory.md` | Append new row to trajectory table |
| `02-analysis/epoch-N-overview.md` | Always new — never modifies a previous epoch |
| `03-dossiers/epoch-N-*.json` | Always new — never modifies previous epoch files |

The amending pattern means a consumer reading `00-index.md` sees the current state of the entire run. Progressively deeper files (epoch overviews, dossiers) are fixed once written and never change.

## Navigation Flow

```
00-index.md
  └─ SOURCES → 01-summary/findings.md
                  └─ SOURCES → 02-analysis/epoch-trajectory.md
                  └─ SOURCES → 02-analysis/epoch-1-overview.md
                                  └─ SOURCES → 03-dossiers/epoch-1-baseline.json
                                  └─ SOURCES → 03-dossiers/epoch-1-validation-edit-1.json
```

A consumer reading the root `00-index.md` can navigate to any epoch or any phase by following SOURCES links directly to the flat dossiers in `03-dossiers/`. No nested 00-index to traverse — the epoch overview IS the entry point for that epoch.

## When to Use

Use the nested pyramid pattern when:
- A system produces 3+ distinct artifact types over time
- Artifacts share a common temporal dimension (epochs, rounds, iterations)
- A consumer needs to understand the system's full output at a glance
- You want to avoid filesystem sprawl from parallel directory structures

## When NOT to Use

- Single-turn research tasks that produce one artifact (use the standard flat pyramid)
- Systems where artifact streams are owned by completely independent subsystems (use composite pyramid synthesis instead)

## Relationship to Composite Pyramid Synthesis

The composite pyramid pattern merges outputs from **multiple independent agents** (e.g., a research pipeline where each subagent produces its own pyramid). The nested pyramid pattern organizes outputs from **a single system** into one hierarchy. They solve different problems:
- Composite: merging independent agent pyramids into a root-level synthesis
- Nested: designing a single system's output tree from scratch

## Implementation Pattern

When building a nested pyramid, use a shared writer function that all phases call after writing their L3 dossiers:

```python
def rebuild_epoch_pyramid(state_dir, epoch):
    """After any phase writes L3 dossiers, rebuild the epoch-level index/summary/analysis."""
    epoch_dir = os.path.join(state_dir, "03-dossiers", f"epoch-{epoch}")
    if not os.path.exists(epoch_dir):
        return
    # Scan for existing L3 dossiers
    # Read board-metadata.json for pass_rate_history
    # Write 00-index.md, 01-summary/findings.md, all 02-analysis/ files
    # Each 02-analysis file has SOURCES linking to its L3 dossiers
```

This centralizes the pyramid writing logic so each phase only produces its raw data (L3), and a single function handles the progressive disclosure wrapper.

## In This Session

This pattern was developed during the SkillOpt artifact-pyramid conversion session (2026-06-03). The pre-existing pattern was scatter (6 parallel directories for baseline, validation, rollout, reflection, proposal, slow-meta). The unified pattern consolidates everything under `<state_dir>/` with epoch-level nesting inside `03-dossiers/epoch-N/`.

# Output Classification Framework

A role-agnostic system for mapping any specialist's outputs into the artifact pyramid, with strict layer-specific content contracts to eliminate duplication.

## The Problem

Every specialist profile (editor, researcher, technical-architect, product-manager, copy-editor, SEO specialist) independently answers the same question: *"Where does my stuff go in the pyramid?"*

Without a shared framework, this produces inconsistent mappings. The same type of output ends up in different layers across profiles. Content duplicates between `00-index.md` and `01-summary/`. The boundary between "navigation" and "verdict" blurs.

This framework fixes that by providing three classification questions any specialist can ask about their outputs, independent of domain.

## The Three Classification Questions

For every distinct type of output a specialist produces, ask:

### 1. Who consumes this?

The consumer determines the layer, because each consumer needs a different level of abstraction.

| Consumer | Example roles | What they need | Layer |
|----------|---------------|----------------|-------|
| **Decider** | Orchestrator, author, pipeline gate | A verdict and priority actions. "Should I ship this? What should I fix first?" | **L1 (Summary)** |
| **Practitioner** | The specialist's counterpart, downstream implementer | Evidence organized by dimension. "Why is this broken? How do I fix it?" | **L2 (Analysis)** |
| **Skeptic** | Verifier, proofreader, challenger, reviewer | Raw material to verify claims. "Is this source real? Show me the test output." | **L3 (Dossiers)** |

**Rule:** If a piece of output serves more than one consumer, it belongs at the layer of the *least* abstract consumer who needs it. If both the decider and practitioner need it, put it in L2 — the decider can still read it there (they just descend one level).

### 2. How often is it consumed?

Consumption frequency confirms the layer assignment and tells you how many files to create.

| Frequency | Pattern | Layer | Cardinality |
|-----------|---------|-------|-------------|
| **Every pipeline handoff** | The first thing anyone reads to decide whether to proceed | **L1** | Exactly one file per project |
| **Per dimension, on demand** | Someone drills into this only when they need that specific dimension | **L2** | One file per analytical dimension |
| **On challenge or deep review** | Rarely accessed — pulled only when someone needs to verify | **L3** | One file per evidence set |

**Rule:** If an output is consumed every time the artifact is touched, it belongs in L1 or the 00-index. If it's consumed only when someone needs a specific question answered, it belongs in L2 or L3.

### 3. What question does it answer?

This is the content boundary check — the rule that prevents duplication.

| Question | Layer | File answers | File does NOT answer |
|----------|-------|--------------|---------------------|
| "What should I do?" | **L1** | The verdict, the score, the priority actions | How the verdict was reached (that's L2), where the raw data lives (that's L3) |
| "Why should I do it?" | **L2** | Evidence, analysis, reasoning per dimension | What to do (that's L1), the primary sources (that's L3) |
| "Is this real / can I prove it?" | **L3** | Raw output, test results, source excerpts, capture metadata | What it means (that's L2), what to do about it (that's L1) |

**Rule:** Every sentence in a file must answer its layer's assigned question. If a sentence answers a different layer's question, it belongs in that other layer. If a sentence answers no layer's question, it doesn't belong in the pyramid at all.

## Applying the Framework

### Step-by-step

1. **Inventory your outputs:** List every distinct type of artifact your specialist produces. (A copy-editor produces: a verdict, mechanical findings, line-edit findings, consistency findings, cross-reference results, a style sheet, a change log, a query list, verified source links.)

2. **For each output, ask the three questions:**
   - *Who consumes this?* → Decider, practitioner, or skeptic?
   - *How often?* → Every handoff, per dimension, or on challenge?
   - *What question does it answer?* → "What should I do?", "Why?", or "Is this real?"

3. **Classify to layer:** The answers determine the layer. If they disagree (e.g., consumed every handoff but answers "why"), the stricter constraint wins — protect the consumer who needs it the least.

4. **Verify with the quality gate:** For each file, check every sentence against its layer's question.

### Worked Example: Copy-Editor

| Output | Who consumes? | How often? | Answers? | Layer |
|--------|-------------|-----------|----------|-------|
| Verdict, key counts, top findings | Decider (orchestrator, author) | Every handoff | "What should I do?" | **L1** |
| Mechanical edit findings | Practitioner (author, writer) | Per dimension | "Why fix this?" | **L2** |
| Line-edit findings | Practitioner (author, writer) | Per dimension | "Why rephrase this?" | **L2** |
| Consistency findings | Practitioner (author, writer) | Per dimension | "Why standardize this?" | **L2** |
| Cross-referencing results | Practitioner (author) | Per dimension | "Which links are broken?" | **L2** |
| Style sheet | Practitioner (proofreader) | Per project, on demand | "What style conventions were applied?" | **L3** |
| Change log | Skeptic (challenger) | On challenge | "What exactly changed?" | **L3** |
| Verified sources | Skeptic (verifier) | On challenge | "Which links actually resolve?" | **L3** |
| Query list | Decider (author) | Every handoff | "What questions remain open?" | **L1** (adjunct) |

### Worked Example: SEO Specialist

| Output | Who consumes? | How often? | Answers? | Layer |
|--------|-------------|-----------|----------|-------|
| Overall verdict + score | Decider (pipeline, author) | Every handoff | "What should I fix?" | **L1** |
| Technical SEO findings | Practitioner (developer) | Per dimension | "Why is this a problem?" | **L2** |
| On-page findings | Practitioner (writer, editor) | Per dimension | "Why optimize this?" | **L2** |
| Schema validation | Practitioner (developer) | Per dimension | "What's broken in my schema?" | **L2** |
| Ghost metadata audit | Practitioner (editor) | Per dimension | "Which fields are missing?" | **L2** |
| AEO analysis | Practitioner (writer, editor) | Per dimension | "Why isn't this LLM-visible?" | **L2** |
| Raw crawl output | Skeptic (verifier) | On challenge | "What actual HTTP responses were seen?" | **L3** |
| Raw schema test results | Skeptic (verifier) | On challenge | "What did the validator say?" | **L3** |

## The 00-index / L1 Boundary

The most common duplication failure is between `00-index.md` and files inside `01-summary/`. The solution is to assign them distinct questions:

| File | Answers | Contains | Does NOT contain |
|------|---------|----------|-----------------|
| **00-index.md** | "Where do I find what I need?" | Navigation tree, project metadata, 1-sentence orientation, SOURCES | Findings, verdict, scores, priority actions, analysis, raw data |
| **01-summary/*** | "What should I do?" | Verdict, score, priority findings, recommended actions | Navigation, metadata, file lists, raw evidence, methodology |

**00-index.md exists only to get the consumer to the right layer file.** It is navigation and provenance, not analysis. If the consumer reads 00-index.md and can decide whether to proceed, they don't need more from the index. If they need to decide *what to do*, they descend to 01-summary.

## Quality Gates

Every file in the pyramid must pass its layer's gate. The gate is a single question:

| Layer | Gate Question | Pass condition |
|-------|---------------|----------------|
| **00-index** | "Is this file only navigation and provenance?" | No findings, verdict, scores, or analysis sentences |
| **L1 (Summary)** | "Does every sentence tell me what to do or what's most important?" | No raw data, no methodology, no file listings |
| **L2 (Analysis)** | "Does every sentence give me evidence organized by dimension?" | No verdict statements, no primary source text |
| **L3 (Dossiers)** | "Does every sentence present raw material without interpreting it?" | No recommendations, no judgments, no analytical framing |

### How to audit

For each file, read every sentence and ask: *"Does this answer my layer's question?"* If the answer is no for any sentence, that sentence belongs in a different layer or doesn't belong at all.

**Common failure patterns:**
- 00-index.md contains "Top 3 Critical Findings" → belongs in 01-summary
- 01-summary/verdict.md contains metadata (author, date, word count) → belongs in 00-index
- 02-analysis/findings.md contains "Recommendation: fix X" → this is a verdict (L1), not evidence (L2)
- 03-dossiers/raw-data.md contains "this suggests that..." → interpretation (L2), not raw material (L3)

## Relationship to Other References

| Reference | How this framework extends it |
|-----------|------------------------------|
| `pipeline-stages.md` | Defines what the layers ARE. This framework defines how to DECIDE what goes WHERE. |
| `methodology-to-pyramid-mapping.md` | Shows how specific roles (technical-architect, researcher) map their domain. This framework is the ROLE-AGNOSTIC abstraction they all follow. |
| `quality-gates.md` | Defines verification for a finished pyramid. This framework defines the CONTENT CONTRACTS that prevent violations from being created in the first place. |

# Pipeline Stages: The Three Layers in Detail

## Layer Numbering Convention

The Artifact Pyramid numbers layers **top-down**, matching the Agent Skills input model:

| | Agent Skills (Input) | Artifact Pyramid (Output) |
|---|---|---|
| Level 1 | Skill metadata (~100 tokens) | **Summary** — the entry point, most distilled |
| Level 2 | Skill instructions (<5000 tokens) | **Analysis Collection** — detail on demand |
| Level 3 | Reference files (loaded as needed) | **Detailed Dossiers** — the full evidentiary base |

---

## Layer 1: Summary

### Purpose
The entry point for every consuming agent. A single file that states the research question, key findings, and most important implications. No evidence, methodology, or supporting data — those live one layer down.

### Format
A single markdown file, typically a few paragraphs to a few pages.

### Contents
- Research question (restated from mission brief)
- Key findings (3-5 bullet points or short paragraphs)
- Implications (what this means for the audience)
- **SOURCES section** at the bottom linking to Layer 2 analysis files:

```
SOURCES (LAYER 2 NAVIGATION)
research/analysis/market-position.md
 -> Competitor mapping and market share analysis

research/analysis/technical-feasibility.md
 -> Architecture evaluation
```

### Who Consumes
- Product-manager agents (strategic orientation, no technical depth needed)
- Executives (headline-level decisions)
- Quick scanners deciding whether to go deeper

### Quality Gate (Gate A)

- [ ] Every claim in the summary links to a specific Layer 2 analysis file
- [ ] The summary contains no unsupported assertions — everything traces downward
- [ ] The summary is self-contained (makes sense without lower layers)
- [ ] Implications are stated explicitly, not buried in findings

---

## Layer 2: Analysis Collection

### Purpose
Self-contained analysis files, each covering a specific dimension of the research. A consumer who needs only one dimension loads that single file and nothing else.

### Format
Individual markdown files, one per research dimension. Each is independently consumable.

### Typical Dimensions
- Market analysis (size, trends, segmentation)
- Competitive landscape (positioning, strengths, gaps)
- Technical feasibility (architecture, constraints, trade-offs)
- Risk assessment (uncertainties, failure modes, mitigations)
- Regulatory analysis (compliance, jurisdictional issues)

### Contents
- Thesis or research question for this dimension
- Analysis narrative with supporting evidence
- Data visualizations or tables as needed
- **SOURCES section** at the bottom linking to Layer 3 dossiers:

```
SOURCES (LAYER 3 NAVIGATION)
research/dossiers/competitor-profiles.md
 -> Market-by-market competitive positioning data

research/dossiers/interview-transcripts.md
 -> Customer interview transcripts referenced in Section 2
```

### Who Consumes
- Domain-specialist agents (data scientist reads data analysis only)
- In-depth researchers (need one dimension, not the full picture)
- Downstream agents composing cross-domain syntheses

### Quality Gate (Gate B)

- [ ] Each analysis file is self-contained (makes sense without surrounding files)
- [ ] Every claim traces to specific Layer 3 sources
- [ ] Analysis adds interpretive value beyond raw data
- [ ] Conflicting evidence is surfaced, not buried
- [ ] Layer 3 navigation (SOURCES) is complete

---

## Layer 3: Detailed Dossiers

### Purpose
The broadest layer. Source excerpts, raw data tables, interview transcripts, methodology notes. A reference library that consuming agents pull from as needed — not intended for linear reading.

### Format
Multiple files of varying types: markdown notes, raw data exports, captured pages, transcript extracts.

### Typical Contents
- Source excerpts (annotated with source metadata)
- Raw data tables
- Interview or meeting transcripts
- Methodology documentation
- Literature review notes
- Dataset descriptions

### Who Consumes
- Validators (fact-checking claims from upper layers)
- Deep-dive researchers (need the full evidentiary base)
- Downstream analysis tools (embedding pipelines, graph databases)

### Quality Gate (Gate C)

- [ ] Every source has attribution (URL, timestamp, author, title)
- [ ] Extracts are faithful to the original (no misrepresentation)
- [ ] Methodology is documented (how was this data collected/processed?)
- [ ] Dossiers are organized for discoverability (not a raw dump)

---

## Production Flow: How the Pyramid Gets Built

The pyramid is not a formatting template applied after research is complete. It is the natural output of a recursive research methodology:

```
Mission Brief
  ↓
1. Mission Interpolation
   Reformulate the brief into explicit research questions,
   scope boundaries, and a register of known unknowns.
  ↓
2. Systematic Gathering
   Use the available retrieval tools: fetch known URLs directly, search for discovery, and synthesize only from the material you can cite.
   Capture sources, extract evidence, organize into preliminary
   drafts of Layer 3 dossiers.
  ↓
3. Draft Layer 2 Analysis
   From dossiers, compose per-dimension analysis files.
   Add SOURCES sections linking back to dossiers.
  ↓
4. Draft Layer 1 Summary
   From analysis files, compose the summary. Add SOURCES
   section linking to analysis files.
  ↓
5. Gap Evaluation
   For each gap, evaluate: is it in-scope? Would filling it
   change any conclusion in layers above? Does it add depth
   or just bulk?
  ↓
6. Recurse (if needed)
   Go deeper on open gaps. Update layers as new evidence arrives.
  ↓
Publish
```

The key insight: **depth is a function of mission complexity, not a fixed template.** A simple technology explanation brief may produce only a summary and two analysis files. A competitive landscape analysis may require all three layers plus multiple files per layer.

---

## Cross-layer Navigation

### Explicit SOURCES Format

Every file carries this at the bottom:

```
SOURCES (LAYER {N} NAVIGATION)
path/to/file.md
 -> One-line description answering "what will I find if I go deeper?"

path/to/another-file.md
 -> Another dimension or supporting evidence
```

### Consumption Flow

```
L1 Summary
  ↓ reads SOURCES, loads only the analysis it needs
L2 market-position.md
  ↓ reads SOURCES, loads dossiers to verify claims
L3 competitor-profiles.md
```

### Production Flow

```
L3 Source capture
  ↓ compose analysis
L2 Analysis Collection
  ↓ synthesize findings
L1 Summary
```

### Auditing

1. **Forward audit**: Start from dossiers → every source is valid → analysis is sound → summary is trustworthy.
2. **Backward audit**: Start from summary → for each claim, find the L2 file → for each claim in L2, find the L3 dossier → for each source, verify.

# Provenance Artifacts — Auxiliary Failover Metadata

The Artifact Pyramid is not just for research outputs. Any system that generates a chain of attempts with variable depth of detail is a natural fit. The auxiliary failover system in an Agent Skills-compatible agent (PR #32411, issue #36797) is a worked example.

## The Problem

The `_FailoverChatCompletions` wrapper walks a fallback chain when the primary provider fails. It logs each attempt but returns a bare result — downstream consumers have no way to tell *which* provider actually served the request without correlating timestamps against log files.

## The Pyramid Shape

The same call chain — a single walk across N providers with one winner — has natural progressive disclosure structure:

### Layer 1 — `served_by` (always present)

```
{
  "served_by": "openrouter/gpt-4o-mini",
  "fallback_chain_used": true,
  "fallback_count": 1
}
```

Every auxiliary response carries this. A single field, no overhead for the common case. UI surfaces and downstream agents that just need to know "who handled this" stop here.

### Layer 2 — Provenance trail (machine-readable, opt-in)

```
{
  "provenance": [
    {
      "provider": "opencode-go",
      "model": "deepseek-v4-flash",
      "status": "failed",
      "failure": "502 Bad Gateway",
      "latency_ms": 3200
    },
    {
      "provider": "openrouter",
      "model": "gpt-4o-mini",
      "status": "success",
      "latency_ms": 890
    }
  ]
}
```

Ordered array of every attempt. Added as an optional response field — consumers opt in by requesting it. No token cost for consumers that don't need it. A downstream RAG agent that needs to reason about extraction quality reads this layer.

### Layer 3 — Full diagnostic trace (debug endpoint only)

```
{
  "trace_id": "aux_comp_172832",
  "attempts": [
    {
      "provider": "opencode-go",
      "model": "deepseek-v4-flash",
      "request": {"messages_len": 4, "estimated_tokens": 24000},
      "error": "APIStatusError: 502 Bad Gateway\nbody: upstream unavailable",
      "started_at": "2026-06-01T12:00:01.234Z",
      "duration_ms": 3200
    },
    {
      "provider": "openrouter",
      "model": "gpt-4o-mini",
      "response": {"choices": [], "usage": {"total_tokens": 485}},
      "started_at": "2026-06-01T12:00:04.500Z",
      "duration_ms": 890
    }
  ]
}
```

Full error payloads, request context, timing watermarks. Gated behind a debug endpoint or explicit request — not attached to the response by default. An operator debugging a production incident reads this layer. Nobody else pays the token cost.

## Why This Fits

The `_FailoverChatCompletions` wrapper already has all three layers of information at runtime:

- **L1**: it knows the winning provider — trivially available
- **L2**: it tracks each attempt's status and failure reason in the `last_error` variable
- **L3**: it catches the full exception objects before deciding to continue or raise

The pyramid is the structured capture of that same walk, organized by consumer depth. The wrapper currently logs it and drops it — this frames what to capture instead.

## Related

- Issue #36797 — feature request for provenance-as-artifact-pyramid
- PR #32411 — adds `_FailoverAuxiliaryClient` with the wrapper that generates this data
- Reference: `references/artifact-pyramid-framework.md` for the general framework

# Quality Gates: Verification at Each Layer

Each layer of the Artifact Pyramid has a quality gate — checks that material must pass before it can flow between layers. Because the pyramid is consumed **top-down** (L1 → L2 → L3) but produced **bottom-up** (L3 → L2 → L1), gates verify both directions: upward production quality and downward navigability.

## How to Use Quality Gates

Run the appropriate gate checklist when:
- Moving from dossiers (L3) into an analysis file (L2) — use Gate C
- Moving from analysis (L2) into the summary (L1) — use Gate B
- Auditing a completed artifact pyramid for publication — use Gate A

Each gate has three categories:
- **Critical** — must pass or the artifact cannot proceed
- **Standard** — should pass; flag exceptions explicitly
- **Advisory** — aspirational; note gaps for future improvement

---

## Gate A: Summary Integrity (L1 — Summary)

Run this before publishing or delivering the summary.

### Critical

- [ ] **Claim traceability**: Every claim in the summary links to a specific Layer 2 analysis file via the SOURCES section. No orphan claims.
- [ ] **Self-containment**: The summary makes sense without reading lower layers. No "as discussed in the analysis" without a link.
- [ ] **Implications stated**: The summary doesn't stop at findings — it states what they mean for the intended audience.

### Standard

- [ ] **Audience fit**: Language, depth, and format match the intended audience (PM agent vs. executive vs. technical lead).
- [ ] **Scope fidelity**: The summary addresses the mission brief's research questions. Out-of-scope findings are flagged as such.
- [ ] **Internal consistency**: The summary doesn't contradict itself.

### Advisory

- [ ] **Version tracking**: The summary has a version identifier.
- [ ] **Downstream navigation**: Every SOURCES reference answers "what will I find if I go deeper?"

---

## Gate B: Analysis Integrity (L2 → L1)

Run this before an analysis file feeds into the summary.

### Critical

- [ ] **Full traceability**: Every claim in the analysis file traces to specific Layer 3 sources via the SOURCES section.
- [ ] **Self-containment**: Each analysis file makes sense on its own — a consumer reading only this file should understand the dimension.
- [ ] **Interpretive value**: The analysis adds value beyond raw data. If it's just reformatted dossiers, it's not analysis.

### Standard

- [ ] **Conflict transparency**: If sources disagree, the analysis surfaces the conflict rather than picking one side.
- [ ] **Narrative structure**: The analysis has a clear thesis, evidence section, and conclusion.
- [ ] **Quantitative precision**: Numbers are preserved and contextualized (not "most showed improvement" but "7 of 12 improved by ≥5%").

### Advisory

- [ ] **Source diversity**: The analysis draws from multiple sources, not a single dossier.
- [ ] **Cross-dimension links**: Analysis files reference each other when findings overlap.

---

## Gate C: Dossier Completeness (L3 → L2)

Run this before dossiers feed into analysis files.

### Critical

- [ ] **Source attribution**: Every dossier entry has source metadata (URL, timestamp, title, author, capture date).
- [ ] **Faithful extraction**: Extracts are faithful to the original. No misrepresentation or cherry-picking.
- [ ] **Methodology documentation**: How was the data collected, processed, or transcribed?

### Standard

- [ ] **Organizational discoverability**: Dossiers are organized for consumption (not a raw dump). Named, attributed, searchable.
- [ ] **Coverage completeness**: The dossier layer covers all sources referenced by analysis files above it.
- [ ] **Contradictory evidence**: Counter-evidence to expected findings is included, not suppressed.

### Advisory

- [ ] **Metadata completeness**: Publication date, author, access timestamp for every source.
- [ ] **Semantic tagging**: Descriptive tags for cross-discovery.

---

## Gate Failure Recovery

| Gate | Failure | Recovery |
|------|---------|----------|
| A | Orphan claim (no L2 link) | Either add the supporting analysis file or remove the claim |
| A | Audience mismatch | Rewrite or re-format for the target audience |
| B | No interpretive value | Re-examine: reformatted dossiers aren't analysis. Find the thesis. |
| B | Orphan claim (no L3 link) | Either add the supporting dossier or remove the claim |
| C | Missing source attribution | Re-extract from source, add metadata |
| C | Cherry-picked evidence | Go back to sources, include contradicting evidence |

# Source index

- **Source repository:** https://github.com/magnus919/hermes-profiles
- **Inspected commit:** `867a555`
- **Imported source directory:** `artifact-pyramids`
- **Porting boundary:** Retained portable methodology, templates, scripts, and references. Removed or generalized Hermes profile, task-orchestration, memory, and rigid response-handoff assumptions.

# Worked Example: How an Artifact Pyramid Works (Synthetic Data)

This example traces a complete research cycle from mission brief to published summary using fabricated data. All names, numbers, and claims are synthetic.

**Mission Brief:** "Assess whether the company should enter the AI-powered code review market. We need competitive positioning, market size, and technical feasibility analysis."

---

## Layer 3: Detailed Dossiers

These are the source materials — raw data, transcripts, and notes that underpin the analysis.

### Dossier: Competitor Profiles

```
source-id: dossier-001
title: Competitor Market Scan — AI Code Review Tools
captured: 2026-05-20
sources:
  - https://example.edu/code-review-market-2026
  - https://example.com/competitor/comparison
```

**Extracted evidence:**

- Graphite: 200K+ developers, Series B, $50M ARR. Core feature: AI review summaries with inline fix suggestions. Enterprise tier $50/seat/mo.
- CodeRabbit: 150K developers, Series A, $20M ARR. Differentiator: fully automated review queue prioritization.
- Reviewpad: 30K developers, bootstrapped. Focus: custom review rules engine.
- Market projected at $800M by 2028, 28% CAGR (MarketResearch 2026).
- Developer survey (N=1500): 67% want AI code review but only 23% are satisfied with current tools (DevSurvey 2026).

### Dossier: Interview Transcript

```
source-id: dossier-002
title: Engineering Leader Interview — AI Review Needs
captured: 2026-05-21
methodology: 30-min semi-structured interview with VP Engineering, mid-size SaaS
```

**Key excerpts:**

- "We tried Graphite. The review summaries are great but the false positive rate is still too high — about 40% of suggestions are stylistic preferences, not bugs."
- "If someone could build a tool that learns our team's style conventions and only flags things that matter, that's worth $30/seat to us."
- "We're currently spending about 4 hours per week per senior dev on code review. Cutting that in half would save us roughly $200K/year."

### Dossier: Technical Notes

```
source-id: dossier-003
title: Architecture Feasibility — AI Review Engine
captured: 2026-05-22
```

- Fine-tuned LLM approach feasible with labeled training data (existing open datasets: CodeSearchNet, CodeReview-25K)
- Latency requirement: <5s per review to avoid blocking developer workflows
- Key technical risk: false positive rate management (survey data suggests >30% FP = tool abandonment)
- Integration surfaces: GitHub App API, GitLab Webhook API, Bitbucket Cloud API

---

## Layer 2: Analysis Collection

Each file covers one dimension of the research. Each links down to its dossiers.

### Analysis: Market Positioning

```
# Analysis: Competitive Landscape — AI Code Review Market

The code review market is projected at $800M by 2028 with strong
growth (28% CAGR). Current leaders have captured ~50% of known
developer users but satisfaction remains low at 23%.

## Key Findings

Graphite and CodeRabbit dominate mindshare but neither has solved
the false positive problem — the top unmet need (67% want it, 40%
FP rate on existing tools). This creates a wedge: a new entrant
with FP rates under 10% could capture the dissatisfied majority.

## Implications
- Entry is viable if we meet the FP rate threshold
- Pricing at $25-35/seat undercuts Graphite's enterprise tier
- Integration with existing workflows (VS Code, GitHub) is table stakes

SOURCES (LAYER 3 NAVIGATION)
research/dossiers/competitor-profiles.md
 -> Market data, ARR figures, developer counts

research/dossiers/interview-transcripts.md
 -> VP Engineering quote on false positive pain point
```

### Analysis: Technical Feasibility

```
# Analysis: Technical Feasibility — AI Review Engine

Building the core review engine is feasible with existing open
datasets and fine-tuning pipelines. The critical risk is not
model quality but false positive rate management.

## Key Findings
- Fine-tuning on CodeReview-25K + team-specific style data works
- Latency target of <5s is achievable with quantization (Q4)
- FP rate management requires reinforcement learning from
  developer corrections — a flywheel, not a one-shot fix

## Key Risk
If FP rate exceeds 30%, tool abandonment follows. The RL-based
approach requires an initial training period where FP rate is
above target. Mitigation: beta cohort with explicit expectations.

SOURCES (LAYER 3 NAVIGATION)
research/dossiers/technical-notes.md
 -> Architecture evaluation, latency targets, dataset references
```

### Analysis: Financial Projection

```
# Analysis: Financial Model — Market Entry

Conservative estimate: Year 1 revenue of $2.5M at 10,000 seats,
growing to $15M by Year 3 at 50,000 seats. Gross margin 75%
(cloud inference costs).

Breakeven at Month 14 with $4M seed round. Unit economics
improve with scale as inference costs drop faster than
customer acquisition costs.

Sensitivity: FP rate is the largest variable. If FP rate
remains above 30%, churn hits 40% and breakeven extends to
Month 24. If FP rate drops below 10%, organic growth from
referrals reduces CAC by 35%.

SOURCES (LAYER 3 NAVIGATION)
research/dossiers/competitor-profiles.md
 -> Market size and growth projections

research/dossiers/interview-transcripts.md
 -> Willingness-to-pay data from interviews
```

---

## Layer 1: Summary

```
# Summary: AI Code Review Market Entry — Assessment

Research question: Should we enter the AI-powered code review market?

## Key Findings
- The market is growing at 28% CAGR to $800M by 2028
- 67% of developers want better AI review tools; only 23% are
  satisfied with current options
- The critical unmet need is false positive rate management —
  existing tools hover at 40% FP, driving dissatisfaction
- Technical feasibility is established; the risk is operational
  (FP rate flywheel takes time to spin up)
- Conservative financial model shows Year 3 revenue of $15M
  with 75% gross margin

## Implications
- Entry is viable and timed well (market before saturation,
  incumbents haven't solved the core problem)
- The bet is on FP rate management as a competitive moat
- Recommended: proceed to pilot with $4M seed round, 5 beta
  customers, 12-month FP rate improvement target

SOURCES (LAYER 2 NAVIGATION)
research/analysis/market-positioning.md
 -> Competitive landscape and market sizing

research/analysis/technical-feasibility.md
 -> Architecture evaluation and risk assessment

research/analysis/financial-projection.md
 -> Revenue model, breakeven, sensitivity analysis
```

---

## Complete Traceability Chain

```
L1 Summary: "67% of developers want better AI review tools"
  ↓ reads SOURCES
L2 market-positioning.md
  ↓ reads SOURCES
L3 competitor-profiles.md
  → source: DevSurvey 2026 (N=1500, cited)
```

Every claim in the summary traces to a specific analysis file, which traces to a specific dossier, which traces to a specific source. A validator can verify any claim by following the chain downward.

#!/usr/bin/env python3
"""extract-atoms.py — Extract atomic statements from source text.

Reads a text file and attempts to split it into atomic claims.
Outputs YAML-formatted atoms suitable for use in an Artifact Pyramid
Layer 1 registry.

Usage:
    extract-atoms.py <source-file> [--source-id <id>] [--domain <domain>] [--output <file>]

The script uses heuristic sentence splitting and claim detection.
It is NOT a replacement for careful human extraction — it's a
first-pass tool that produces candidate atoms for review.

Examples:
    extract-atoms.py paper.txt --source-id source-001 --domain scaling-laws
    extract-atoms.py transcript.log --domain meetings --output atoms.yaml
"""

import sys
import re
import argparse
from pathlib import Path
from datetime import date
import textwrap


def split_sentences(text: str) -> list[str]:
    """Split text into sentences using a simple heuristic."""
    # Normalize whitespace
    text = re.sub(r'\s+', ' ', text).strip()

    # Split on sentence boundaries
    # Use a simpler approach without variable-length lookbehind
    sentences = []
    current = []
    for word in text.split(' '):
        current.append(word)
        if word.endswith('.') or word.endswith('!') or word.endswith('?'):
            # Check if it's an abbreviation
            abbrev = {'Mr', 'Ms', 'Mrs', 'Dr', 'Prof', 'Sr', 'Jr', 'St',
                      'vs', 'etc', 'dept', 'est', 'approx', 'Fig', 'Table',
                      'Section', 'Eq', 'Inc', 'Ltd', 'Co'}
            base = word.rstrip('.!?')
            if base not in abbrev and not base.isdigit():
                sentences.append(' '.join(current))
                current = []

    if current:
        sentences.append(' '.join(current))

    return [s.strip() for s in sentences if s.strip()]


def is_claim_candidate(sentence: str) -> bool:
    """Heuristic to determine if a sentence is likely a factual claim.

    Returns True for sentences that state facts, findings, or results
    rather than being meta-commentary or formatting.
    """
    # Too short
    if len(sentence.split()) < 4:
        return False

    lower = sentence.lower()

    # Skip formatting/header-like sentences
    skip_patterns = [
        r'^abstract$', r'^introduction$', r'^methodology$',
        r'^results$', r'^discussion$', r'^conclusion$',
        r'^references$', r'^appendix', r'^figure\s+\d+',
        r'^table\s+\d+', r'^section\s+\d+',
        r'^\d+\.\s+(introduction|method|result|discussion)',
    ]
    for pat in skip_patterns:
        if re.match(pat, lower):
            return False

    # Skip questions
    if sentence.strip().endswith('?') or '?' in sentence:
        return False

    # Skip very long sentences (likely a list or paragraph artifact)
    if len(sentence.split()) > 60:
        return False

    # Indicators of a claim
    claim_indicators = [
        r'\b(is|are|was|were|has|have|had|shows|showed|demonstrates'
        r'|demonstrated|finds|found|achieves|achieved|reaches|reached'
        r'|increases|increased|decreases|decreased|improves|improved'
        r'|reduces|reduced|enables|enabled|requires|required'
        r'|suggests|suggested|indicates|indicated|reports|reported'
        r'|results\s+(in|show|indicate)|we\s+(find|show|demonstrate|report)'
        r'|our\s+(results|findings|analysis|experiments)'
        r'|according\s+to|as\s+(shown|measured|observed)'
        r'|represents|consists|comprises|contains'
        r'|scales|depends|varies|correlates'
        r'|marginally|significantly|substantially'
        r'|achieves?\s+\d+[%x]|reaches?\s+\d+'
        r'|\d+[%x]\s+(improvement|reduction|increase|decrease)'
        r')\b',
    ]
    for pat in claim_indicators:
        if re.search(pat, lower):
            return True

    return False


def extract_domain(sentence: str, fallback: str) -> str:
    """Heuristic domain detection based on keywords."""
    lower = sentence.lower()
    domain_keywords = {
        'natural-language-processing': ['language model', 'nlp', 'transformer', 'token', 'attention',
                                          'bert', 'gpt', 'llm', 'text generation'],
        'computer-vision': ['image', 'vision', 'convolution', 'detection', 'segmentation',
                            'cnn', 'visual', 'pixel'],
        'reinforcement-learning': ['reinforcement', 'reward', 'agent', 'policy', 'value function',
                                   'rl', 'temporal difference', 'q-learning'],
        'data-quality': ['data quality', 'curation', 'filtering', 'clean data', 'noise',
                         'curriculum', 'sample quality'],
        'scaling-laws': ['scale', 'scaling', 'parameter count', 'compute', 'power-law',
                         'large model', 'bigger model', 'capability threshold'],
        'hardware': ['gpu', 'memory', 'throughput', 'latency', 'flops', 'energy',
                     'efficiency', 'inference speed'],
        'privacy': ['privacy', 'differential', 'federated', 'encryption', 'anonymization',
                    'data protection'],
    }

    scores = {}
    for domain, keywords in domain_keywords.items():
        score = sum(1 for kw in keywords if kw in lower)
        if score > 0:
            scores[domain] = score

    if scores:
        return max(scores, key=scores.get)
    return fallback


def extract_atoms(text: str, source_id: str = "unknown", domain: str = "general") -> list[dict]:
    """Extract candidate atoms from text."""
    sentences = split_sentences(text)
    atoms = []

    for i, sentence in enumerate(sentences):
        if is_claim_candidate(sentence):
            atom = {
                "content": sentence,
                "type": "claim",
                "domain": extract_domain(sentence, domain),
                "source": source_id,
                "source_location": f"Extracted sentence {i+1}",
                "tags": [],
                "contradictions": [],
                "extracted_at": str(date.today()),
            }
            atoms.append(atom)

    return atoms


def atoms_to_yaml(atoms: list[dict]) -> str:
    """Format atoms as YAML."""
    lines = ["# Atoms extracted by extract-atoms.py",
             f"# Extracted: {date.today()}",
             f"# Count: {len(atoms)}",
             ""]

    for i, atom in enumerate(atoms):
        atom_id = f"atom-{i+1:03d}"
        lines.append(f"{atom_id}:")
        lines.append(f'  content: "{atom["content"]}"')
        lines.append(f'  type: {atom["type"]}')
        lines.append(f'  domain: {atom["domain"]}')
        lines.append(f"  source: {atom['source']}")
        lines.append(f"  source_location: \"{atom['source_location']}\"")
        lines.append(f"  tags: [{', '.join(atom['tags'])}]")
        lines.append(f"  contradictions: [{', '.join(atom['contradictions'])}]")
        lines.append(f"  extracted_at: {atom['extracted_at']}")
        lines.append("")

    return "\n".join(lines)


def main():
    parser = argparse.ArgumentParser(
        description="Extract atomic statements from source text",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog=textwrap.dedent("""\
            Examples:
              extract-atoms.py paper.txt --source-id paper-001 --domain scaling-laws
              extract-atoms.py transcript.log --output atoms.yaml
        """))
    parser.add_argument("source_file", help="Path to the source text file")
    parser.add_argument("--source-id", default="unknown",
                        help="Identifier for the source (used in atom.source field)")
    parser.add_argument("--domain", default="general",
                        help="Default domain classification for atoms")
    parser.add_argument("--output", "-o", default=None,
                        help="Output file (default: stdout)")

    args = parser.parse_args()

    source_path = Path(args.source_file)
    if not source_path.exists():
        print(f"Error: file not found: {args.source_file}", file=sys.stderr)
        sys.exit(1)

    text = source_path.read_text(encoding="utf-8", errors="replace")
    source_id = args.source_id if args.source_id != "unknown" else source_path.stem
    atoms = extract_atoms(text, source_id=source_id, domain=args.domain)

    yaml_output = atoms_to_yaml(atoms)

    if args.output:
        output_path = Path(args.output)
        output_path.write_text(yaml_output, encoding="utf-8")
        print(f"Wrote {len(atoms)} atoms to {args.output}")
    else:
        print(yaml_output)


if __name__ == "__main__":
    main()

#!/usr/bin/env bash
# pyramid-status.sh — Audit a research project directory for Artifact Pyramid coverage
#
# Usage:
#   pyramid-status.sh [--layer 1|2|3] [--json] <project-directory>
#
# Scans the directory for:
#   Layer 1: files matching 01-*, *summary*, *dossier*
#   Layer 2: files matching 02-*, *analysis*, *market*, *competitive*, *technical*
#   Layer 3: files matching 03-*, *dossier*, *source*, *transcript*, *raw*, *data*
#
# Checks naming convention, content structure, and cross-references.

set -euo pipefail

# --- Colors ---
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'

# --- Defaults ---
LAYER_FILTER=""
JSON_MODE=false

# --- Parse arguments ---
while [[ $# -gt 0 ]]; do
    case "$1" in
        --layer)
            LAYER_FILTER="$2"
            shift 2
            ;;
        --json)
            JSON_MODE=true
            shift
            ;;
        --help|-h)
            echo "Usage: pyramid-status.sh [--layer 1|2|3] [--json] <project-directory>"
            exit 0
            ;;
        *)
            PROJECT_DIR="$1"
            shift
            ;;
    esac
done

if [[ -z "${PROJECT_DIR:-}" ]]; then
    echo "Error: project directory required"
    echo "Usage: pyramid-status.sh [--layer 1|2|3] [--json] <project-directory>"
    exit 1
fi

if [[ ! -d "$PROJECT_DIR" ]]; then
    echo "Error: '$PROJECT_DIR' is not a directory"
    exit 1
fi

# --- Search patterns by layer ---
L1_PATTERNS=("01-*" "1-*" "*summary*" "*dossier*")
L2_PATTERNS=("02-*" "2-*" "*analysis*" "*market*" "*competitive*" "*technical*" "*feasibility*")
L3_PATTERNS=("03-*" "3-*" "*dossier*" "*source*" "*transcript*" "*raw*" "*data*")

count_layer() {
    local search_dir="$1"
    shift
    local files=()
    for pattern in "$@"; do
        while IFS= read -r -d '' f; do
            files+=("$f")
        done < <(find "$search_dir" -maxdepth 3 -type f -name "$pattern" -not -path '*/\.*' -print0 2>/dev/null || true)
    done
    if [[ ${#files[@]} -eq 0 ]]; then
        echo "0"
        return 0
    fi
    declare -a unique=()
    local f rp u
    for f in "${files[@]}"; do
        rp=$(realpath "$f" 2>/dev/null || echo "$f")
        local seen=false
        for u in "${unique[@]-}"; do
            if [[ "$u" == "$rp" ]]; then
                seen=true
                break
            fi
        done
        if ! $seen; then
            unique+=("$rp")
        fi
    done
    echo "${#unique[@]}"
    return 0
}

check_md_quality() {
    local dir="$1"
    local issues=0
    local findings=""
    while IFS= read -r -d '' f; do
        local has_yaml=false
        if head -1 "$f" 2>/dev/null | grep -q '^---$'; then
            has_yaml=true
        fi
        if ! $has_yaml; then
            local lines
            lines=$(wc -l < "$f" 2>/dev/null || echo 0)
            if [[ "$lines" -gt 5 ]]; then
                findings="${findings}  ${f##*/}: no frontmatter (${lines} lines, ${f})\n"
                issues=$((issues + 1))
            fi
        fi
    done < <(find "$dir" -maxdepth 3 -name '*.md' -not -name 'README.md' -print0 2>/dev/null || true)
    printf '%s\n' "$findings"
    echo "$issues"
}

check_cross_references() {
    local dir="$1"
    local missing_refs=0
    local findings=""
    while IFS= read -r -d '' f; do
        while IFS= read -r match; do
            local ref
            ref=$(echo "$match" | sed -n 's/.*\[\[\(.*\)\]\].*/\1/p' 2>/dev/null || true)
            if [[ -z "$ref" ]]; then
                ref=$(echo "$match" | sed -n 's/.*\[\(atom-[0-9]*\)\].*/\1/p' 2>/dev/null || true)
            fi
            if [[ -n "$ref" ]]; then
                local found_ref
                found_ref=$(grep -rl "$ref" "$dir" --include='*.md' 2>/dev/null | head -1 || true)
                if [[ -z "$found_ref" ]]; then
                    findings="${findings}  ${f##*/} references '${ref}' which is not found\n"
                    missing_refs=$((missing_refs + 1))
                fi
            fi
        done < <(grep -oP '\[\[.*?\]\]|\[atom-\d+\]' "$f" 2>/dev/null || true)
    done < <(find "$dir" -maxdepth 3 -name '*.md' -print0 2>/dev/null || true)
    printf '%s\n' "$findings"
    echo "$missing_refs"
}

# --- Gather stats ---
L1_COUNT=$(count_layer "$PROJECT_DIR" "${L1_PATTERNS[@]}")
L2_COUNT=$(count_layer "$PROJECT_DIR" "${L2_PATTERNS[@]}")
L3_COUNT=$(count_layer "$PROJECT_DIR" "${L3_PATTERNS[@]}")

L1_RESULTS=$(check_md_quality "$PROJECT_DIR" 2>/dev/null || true)
L1_FINDINGS="${L1_RESULTS%$'\n'*}"
L1_ISSUES="${L1_RESULTS##*$'\n'}"

REF_RESULTS=$(check_cross_references "$PROJECT_DIR" 2>/dev/null || true)
REF_FINDINGS="${REF_RESULTS%$'\n'*}"
MISSING_REFS="${REF_RESULTS##*$'\n'}"

# Fallback if parsing fails
if ! [[ "$L1_ISSUES" =~ ^[0-9]+$ ]]; then L1_ISSUES=0; fi
if ! [[ "$MISSING_REFS" =~ ^[0-9]+$ ]]; then MISSING_REFS=0; fi

if [[ "$L1_COUNT" -gt 0 ]]; then L1_PASS=true; else L1_PASS=false; fi
if [[ "$L2_COUNT" -gt 0 ]]; then L2_PASS=true; else L2_PASS=false; fi
if [[ "$L3_COUNT" -gt 0 ]]; then L3_PASS=true; else L3_PASS=false; fi

# --- JSON output ---
if $JSON_MODE; then
    cat <<EOF
{
  "project": "$(basename "$PROJECT_DIR")",
  "layers": {
    "1": { "name": "Summary", "count": $L1_COUNT, "present": $L1_PASS, "quality_issues": $L1_ISSUES },
    "2": { "name": "Analysis Collection", "count": $L2_COUNT, "present": $L2_PASS, "quality_issues": 0 },
    "3": { "name": "Detailed Dossiers", "count": $L3_COUNT, "present": $L3_PASS, "quality_issues": 0 }
  },
  "missing_references": $MISSING_REFS,
  "pyramid_health": $(if $L1_PASS && $L2_PASS && $L3_PASS; then echo '"complete"'; else echo '"incomplete"'; fi)
}
EOF
    exit 0
fi

# --- Human output ---
echo ""
echo "╔══════════════════════════════════════════════════╗"
echo "║  Artifact Pyramid Status Report                  ║"
echo "╚══════════════════════════════════════════════════╝"
echo ""
echo "  Project: $(basename "$PROJECT_DIR")"
echo ""
echo "  Layer 1 — Summary"
if $L1_PASS; then
    echo -e "    ${GREEN}✓ Present${NC} (${L1_COUNT} file(s))"
else
    echo -e "    ${RED}✗ Missing${NC}"
fi
if [[ "$L1_ISSUES" -gt 0 ]]; then
    echo -e "    ${YELLOW}⚠  ${L1_ISSUES} quality issue(s)${NC}"
fi
echo ""
echo "  Layer 2 — Analysis Collection"
if $L2_PASS; then
  echo -e "    ${GREEN}✓ Present${NC} (${L2_COUNT} file(s))"
else
  echo -e "    ${RED}✗ Missing${NC}"
fi
echo ""
echo "  Layer 3 — Detailed Dossiers"
if $L3_PASS; then
    echo -e "    ${GREEN}✓ Present${NC} (${L3_COUNT} file(s))"
else
    echo -e "    ${RED}✗ Missing${NC}"
fi
echo ""
echo "  Cross-Reference Health"
if [[ "$MISSING_REFS" -gt 0 ]]; then
    echo -e "    ${YELLOW}⚠  ${MISSING_REFS} broken reference(s)${NC}"
else
    echo -e "    ${GREEN}✓ All references resolve${NC}"
fi
echo ""
echo "  Overall Pyramid Health"
if $L1_PASS && $L2_PASS && $L3_PASS; then
    echo -e "    ${GREEN}✓ Complete — all three layers present${NC}"
else
    echo -e "    ${YELLOW}◐ Partial — missing: $(if ! $L1_PASS; then echo -n 'L1 '; fi)$(if ! $L2_PASS; then echo -n 'L2 '; fi)$(if ! $L3_PASS; then echo -n 'L3 '; fi)${NC}"
fi
echo ""