interface:
  display_name: "CodeQL Analysis"
  short_description: "Analyze code for security vulnerabilities with CodeQL"
  icon_small: "assets/trail-of-bits-mark.svg"
  icon_large: "assets/trail-of-bits-mark.svg"
  brand_color: "#D83A34"

# Build Fixes

Fixes to apply when a CodeQL database build method fails. Try these in order, then retry the current build method. **Log each fix attempt.**

Every block sources `build_log.sh` for itself. A helper defined in an earlier Bash call is
gone by the next one, and `log_step` then exits 127, which the ladder reads as a failed method.

## 1. Clean existing state

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "Applying fix: clean existing state"
rm -rf "$DB_NAME"
log_result "Removed $DB_NAME"
```

## 2. Clean build cache

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "Applying fix: clean build cache"
CLEANED=""
make clean 2>/dev/null && CLEANED="$CLEANED make"
rm -rf build CMakeCache.txt CMakeFiles 2>/dev/null && CLEANED="$CLEANED cmake-artifacts"
./gradlew clean 2>/dev/null && CLEANED="$CLEANED gradle"
mvn clean 2>/dev/null && CLEANED="$CLEANED maven"
cargo clean 2>/dev/null && CLEANED="$CLEANED cargo"
log_result "Cleaned: $CLEANED"
```

## 3. Install missing dependencies

> **Note:** The commands below install the *target project's* dependencies so CodeQL can trace the build. Use whatever package manager the target project expects (`pip`, `npm`, `go mod`, etc.) — these are not the skill's own tooling preferences.

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "Applying fix: install dependencies"
FAILED_INSTALLS=()

# Python — use target project's package manager (pip/uv/poetry)
# allow-legacy-python: installs the analysed project's own deps; forcing uv could change its build.
if [ -f requirements.txt ]; then
  run_logged pip install -r requirements.txt || FAILED_INSTALLS+=("pip install -r requirements.txt")
fi
if [ -f setup.py ] || [ -f pyproject.toml ]; then
  run_logged pip install -e . || FAILED_INSTALLS+=("pip install -e .")
fi

# Node
if [ -f package.json ]; then
  run_logged npm install || FAILED_INSTALLS+=("npm install")
fi

# Go
if [ -f go.mod ]; then
  run_logged go mod download || FAILED_INSTALLS+=("go mod download")
fi

# Java
if [ -f build.gradle ] || [ -f build.gradle.kts ]; then
  run_logged ./gradlew dependencies --refresh-dependencies || FAILED_INSTALLS+=("gradlew dependencies")
fi
if [ -f pom.xml ]; then
  run_logged mvn dependency:resolve || FAILED_INSTALLS+=("mvn dependency:resolve")
fi

# Rust
if [ -f Cargo.toml ]; then
  run_logged cargo fetch || FAILED_INSTALLS+=("cargo fetch")
fi

if [ ${#FAILED_INSTALLS[@]} -gt 0 ]; then
  log_result "Dependency installation FAILED: ${FAILED_INSTALLS[*]}"
  echo "WARNING: ${#FAILED_INSTALLS[@]} dependency step(s) failed — a retry will likely" \
       "fail the same way. Report which, rather than retrying blind." >&2
else
  log_result "Dependencies installed"
fi
```

## 4. Handle private registries

If dependencies require authentication, ask user:
```
AskUserQuestion: "Build requires private registry access. Options:"
  1. "I'll configure auth and retry"
  2. "Skip these dependencies"
  3. "Show me what's needed"
```

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

# Log authentication setup if performed
log_step "Private registry authentication configured"
log_result "Registry: <REGISTRY_URL>, Method: <AUTH_METHOD>"
```

**After fixes:** Retry current build method. If still fails, move to next method.

# Diagnostic Query Templates

Language-specific QL queries for enumerating sources and sinks recognized by CodeQL. Used during the data extensions creation process.

## Source Enumeration Query

All languages use the class `RemoteFlowSource`. The import differs per language.

### Import Reference

| Language | Imports | Class |
|----------|---------|-------|
| Python | `import python` + `import semmle.python.dataflow.new.RemoteFlowSources` | `RemoteFlowSource` |
| JavaScript | `import javascript` | `RemoteFlowSource` |
| Java | `import java` + `import semmle.code.java.dataflow.FlowSources` | `RemoteFlowSource` |
| Go | `import go` | `RemoteFlowSource` |
| C/C++ | `import cpp` + `import semmle.code.cpp.security.FlowSources` | `RemoteFlowSource` |
| C# | `import csharp` + `import semmle.code.csharp.security.dataflow.flowsources.Remote` | `RemoteFlowSource` |
| Ruby | `import ruby` + `import codeql.ruby.dataflow.RemoteFlowSources` | `RemoteFlowSource` |

### Template (Python — swap imports per table above)

```ql
/**
 * @name List recognized dataflow sources
 * @description Enumerates all locations CodeQL recognizes as dataflow sources
 * @kind problem
 * @id custom/list-sources
 */
import python
import semmle.python.dataflow.new.RemoteFlowSources

from RemoteFlowSource src
select src,
  src.getSourceType()
    + " | " + src.getLocation().getFile().getRelativePath()
    + ":" + src.getLocation().getStartLine().toString()
```

**Note:** `getSourceType()` is available on Python, Java, and C#. For Go, JavaScript, Ruby, and C++ replace the select with:
```ql
select src,
  src.getLocation().getFile().getRelativePath()
    + ":" + src.getLocation().getStartLine().toString()
```

---

## Sink Enumeration Queries

The Concepts API differs significantly across languages. Use the correct template.

**Java, C/C++, and C# need their own pack.** Those three have no unified Concepts module,
so their queries import the language library directly and will not compile without a
`qlpack.yml` beside them. Create it once, substituting the language, then run
`codeql pack install` in the diagnostics directory before executing any query:

```yaml
# $DIAG_DIR/qlpack.yml — <lang> is java, cpp, or csharp
name: custom/diagnostics
version: 0.0.1
dependencies:
  codeql/<lang>-all: "*"
```

### Concept Class Reference

| Concept | Python | JavaScript | Go | Ruby |
|---------|--------|------------|-----|------|
| SQL | `SqlExecution.getSql()` | `DatabaseAccess.getAQueryArgument()` | `SQL::QueryString` (is-a Node) | `SqlExecution.getSql()` |
| Command exec | `SystemCommandExecution.getCommand()` | `SystemCommandExecution.getACommandArgument()` | `SystemCommandExecution.getCommandName()` | `SystemCommandExecution.getAnArgument()` |
| File access | `FileSystemAccess.getAPathArgument()` | `FileSystemAccess.getAPathArgument()` | `FileSystemAccess.getAPathArgument()` | `FileSystemAccess.getAPathArgument()` |
| HTTP client | `Http::Client::Request.getAUrlPart()` | — | — | — |
| Decoding | `Decoding.getAnInput()` | — | — | — |
| XML parsing | — | — | — | `XmlParserCall.getAnInput()` |

### Python

```ql
/**
 * @name List recognized dataflow sinks
 * @description Enumerates security-relevant sinks CodeQL recognizes
 * @kind problem
 * @id custom/list-sinks
 */
import python
import semmle.python.Concepts

from DataFlow::Node sink, string kind
where
  exists(SqlExecution e | sink = e.getSql() and kind = "sql-execution")
  or
  exists(SystemCommandExecution e |
    sink = e.getCommand() and kind = "command-execution"
  )
  or
  exists(FileSystemAccess e |
    sink = e.getAPathArgument() and kind = "file-access"
  )
  or
  exists(Http::Client::Request r |
    sink = r.getAUrlPart() and kind = "http-request"
  )
  or
  exists(Decoding d | sink = d.getAnInput() and kind = "decoding")
  or
  exists(CodeExecution e | sink = e.getCode() and kind = "code-execution")
select sink,
  kind
    + " | " + sink.getLocation().getFile().getRelativePath()
    + ":" + sink.getLocation().getStartLine().toString()
```

### JavaScript / TypeScript

```ql
/**
 * @name List recognized dataflow sinks
 * @description Enumerates security-relevant sinks CodeQL recognizes
 * @kind problem
 * @id custom/list-sinks-js
 */
import javascript

from DataFlow::Node sink, string kind
where
  exists(DatabaseAccess e |
    sink = e.getAQueryArgument() and kind = "database-access"
  )
  or
  exists(SystemCommandExecution e |
    sink = e.getACommandArgument() and kind = "command-execution"
  )
  or
  exists(FileSystemAccess e |
    sink = e.getAPathArgument() and kind = "file-access"
  )
select sink,
  kind
    + " | " + sink.getLocation().getFile().getRelativePath()
    + ":" + sink.getLocation().getStartLine().toString()
```

### Go

```ql
/**
 * @name List recognized dataflow sinks
 * @description Enumerates security-relevant sinks CodeQL recognizes
 * @kind problem
 * @id custom/list-sinks-go
 */
import go
import semmle.go.frameworks.SQL

from DataFlow::Node sink, string kind
where
  sink instanceof SQL::QueryString and kind = "sql-query"
  or
  exists(SystemCommandExecution e |
    sink = e.getCommandName() and kind = "command-execution"
  )
  or
  exists(FileSystemAccess e |
    sink = e.getAPathArgument() and kind = "file-access"
  )
select sink,
  kind
    + " | " + sink.getLocation().getFile().getRelativePath()
    + ":" + sink.getLocation().getStartLine().toString()
```

### Ruby

```ql
/**
 * @name List recognized dataflow sinks
 * @description Enumerates security-relevant sinks CodeQL recognizes
 * @kind problem
 * @id custom/list-sinks-ruby
 */
import ruby
import codeql.ruby.Concepts

from DataFlow::Node sink, string kind
where
  exists(SqlExecution e | sink = e.getSql() and kind = "sql-execution")
  or
  exists(SystemCommandExecution e |
    sink = e.getAnArgument() and kind = "command-execution"
  )
  or
  exists(FileSystemAccess e |
    sink = e.getAPathArgument() and kind = "file-access"
  )
  or
  exists(CodeExecution e | sink = e.getCode() and kind = "code-execution")
select sink,
  kind
    + " | " + sink.getLocation().getFile().getRelativePath()
    + ":" + sink.getLocation().getStartLine().toString()
```

### Java

Per-vulnerability sink classes rather than Concepts. Needs the `codeql/java-all` pack above.

```ql
/**
 * @name List recognized dataflow sinks
 * @description Enumerates security-relevant sinks CodeQL recognizes
 * @kind problem
 * @id custom/list-sinks
 */
import java
import semmle.code.java.dataflow.DataFlow
import semmle.code.java.security.QueryInjection
import semmle.code.java.security.CommandLineQuery
import semmle.code.java.security.TaintedPathQuery
import semmle.code.java.security.XSS
import semmle.code.java.security.RequestForgery
import semmle.code.java.security.Xxe

from DataFlow::Node sink, string kind
where
  sink instanceof QueryInjectionSink and kind = "sql-injection"
  or
  sink instanceof CommandInjectionSink and kind = "command-injection"
  or
  sink instanceof TaintedPathSink and kind = "path-injection"
  or
  sink instanceof XssSink and kind = "xss"
  or
  sink instanceof RequestForgerySink and kind = "ssrf"
  or
  sink instanceof XxeSink and kind = "xxe"
select sink,
  kind
    + " | " + sink.getLocation().getFile().getRelativePath()
    + ":" + sink.getLocation().getStartLine().toString()
```

### C / C++

Matches on called function names rather than sink classes. Needs the `codeql/cpp-all` pack above.

```ql
/**
 * @name List recognized dataflow sinks
 * @description Enumerates security-relevant sinks CodeQL recognizes
 * @kind problem
 * @id custom/list-sinks-cpp
 */
import cpp
import semmle.code.cpp.dataflow.DataFlow
import semmle.code.cpp.security.CommandExecution
import semmle.code.cpp.security.FileAccess
import semmle.code.cpp.security.BufferWrite

from DataFlow::Node sink, string kind
where
  exists(FunctionCall call |
    sink.asExpr() = call.getAnArgument() and
    call.getTarget().hasGlobalOrStdName("system") and
    kind = "command-injection"
  )
  or
  exists(FunctionCall call |
    sink.asExpr() = call.getAnArgument() and
    call.getTarget().hasGlobalOrStdName(["fopen", "open", "freopen"]) and
    kind = "file-access"
  )
  or
  exists(FunctionCall call |
    sink.asExpr() = call.getAnArgument() and
    call.getTarget().hasGlobalOrStdName(["sprintf", "strcpy", "strcat", "gets"]) and
    kind = "buffer-write"
  )
  or
  exists(FunctionCall call |
    sink.asExpr() = call.getAnArgument() and
    call.getTarget().hasGlobalOrStdName(["execl", "execle", "execlp", "execv", "execvp", "execvpe", "popen"]) and
    kind = "command-execution"
  )
select sink,
  kind
    + " | " + sink.getLocation().getFile().getRelativePath()
    + ":" + sink.getLocation().getStartLine().toString()
```

### C\#

Per-vulnerability sink classes. Needs the `codeql/csharp-all` pack above.

```ql
/**
 * @name List recognized dataflow sinks
 * @description Enumerates security-relevant sinks CodeQL recognizes
 * @kind problem
 * @id custom/list-sinks-csharp
 */
import csharp
import semmle.code.csharp.dataflow.DataFlow
import semmle.code.csharp.security.dataflow.SqlInjectionQuery
import semmle.code.csharp.security.dataflow.CommandInjectionQuery
import semmle.code.csharp.security.dataflow.TaintedPathQuery
import semmle.code.csharp.security.dataflow.XSSQuery

from DataFlow::Node sink, string kind
where
  sink instanceof SqlInjection::Sink and kind = "sql-injection"
  or
  sink instanceof CommandInjection::Sink and kind = "command-injection"
  or
  sink instanceof TaintedPath::Sink and kind = "path-injection"
  or
  sink instanceof XSS::Sink and kind = "xss"
select sink,
  kind
    + " | " + sink.getLocation().getFile().getRelativePath()
    + ":" + sink.getLocation().getStartLine().toString()
```

# Data Extension YAML Format

YAML format for CodeQL data extension files. Used by the create-data-extensions workflow to model project-specific sources, sinks, and flow summaries.

## Structure

All extension files follow this structure:

```yaml
extensions:
  - addsTo:
      pack: codeql/<language>-all  # Target library pack
      extensible: <model-type>      # sourceModel, sinkModel, summaryModel, neutralModel
    data:
      - [<columns>]
```

## Source Models

Columns: `[package, type, subtypes, name, signature, ext, output, kind, provenance]`

| Column | Description | Example |
|--------|-------------|---------|
| package | Module/package path | `myapp.auth` |
| type | Class or module name | `AuthManager` |
| subtypes | Include subclasses | `True` (Java: capitalized) / `true` (Python/JS/Go) |
| name | Method name | `get_token` |
| signature | Method signature (optional) | `""` (Python/JS), `"(String,int)"` (Java) |
| ext | Extension (optional) | `""` |
| output | What is tainted | `ReturnValue`, `Parameter[0]` (Java) / `Argument[0]` (Python/JS/Go) |
| kind | Source category | `remote`, `local`, `file`, `environment`, `database` |
| provenance | How model was created | `manual` |

**Java-specific format differences:**
- **subtypes**: Use `True` / `False` (capitalized, Python-style), not `true` / `false`
- **output for parameters**: Use `Parameter[N]` (not `Argument[N]`) to mark method parameters as sources
- **signature**: Required for disambiguation — use Java type syntax: `"(String)"`, `"(String,int)"`
- **Parameter ranges**: Use `Parameter[0..2]` to mark multiple consecutive parameters

Example (Python):

```yaml
# $OUTPUT_DIR/extensions/sources.yml
extensions:
  - addsTo:
      pack: codeql/python-all
      extensible: sourceModel
    data:
      - ["myapp.http", "Request", true, "get_param", "", "", "ReturnValue", "remote", "manual"]
      - ["myapp.http", "Request", true, "get_header", "", "", "ReturnValue", "remote", "manual"]
```

Example (Java — note `True`, `Parameter[N]`, and signature):

```yaml
# $OUTPUT_DIR/extensions/sources.yml
extensions:
  - addsTo:
      pack: codeql/java-all
      extensible: sourceModel
    data:
      - ["com.myapp.controller", "ApiController", True, "search", "(String)", "", "Parameter[0]", "remote", "manual"]
      - ["com.myapp.service", "FileService", True, "upload", "(String,String)", "", "Parameter[0..1]", "remote", "manual"]
```

## Sink Models

Columns: `[package, type, subtypes, name, signature, ext, input, kind, provenance]`

Note: column 7 is `input` (which argument receives tainted data), not `output`.

| Kind | Vulnerability |
|------|---------------|
| `sql-injection` | SQL injection |
| `command-injection` | Command injection |
| `path-injection` | Path traversal |
| `xss` | Cross-site scripting |
| `code-injection` | Code injection |
| `ssrf` | Server-side request forgery |
| `unsafe-deserialization` | Insecure deserialization |

Example (Python):

```yaml
# $OUTPUT_DIR/extensions/sinks.yml
extensions:
  - addsTo:
      pack: codeql/python-all
      extensible: sinkModel
    data:
      - ["myapp.db", "Connection", true, "raw_query", "", "", "Argument[0]", "sql-injection", "manual"]
      - ["myapp.shell", "Runner", false, "execute", "", "", "Argument[0]", "command-injection", "manual"]
```

Java sinks take the same `True`/signature conventions as the source example above, with
`Argument[N]` for the input rather than `Parameter[N]`.

## Summary Models

Columns: `[package, type, subtypes, name, signature, ext, input, output, kind, provenance]`

| Kind | Description |
|------|-------------|
| `taint` | Data flows through, still tainted |
| `value` | Data flows through, exact value preserved |

Example:

```yaml
# $OUTPUT_DIR/extensions/summaries.yml
extensions:
  # Pass-through: taint propagates
  - addsTo:
      pack: codeql/python-all
      extensible: summaryModel
    data:
      - ["myapp.cache", "Cache", true, "get", "", "", "Argument[0]", "ReturnValue", "taint", "manual"]
      - ["myapp.utils", "JSON", false, "parse", "", "", "Argument[0]", "ReturnValue", "taint", "manual"]

```

## Neutral Models

Columns: `[package, type, name, signature, kind, provenance]` (6 columns, NOT the 10-column `summaryModel` format).

Example:

```yaml
  - addsTo:
      pack: codeql/python-all
      extensible: neutralModel
    data:
      - ["myapp.security", "Sanitizer", "escape_html", "", "summary", "manual"]
```

**`neutralModel` vs no model:** If a function has no model at all, CodeQL may still infer flow through it. Use `neutralModel` to explicitly block taint propagation through known-safe functions.

## Language-Specific Notes

**Python:** Use dotted module paths for `package` (e.g., `myapp.db`).

**JavaScript:** `package` is often `""` for project-local code. Use the import path for npm packages.

**Go:** Use full import paths (e.g., `myapp/internal/db`). `type` is often `""` for package-level functions.

**Java:** Use fully qualified package names (e.g., `com.myapp.db`).

**C/C++:** Use `""` for package, put the namespace in `type`.

## Deploying Extensions

**Known limitation:** `--additional-packs` and `--model-packs` flags do not work with pre-compiled query packs (bundled CodeQL distributions that cache `java-all` inside `.codeql/libraries/`). Extensions placed in a standalone model pack directory will be resolved by `codeql resolve qlpacks` but silently ignored during `codeql database analyze`.

**Workaround — copy extensions into the library pack's `ext/` directory:**

> **Warning:** Files copied into the `ext/` directory live inside CodeQL's managed pack cache. They will be **lost** when packs are updated via `codeql pack download` or version upgrades. After any pack update, re-run this deployment step to restore the extensions.

```bash
# Find the java-all ext directory used by the query pack
JAVA_ALL_EXT=$(find "$(codeql resolve qlpacks 2>/dev/null | grep 'java-queries' | awk '{print $NF}' | tr -d '()')" \
  -path '*/.codeql/libraries/codeql/java-all/*/ext' -type d 2>/dev/null | head -1)

if [ -n "$JAVA_ALL_EXT" ]; then
  PROJECT_NAME=$(basename "$(pwd)")
  cp "$OUTPUT_DIR/extensions/sources.yml" "$JAVA_ALL_EXT/${PROJECT_NAME}.sources.model.yml"
  [ -f "$OUTPUT_DIR/extensions/sinks.yml" ] && cp "$OUTPUT_DIR/extensions/sinks.yml" "$JAVA_ALL_EXT/${PROJECT_NAME}.sinks.model.yml"
  [ -f "$OUTPUT_DIR/extensions/summaries.yml" ] && cp "$OUTPUT_DIR/extensions/summaries.yml" "$JAVA_ALL_EXT/${PROJECT_NAME}.summaries.model.yml"

  # Verify deployment — confirm files landed correctly
  DEPLOYED=$(ls "$JAVA_ALL_EXT/${PROJECT_NAME}".*.model.yml 2>/dev/null | wc -l)
  if [ "$DEPLOYED" -gt 0 ]; then
    echo "Extensions deployed to $JAVA_ALL_EXT ($DEPLOYED files):"
    ls -la "$JAVA_ALL_EXT/${PROJECT_NAME}".*.model.yml
  else
    echo "ERROR: Files were copied but verification failed. Check path: $JAVA_ALL_EXT"
  fi
else
  echo "WARNING: Could not find java-all ext directory. Extensions may not load."
  echo "Attempted path lookup from: codeql resolve qlpacks | grep java-queries"
  echo "Run 'codeql resolve qlpacks' manually to debug."
fi
```

**For Python/JS/Go:** The same limitation may apply. Locate the `<lang>-all` pack's `ext/` directory and copy extensions there.

**Alternative (if query packs are NOT pre-compiled):** Use `--additional-packs=./codeql-extensions` with a proper model pack `qlpack.yml`:

```yaml
# $OUTPUT_DIR/extensions/qlpack.yml
name: custom/<project>-extensions
version: 0.0.1
library: true
extensionTargets:
  codeql/<lang>-all: "*"
dataExtensions:
  - sources.yml
  - sinks.yml
  - summaries.yml
```

# Important-Only Query Suite

In important-only mode, generate a custom `.qls` query suite file at runtime. This applies the same precision/severity filtering to **all** packs (official + third-party).

## Why a Custom Suite

The built-in `security-extended` suite only applies to the official `codeql/<lang>-queries` pack. Third-party packs (Trail of Bits, Community Packs) run unfiltered when passed directly to `codeql database analyze`. A custom `.qls` suite loads queries from all packs and applies a single set of `include`/`exclude` filters uniformly.

## Metadata Criteria

Two-phase filtering: the **suite** selects candidate queries (broad), then a **post-analysis jq filter** removes low-severity medium-precision results from the SARIF output.

### Phase 1: Suite selection (which queries run)

Queries are included if they match **any** of these blocks (OR logic across blocks, AND logic within):

| Block | kind | precision | problem.severity | tags |
|-------|------|-----------|-----------------|------|
| 1 | `problem`, `path-problem` | `high`, `very-high` | *(any)* | must contain `security` |
| 2 | `problem`, `path-problem` | `medium` | *(any)* | must contain `security` |

### Phase 2: Post-analysis filter (which results are reported)

After `codeql database analyze` completes, filter the SARIF output:

| precision | security-severity | Action |
|-----------|-------------------|--------|
| high / very-high | *(any)* | **Keep** |
| medium | >= 6.0 | **Keep** |
| medium | < 6.0 or missing | **Drop** |

This ensures medium-precision queries with meaningful security impact (e.g., `cpp/path-injection` at 7.5, `cpp/world-writable-file-creation` at 7.8) are included, while noisy low-severity medium-precision findings are filtered out.

Excluded: deprecated queries, model editor/generator queries. Experimental queries are **included**.

**Key difference from `security-extended`:** The `security-extended` suite includes medium-precision queries at any severity. Important-only mode adds a security-severity threshold to reduce noise from medium-precision queries that flag low-impact issues.

## Suite Template

What the generation script writes, shown so the filter semantics are readable. Do not
hand-write this file — the script adds the installed third-party packs and verifies the
result, and `test_generation_scripts.py` fails if this block and the script disagree:

```yaml
- description: Important-only — security vulnerabilities, medium-high confidence
# Official queries
- queries: .
  from: codeql/<CODEQL_LANG>-queries
# Third-party packs (include only if installed, one entry per pack)
# - queries: .
#   from: trailofbits/<CODEQL_LANG>-queries
# - queries: .
#   from: GitHubSecurityLab/CodeQL-Community-Packs-<CODEQL_LANG>
# Filtering: security only, high/very-high precision (any severity),
# medium precision (any severity — low-severity filtered post-analysis by security-severity score).
# Experimental queries included.
- include:
    kind:
      - problem
      - path-problem
    precision:
      - high
      - very-high
    tags contain:
      - security
- include:
    kind:
      - problem
      - path-problem
    precision:
      - medium
    tags contain:
      - security
- exclude:
    deprecated: //
- exclude:
    tags contain:
      - modeleditor
      - modelgenerator
```

> **Post-analysis step required:** After running the analysis, apply the post-analysis jq filter (defined in the run-analysis workflow Step 5) to remove medium-precision results with `security-severity` < 6.0.

## Generation Script

The suite is generated from the installed packs, not copied from the template above:

```bash
# `set -e` and the trailing script call are both load-bearing: an assignment placed last
# would overwrite the script's exit status, and the run would proceed to analysis with no
# suite — or with a stale one from an earlier run.
set -euo pipefail
SUITE_FILE="$OUTPUT_DIR/raw/important-only.qls"
CODEQL_LANG="${CODEQL_LANG:-}" OUTPUT_DIR="${OUTPUT_DIR:-}" \
  INSTALLED_THIRD_PARTY_PACKS="${INSTALLED_THIRD_PARTY_PACKS:-}" \
  {baseDir}/scripts/generate_suite.sh important-only
```

`codeql database analyze` accepts a suite that resolves to zero queries. It writes an empty
SARIF and the run reports "0 findings". The script runs `verify_query_suite.py`, which
exits non-zero on zero queries, on a CodeQL error, and on malformed output, so the run
stops before analysis rather than after it.

## How Filtering Works on Third-Party Queries

CodeQL query suite filters match on query metadata (`@precision`, `@problem.severity`, `@tags`). Third-party queries that:

- **Have proper metadata**: Filtered normally (kept if they match the include criteria)
- **Lack `@precision`**: Excluded by `include` blocks (they require precision to match). This is correct — if a query doesn't declare its precision, we cannot assess its confidence.
- **Lack `@tags security`**: Excluded. Non-security queries are not relevant to important-only mode.

This is a stricter-than-necessary filter for third-party packs, but it ensures only well-annotated security queries run in important-only mode. The post-analysis jq filter then further narrows medium-precision results to those with `security-severity` >= 6.0.

# Language-Specific Guidance

Commands below assume `$DB_NAME` and `$CODEQL_LANG` from the build-database workflow. They
are written as `"$DB_NAME"` rather than a literal path on purpose: everything a run
produces belongs under `$OUTPUT_DIR`, and a database in the working directory is invisible
to the discovery step that looks there next time.

## No Build Required

### Python

```bash
codeql database create "$DB_NAME" --language=python --source-root=.
```

**Framework Support:**
- Django, Flask, FastAPI: Built-in models
- Tornado, Pyramid: Partial support
- Custom frameworks: May need data extensions

**Common Issues:**
| Issue | Fix |
|-------|-----|
| Missing Django models | Ensure `settings.py` is at expected location |
| Virtual env included | Use `paths-ignore` in config |
| Type stubs missing | Install `types-*` packages before extraction |

### JavaScript/TypeScript

```bash
codeql database create "$DB_NAME" --language=javascript --source-root=.
```

**Framework Support:**
- React, Vue, Angular: Built-in models
- Express, Koa, Fastify: HTTP source/sink models
- Next.js, Nuxt: Partial SSR support

**Common Issues:**
| Issue | Fix |
|-------|-----|
| node_modules bloat | Already excluded by default |
| TypeScript not parsed | Ensure `tsconfig.json` is valid |
| Monorepo issues | Use `--source-root` for specific package |

### Ruby

```bash
codeql database create "$DB_NAME" --language=ruby --source-root=.
```

**Framework Support:**
- Rails: Full support (controllers, models, views)
- Sinatra: Built-in support
- Hanami: Partial support

**Common Issues:**
| Issue | Fix |
|-------|-----|
| Bundler issues | Run `bundle install` first |
| Rails engines | May need multiple database passes |

## Build Required

### Go

Compiled, and **rejects `--build-mode=none`** — autobuild or a manual command only. The
command below looks build-free but runs autobuild, which invokes the Go toolchain; it
fails if Go is absent or the module does not build, and there is no no-build fallback.

```bash
codeql database create "$DB_NAME" --language=go --source-root=.
```

**Framework Support:**
- net/http, Gin, Echo, Chi: Built-in models
- gRPC: Partial support
- Custom routers: May need data extensions

**Common Issues:**
| Issue | Fix |
|-------|-----|
| Missing dependencies | Run `go mod download` first |
| Vendor directory | CodeQL handles automatically |
| CGO code | Requires `--command='go build'` with CGO enabled |
| Build fails | Fix it. `--build-mode=none` is rejected for Go, so there is no fallback |

### C/C++

```bash
# Make
codeql database create "$DB_NAME" --language=cpp --command='make -j8'

# CMake
codeql database create "$DB_NAME" --language=cpp \
  --source-root=/path/to/src \
  --command='cmake --build build'

# Ninja
codeql database create "$DB_NAME" --language=cpp \
  --command='ninja -C build'
```

**Build System Tips:**
| Build System | Command |
|--------------|---------|
| Make | `make clean && make -j"$(nproc 2>/dev/null || sysctl -n hw.ncpu)"` |
| CMake | `cmake -B build && cmake --build build` |
| Meson | `meson setup build && ninja -C build` |
| Bazel | `bazel build //...` |

**Common Issues:**
| Issue | Fix |
|-------|-----|
| Partial extraction | Ensure `make clean` before CodeQL build |
| Header-only libraries | Use `--extractor-option cpp_trap_headers=true` |
| Cross-compilation | Set `CODEQL_EXTRACTOR_CPP_TARGET_ARCH` |

### Java/Kotlin

```bash
# Gradle
codeql database create "$DB_NAME" --language=java --command='./gradlew build -x test'

# Maven
codeql database create "$DB_NAME" --language=java --command='mvn compile -DskipTests'
```

**Framework Support:**
- Spring Boot: Full support
- Jakarta EE: Built-in models
- Android: Requires Android SDK

**Common Issues:**
| Issue | Fix |
|-------|-----|
| Missing dependencies | Run `./gradlew dependencies` first |
| Kotlin mixed projects | Use `--language=java` (covers both) |
| Annotation processors | Ensure they run during CodeQL build |

### Rust

```bash
codeql database create "$DB_NAME" --language=rust --command='cargo build'
```

**Common Issues:**
| Issue | Fix |
|-------|-----|
| Proc macros | May require special handling |
| Workspace projects | Use `--source-root` for specific crate |
| Build script failures | Ensure native dependencies are available |

### C#

```bash
# .NET Core
codeql database create "$DB_NAME" --language=csharp --command='dotnet build'

# MSBuild
codeql database create "$DB_NAME" --language=csharp --command='msbuild /t:rebuild'
```

**Framework Support:**
- ASP.NET Core: Full support
- Entity Framework: Database query models
- Blazor: Partial support

**Common Issues:**
| Issue | Fix |
|-------|-----|
| NuGet restore | Run `dotnet restore` first |
| Multiple solutions | Specify solution file in command |

### Swift

```bash
# Xcode project
codeql database create "$DB_NAME" --language=swift \
  --command='xcodebuild -project MyApp.xcodeproj -scheme MyApp build'

# Swift Package Manager
codeql database create "$DB_NAME" --language=swift --command='swift build'
```

**Requirements:**
- macOS only
- Xcode Command Line Tools

**Common Issues:**
| Issue | Fix |
|-------|-----|
| Code signing | Add `CODE_SIGN_IDENTITY=- CODE_SIGNING_REQUIRED=NO` |
| Simulator target | Add `-sdk iphonesimulator` |

## Extractor Options

Set via environment variables: `CODEQL_EXTRACTOR_<LANG>_OPTION_<NAME>=<VALUE>`

### C/C++ Options

| Option | Description |
|--------|-------------|
| `trap_headers=true` | Include header file analysis |
| `target_arch=x86_64` | Target architecture |

### Java Options

| Option | Description |
|--------|-------------|
| `jdk_version=17` | JDK version for analysis |

### Python Options

| Option | Description |
|--------|-------------|
| `python_executable=/path/to/python` | Specific Python interpreter |

# macOS arm64e Workaround

Methods for building CodeQL databases on macOS Apple Silicon when the `arm64e`/`arm64` architecture mismatch causes SIGKILL (exit code 137) during build tracing.

**Use when `IS_MACOS_ARM64E=true`** (detected in build-database workflow Step 2a). These replace Methods 1 and 2 on affected systems.

The strategy is to use Homebrew-installed tools (plain `arm64`, not `arm64e`) so `libtrace.dylib` can be injected successfully. Try sub-methods in order:

> Each sub-method sources `build_log.sh` itself. That defines the helpers, which are gone by
> the next Bash call, and sets `pipefail`. These sub-methods branch on exit code 137, so
> without `pipefail` they read tee's status instead of the build's.

## Sub-method 2m-a: Homebrew clang/gcc with multi-step tracing

Trace only the compiler invocations individually, avoiding system tools (`/usr/bin/ar`, `/bin/mkdir`) that would be killed. This requires a multi-step build: init → trace each compiler call → finalize.

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "METHOD 2m-a: macOS arm64 — Homebrew compiler with multi-step tracing"

# 1. Find Homebrew C/C++ compiler (arm64, not arm64e)
BREW_CC=""
# Prefer Homebrew clang
if [ -x "/opt/homebrew/opt/llvm/bin/clang" ]; then
  BREW_CC="/opt/homebrew/opt/llvm/bin/clang"
# Try Homebrew GCC (e.g. gcc-14, gcc-13)
elif command -v gcc-14 >/dev/null 2>&1; then
  BREW_CC="$(command -v gcc-14)"
elif command -v gcc-13 >/dev/null 2>&1; then
  BREW_CC="$(command -v gcc-13)"
fi

if [ -z "$BREW_CC" ]; then
  log_result "No Homebrew C/C++ compiler found — skipping 2m-a"
  # Fall through to 2m-b
else
  # Verify it's arm64 (not arm64e)
  BREW_CC_ARCH=$(lipo -archs "$BREW_CC" 2>/dev/null)
  if [[ "$BREW_CC_ARCH" == *"arm64e"* ]]; then
    log_result "Homebrew compiler is arm64e — skipping 2m-a"
  else
    log_step "Using Homebrew compiler: $BREW_CC (arch: $BREW_CC_ARCH)"

    # 2. Run the build normally (without tracing) to create build dirs and artifacts
    #    Use Homebrew make (gmake) if available, otherwise system make outside tracer
    if command -v gmake >/dev/null 2>&1; then
      MAKE_CMD="gmake"
    else
      MAKE_CMD="make"
    fi
    $MAKE_CMD clean 2>/dev/null || true
    run_logged "$MAKE_CMD" CC="$BREW_CC"

    # 3. Extract compiler commands from the Makefile / build system
    #    Use make's dry-run mode to get the exact compiler invocations
    $MAKE_CMD clean 2>/dev/null || true
    COMPILE_CMDS=$($MAKE_CMD CC="$BREW_CC" --dry-run 2>/dev/null \
      | grep -E "^\s*$BREW_CC\b.*\s-c\s" \
      | sed 's/^[[:space:]]*//')

    if [ -z "$COMPILE_CMDS" ]; then
      log_result "Could not extract compile commands from dry-run — skipping 2m-a"
    else
      # 4. Init database
      run_logged codeql database init "$DB_NAME" \
        --language=cpp --source-root=. --overwrite

      # 5. Ensure build directories exist (outside tracer — avoids arm64e mkdir)
      $MAKE_CMD clean 2>/dev/null || true
      #    Parse -o flags to find output dirs, or just create common dirs
      echo "$COMPILE_CMDS" | sed -n 's/.*-o[[:space:]]\{1,\}\([^[:space:]]\{1,\}\).*/\1/p' | xargs -I{} dirname {} \
        | sort -u | xargs mkdir -p 2>/dev/null || true

      # 6. Trace each compiler invocation individually
      TRACE_OK=true
      while IFS= read -r cmd; do
        [ -z "$cmd" ] && continue
        # shellcheck disable=SC2086 # $cmd is a compiler invocation that must word-split
        if ! run_logged codeql database trace-command "$DB_NAME" -- $cmd; then
          log_result "FAILED on: $cmd"
          TRACE_OK=false
          break
        fi
      done <<< "$COMPILE_CMDS"

      if $TRACE_OK; then
        # 7. Finalize
        run_logged codeql database finalize "$DB_NAME"
        if codeql resolve database -- "$DB_NAME" >/dev/null 2>&1; then
          log_result "SUCCESS (macOS arm64 multi-step)"
          # Done — skip to Step 4
        else
          log_result "FAILED (finalize failed)"
        fi
      fi
    fi
  fi
fi
```

## Sub-method 2m-b: Rosetta x86_64 emulation

Force the entire CodeQL pipeline to run under Rosetta, which uses the `x86_64` slice of both `libtrace.dylib` and system tools — no `arm64e` mismatch.

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "METHOD 2m-b: macOS arm64 — Rosetta x86_64 emulation"

# Check if Rosetta is available
if ! arch -x86_64 /usr/bin/true 2>/dev/null; then
  log_result "Rosetta not available — skipping 2m-b"
else
  BUILD_CMD="<BUILD_CMD>"  # e.g. "make clean && make -j4"

  run_logged arch -x86_64 codeql database create "$DB_NAME" \
    --language="$CODEQL_LANG" --source-root=. \
    --command="$BUILD_CMD" --overwrite

  if codeql resolve database -- "$DB_NAME" >/dev/null 2>&1; then
    log_result "SUCCESS (Rosetta x86_64)"
  else
    log_result "FAILED (Rosetta)"
  fi
fi
```

## Sub-method 2m-c: System compiler (direct attempt)

As a verification step, try the standard autobuild with the system compiler. This will likely fail with exit code 137 on affected systems, but confirms the arm64e issue is the cause.

> **This sub-method is optional.** Skip it if arm64e incompatibility was already confirmed in Step 2a.

> This sub-method reads `EXIT_CODE` directly, so sourcing the log helpers is not optional
> here: without `pipefail` the value is always 0 and 137 is never seen.

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "METHOD 2m-c: System compiler (expected to fail on arm64e)"

run_logged codeql database create "$DB_NAME" \
  --language="$CODEQL_LANG" --source-root=. --overwrite

EXIT_CODE=$?
if [ $EXIT_CODE -eq 137 ] || [ $EXIT_CODE -eq 134 ]; then
  log_result "FAILED: exit code $EXIT_CODE confirms arm64e/libtrace incompatibility"
elif codeql resolve database -- "$DB_NAME" >/dev/null 2>&1; then
  log_result "SUCCESS (unexpected — system compiler worked)"
else
  log_result "FAILED (exit code: $EXIT_CODE)"
fi
```

## Sub-method 2m-d: Ask user

If all macOS workarounds fail, present options:

```
AskUserQuestion:
  header: "macOS Build"
  question: "Build tracing failed due to macOS arm64e incompatibility. How to proceed?"
  multiSelect: false
  options:
    - label: "Use build-mode=none (Recommended)"
      description: "Source-level analysis only. Misses some interprocedural data flow but catches most C/C++ vulnerabilities (format strings, buffer overflows, unsafe functions)."
    - label: "Install arm64 tools and retry"
      description: "Run: brew install llvm make — then retry with Homebrew toolchain"
    - label: "Install Rosetta and retry"
      description: "Run: softwareupdate --install-rosetta — then retry under x86_64 emulation"
    - label: "Abort"
      description: "Stop database creation"
```

**If "Use build-mode=none":** Proceed to Method 4.

**If "Install arm64 tools and retry":**
```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "Installing Homebrew arm64 toolchain"
run_logged brew install llvm make || {
  log_result "FAILED: brew install did not complete — do not retry 2m-a, it will fail identically"
  exit 1
}
# Retry Sub-method 2m-a
```

**If "Install Rosetta and retry":**
```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "Installing Rosetta"
run_logged softwareupdate --install-rosetta --agree-to-license || {
  log_result "FAILED: Rosetta did not install — do not retry 2m-b, it will fail identically"
  exit 1
}
# Retry Sub-method 2m-b
```

# Performance Tuning

## Memory, Threads, and Timeouts

All three are set on `codeql database analyze "$DB_NAME"`. `CODEQL_RAM` is an environment
variable in MB; the other two are flags.

| Setting | Value | When |
|---------|-------|------|
| `CODEQL_RAM` | `4000`–`8000` | Small codebase, under 100K LoC |
| | `8000`–`16000` | Medium, 100K–1M LoC |
| | `32000`–`64000` | Large, 1M+ LoC |
| `--threads` | `0` | Use every core — the default choice |
| | `8` | Shared machine; leave headroom for other work |
| `--timeout` | `600000` | Milliseconds. Ten minutes catches a runaway query without killing legitimate deep taint tracking |

## Evaluator Diagnostics

When analysis is slow, `--evaluator-log` identifies which queries consume the time:

```bash
codeql database analyze "$DB_NAME" \
  --evaluator-log="$OUTPUT_DIR/evaluator.log" \
  --format=sarif-latest \
  --output="$OUTPUT_DIR/raw/results.sarif" \
  -- "$SUITE_FILE"

codeql generate log-summary "$OUTPUT_DIR/evaluator.log" --format=text
```

The summary shows per-query timing and tuple counts. Queries producing millions of tuples are likely the bottleneck.

## Disk Space

| Phase | Typical Size | Notes |
|-------|-------------|-------|
| Database creation | 2-10x source size | Compiled languages are larger due to build tracing |
| Analysis cache | 1-5 GB | Stored in database directory |
| SARIF output | 1-50 MB | Depends on finding count |

Check available space before starting:

```bash
df -h .
du -sh "$OUTPUT_DIR"/*.db 2>/dev/null
```

## Caching Behavior

CodeQL caches query evaluation results inside the database directory. Subsequent runs of the same queries skip re-evaluation.

| Scenario | Cache Effect |
|----------|-------------|
| Re-run same packs | Fast — uses cached results |
| Add new query pack | Only new queries evaluate |
| `codeql database cleanup` | Clears cache — forces full re-evaluation |
| `--rerun` flag | Ignores cache for this run |

**When to clear cache:**
- After deploying new data extensions (cache may hold stale results)
- When investigating unexpected zero-finding results
- Before benchmark comparisons (ensures consistent timing)

```bash
# Clear evaluation cache
codeql database cleanup "$DB_NAME"
```

## Troubleshooting Performance

| Symptom | Likely Cause | Solution |
|---------|--------------|----------|
| OOM during analysis | Not enough RAM | Increase `CODEQL_RAM` |
| Slow database creation | Complex build | Use `--threads`, simplify build |
| Slow query execution | Large codebase | Reduce query scope, add RAM |
| Database too large | Too many files | Use exclusion config (`codeql-config.yml` with `paths-ignore`) |
| Single query hangs | Runaway evaluation | Use `--timeout` and check `--evaluator-log` |
| Repeated runs still slow | Cache not used | Check you're using same database path |

# Quality Assessment

How to assess and improve CodeQL database quality after a successful build.

## Collect Metrics

One call produces every metric and enforces the thresholds. Nothing else recomputes any of
them: a second hand-written pipeline drifts from the script and logs a contradicting number,
which is how this file came to report 202 project files where the script said 2.

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "Assessing database quality"

# Capture the status into a variable. Inside `if ! cmd; then`, `$?` is the *negated*
# status and always reads 0, so the log would record every failure as a success.
QUALITY_JSON=$(uv run {baseDir}/scripts/check_db_quality.py "$DB_NAME" --format=json)
QUALITY_STATUS=$?
if [ "$QUALITY_STATUS" -ne 0 ]; then
  log_result "Quality gate failed (exit $QUALITY_STATUS) — see Enforce the Thresholds below"
  exit "$QUALITY_STATUS"
fi

printf '%s' "$QUALITY_JSON" | jq -r '
  "Baseline LoC: \(.baseline_loc)",
  "Project source files: \(.project_files)",
  "Total archive files: \(.archive_files) (system headers included for compiled languages)",
  "Extractor errors: \(.extractor_errors) (\(.error_ratio)%)",
  "Finalised: \(.finalised)"' | tee -a "$LOG_FILE"

# Not derived from the database, so the script cannot report it.
DIAG_TEXT=$(codeql database export-diagnostics --format=text -- "$DB_NAME" 2>/dev/null || true)
if [ -n "$DIAG_TEXT" ]; then
  echo "Diagnostics: $DIAG_TEXT"
fi
```

## Compare Against Expected Source

The one number the checker cannot produce: how many source files the working tree holds.
Compare it against `.project_files`, never against `.archive_files` — for C/C++ the archive
runs 10-20x larger because it carries the SDK headers (690 against 473 on a real mbedtls
database).

```bash
# `fd` is not in the Quick Start preflight, and a missing fd exits non-zero into `wc -l`,
# which prints 0 — so this would read as "extraction met expectations" on a machine that
# simply lacks the tool.
if command -v fd >/dev/null 2>&1; then
  EXPECTED=$(fd -t f -e c -e cpp -e h -e hpp -e java -e kt -e py -e js -e ts \
    --exclude 'codeql_*.db' --exclude node_modules --exclude vendor --exclude .git . \
    | wc -l)
else
  EXPECTED=$(find . -type f \( -name '*.c' -o -name '*.cpp' -o -name '*.h' -o -name '*.hpp' \
    -o -name '*.java' -o -name '*.kt' -o -name '*.py' -o -name '*.js' -o -name '*.ts' \) \
    -not -path './.git/*' -not -path './node_modules/*' -not -path './vendor/*' \
    -not -path './codeql_*.db/*' | wc -l)
fi
echo "Expected source files: $EXPECTED"
```

## Enforce the Thresholds

The numbers above are only useful if something compares them to a threshold, which the
call in Collect Metrics already does. Its two failure exits are not equivalent:

| Exit | Meaning | What to do |
|------|---------|------------|
| `1` | Nothing to analyse — no baseline LoC, or no project files in the source archive | Stop. Fix the build; do not analyse. Not overridable |
| `3` | Extractor error ratio above 5% | Judgement call. See below |
| `4` | Diagnostics format changed — the checker needs updating | Report it; the database itself may be fine |

Exit `2` is argparse's usage error, so a mistyped flag can never be mistaken for a
threshold decision.

Zero project files means build tracing captured nothing. A database in that state still
analyses without error and reports zero findings, so exit 1 has to stop the run rather
than leave it to be noticed later.

Exit 3 is a heuristic, and partial C/C++ extraction over vendored dependencies or
generated code exceeds it legitimately. Look at which files failed before deciding: if
the errors are confined to code that does not need analysing, re-run with a raised
threshold and record the reason in the log.

The log line goes inside the `if`. After it, a re-run that still fails writes "Raised
threshold to 15%" as though the override took, and the block exits 0 — `log_result`'s status.

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

if uv run {baseDir}/scripts/check_db_quality.py "$DB_NAME" --max-error-ratio 15; then
  log_result "Raised error-ratio threshold to 15%: failures are all in third_party/, not project source"
else
  log_result "Still failing at a 15% error ratio — the failures are not confined to third_party/"
  exit 1
fi
```

## Quality Criteria

Every metric below comes from the single call in Collect Metrics. The gate already fails on
the first three; the rest are for reading the result.

| Metric | JSON key | Good | Poor |
|--------|----------|------|------|
| Baseline LoC | `.baseline_loc` | > 0, proportional to project size | 0 or far below expected |
| Project source files | `.project_files` | Close to the expected count | 0 or < 50% of expected |
| Extractor errors | `.error_ratio` | < 5% of project files | > 5% |
| Total archive files | `.archive_files` | 10-20x `.project_files` for C/C++, ≈ equal for interpreted | equal to `.project_files` for C/C++ (no toolchain traced) |
| Finalised | `.finalised` | `true` | `false` or absent (interrupted build) |
| "No source code seen" | build log | Absent | Present (cached build, compiled languages) |

A small number of extractor errors is normal. Baseline LoC of 0, or an archive with no
project files, means the database is empty: a cached build for a compiled language, or the
wrong `--source-root`.

---

## Improve Quality (if poor)

Try these improvements, re-assess after each. **Log all improvements:**

### 1. Adjust source root

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "Quality improvement: adjust source root"
NEW_ROOT="./src"  # or detected subdirectory
# For interpreted: add --codescanning-config=codeql-config.yml
# For compiled: omit config flag
run_logged codeql database create "$DB_NAME" \
  --language="$CODEQL_LANG" --source-root="$NEW_ROOT" --overwrite
log_result "Changed source-root to: $NEW_ROOT"
```

### 2. Fix "no source code seen" (cached build - compiled languages only)

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "Quality improvement: force rebuild (cached build detected)"
# The rebuild is only worth running if the clean succeeded. Against a still-cached tree it
# re-extracts the same empty database, and the log would record that as a fix.
if make clean; then
  run_logged codeql database create "$DB_NAME" --language="$CODEQL_LANG" --overwrite
  log_result "Forced clean rebuild"
else
  log_result "SKIPPED: make clean failed, so the build is still cached"
fi
```

### 3. Install type stubs / dependencies

> **Note:** These install into the *target project's* environment to improve CodeQL extraction quality.

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "Quality improvement: install type stubs/additional deps"

# Python type stubs — install into target project's environment
# allow-legacy-python: installs into the analysed project's environment, which may not be uv-managed.
STUBS_INSTALLED=""
for stub in types-requests types-PyYAML types-redis; do
  if pip install "$stub" 2>/dev/null; then
    STUBS_INSTALLED="$STUBS_INSTALLED $stub"
  fi
done
log_result "Installed type stubs:$STUBS_INSTALLED"

# Additional project dependencies
# allow-legacy-python: the analysed project's own editable install.
run_logged pip install -e . || log_result "WARNING: pip install -e . failed — extraction may stay incomplete"
```

### 4. Adjust extractor options

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "Quality improvement: adjust extractor options"

# C/C++: Include headers
export CODEQL_EXTRACTOR_CPP_OPTION_TRAP_HEADERS=true
log_result "Set CODEQL_EXTRACTOR_CPP_OPTION_TRAP_HEADERS=true"

# Java: Specific JDK version
export CODEQL_EXTRACTOR_JAVA_OPTION_JDK_VERSION=17
log_result "Set CODEQL_EXTRACTOR_JAVA_OPTION_JDK_VERSION=17"

# Then rebuild with current method
```

**After each improvement:** Re-assess quality. If no improvement possible, move to next build method.

# Ruleset Catalog

## Official CodeQL Suites

| Suite | False Positives | Use Case |
|-------|-----------------|----------|
| `security-extended` | Low | **Default** - Security audits |
| `security-and-quality` | Medium | Comprehensive review (stable security + code quality) |
| `security-experimental` | Higher | Research, vulnerability hunting (stable security + experimental security) |

> **Suite hierarchy:** `security-and-quality` and `security-experimental` are complementary. `security-and-quality` excludes `experimental/` query paths. `security-experimental` includes them but excludes code quality queries. For maximum coverage (run-all mode), import both.

**Usage:** `codeql/<lang>-queries:codeql-suites/<lang>-security-extended.qls`

**Languages:** `cpp`, `csharp`, `go`, `java`, `javascript`, `python`, `ruby`, `swift`

---

## Trail of Bits Packs

| Pack | Language | Focus |
|------|----------|-------|
| `trailofbits/cpp-queries` | C/C++ | Memory safety, integer overflows |
| `trailofbits/go-queries` | Go | Concurrency, error handling |
| `trailofbits/java-queries` | Java | Security, code quality |

**Install:**
```bash
codeql pack download trailofbits/cpp-queries
codeql pack download trailofbits/go-queries
codeql pack download trailofbits/java-queries
```

---

## CodeQL Community Packs

| Pack | Language |
|------|----------|
| `GitHubSecurityLab/CodeQL-Community-Packs-JavaScript` | JavaScript/TypeScript |
| `GitHubSecurityLab/CodeQL-Community-Packs-Python` | Python |
| `GitHubSecurityLab/CodeQL-Community-Packs-Go` | Go |
| `GitHubSecurityLab/CodeQL-Community-Packs-Java` | Java |
| `GitHubSecurityLab/CodeQL-Community-Packs-CPP` | C/C++ |
| `GitHubSecurityLab/CodeQL-Community-Packs-CSharp` | C# |
| `GitHubSecurityLab/CodeQL-Community-Packs-Ruby` | Ruby |

**Install:**
```bash
codeql pack download GitHubSecurityLab/CodeQL-Community-Packs-<Lang>
```

**Source:** [github.com/GitHubSecurityLab/CodeQL-Community-Packs](https://github.com/GitHubSecurityLab/CodeQL-Community-Packs)

---

## Verify Installation

```bash
# List all installed packs
codeql resolve qlpacks

# Check specific packs
codeql resolve qlpacks | grep -E "(trailofbits|GitHubSecurityLab)"
```

# Run-All Query Suite

In run-all mode, generate a custom `.qls` query suite file at runtime. It runs the `security-and-quality` and `security-experimental` suites of every installed pack, which is a much wider selection than the code-scanning default each pack would otherwise apply — but it is not every query in those packs. See [What "run all" does and does not cover](#what-run-all-does-and-does-not-cover) before reporting coverage to anyone.

## Why a Custom Suite

When you pass a pack name directly to `codeql database analyze` (e.g., `-- codeql/cpp-queries`), CodeQL uses the pack's `defaultSuiteFile` field from `qlpack.yml`. For official packs, this is typically `codeql-suites/<lang>-code-scanning.qls`, which applies strict precision and severity filters. This drops many queries and can produce zero results for small codebases.

The run-all suite explicitly imports both `security-and-quality` and `security-experimental` from official packs, plus third-party packs with minimal filtering.

> **Why both suites?** `security-and-quality` = stable security + code quality (excludes `experimental/` paths). `security-experimental` = stable security + experimental security (re-includes `experimental/` paths tagged `security`). They are complementary — importing both is safe since CodeQL deduplicates shared queries automatically.

## Suite Template

What the generation script writes, shown so the imports and filters are readable. Do not
hand-write this file — the script adds the installed third-party packs and verifies the
result, and `test_generation_scripts.py` fails if this block and the script disagree:

```yaml
- description: Run-all — the security-and-quality and security-experimental suites from all installed packs, not every query in them; see run-all-suite.md
# Official queries: import BOTH suites (they are complementary, not hierarchical)
# security-and-quality = stable security + code quality (excludes experimental/ paths)
# security-experimental = stable security + experimental security (re-includes experimental/ with security tag)
- import: codeql-suites/<CODEQL_LANG>-security-and-quality.qls
  from: codeql/<CODEQL_LANG>-queries
- import: codeql-suites/<CODEQL_LANG>-security-experimental.qls
  from: codeql/<CODEQL_LANG>-queries
# Third-party packs (include only if installed, one entry per pack)
# - queries: .
#   from: trailofbits/<CODEQL_LANG>-queries
# - queries: .
#   from: GitHubSecurityLab/CodeQL-Community-Packs-<CODEQL_LANG>
# Minimal filtering — only select alert-type queries
- include:
    kind:
      - problem
      - path-problem
- exclude:
    deprecated: //
- exclude:
    tags contain:
      - modeleditor
      - modelgenerator
```

## Generation Script

```bash
# `set -e` and the trailing script call are both load-bearing: an assignment placed last
# would overwrite the script's exit status, and the run would proceed to analysis with no
# suite — or with a stale one from an earlier run.
set -euo pipefail
SUITE_FILE="$OUTPUT_DIR/raw/run-all.qls"
CODEQL_LANG="${CODEQL_LANG:-}" OUTPUT_DIR="${OUTPUT_DIR:-}" \
  INSTALLED_THIRD_PARTY_PACKS="${INSTALLED_THIRD_PARTY_PACKS:-}" \
  {baseDir}/scripts/generate_suite.sh run-all
```

Run-all imports whole upstream suites. A typo in `$CODEQL_LANG` gives a suite that resolves
to nothing instead of one that errors, so without the script's `verify_query_suite.py` call
the run would continue and report no findings.

## What "run all" does and does not cover

Measured against `codeql/cpp-queries` 1.8.0 with CodeQL 2.25.6:

| | cpp queries |
|---|---|
| `cpp-security-and-quality.qls` | 182 |
| `cpp-security-experimental.qls` | 135 |
| This template (both imported) | **219** — the exact union |
| Alert queries in the whole pack | 515 |

Two consequences the mode name hides:

- **"Run all" is not every query in the pack.** Of its 219, only 208 raise alerts; the
  other 11 are `Diagnostics/`, `Summary/`, and `Telemetry/` queries about the extraction
  itself. So 307 of the pack's 515 alert queries never run. Most are the coding-standard
  packs the official suites deliberately exclude (`jsf` 137, `JPL_C` 42, `Power of 10`
  23), but 13 are `Security/CWE/` queries — `ArithmeticTainted`, `IntegerOverflowTainted`,
  and `ImproperArrayIndexValidation` among them — and 20 are `Critical/` resource-leak and
  initialization queries. Skipping the standards packs in a security scan is reasonable;
  it is still a choice, not total coverage.
- **important-only is not a subset of run-all.** The modes select differently: run-all
  `import:`s two official suites, while important-only takes `queries: .` (the whole pack)
  and filters on precision. So important-only selects a few queries run-all does not —
  three for cpp, including `SuspiciousCallToMemset.ql`.

Re-check on your own packs after a pack upgrade; these counts move:
`codeql resolve queries "$OUTPUT_DIR/raw/run-all.qls" --format=json | jq length`.

## How This Differs From Important-Only

| Aspect | Run all | Important only |
|--------|---------|----------------|
| Official pack suites | `security-and-quality` + `security-experimental` (stable security + code quality + experimental security) | All queries loaded, filtered by precision |
| Third-party packs | All `problem`/`path-problem` queries | Only `security`-tagged queries with precision metadata |
| Precision filter | None | high/very-high always; medium only if security-severity >= 6.0 |
| Post-analysis filter | None | Drops medium-precision results with security-severity < 6.0 |

# SARIF Processing

jq commands for processing CodeQL SARIF output. Used in the run-analysis workflow Step 5.

> **SARIF structure note:** `security-severity` and `level` are stored on rule definitions (`.runs[].tool.driver.rules[]`), NOT on individual result objects. Results reference rules by `ruleIndex`. The jq commands below join results with their rule metadata.
>
> **Portability note:** These jq patterns assume CodeQL SARIF output where `ruleIndex` is populated. For SARIF from other tools (e.g., Semgrep), use `ruleId`-based lookups instead.

> **Directory convention:** Unfiltered output lives in `$RAW_DIR` (`$OUTPUT_DIR/raw`). Final results live in `$RESULTS_DIR` (`$OUTPUT_DIR/results`). The summary commands below operate on `$RESULTS_DIR/results.sarif` (the final output).

## Count Findings

```bash
jq '.runs[].results | length' "$RESULTS_DIR/results.sarif"
```

## Summary by SARIF Level

```bash
jq -r '
  .runs[] |
  . as $run |
  .results[] |
  ($run.tool.driver.rules[.ruleIndex].defaultConfiguration.level // "unknown")
' "$RESULTS_DIR/results.sarif" \
  | sort | uniq -c | sort -rn
```

## Summary by Security Severity (most useful for triage)

```bash
jq -r '
  .runs[] |
  . as $run |
  .results[] |
  ($run.tool.driver.rules[.ruleIndex].properties["security-severity"] // "none") + " | " +
  .ruleId + " | " +
  (.locations[0].physicalLocation.artifactLocation.uri // "?") + ":" +
  ((.locations[0].physicalLocation.region.startLine // 0) | tostring) + " | " +
  (.message.text // "no message" | .[0:80])
' "$RESULTS_DIR/results.sarif" | sort -rn | head -20
```

## Summary by Rule

```bash
jq -r '.runs[].results[] | .ruleId' "$RESULTS_DIR/results.sarif" \
  | sort | uniq -c | sort -rn
```

## Important-Only Post-Filter

If scan mode is "important only", filter out medium-precision results with `security-severity` < 6.0 from the report. The suite includes all medium-precision security queries to let CodeQL evaluate them, but low-severity medium-precision findings are noise.

The filter reads from `$RAW_DIR/results.sarif` (unfiltered) and writes to `$RESULTS_DIR/results.sarif` (final). The raw file is preserved unmodified.

```bash
# Filter important-only results: drop medium-precision findings with security-severity < 6.0
# Medium-precision queries without a security-severity score default to 0.0 (excluded).
# Non-medium queries are always kept regardless of security-severity.
# Reads from raw/, writes to results/ — preserving the unfiltered original.
RAW_DIR="$OUTPUT_DIR/raw"
RESULTS_DIR="$OUTPUT_DIR/results"
jq '
  .runs[] |= (
    . as $run |
    .results = [
      .results[] |
      ($run.tool.driver.rules[.ruleIndex].properties.precision // "unknown") as $prec |
      ($run.tool.driver.rules[.ruleIndex].properties["security-severity"] // null) as $raw_sev |
      (if $prec == "medium" then ($raw_sev // "0" | tonumber) else 10 end) as $sev |
      select(
        ($prec == "high") or ($prec == "very-high") or ($prec == "unknown") or
        ($prec == "medium" and $sev >= 6.0)
      )
    ]
  )
' "$RAW_DIR/results.sarif" > "$RESULTS_DIR/results.sarif"
```

# Threat Models Reference

Control which source categories are active during CodeQL analysis. By default, only `remote` sources are tracked.

## Available Models

| Model | Sources Included | When to Enable | False Positive Impact |
|-------|------------------|----------------|----------------------|
| `remote` | HTTP requests, network input | Always (default). Covers web services, APIs, network-facing code. | Low — these are the most common attack vectors. |
| `local` | Command line args, local files | CLI tools, batch processors, desktop apps where local users are untrusted. | Medium — generates noise for web-only services where CLI args are developer-controlled. |
| `environment` | Environment variables | Apps that read config from env vars at runtime (12-factor apps, containers). Skip for apps that only read env at startup into validated config objects. | Medium — many env reads are startup-only config, not runtime-tainted data. |
| `database` | Database query results | Second-order injection scenarios: stored XSS, data from shared databases where other writers are untrusted. | High — most apps trust their own database. Only enable when auditing for stored/second-order attacks. |
| `file` | File contents | File upload processors, log parsers, config file readers that accept user-provided files. | Medium — triggers on all file reads including trusted config files. |

## Default Behavior

With no `--threat-model` flag, CodeQL uses `remote` only (the `default` group). This is correct for most web applications and APIs. Expanding beyond `remote` is useful when the application's trust boundary extends to local inputs.

## Usage

Enable additional threat models with the `--threat-model` flag (singular, NOT `--threat-models`):

| Application | Flags to add to `codeql database analyze "$DB_NAME"` |
|---|---|
| Web service | none — `remote` is the default |
| CLI tool, local users untrusted | `--threat-model local` |
| Container reading env from an untrusted orchestrator | `--threat-model local --threat-model environment` |
| Audit mode, every input vector | `--threat-model all` |
| Everything except stored data, to cut noise | `--threat-model all --threat-model '!database'` |

The `--threat-model` flag can be repeated. Each invocation adds (or removes with `!` prefix) a threat model group. The `remote` group is always enabled by default — use `--threat-model '!default'` to disable it (rare). The `all` group enables everything, and `!<name>` disables a specific model.

Multiple models can be combined. Each additional model expands the set of sources CodeQL considers tainted, increasing coverage but potentially increasing false positives. Start with the narrowest set that matches the application's actual threat model, then expand if needed.

#!/usr/bin/env bash
# Logging helpers shared by the build-database workflow and its reference docs.
# Source it: . "{baseDir}/scripts/build_log.sh" || exit 1
#
# The `|| exit 1` is not decoration. The guard below returns 1 without defining the helpers,
# and the blocks that source this do not set -e, so without it the diagnostic scrolls past and
# the next line is `log_step: command not found` — 127, which the caller reads as a failed
# build rather than an unwritable log.
#
# Defaulted, so sourcing this in a standalone block is safe: an unset LOG_FILE makes
# `tee -a ""` fail, which under pipefail reports a successful build as failed.
LOG_FILE="${LOG_FILE:-${OUTPUT_DIR:-.}/build.log}"

# Fail here while the cause is legible: an unwritable log makes every build method report
# failure and walks the ladder to --build-mode=none after a build that succeeded.
if ! : >>"$LOG_FILE" 2>/dev/null; then
  echo "ERROR: cannot write build log $LOG_FILE." >&2
  echo "  Usually \$OUTPUT_DIR was never created: mkdir -p \"\$OUTPUT_DIR\"." >&2
  echo "  Otherwise set LOG_FILE or OUTPUT_DIR to a writable path before sourcing this." >&2
  # `return` when sourced (the documented use), `exit` when run directly. shellcheck
  # cannot tell which, so it reads the second path as dead.
  # shellcheck disable=SC2317
  return 1 2>/dev/null || exit 1
fi

# `cmd | tee` reports tee's exit status. Without pipefail a failed build looks like a
# success, and callers that branch on the result take the wrong branch.
set -o pipefail

# Deliberately no `set -e`: the method ladder must survive each failed method to reach the
# next. Blocks that should abort on first error set `-e` themselves.

log_step() { echo "[$(date -Iseconds)] $1" >>"$LOG_FILE"; }
log_cmd() { echo "[$(date -Iseconds)] COMMAND: $1" >>"$LOG_FILE"; }
log_result() {
  echo "[$(date -Iseconds)] RESULT: $1" >>"$LOG_FILE"
  echo "" >>"$LOG_FILE"
}

# Run a command, log its output, keep its exit status. Arguments as a list: a command built
# into a string and run unquoted word-splits on any path containing a space.
run_logged() {
  log_cmd "$*"
  "$@" 2>&1 | tee -a "$LOG_FILE"
}

# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
"""Fail unless a CodeQL database extracted enough source to be worth analysing.

An empty database analyses without error and reports "0 findings", the same output a
clean codebase produces. These thresholds are enforced rather than printed.

Two kinds of failure, two exit codes, because they do not deserve the same response:

    1  Nothing to analyse. No baseline lines of code, no project files in the source
       archive, or the database is unreadable. Not a judgement call and not overridable —
       analysing would report zero findings for source nobody read.
    3  Extractor error ratio above the threshold. A heuristic. Partial C/C++ extraction
       over vendored dependencies or generated code exceeds it legitimately, so this one
       is a decision to record rather than a wall: re-run with --max-error-ratio once you
       have established the errors are confined to code you do not need analysed.
    4  The diagnostics format changed and this checker needs updating. Distinct from 1
       because the database itself may be fine.

Note the gap: argparse exits 2 on a usage error, so a typo'd flag must not be readable as
a threshold decision.
"""

from __future__ import annotations

import argparse
import json
import re
import sys
import zipfile
from pathlib import Path

# Toolchain paths in src.zip. For compiled languages these outnumber project files
# 10-20x, so counting them would mask an extraction that captured nothing.
#
# Only used for a database whose codeql-database.yml records no source root. A blocklist
# cannot be complete — this one knows where macOS and Debian keep their toolchains, and a
# nix, Homebrew, or vendored-SDK build walks straight past it.
NON_PROJECT_PREFIXES = (
    "Applications/",
    "Library/",
    "Program Files/",
    "System/",
    "nix/store/",
    "opt/",
    "usr/",
)

SOURCE_LOCATION_PREFIX = re.compile(r"^sourceLocationPrefix:\s*(.+?)\s*$", re.MULTILINE)
# CodeQL spells it the British way. Checked on 2.25.6, where an interrupted build leaves
# the key absent rather than false.
FINALISED = re.compile(r"^finalised:\s*(\S+)\s*$", re.MULTILINE)

DEFAULT_MAX_ERROR_RATIO = 5.0


NOTHING_TO_ANALYSE = 1
# 2 belongs to argparse: `--max-error-ratio abc` exits 2, and a caller that reads 2 as
# "ratio exceeded" would raise the threshold for a database it never inspected.
ERROR_RATIO_EXCEEDED = 3
DIAGNOSTICS_FORMAT_CHANGED = 4


class QualityFailure(Exception):
    """A database that must not be analysed.

    `exit_code` separates the unarguable cases from the threshold. Callers that only
    check for non-zero keep working; the build workflow branches on the value.
    """

    def __init__(self, message: str, exit_code: int = NOTHING_TO_ANALYSE) -> None:
        super().__init__(message)
        self.exit_code = exit_code


def baseline_lines_of_code(database: Path) -> int:
    """Total baseline LoC across languages, from the database's own metadata."""
    info = database / "baseline-info.json"
    if not info.is_file():
        raise QualityFailure(
            f"{info} is missing. `codeql database create` writes it when it finalizes, "
            f"so the build did not get that far. (`print-baseline` prints a count to "
            f"stdout; it does not create this file.)"
        )
    try:
        data = json.loads(info.read_text())
    except (json.JSONDecodeError, OSError, UnicodeDecodeError) as error:
        raise QualityFailure(f"Could not read {info}: {error}") from error

    languages = data.get("languages")
    if not isinstance(languages, dict) or not languages:
        raise QualityFailure(f"{info} lists no languages — the database extracted nothing.")

    total = 0
    for language, entry in languages.items():
        if not isinstance(entry, dict):
            raise QualityFailure(f"{info}: entry for '{language}' is malformed.")
        try:
            total += int(entry.get("linesOfCode", 0))
        except (TypeError, ValueError) as error:
            raise QualityFailure(
                f"{info}: linesOfCode for '{language}' is not a number "
                f"({entry.get('linesOfCode')!r})."
            ) from error
    return total


def _archive_source_root(database: Path) -> str | None:
    """Where project source sits inside src.zip, from the database's own metadata.

    `codeql-database.yml` records the absolute source root, and src.zip stores every file
    at its absolute path minus the leading separator — so everything under that prefix is
    project source and everything else is toolchain, on any platform.

    Verified against a cpp database built by CodeQL 2.25.6: `sourceLocationPrefix: /…/proj`
    in the yml, `private/…/proj/src/main.c` in the archive, and the SDK headers alongside
    it under `Applications/Xcode.app/…`.
    """
    try:
        text = (database / "codeql-database.yml").read_text()
    except (OSError, UnicodeDecodeError):
        return None

    match = SOURCE_LOCATION_PREFIX.search(text)
    if not match:
        return None
    root = match.group(1).strip().strip("'\"").replace("\\", "/").strip("/")
    return f"{root}/" if root else None


def archive_file_count(database: Path) -> int:
    """Every file in the source archive, toolchain included.

    Reported so a caller does not have to run its own `unzip -Z1 | wc -l`. For a compiled
    language this is 10-20x project_file_count — 690 against 66 for an mbedtls cpp
    database on 2.25.6 — which is why it is not the quality signal on its own.
    """
    return len(_archive_files(database))


def _archive_files(database: Path) -> list[str]:
    """Non-directory entries in src.zip."""
    archive = database / "src.zip"
    if not archive.is_file():
        raise QualityFailure(f"{archive} is missing — the database has no source archive.")
    try:
        with zipfile.ZipFile(archive) as handle:
            names = handle.namelist()
    except (zipfile.BadZipFile, OSError) as error:
        raise QualityFailure(f"Could not read {archive}: {error}") from error
    return [name for name in names if not name.endswith("/")]


def is_finalised(database: Path) -> bool:
    """Whether `codeql database finalize` completed.

    A database that is not finalised resolves and analyses, and reports nothing.
    """
    try:
        text = (database / "codeql-database.yml").read_text()
    except (OSError, UnicodeDecodeError):
        return False
    match = FINALISED.search(text)
    return bool(match) and match.group(1).strip().strip("'\"").lower() == "true"


def project_file_count(database: Path) -> int:
    """Files in the source archive that belong to the project, not the toolchain."""
    archive = database / "src.zip"
    files = _archive_files(database)
    root = _archive_source_root(database)
    if root is None:
        return sum(1 for name in files if not name.startswith(NON_PROJECT_PREFIXES))

    under_root = sum(1 for name in files if name.startswith(root))
    if under_root == 0 and any(not name.startswith(NON_PROJECT_PREFIXES) for name in files):
        # Toolchain-only is a real answer, handled by the caller. Non-toolchain files that
        # sit outside the recorded root are not: the archive is laid out in some way this
        # check does not model, and guessing would report a number nobody can trust.
        raise QualityFailure(
            f"No file in {archive} is under the recorded source root ({root}), yet the "
            f"archive holds files that are not toolchain paths either. Its layout does not "
            f"match codeql-database.yml, so project source cannot be told from toolchain."
        )
    return under_root


def _load_records(path: Path) -> list[dict]:
    """Diagnostic records from one file.

    The `.jsonl` extension is not reliable. CodeQL writes some of these as a single
    pretty-printed object spanning many lines, and the directory also holds plain `.json`
    files. Try whole-file JSON first, then line-delimited.
    """
    try:
        text = path.read_text()
    except (OSError, UnicodeDecodeError) as error:
        raise QualityFailure(f"Could not read extractor diagnostics {path}: {error}") from error

    try:
        data = json.loads(text)
    except json.JSONDecodeError:
        records = []
        for line in text.splitlines():
            stripped = line.strip()
            if not stripped:
                continue
            try:
                records.append(json.loads(stripped))
            except json.JSONDecodeError:
                # Neither whole-file JSON nor JSONL. Skipping would under-count, so treat
                # it as a format change that needs handling rather than as zero errors.
                raise QualityFailure(
                    f"{path} is neither a JSON object nor JSON Lines. The diagnostic "
                    f"format has changed and this check needs updating.",
                    DIAGNOSTICS_FORMAT_CHANGED,
                ) from None
        return [r for r in records if isinstance(r, dict)]

    if isinstance(data, list):
        return [r for r in data if isinstance(r, dict)]
    return [data] if isinstance(data, dict) else []


def _errors_from_one_extractor(records: list[dict]) -> int:
    """CodeQL's own tally if this extractor wrote one, else its severity-`error` records."""
    reported: list[int] = []
    for record in records:
        attributes = record.get("attributes")
        if isinstance(attributes, dict) and "extractor-failures" in attributes:
            try:
                reported.append(int(attributes["extractor-failures"]))
            except (TypeError, ValueError):
                continue
    if reported:
        return sum(reported)

    return sum(1 for record in records if str(record.get("severity", "")).lower() == "error")


def _extractor_of(diagnostics: Path, path: Path) -> str:
    """Which extractor a diagnostic file belongs to.

    The layout is `diagnostic/extractors/<language>/…`. Files sitting directly in
    `extractors/` share one group, since they cannot be attributed further.
    """
    parts = path.relative_to(diagnostics).parts
    return parts[0] if len(parts) > 1 else ""


def extractor_error_count(database: Path) -> int:
    """Extractor failures recorded during the build, summed over extractors.

    Prefers CodeQL's own tally in `summary.jsonl` (`attributes.extractor-failures`), and
    falls back to counting records whose `severity` is `error`. That preference is
    resolved per extractor, not once for the tree: extractors are separate binaries that
    gained summary reporting in different releases, so in a multi-language database one
    may write a summary while another emits only error records. Deciding globally let a
    single `extractor-failures: 0` suppress the fallback for every other language,
    reporting a partially-failed database as clean.

    Counting lines that start with `{` reported 5 errors for a database where CodeQL
    recorded `extractor-failures: 0` and every record was severity `note`: the files are
    pretty-printed despite the `.jsonl` name, so nested braces each counted as a record.

    A missing directory genuinely means zero errors: the extractor creates it only when
    it has something to report.
    """
    diagnostics = database / "diagnostic" / "extractors"
    if not diagnostics.is_dir():
        return 0

    # Only the JSON-family files. `rglob("*")` handed every file to the parser, so a `.log`
    # or `.txt` dropped into this tree by a future release failed the whole gate as a
    # format change — on a database whose LoC and file counts were fine.
    by_extractor: dict[str, list[dict]] = {}
    for path in sorted(p for p in diagnostics.rglob("*.json*") if p.is_file()):
        by_extractor.setdefault(_extractor_of(diagnostics, path), []).extend(_load_records(path))

    return sum(_errors_from_one_extractor(records) for records in by_extractor.values())


def assess(database: Path, max_error_ratio: float = DEFAULT_MAX_ERROR_RATIO) -> dict[str, object]:
    """Return quality metrics, raising QualityFailure if the database is unusable."""
    if not (database / "codeql-database.yml").is_file():
        raise QualityFailure(
            f"{database} is not a CodeQL database (no codeql-database.yml marker)."
        )

    loc = baseline_lines_of_code(database)
    files = project_file_count(database)
    errors = extractor_error_count(database)

    if loc == 0:
        raise QualityFailure(
            "Baseline lines of code is 0 — the database contains no analysable source. "
            "Analysing it would report zero findings for a codebase that was never read."
        )
    if files == 0:
        raise QualityFailure(
            "Zero project files in the source archive (only toolchain paths). "
            "This is the strongest signal that build tracing captured nothing."
        )

    ratio = errors / files * 100
    if ratio > max_error_ratio:
        raise QualityFailure(
            f"Extractor error ratio {ratio:.1f}% exceeds the {max_error_ratio:.1f}% "
            f"threshold ({errors} errors across {files} project files). Findings from "
            f"this database would be incomplete in ways the results cannot show. If the "
            f"failures are confined to code that does not need analysing, re-run with "
            f"--max-error-ratio and record why.",
            ERROR_RATIO_EXCEEDED,
        )

    return {
        "baseline_loc": loc,
        "project_files": files,
        "archive_files": archive_file_count(database),
        "extractor_errors": errors,
        "error_ratio": round(ratio, 1),
        "finalised": is_finalised(database),
    }


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("database", type=Path, help="Path to the CodeQL database directory")
    parser.add_argument(
        "--max-error-ratio",
        type=float,
        default=DEFAULT_MAX_ERROR_RATIO,
        help="Percent of project files that may fail extraction "
        f"(default: {DEFAULT_MAX_ERROR_RATIO})",
    )
    parser.add_argument(
        "--format",
        choices=("text", "json"),
        default="text",
        help="json emits the metrics for a caller to read, so nothing has to recompute "
        "them with a second, differently-written shell pipeline",
    )
    args = parser.parse_args(argv)

    try:
        metrics = assess(args.database, args.max_error_ratio)
    except QualityFailure as error:
        print(f"ERROR: {error}", file=sys.stderr)
        return error.exit_code

    if args.format == "json":
        print(json.dumps(metrics))
        return 0

    print(
        f"Database OK: {metrics['baseline_loc']} baseline LoC, "
        f"{metrics['project_files']} project files, "
        f"{metrics['extractor_errors']} extractor errors ({metrics['error_ratio']}%)"
    )
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env bash
# Print every real CodeQL database under the given roots, one path per line.
#
#   find_databases.sh [root ...]      # defaults to "$OUTPUT_DIR" then "."
#
# Callers build their own array from this output, in their own block: an array does not
# survive into a later Bash call, and an empty one reads as "no database".
#
# Read it with command substitution, not `done < <(...)`: a process substitution discards
# the exit status, and exit 2 below (no codeql on this shell's PATH) then looks exactly
# like an empty result — the one confusion this script exists to prevent.
#
# `codeql resolve database` is the filter that matters. The codeql-database.yml marker is
# written before the build finishes, so a failed build leaves one a bare `find` would take.
set -uo pipefail

# Without codeql the filter below rejects everything, and the script exits 0 having printed
# nothing — indistinguishable from a project with no databases. SKILL.md's auto-detection
# routes that to "build the whole pipeline", so a machine with three good databases and no
# codeql on this shell's PATH rebuilds instead of being told what is actually wrong.
if ! command -v codeql >/dev/null 2>&1; then
  echo "ERROR: codeql not found on PATH — cannot tell a database from the marker a failed build leaves." >&2
  exit 2
fi

if [ "$#" -eq 0 ]; then
  set -- "${OUTPUT_DIR:-.}" "."
fi

seen=""
for root in "$@"; do
  [ -d "$root" ] || continue
  # Absolute physical path first: $OUTPUT_DIR is usually inside ".", so searched as written
  # one database surfaces twice under two spellings and defeats the dedup below.
  root=$(cd "$root" 2>/dev/null && pwd -P) || continue
  # Dotted directories are pruned by name rather than excluded with `-not -path '*/.*'`.
  # `find` matches -path against the whole path, and the root above is absolute, so that
  # pattern also matched dotted *ancestors*: a project anywhere under ~/.cache, ~/.local or
  # a dotted checkout reported no databases at all, and the caller rebuilt from scratch.
  # -mindepth 1 keeps the prune off the root itself, which may legitimately be dotted.
  while IFS= read -r marker; do
    db=$(dirname "$marker")
    case "$seen" in
      *"|$db|"*) continue ;;
    esac
    if codeql resolve database -- "$db" >/dev/null 2>&1; then
      seen="$seen|$db|"
      printf '%s\n' "$db"
    fi
  done < <(find "$root" -mindepth 1 -maxdepth 3 \
    \( -type d -name '.*' -prune \) -o \
    \( -type f -name codeql-database.yml -print \) 2>/dev/null)
done

#!/usr/bin/env bash
# Write the query suite for one scan mode, and prove it resolves to at least one query.
#
#   generate_suite.sh run-all|important-only
#
# Reads OUTPUT_DIR, CODEQL_LANG, and INSTALLED_THIRD_PARTY_PACKS from the environment;
# writes $OUTPUT_DIR/raw/<mode>.qls.
#
# The modes differ only in header entries and include filters; everything else was
# duplicated across two reference docs, where `make shell` could not see it.
set -euo pipefail

MODE="${1:-}"
case "$MODE" in
  run-all | important-only) ;;
  *)
    echo "usage: generate_suite.sh run-all|important-only" >&2
    exit 2
    ;;
esac

# Before the first heredoc: unset, ${CODEQL_LANG} names the pack `codeql/-queries`, and the
# broken suite is on disk for a later step to pick up.
: "${CODEQL_LANG:?ERROR: CODEQL_LANG must be set before generating the suite}"
: "${OUTPUT_DIR:?ERROR: OUTPUT_DIR must be set}"
INSTALLED_THIRD_PARTY_PACKS="${INSTALLED_THIRD_PARTY_PACKS:-}"

SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
RAW_DIR="$OUTPUT_DIR/raw"
mkdir -p "$RAW_DIR"
SUITE_FILE="$RAW_DIR/$MODE.qls"

# Unquoted heredocs here: ${CODEQL_LANG} must expand.
if [ "$MODE" = run-all ]; then
  cat >"$SUITE_FILE" <<HEADER
- description: Run-all — the security-and-quality and security-experimental suites from all installed packs, not every query in them; see run-all-suite.md
- import: codeql-suites/${CODEQL_LANG}-security-and-quality.qls
  from: codeql/${CODEQL_LANG}-queries
- import: codeql-suites/${CODEQL_LANG}-security-experimental.qls
  from: codeql/${CODEQL_LANG}-queries
HEADER
else
  cat >"$SUITE_FILE" <<HEADER
- description: Important-only — security vulnerabilities, medium-high confidence
- queries: .
  from: codeql/${CODEQL_LANG}-queries
HEADER
fi

# Word splitting is the point: the variable holds a space-separated pack list.
# shellcheck disable=SC2086
for PACK in $INSTALLED_THIRD_PARTY_PACKS; do
  cat >>"$SUITE_FILE" <<PACK_ENTRY
- queries: .
  from: ${PACK}
PACK_ENTRY
done

# Quoted heredocs below: these are literal, nothing to expand.
if [ "$MODE" = run-all ]; then
  # Minimal filtering — select alert queries and nothing else.
  cat >>"$SUITE_FILE" <<'FILTERS'
- include:
    kind:
      - problem
      - path-problem
FILTERS
else
  # Security tag required. High and very-high run at any severity; medium is narrowed
  # after the run by the security-severity filter in run-analysis Step 5.
  cat >>"$SUITE_FILE" <<'FILTERS'
- include:
    kind:
      - problem
      - path-problem
    precision:
      - high
      - very-high
    tags contain:
      - security
- include:
    kind:
      - problem
      - path-problem
    precision:
      - medium
    tags contain:
      - security
FILTERS
fi

cat >>"$SUITE_FILE" <<'SHARED_FILTERS'
- exclude:
    deprecated: //
- exclude:
    tags contain:
      - modeleditor
      - modelgenerator
SHARED_FILTERS

# Fails on zero resolved queries, a CodeQL error, or malformed output. Not
# `codeql resolve queries | wc -l`, which reports wc's status whatever CodeQL did.
#
# Delete an unverified suite: left on disk it is indistinguishable from a good one, and
# run-analysis Step 4 derives the same path and would pick it up.
if ! uv run "$SCRIPT_DIR/verify_query_suite.py" "$SUITE_FILE"; then
  rm -f "$SUITE_FILE"
  echo "Removed $SUITE_FILE: it did not resolve to any queries." >&2
  exit 1
fi

echo "Suite generated: $SUITE_FILE"

# Deliberately no [tool.ruff] section. Ruff configures each file from the nearest
# ancestor config that has one, so adding it here would replace the repo's ruff.toml —
# and its select list — with these two lines, quietly linting this directory less than
# every other plugin. Both scripts run standalone under `uv run` with PEP 723 headers;
# this file exists for editors and type checkers.
[project]
name = "codeql-skill-scripts"
version = "0.1.0"
description = "Quality gate and suite verifier for the static-analysis codeql skill"
requires-python = ">=3.11"
dependencies = []

"""Run build_log.sh, rather than reading it.

`make shell` checks this file's syntax and test_shell_blocks.py checks the markdown that
calls it. Neither executes it — so the claim the whole build-method ladder rests on, that
`run_logged` returns the command's exit status and not tee's, was never demonstrated.
"""

from __future__ import annotations

import subprocess
from pathlib import Path

BUILD_LOG = Path(__file__).resolve().parent / "build_log.sh"


def _bash(script: str, cwd: Path) -> subprocess.CompletedProcess[str]:
    return subprocess.run(
        ["bash", "-c", f'. "{BUILD_LOG}"\n{script}'],
        cwd=cwd,
        capture_output=True,
        text=True,
    )


# --- the exit-status claim ---------------------------------------------------------


def test_run_logged_reports_the_commands_failure(tmp_path: Path) -> None:
    """The bug this file was written to fix, in its simplest form.

    Without pipefail `false | tee` exits 0, so the ladder recorded a failed build as a
    success and never tried the next method.
    """
    result = _bash('run_logged false\necho "status=$?"', tmp_path)
    assert "status=1" in result.stdout


def test_run_logged_reports_success(tmp_path: Path) -> None:
    """The other direction: a working build must not look like a failure."""
    result = _bash('run_logged true\necho "status=$?"', tmp_path)
    assert "status=0" in result.stdout


def test_sourcing_sets_pipefail_for_the_caller(tmp_path: Path) -> None:
    """Blocks that pipe directly, rather than through the wrapper, inherit the setting."""
    result = _bash('false | true\necho "status=$?"', tmp_path)
    assert "status=1" in result.stdout


def test_run_logged_keeps_arguments_apart(tmp_path: Path) -> None:
    """argv, not a string: a path with a space must survive as one argument."""
    result = _bash('run_logged printf "[%s]" "two words"', tmp_path)
    assert "[two words]" in result.stdout


# --- where the log goes ------------------------------------------------------------


def test_output_is_written_to_the_log(tmp_path: Path) -> None:
    _bash("run_logged echo hello-from-the-build", tmp_path)
    assert "hello-from-the-build" in (tmp_path / "build.log").read_text()


def test_log_file_defaults_under_output_dir(tmp_path: Path) -> None:
    scans = tmp_path / "scans"
    scans.mkdir()
    result = subprocess.run(
        ["bash", "-c", f'OUTPUT_DIR="{scans}"\n. "{BUILD_LOG}"\nrun_logged echo hi'],
        cwd=tmp_path,
        capture_output=True,
        text=True,
    )
    assert result.returncode == 0
    assert "hi" in (scans / "build.log").read_text()


def test_an_unwritable_log_fails_at_source_time(tmp_path: Path) -> None:
    """Not four rungs later, as a CodeQL problem.

    tee's failure is the pipeline's failure under pipefail, so every method would report
    failure and the run would end at --build-mode=none blaming the build.
    """
    missing = tmp_path / "no-such-dir" / "build.log"
    result = subprocess.run(
        ["bash", "-c", f'LOG_FILE="{missing}"\n. "{BUILD_LOG}"\nrun_logged true'],
        cwd=tmp_path,
        capture_output=True,
        text=True,
    )
    assert result.returncode != 0
    assert "cannot write build log" in result.stderr

"""Keep the build-mode claims in the docs consistent with each other.

The docs classified Go as interpreted and no-build. Go is compiled and rejects
`--build-mode=none` outright:

    A fatal error occurred: Go does not support the none build mode.
    Please try using one of the following build modes instead: autobuild, manual.

They also listed C# and Java as requiring tracing without mentioning that both accept
`none`. Verified against CodeQL 2.25.6.

These checks are hermetic: they read the markdown, not the CLI, so they run without a
CodeQL install. Re-verify the table by hand when bumping CodeQL, since supported modes
change between releases.
"""

from __future__ import annotations

import re
from pathlib import Path

import pytest

SKILL_ROOT = Path(__file__).resolve().parent.parent
BUILD_DB = SKILL_ROOT / "workflows" / "build-database.md"
LANG_DETAILS = SKILL_ROOT / "references" / "language-details.md"

# Verified with `codeql database create --build-mode=none --language=<lang>` on 2.25.6.
# Note the CLI's own --help omits C/C++ from its `none` list, but cpp does support it.
NO_NONE_MODE = ("Go", "Swift")


def test_go_is_not_listed_as_needing_no_build() -> None:
    """The original bug: Go grouped with Python and Ruby."""
    text = BUILD_DB.read_text(encoding="utf-8")
    section = re.search(r"### No build needed \(([^)]*)\)", text)
    assert section, "build-database.md no longer has a 'No build needed' section"
    assert "Go" not in section.group(1), (
        f"Go is listed as needing no build: '{section.group(1)}'. Go rejects "
        f"--build-mode=none and requires autobuild or a manual command."
    )


@pytest.mark.parametrize("language", NO_NONE_MODE)
def test_languages_without_a_none_mode_are_flagged(language: str) -> None:
    """A reader must not be sent to Method 4 for a language that rejects it."""
    text = BUILD_DB.read_text(encoding="utf-8")
    assert re.search(rf"{language}.*[Rr]eject", text), (
        f"{language} rejects --build-mode=none, but build-database.md does not say so. "
        f"Method 4 would be attempted and fail."
    )


def test_method_4_names_the_languages_it_does_not_apply_to() -> None:
    text = BUILD_DB.read_text(encoding="utf-8")
    method4 = text[text.index("#### Method 4") :]
    for language in NO_NONE_MODE:
        assert language in method4[:800], (
            f"Method 4 does not mention that {language} rejects --build-mode=none"
        )


def test_language_table_covers_both_axes() -> None:
    """The table must state build need and none-support, not 'Interpreted/Compiled'.

    That framing is what produced the error: Go is compiled but was called interpreted,
    and 'compiled' was treated as implying no `none` mode.
    """
    text = BUILD_DB.read_text(encoding="utf-8")
    assert "| Language | `--language=` | Build needed | `--build-mode=none` |" in text, (
        "the language table lost its build-mode columns"
    )
    assert "| Type |" not in text, "the Interpreted/Compiled column is back"


def test_go_sits_under_build_required_in_language_details() -> None:
    text = LANG_DETAILS.read_text(encoding="utf-8")
    build_required = text.index("## Build Required")
    go = text.index("### Go")
    assert go > build_required, "language-details.md lists Go under 'No Build Required'"


def test_claims_name_the_verified_codeql_version() -> None:
    """A version-specific claim with no version is unfalsifiable later."""
    assert "2.25.6" in BUILD_DB.read_text(encoding="utf-8"), (
        "build-database.md no longer says which CodeQL version the table was verified "
        "against, so a reader cannot tell whether it has gone stale"
    )

"""Regression tests for the database quality gate.

The workflow used to compute these metrics and print them, special-casing zero files to
"N/A (no files)" — so a database that extracted nothing still got analysed. These pin
the thresholds that now stop it.
"""

from __future__ import annotations

import json
import zipfile
from pathlib import Path

import pytest
from check_db_quality import (
    DIAGNOSTICS_FORMAT_CHANGED,
    ERROR_RATIO_EXCEEDED,
    QualityFailure,
    assess,
    is_finalised,
    main,
)

PROJECT_FILES = ["src/app.c", "src/util.c", "include/app.h"]
TOOLCHAIN_FILES = ["usr/include/stdio.h", "Library/Developer/SDK/string.h", "System/x.h"]


def _database(
    tmp_path: Path,
    *,
    loc: int = 1200,
    project_files: list[str] | None = None,
    toolchain_files: list[str] | None = None,
    source_prefix: str | None = None,
    extractor_errors: int = 0,
    baseline: bool = True,
    archive: bool = True,
) -> Path:
    """Build a directory shaped like a finalized CodeQL database.

    With `source_prefix`, the yml records a source root and the archive stores project
    files beneath it, as CodeQL does. Without one, it falls back to the toolchain
    blocklist — which is what a database from a CLI old enough to omit the key gets.
    """
    database = tmp_path / "codeql.db"
    database.mkdir(parents=True, exist_ok=True)
    marker = "primaryLanguage: cpp\nfinalised: true\n"
    if source_prefix:
        marker += f"sourceLocationPrefix: {source_prefix}\n"
    (database / "codeql-database.yml").write_text(marker)

    if baseline:
        files = project_files if project_files is not None else PROJECT_FILES
        (database / "baseline-info.json").write_text(
            json.dumps({"languages": {"cpp": {"linesOfCode": loc, "files": files}}})
        )

    if archive:
        root = (source_prefix or "").replace("\\", "/").strip("/")
        sources = project_files if project_files is not None else PROJECT_FILES
        names = [f"{root}/{name}" if root else name for name in sources]
        names += toolchain_files if toolchain_files is not None else TOOLCHAIN_FILES
        with zipfile.ZipFile(database / "src.zip", "w") as handle:
            for name in names:
                handle.writestr(name, "// source\n")

    if extractor_errors:
        diagnostics = database / "diagnostic" / "extractors" / "cpp"
        diagnostics.mkdir(parents=True)
        (diagnostics / "errors.jsonl").write_text(
            "".join(f'{{"severity":"error","n":{i}}}\n' for i in range(extractor_errors))
        )

    return database


def _diag(database: Path, name: str, payload: str) -> Path:
    directory = database / "diagnostic" / "extractors" / "cpp"
    directory.mkdir(parents=True, exist_ok=True)
    path = directory / name
    path.write_text(payload)
    return path


# --- the thresholds that did not previously exist ---------------------------------


def test_zero_baseline_loc_fails(tmp_path: Path) -> None:
    """No analysable source must halt the run, not produce a clean report."""
    with pytest.raises(QualityFailure, match="Baseline lines of code is 0"):
        assess(_database(tmp_path, loc=0))


def test_zero_project_files_fails(tmp_path: Path) -> None:
    """Only toolchain paths in src.zip means build tracing captured nothing.

    The case the old shell forgave with "N/A (no files)".
    """
    with pytest.raises(QualityFailure, match="Zero project files"):
        assess(_database(tmp_path, project_files=[]))


def test_error_ratio_over_threshold_fails(tmp_path: Path) -> None:
    with pytest.raises(QualityFailure, match="error ratio"):
        assess(_database(tmp_path, extractor_errors=3))  # 3/3 files = 100%


def test_error_ratio_under_threshold_passes(tmp_path: Path) -> None:
    many = [f"src/f{i}.c" for i in range(100)]
    metrics = assess(_database(tmp_path, project_files=many, extractor_errors=2))
    assert metrics["error_ratio"] == 2.0


def test_threshold_is_configurable(tmp_path: Path) -> None:
    database = _database(tmp_path, extractor_errors=3)
    with pytest.raises(QualityFailure):
        assess(database, max_error_ratio=5.0)
    assert assess(database, max_error_ratio=100.0)["extractor_errors"] == 3


# --- toolchain files must not be mistaken for project source ----------------------


def test_toolchain_files_do_not_count_as_project_source(tmp_path: Path) -> None:
    """Counting SDK headers would let an extraction that captured only headers pass."""
    database = _database(
        tmp_path,
        project_files=[],
        toolchain_files=[f"usr/include/h{i}.h" for i in range(200)],
    )
    with pytest.raises(QualityFailure, match="Zero project files"):
        assess(database)


def test_the_source_root_beats_the_toolchain_blocklist(tmp_path: Path) -> None:
    """A Linux toolchain under `nix/store/` is not project source, and no list knows that.

    The blocklist was macOS-shaped, so these 200 headers counted as project files: the
    error-ratio denominator inflated and the gate softened. The recorded source root
    settles it without the list having to name every platform's toolchain location.
    """
    database = _database(
        tmp_path,
        source_prefix="/srv/build/proj",
        project_files=["src/app.c", "src/util.c"],
        toolchain_files=[f"nix/store/x-gcc-13/include/h{i}.h" for i in range(200)],
    )
    assert assess(database)["project_files"] == 2


def test_real_archive_layout_from_codeql_2_25_6(tmp_path: Path) -> None:
    """Pin the contract, copied from a cpp database this skill's own workflow built.

    src.zip stores each file at its absolute path minus the leading slash, so the source
    root from the yml is a literal prefix of the project's entries and of nothing else.
    """
    database = _database(
        tmp_path,
        source_prefix="/private/tmp/scan/proj",
        project_files=["src/main.c"],
        toolchain_files=[
            "Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/"
            "Developer/SDKs/MacOSX.sdk/usr/include/stdio.h"
        ],
    )
    with zipfile.ZipFile(database / "src.zip") as handle:
        assert "private/tmp/scan/proj/src/main.c" in handle.namelist()
    assert assess(database)["project_files"] == 1


def test_toolchain_only_archive_fails_even_with_a_source_root(tmp_path: Path) -> None:
    """Nothing under the root is the signal the gate exists for, not a layout problem."""
    database = _database(
        tmp_path,
        source_prefix="/srv/build/proj",
        project_files=[],
        toolchain_files=[f"usr/include/h{i}.h" for i in range(50)],
    )
    with pytest.raises(QualityFailure, match="Zero project files"):
        assess(database)


def test_archive_that_ignores_the_source_root_fails_loudly(tmp_path: Path) -> None:
    """Files outside the root that are not toolchain paths mean an unmodelled layout.

    Counting them as project source, or as zero, would both report a number nobody can
    check. Windows drive letters are the case most likely to land here.
    """
    database = _database(tmp_path, project_files=["src/app.c"], toolchain_files=[])
    (database / "codeql-database.yml").write_text(
        "primaryLanguage: cpp\nfinalised: true\nsourceLocationPrefix: /srv/build/proj\n"
    )
    with pytest.raises(QualityFailure, match="layout does not match"):
        assess(database)


def test_healthy_database_passes(tmp_path: Path) -> None:
    metrics = assess(_database(tmp_path))
    assert metrics["baseline_loc"] == 1200
    assert metrics["project_files"] == len(PROJECT_FILES)
    assert metrics["extractor_errors"] == 0


# --- malformed inputs are failures, not zeros -------------------------------------


def test_missing_marker_fails(tmp_path: Path) -> None:
    plain = tmp_path / "not-a-db"
    plain.mkdir()
    with pytest.raises(QualityFailure, match="not a CodeQL database"):
        assess(plain)


def test_missing_baseline_info_fails(tmp_path: Path) -> None:
    with pytest.raises(QualityFailure, match="baseline-info.json"):
        assess(_database(tmp_path, baseline=False))


def test_missing_source_archive_fails(tmp_path: Path) -> None:
    with pytest.raises(QualityFailure, match="src.zip"):
        assess(_database(tmp_path, archive=False))


def test_empty_language_map_fails(tmp_path: Path) -> None:
    database = _database(tmp_path)
    (database / "baseline-info.json").write_text(json.dumps({"languages": {}}))
    with pytest.raises(QualityFailure, match="no languages"):
        assess(database)


def test_parses_baseline_info_as_codeql_actually_writes_it(tmp_path: Path) -> None:
    """Pin the real contract, copied verbatim from a database built by CodeQL 2.25.6.

    `codeql database create` writes baseline-info.json when it finalizes. `print-baseline`
    prints a count to stdout and creates nothing, so the file is the only machine-readable
    source. Extra keys (displayName, name) must not break parsing.
    """
    database = _database(tmp_path, baseline=False)
    (database / "baseline-info.json").write_text(
        json.dumps(
            {
                "languages": {
                    "python": {
                        "displayName": "Python",
                        "files": ["app.py"],
                        "linesOfCode": 5,
                        "name": "python",
                    }
                }
            }
        )
    )
    assert assess(database)["baseline_loc"] == 5


def test_sums_lines_of_code_across_languages(tmp_path: Path) -> None:
    """`languages` is a map, so a multi-language database has more than one entry."""
    database = _database(tmp_path, baseline=False)
    (database / "baseline-info.json").write_text(
        json.dumps(
            {
                "languages": {
                    "python": {"linesOfCode": 5, "files": ["a.py"]},
                    "cpp": {"linesOfCode": 7, "files": ["b.c"]},
                }
            }
        )
    )
    assert assess(database)["baseline_loc"] == 12


# --- diagnostics: severity, not brace count ---------------------------------------


def test_pretty_printed_notes_are_not_errors(tmp_path: Path) -> None:
    """The regression. Verbatim from a healthy database built by CodeQL 2.25.6.

    Despite the `.jsonl` name this is one pretty-printed object. Counting lines starting
    with `{` scored its nested braces as records and failed the database at a 500% error
    ratio, while CodeQL reported extractor-failures: 0.
    """
    database = _database(tmp_path)
    _diag(
        database,
        "autobuilder-41150.jsonl",
        json.dumps(
            {
                "timestamp": "2026-07-30T23:47:17Z",
                "source": {
                    "id": "cpp/autobuilder/buildless/mode-active",
                    "name": "C/C++ was extracted with build-mode set to 'none'",
                    "extractorName": "cpp",
                },
                "severity": "note",
                "visibility": {"statusPage": False, "telemetry": True},
                "attributes": {"tried": []},
            },
            indent=2,
        ),
    )
    assert assess(database)["extractor_errors"] == 0


def test_codeql_own_failure_count_wins(tmp_path: Path) -> None:
    """summary.jsonl carries `extractor-failures`, which is authoritative."""
    database = _database(tmp_path, project_files=[f"src/f{i}.c" for i in range(100)])
    _diag(
        database,
        "summary.jsonl",
        json.dumps({"attributes": {"extractor-failures": 3, "extractor-successes": 97}}, indent=4),
    )
    assert assess(database)["extractor_errors"] == 3


def test_failure_counts_sum_across_extractors(tmp_path: Path) -> None:
    database = _database(tmp_path, project_files=[f"src/f{i}.c" for i in range(100)])
    for lang, failures in (("cpp", 2), ("python", 1)):
        directory = database / "diagnostic" / "extractors" / lang
        directory.mkdir(parents=True, exist_ok=True)
        (directory / "summary.jsonl").write_text(
            json.dumps({"attributes": {"extractor-failures": failures}})
        )
    assert assess(database)["extractor_errors"] == 3


def test_a_summary_from_one_extractor_does_not_mask_another(tmp_path: Path) -> None:
    """The summary-vs-severity preference is per extractor, not per database.

    Resolved globally, cpp's `extractor-failures: 0` returned 0 for the whole tree and
    python's four error records vanished — a database whose python extraction largely
    failed passed the ratio gate at 0.0%.
    """
    database = _database(tmp_path, project_files=[f"src/f{i}.c" for i in range(100)])
    extractors = database / "diagnostic" / "extractors"
    (extractors / "cpp").mkdir(parents=True)
    (extractors / "cpp" / "summary.jsonl").write_text(
        json.dumps({"attributes": {"extractor-failures": 0, "extractor-successes": 40}})
    )
    (extractors / "python").mkdir(parents=True)
    (extractors / "python" / "errors.jsonl").write_text(
        "\n".join(json.dumps({"severity": "error", "n": i}) for i in range(4))
    )
    assert assess(database)["extractor_errors"] == 4


def test_summary_and_severity_extractors_are_both_counted(tmp_path: Path) -> None:
    """Each extractor contributes by whichever measure it recorded."""
    database = _database(tmp_path, project_files=[f"src/f{i}.c" for i in range(100)])
    extractors = database / "diagnostic" / "extractors"
    (extractors / "cpp").mkdir(parents=True)
    (extractors / "cpp" / "summary.jsonl").write_text(
        json.dumps({"attributes": {"extractor-failures": 2}})
    )
    (extractors / "python").mkdir(parents=True)
    (extractors / "python" / "errors.jsonl").write_text(
        "\n".join(json.dumps({"severity": s}) for s in ("error", "note", "error", "error"))
    )
    assert assess(database)["extractor_errors"] == 5


def test_severity_fallback_counts_only_errors(tmp_path: Path) -> None:
    """With no summary present, count severity `error` and ignore note/warning."""
    database = _database(tmp_path, project_files=[f"src/f{i}.c" for i in range(100)])
    _diag(
        database,
        "records.jsonl",
        "\n".join(
            json.dumps({"severity": s}) for s in ("note", "error", "warning", "error", "note")
        ),
    )
    assert assess(database)["extractor_errors"] == 2


def test_plain_json_diagnostics_are_read_too(tmp_path: Path) -> None:
    """The directory holds `.json` files as well; globbing `*.jsonl` skipped them."""
    database = _database(tmp_path, project_files=[f"src/f{i}.c" for i in range(100)])
    _diag(database, "standalone-extraction.0.json", json.dumps({"severity": "error"}))
    assert assess(database)["extractor_errors"] == 1


def test_unparseable_diagnostics_fail_rather_than_count_zero(tmp_path: Path) -> None:
    """A format change must be visible, not silently reported as a healthy database."""
    database = _database(tmp_path)
    _diag(database, "weird.jsonl", "not json at all\nnor this")
    with pytest.raises(QualityFailure, match="format has changed"):
        assess(database)


def test_absent_extractor_diagnostics_mean_zero_errors(tmp_path: Path) -> None:
    """Absent is genuinely zero here, unlike the cases above where absent data fails.

    Pinned so nobody "fixes" it into a spurious error.
    """
    database = _database(tmp_path)
    assert not (database / "diagnostic").exists()
    assert assess(database)["extractor_errors"] == 0


# --- CLI ---------------------------------------------------------------------------


def test_cli_exits_nonzero_on_bad_database(tmp_path: Path) -> None:
    assert main([str(_database(tmp_path, loc=0))]) == 1


def test_nothing_to_analyse_and_ratio_exceeded_get_different_exits(tmp_path: Path) -> None:
    """The build workflow branches on these, so they must not collapse to "non-zero".

    Exit 1 is unarguable — there is no source to analyse. Exit 2 is a threshold a
    legitimate partial extraction can exceed, and the workflow is allowed to override it
    with --max-error-ratio. Reporting both as 1 would make that override look like the
    same decision as ignoring an empty database.
    """
    assert main([str(_database(tmp_path, project_files=[]))]) == 1
    assert main([str(_database(tmp_path, extractor_errors=3))]) == ERROR_RATIO_EXCEEDED


def test_raising_the_threshold_clears_only_the_ratio_failure(tmp_path: Path) -> None:
    """The escape hatch must not also wave through a database with nothing in it."""
    assert main([str(_database(tmp_path, extractor_errors=3)), "--max-error-ratio", "100"]) == 0
    assert main([str(_database(tmp_path, loc=0)), "--max-error-ratio", "100"]) == 1


def test_json_output_carries_the_metrics_the_docs_read(tmp_path: Path, capsys) -> None:
    """quality-assessment.md reads these keys with jq instead of recomputing them.

    A second hand-written pipeline is how the doc came to log 202 project files while the
    script reported 2 — it filtered a macOS-shaped prefix list rather than the source root.
    """
    assert main([str(_database(tmp_path)), "--format=json"]) == 0
    metrics = json.loads(capsys.readouterr().out)
    assert set(metrics) == {
        "baseline_loc",
        "project_files",
        "archive_files",
        "extractor_errors",
        "error_ratio",
        "finalised",
    }
    assert metrics["project_files"] == len(PROJECT_FILES)
    # archive_files counts the toolchain too, so it is the larger of the two. The doc used
    # to get this from its own `unzip -Z1 | wc -l`.
    assert metrics["archive_files"] > metrics["project_files"]
    assert metrics["finalised"] is True


def test_cli_exits_zero_on_healthy_database(tmp_path: Path) -> None:
    assert main([str(_database(tmp_path))]) == 0


def test_cli_accepts_max_error_ratio(tmp_path: Path) -> None:
    database = _database(tmp_path, extractor_errors=3)
    assert main([str(database)]) == ERROR_RATIO_EXCEEDED
    assert main([str(database), "--max-error-ratio", "100"]) == 0


def test_exit_codes_avoid_argparses_usage_status(tmp_path: Path) -> None:
    """argparse exits 2 on a bad command line, so no verdict of ours may use 2.

    build-database.md tells the reader that the ratio exit means "re-run with a raised
    --max-error-ratio". Sharing 2 with argparse made `--max-error-ratio abc` — which
    inspects nothing — indistinguishable from that verdict.
    """
    assert 2 not in {ERROR_RATIO_EXCEEDED, DIAGNOSTICS_FORMAT_CHANGED}
    with pytest.raises(SystemExit) as usage_error:
        main([str(_database(tmp_path)), "--max-error-ratio", "not-a-number"])
    assert usage_error.value.code == 2


def test_a_non_json_file_in_the_diagnostics_tree_is_ignored(tmp_path: Path) -> None:
    """A future release dropping a .log there must not fail an otherwise good database.

    Every file under diagnostic/extractors used to be handed to the JSON parser, and the
    resulting failure carried the non-overridable "nothing to analyse" code.
    """
    database = _database(tmp_path, project_files=[f"src/f{i}.c" for i in range(100)])
    _diag(database, "records.jsonl", json.dumps({"severity": "error"}))
    (database / "diagnostic" / "extractors" / "cpp" / "extractor.log").write_text(
        "2026-07-31 building ...\nnot json at all\n"
    )
    assert assess(database)["extractor_errors"] == 1


def test_a_malformed_json_diagnostic_still_fails_with_its_own_code(tmp_path: Path) -> None:
    """Narrowing the glob must not silence the format-change signal it was added for."""
    database = _database(tmp_path)
    _diag(database, "weird.jsonl", "not json at all\nnor this")
    assert main([str(database)]) == DIAGNOSTICS_FORMAT_CHANGED


def test_an_unfinalised_database_is_reported_as_such(tmp_path: Path) -> None:
    """An interrupted build leaves the key absent, not false.

    `codeql database finalize` after a failed trace-command is the case: the database
    resolves, analyses, and reports nothing.
    """
    database = _database(tmp_path)
    marker = database / "codeql-database.yml"
    marker.write_text(marker.read_text().replace("finalised: true\n", ""))
    assert is_finalised(database) is False
    assert assess(database)["finalised"] is False

"""Run find_databases.sh, which exists because bash arrays cannot cross Bash calls.

Discovery used to be a loop in SKILL.md whose `FOUND_DBS` array a *different* markdown
block then read. Each block runs in a fresh shell, so the consumer saw an empty array and
reported "No CodeQL database found" for a project that had one. Both callers now run this
and build their own array in their own block.

`codeql resolve database` is stubbed, so no CodeQL install is needed. The stub is what
separates a real database from the marker file a failed build leaves behind, which is the
whole point of the script.
"""

from __future__ import annotations

import stat
import subprocess
from pathlib import Path

SCRIPT = Path(__file__).resolve().parent / "find_databases.sh"


def _marker(directory: Path) -> Path:
    """A codeql-database.yml, as `codeql database create` writes it before the build."""
    directory.mkdir(parents=True, exist_ok=True)
    marker = directory / "codeql-database.yml"
    marker.write_text("primaryLanguage: cpp\n")
    return marker


def _fake_codeql(bin_dir: Path, *, valid: list[str]) -> None:
    """`codeql resolve database -- DIR` succeeds only for the named directories."""
    bin_dir.mkdir(parents=True, exist_ok=True)
    script = bin_dir / "codeql"
    checks = "\n".join(f"  {name!r}) exit 0 ;;".replace("'", '"') for name in valid)
    script.write_text(
        "#!/bin/sh\n"
        'for arg in "$@"; do db="$arg"; done\n'
        'case "$(basename "$db")" in\n'
        f"{checks}\n"
        "  *) exit 1 ;;\n"
        "esac\n"
    )
    script.chmod(script.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH)


def _invoke(tmp_path: Path, *roots: str) -> subprocess.CompletedProcess[str]:
    env = {"PATH": f"{tmp_path / 'bin'}:/usr/bin:/bin", "HOME": str(tmp_path)}
    return subprocess.run(
        [str(SCRIPT), *roots], capture_output=True, text=True, cwd=tmp_path, env=env
    )


def _run(tmp_path: Path, *roots: str) -> list[str]:
    result = _invoke(tmp_path, *roots)
    assert result.returncode == 0, result.stderr
    return result.stdout.split()


def test_a_failed_builds_marker_is_not_reported_as_a_database(tmp_path: Path) -> None:
    """`codeql database create` writes the marker before the build finishes.

    A bare `find` therefore reports a database that does not exist, and the run analyses
    it and finds nothing. Paths come back absolute, so a caller in a different working
    directory can still use them.
    """
    _marker(tmp_path / "good.db")
    _marker(tmp_path / "half-built.db")
    _fake_codeql(tmp_path / "bin", valid=["good.db"])

    assert _run(tmp_path, ".") == [str(tmp_path / "good.db")]


def test_every_database_is_reported_not_just_the_first(tmp_path: Path) -> None:
    """SKILL.md offers the user a choice, so discovery must not stop at one."""
    for name in ("first.db", "second.db", "third.db"):
        _marker(tmp_path / name)
    _fake_codeql(tmp_path / "bin", valid=["first.db", "second.db", "third.db"])

    assert len(_run(tmp_path, ".")) == 3


def test_overlapping_roots_do_not_produce_duplicates(tmp_path: Path) -> None:
    """Callers pass "$OUTPUT_DIR" and "." — the first is usually inside the second.

    Searched as written, the same database appears once as an absolute path and once as
    ./out/codeql.db, which a string dedup cannot collapse; the user is then offered the
    same database twice. Roots are resolved with `pwd -P` before the search.
    """
    _marker(tmp_path / "out" / "codeql.db")
    _fake_codeql(tmp_path / "bin", valid=["codeql.db"])

    assert _run(tmp_path, str(tmp_path / "out"), ".") == [str(tmp_path / "out" / "codeql.db")]


def test_a_database_under_a_dotted_ancestor_is_still_found(tmp_path: Path) -> None:
    """Roots are absolutised, so the old `-not -path '*/.*'` matched dotted *ancestors* too.

    `find` tests -path against the whole path. Once the root became absolute, a checkout
    under ~/.cache, ~/.local, or any dotted parent had every database filtered out: the
    script printed nothing and exited 0, and SKILL.md's auto-detection reads that as "no
    databases found" and rebuilds from scratch.
    """
    root = tmp_path / ".cache" / "builds" / "proj"
    database = root / "static_analysis_codeql_1" / "codeql.db"
    _marker(database)
    _fake_codeql(tmp_path / "bin", valid=["codeql.db"])

    assert _run(tmp_path, str(root)) == [str(database)]


def test_a_dot_directory_below_the_root_is_still_skipped(tmp_path: Path) -> None:
    """The fix must not widen into "search dot-directories".

    A database inside .git or .venv is a stray copy, not the project's own. Offering it in
    the selection prompt spends one of AskUserQuestion's four options on a database nobody
    asked to build.
    """
    _marker(tmp_path / ".git" / "stray.db")
    _marker(tmp_path / "real.db")
    _fake_codeql(tmp_path / "bin", valid=["stray.db", "real.db"])

    assert _run(tmp_path, ".") == [str(tmp_path / "real.db")]


def test_no_databases_prints_nothing_and_succeeds(tmp_path: Path) -> None:
    """The caller decides what an empty list means; this is not an error here."""
    _fake_codeql(tmp_path / "bin", valid=[])
    assert _run(tmp_path, ".") == []


def test_a_missing_codeql_is_an_error_not_an_empty_list(tmp_path: Path) -> None:
    """Otherwise "codeql is not installed" and "this project has no databases" look alike.

    The filter is `codeql resolve database`, so without the binary every candidate is
    rejected and the script prints nothing. SKILL.md's auto-detection table sends an empty
    result to "run the full pipeline", so the user gets a rebuild attempt rather than the
    one line that would have told them what to fix.
    """
    _marker(tmp_path / "good.db")
    (tmp_path / "bin").mkdir(parents=True, exist_ok=True)  # on PATH, but holding no codeql

    result = _invoke(tmp_path, ".")
    assert result.returncode != 0, "a missing codeql must not report success"
    assert "codeql not found" in result.stderr
    assert result.stdout == "", "no database may be reported when none could be validated"

"""Run generate_suite.sh for real, in both modes.

It writes the `.qls` that decides which queries run. Both modes previously lived as
copy-pasted bash inside two reference docs, and both ended with
`if ! codeql resolve queries "$SUITE_FILE" | wc -l; then` — a condition that can never
be true, since `wc` succeeding masks any CodeQL failure.

Runs against a fake `codeql` on PATH, so no CodeQL install is needed and the real
control flow is still exercised.
"""

from __future__ import annotations

import os
import re
import stat
import subprocess
from pathlib import Path

import pytest

SKILL_ROOT = Path(__file__).resolve().parent.parent
BASH_BLOCK = re.compile(r"^```bash\n(.*?)^```", re.MULTILINE | re.DOTALL)

GENERATE_SUITE = SKILL_ROOT / "scripts" / "generate_suite.sh"
YAML_BLOCK = re.compile(r"^```yaml\n(.*?)^```", re.MULTILINE | re.DOTALL)


def _significant(text: str) -> list[str]:
    """Suite lines that carry meaning — comments and blanks are presentation."""
    return [
        ln.rstrip() for ln in text.splitlines() if ln.strip() and not ln.lstrip().startswith("#")
    ]


def _invocation_block(doc: Path) -> str:
    """The bash block that calls generate_suite.sh."""
    blocks = [
        b for b in BASH_BLOCK.findall(doc.read_text(encoding="utf-8")) if "generate_suite.sh" in b
    ]
    if len(blocks) != 1:
        pytest.fail(f"expected one invocation block in {doc.name}, found {len(blocks)}")
    return blocks[0]


# The doc that documents each mode. Both must invoke the script rather than inlining it
# again, or the duplication this script replaced grows back unnoticed.
SUITES = {
    "important-only": SKILL_ROOT / "references" / "important-only-suite.md",
    "run-all": SKILL_ROOT / "references" / "run-all-suite.md",
}


def test_the_docs_invoke_the_script_rather_than_inlining_it() -> None:
    """Guard the guard: these tests exercise the script, not what the docs tell an agent.

    If a doc grows its own `cat > "$SUITE_FILE" << HEADER` again, everything below still
    passes while the skill runs unverified bash.
    """
    for name, doc in sorted(SUITES.items()):
        text = doc.read_text(encoding="utf-8")
        assert f"generate_suite.sh {name}" in text, f"{doc.name} no longer calls the script"
        inlined = [b for b in BASH_BLOCK.findall(text) if "<< HEADER" in b or "<<HEADER" in b]
        assert not inlined, f"{doc.name} inlines suite generation again: {inlined[0][:80]}"


@pytest.mark.parametrize("name", sorted(SUITES))
def test_the_invocation_block_propagates_failure(name: str) -> None:
    """A failed generation must fail the block an agent runs, not just the script.

    `generate_suite.sh` exits non-zero on an unset language and on a suite that resolves
    to nothing. Written with the `SUITE_FILE=` assignment last, the block discards that
    status and returns 0, and the run proceeds to analysis with no suite — or with a
    stale one from an earlier run. Rediscovered here after exactly that shipped.
    """
    block = _invocation_block(SUITES[name])
    assert "set -e" in block, f"{SUITES[name].name}: invocation block does not set -e"
    last = [
        ln.strip() for ln in block.splitlines() if ln.strip() and not ln.strip().startswith("#")
    ][-1]
    assert not re.match(r"[A-Z_]+=", last), (
        f"{SUITES[name].name}: the block ends with the assignment `{last}`, which "
        f"overwrites the script's exit status. Put the script call last."
    )


@pytest.mark.parametrize("name", sorted(SUITES))
def test_the_documented_template_matches_what_the_script_writes(name: str, tmp_path: Path) -> None:
    """The yaml template is a copy of logic that now lives in the script.

    Nothing else compares them: test_suite_templates.py checks the template's shape, so a
    filter changed in one place leaves every test green while the doc documents filters
    nobody runs. Neither resolves against the real CLI — that is a manual check now.
    """
    result = _run(name, tmp_path, env_extra={"CODEQL_LANG": "cpp"})
    assert result.returncode == 0, result.stderr

    blocks = YAML_BLOCK.findall(SUITES[name].read_text(encoding="utf-8"))
    assert len(blocks) == 1, f"expected one yaml template in {SUITES[name].name}"
    documented = _significant(blocks[0].replace("<CODEQL_LANG>", "cpp"))
    written = _significant(_suite_text(tmp_path, name))
    assert documented == written, (
        f"{SUITES[name].name}'s template no longer matches generate_suite.sh.\n"
        f"documented: {documented}\nwritten:    {written}"
    )


@pytest.mark.parametrize("name", sorted(SUITES))
def test_the_doc_names_the_path_the_script_writes(name: str, tmp_path: Path) -> None:
    """The doc assigns SUITE_FILE literally; the script derives it. Two definitions."""
    _run(name, tmp_path, env_extra={"CODEQL_LANG": "cpp"})
    written = next((tmp_path / "out" / "raw").glob("*.qls"))
    documented = f'SUITE_FILE="$OUTPUT_DIR/raw/{name}.qls"'
    assert documented in SUITES[name].read_text(encoding="utf-8"), (
        f"{SUITES[name].name} does not assign {documented}"
    )
    assert written.name == f"{name}.qls", (
        f"script wrote {written.name}, but the doc points at {name}.qls"
    )


def _fake_codeql(directory: Path, *, queries: int = 3, exit_code: int = 0) -> Path:
    """A stand-in for the CodeQL CLI that reports a controlled query count.

    It reads the suite it is handed and rejects one that is missing or empty, the way
    the real CLI would. An earlier version ignored its arguments entirely, so these
    tests would have passed even if the generator had written nothing at all.
    """
    directory.mkdir(parents=True, exist_ok=True)
    script = directory / "codeql"
    payload = "[" + ", ".join(f'"q{i}.ql"' for i in range(queries)) + "]"
    script.write_text(
        "#!/bin/sh\n"
        "# Last argument is the suite path, as `codeql resolve queries ... -- FILE`.\n"
        'for arg in "$@"; do suite="$arg"; done\n'
        'if [ ! -s "$suite" ]; then\n'
        '  echo "fake codeql: no such suite or empty: $suite" >&2\n'
        "  exit 2\n"
        "fi\n"
        'if ! grep -q "^- " "$suite"; then\n'
        '  echo "fake codeql: $suite has no suite entries" >&2\n'
        "  exit 2\n"
        "fi\n"
        f"if [ {exit_code} -ne 0 ]; then\n"
        '  echo "fake codeql failure" >&2\n'
        f"  exit {exit_code}\n"
        "fi\n"
        f"echo '{payload}'\n"
    )
    script.chmod(script.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH)
    return script


def _fake_uv(directory: Path) -> Path:
    """`uv run SCRIPT ARGS...` -> `python3 SCRIPT ARGS...`.

    The generation scripts invoke the guard via `uv run`. Relying on the real uv would
    make these tests depend on the ambient environment: CI runs pytest inside `uv run`,
    where uv is not on the child's PATH, so the suite passed locally and failed there.
    """
    directory.mkdir(parents=True, exist_ok=True)
    script = directory / "uv"
    # allow-legacy-python: a test stub standing in for uv itself, run with a controlled PATH.
    script.write_text(
        "#!/bin/sh\n"
        '[ "$1" = "run" ] && shift\n'
        "# Drop uv-only flags so the remaining argv is the script and its arguments.\n"
        "while [ $# -gt 0 ]; do\n"
        '  case "$1" in --no-project|--quiet|-q) shift ;; --with) shift 2 ;; *) break ;; esac\n'
        "done\n"
        'exec python3 "$@"\n'
    )
    script.chmod(script.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH)
    return script


def _run(
    mode: str, tmp_path: Path, *, env_extra: dict[str, str], **fake
) -> subprocess.CompletedProcess[str]:
    bin_dir = tmp_path / "bin"
    _fake_codeql(bin_dir, **fake)
    _fake_uv(bin_dir)
    output_dir = tmp_path / "out"
    (output_dir / "raw").mkdir(parents=True, exist_ok=True)

    env = {
        "PATH": f"{bin_dir}{os.pathsep}{os.environ['PATH']}",
        "HOME": str(tmp_path),
        "OUTPUT_DIR": str(output_dir),
        **env_extra,
    }
    return subprocess.run(
        [str(GENERATE_SUITE), mode], capture_output=True, text=True, env=env, cwd=tmp_path
    )


def _suite_text(tmp_path: Path, name: str) -> str:
    matches = list((tmp_path / "out" / "raw").glob("*.qls"))
    assert matches, f"{name}: no .qls was written"
    return matches[0].read_text()


@pytest.mark.parametrize("name", sorted(SUITES))
def test_generates_a_suite_naming_the_language(name: str, tmp_path: Path) -> None:
    result = _run(name, tmp_path, env_extra={"CODEQL_LANG": "cpp"})
    assert result.returncode == 0, f"{name} generation failed:\n{result.stderr}"

    text = _suite_text(tmp_path, name)
    assert "codeql/cpp-queries" in text, f"{name}: CODEQL_LANG did not expand"
    assert "${CODEQL_LANG}" not in text, f"{name}: placeholder left unexpanded"


@pytest.mark.parametrize("name", sorted(SUITES))
def test_third_party_packs_are_included(name: str, tmp_path: Path) -> None:
    result = _run(
        name,
        tmp_path,
        env_extra={
            "CODEQL_LANG": "cpp",
            "INSTALLED_THIRD_PARTY_PACKS": "trailofbits/cpp-queries acme/cpp-queries",
        },
    )
    assert result.returncode == 0, result.stderr

    text = _suite_text(tmp_path, name)
    assert "trailofbits/cpp-queries" in text
    assert "acme/cpp-queries" in text, "second pack dropped — the loop only handled one"


@pytest.mark.parametrize("name", sorted(SUITES))
def test_generation_aborts_when_codeql_fails(name: str, tmp_path: Path) -> None:
    """The regression guard. This failed before the `| wc -l` pipeline was replaced."""
    result = _run(
        name,
        tmp_path,
        env_extra={"CODEQL_LANG": "cpp"},
        exit_code=2,
    )
    assert result.returncode != 0, (
        f"{name}: CodeQL failed but the script exited 0. A suite that cannot be resolved "
        f"will produce an empty SARIF that reads as a clean scan."
    )


@pytest.mark.parametrize("name", sorted(SUITES))
def test_generation_aborts_when_suite_resolves_to_zero_queries(name: str, tmp_path: Path) -> None:
    """Zero queries is the failure Essential Principle #3 exists to prevent."""
    result = _run(
        name,
        tmp_path,
        env_extra={"CODEQL_LANG": "cpp"},
        queries=0,
    )
    assert result.returncode != 0, (
        f"{name}: suite resolved to zero queries and the script exited 0 — analysis would "
        f"run and report no findings for a codebase it never examined."
    )


@pytest.mark.parametrize("name", sorted(SUITES))
def test_a_suite_that_fails_verification_is_deleted(name: str, tmp_path: Path) -> None:
    """Left on disk it is indistinguishable from a verified one.

    run-analysis Step 4 derives the same path, and the docs tell the model the generator
    already verified the suite — so the file that just failed gets analysed with, and the
    empty SARIF reads as a clean codebase.
    """
    result = _run(name, tmp_path, env_extra={"CODEQL_LANG": "cpp"}, queries=0)
    assert result.returncode != 0

    left = list((tmp_path / "out" / "raw").glob("*.qls"))
    assert not left, f"{name}: unverifiable suite left behind at {left}"


@pytest.mark.parametrize("name", sorted(SUITES))
def test_unset_language_aborts_before_writing_a_suite(name: str, tmp_path: Path) -> None:
    """The guard must precede the heredoc.

    Checked afterwards, an unset CODEQL_LANG leaves a suite naming `codeql/-queries` on
    disk for a later step to pick up.
    """
    result = _run(name, tmp_path, env_extra={})
    assert result.returncode != 0, f"{name}: unset CODEQL_LANG did not abort"

    for written in (tmp_path / "out" / "raw").glob("*.qls"):
        assert "codeql/-queries" not in written.read_text(), (
            f"{name}: wrote a suite naming the empty pack `codeql/-queries` before the "
            f"guard fired — the guard is still after the heredoc"
        )

"""Every flag the docs pass to this plugin's own scripts must be a flag they accept.

`codeql-build.js` shipped `check_db_quality.py --json`. The script takes
`--format {text,json}`, so argparse exited 2 on the Assess phase's first command --
and 2 is the one exit code the workflow's schema does not describe, so the failure
read as an unknown status rather than a typo.

Nothing caught it because the two call sites live in different trees:
`test_shell_blocks.py` lints markdown under the skill, and `codeql-build.js` sits
outside it under the plugin's `workflows/`. This check scans the whole plugin, both
file types, and asks the scripts themselves what they accept.

Hermetic: the scripts are stdlib-only, so `--help` runs under the ambient
interpreter. No CodeQL, no uv, no network.
"""

from __future__ import annotations

import re
import subprocess
import sys
from pathlib import Path

import pytest

SCRIPTS_DIR = Path(__file__).resolve().parent
PLUGIN_ROOT = SCRIPTS_DIR.parent.parent.parent
SKIP_DIRS = {".pytest_cache", "__pycache__", ".venv", ".ruff_cache"}

# Anything after one of these belongs to the next command, not to the script.
COMMAND_END = re.compile(r"[|;)`]|&&|\|\|")
LONG_FLAG = re.compile(r"--[A-Za-z][A-Za-z0-9-]*")


def _argparse_scripts() -> list[Path]:
    """The scripts that parse flags, discovered rather than listed.

    A hardcoded list stops covering a script the moment someone adds one.
    """
    return sorted(
        p
        for p in SCRIPTS_DIR.glob("*.py")
        if not p.name.startswith("test_") and "add_argument" in p.read_text(encoding="utf-8")
    )


def _accepted_flags(script: Path) -> set[str]:
    """What the script really takes, from its own --help."""
    result = subprocess.run(
        [sys.executable, str(script), "--help"],
        capture_output=True,
        text=True,
        timeout=60,
        check=False,
    )
    assert result.returncode == 0, (
        f"{script.name} --help exited {result.returncode}, so its accepted flags could "
        f"not be read:\n{result.stderr}"
    )
    return set(LONG_FLAG.findall(result.stdout))


ACCEPTED = {script.name: _accepted_flags(script) for script in _argparse_scripts()}


def _doc_files() -> list[Path]:
    return sorted(
        p
        for p in PLUGIN_ROOT.rglob("*")
        if p.suffix in {".md", ".js"}
        and not SKIP_DIRS.intersection(p.parts)
        and not p.name.startswith("test_")
    )


# An invocation names the script by path — `{baseDir}/scripts/x.py`, `${SKILL_DIR}/scripts/x.py`.
# A bare mention is prose: a comment reading "the workflow passed check_db_quality.py a --json
# flag" is not a call site, and reporting it as one made this check fail on its own changelog.
INVOCATION = re.compile(r"/(" + "|".join(re.escape(n) for n in ACCEPTED) + r")(?=\s|$)")


def _invocations() -> list[tuple[Path, int, str, str]]:
    """Every (file, line, script, flag) the plugin documents.

    Flags are read from the text between the script name and the end of that command,
    so the `jq -r` in `$(… --format=json | jq -r '.x')` is not mistaken for the
    script's own.
    """
    found: list[tuple[Path, int, str, str]] = []
    for path in _doc_files():
        for lineno, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
            for match in INVOCATION.finditer(line):
                tail = line[match.end() :]
                end = COMMAND_END.search(tail)
                if end:
                    tail = tail[: end.start()]
                for flag in LONG_FLAG.findall(tail):
                    found.append((path, lineno, match.group(1), flag))
    return found


ALL_INVOCATIONS = _invocations()


def test_scripts_were_discovered() -> None:
    """Guard the guard: no scripts found turns every check below into a no-op."""
    assert len(ACCEPTED) >= 2, (
        f"only {len(ACCEPTED)} argparse scripts found in {SCRIPTS_DIR} — discovery is "
        f"broken, not the skill"
    )
    for name, flags in ACCEPTED.items():
        assert flags, f"{name} --help listed no long flags, so nothing can be checked against it"


def test_flagged_invocations_were_found() -> None:
    """A regex that matches nothing would pass this file silently.

    Deliberately low: deduplicating the docs legitimately removes call sites, and this
    guard is here to catch a scanner that finds nothing at all, not to pin a count.
    """
    assert len(ALL_INVOCATIONS) >= 2, (
        f"only {len(ALL_INVOCATIONS)} flagged script invocations extracted from "
        f"{PLUGIN_ROOT} — the scanner is broken, not the docs"
    )


def test_the_workflow_is_in_scope() -> None:
    """The file the original bug shipped in must be one of the files scanned.

    It lives outside the skill tree, which is why the markdown linters missed it.
    """
    workflow = PLUGIN_ROOT / "workflows" / "codeql-build.js"
    assert workflow.exists(), f"{workflow} is gone — update this check to match"
    assert workflow in _doc_files(), (
        f"{workflow.name} is no longer scanned, so a bad flag in the workflow would ship again"
    )


@pytest.mark.parametrize(
    ("path", "line", "script", "flag"),
    ALL_INVOCATIONS,
    ids=[f"{p.relative_to(PLUGIN_ROOT)}:{ln}:{flag}" for p, ln, _, flag in ALL_INVOCATIONS],
)
def test_documented_flag_exists(path: Path, line: int, script: str, flag: str) -> None:
    accepted = ACCEPTED[script]
    assert flag in accepted, (
        f"{path.relative_to(PLUGIN_ROOT)}:{line} passes {flag} to {script}, which does "
        f"not accept it. argparse exits 2 on an unrecognised flag. Accepted: "
        f"{', '.join(sorted(accepted))}"
    )


def test_the_scanner_tells_an_invocation_from_a_mention() -> None:
    """Both directions, because getting either wrong disables the check quietly.

    A prose mention counted as a call site makes the check fail on documentation that
    merely discusses a flag. A real call site missed makes it pass on a broken command.
    """
    script = next(iter(ACCEPTED))
    assert not INVOCATION.search(f"the workflow passed {script} a --json flag"), (
        "a bare mention in prose is being read as an invocation"
    )
    for real in (
        f'uv run {{baseDir}}/scripts/{script} "$DB_NAME" --format=json',
        f'  uv run ${{SKILL_DIR}}/scripts/{script} "$DB_NAME" --format=json',
    ):
        assert INVOCATION.search(real), f"a real invocation is not being scanned: {real}"

"""A pointer to a section must land on a section that exists.

Deduplicating the docs replaced inline copies with pointers: the workflow says
`Follow "Method 1: Autobuild" in .../build-database.md` instead of restating the
command, and build-database.md links back to SKILL.md for the fresh-shell rule
rather than repeating it. That trade is only worth making if the pointers are real.

The repo validator resolves file paths, not section names or anchors, so a heading
renamed by one character leaves a pointer aimed at nothing and every check still
green. Both forms are covered here:

    [text](../SKILL.md#each-bash-call-is-a-fresh-shell)   markdown anchor
    Follow "Method 1: Autobuild" in <path>/build-database.md   prose pointer
"""

from __future__ import annotations

import re
from pathlib import Path

import pytest

SCRIPTS_DIR = Path(__file__).resolve().parent
SKILL_ROOT = SCRIPTS_DIR.parent
PLUGIN_ROOT = SKILL_ROOT.parent.parent
SKIP_DIRS = {".pytest_cache", "__pycache__", ".venv", ".ruff_cache"}

WORKFLOW_JS = PLUGIN_ROOT / "workflows" / "codeql-build.js"

# `codeql-build.js` cannot know where the skill is installed, so its pointers are written
# against a variable the Detect agent resolves at runtime — `${SKILL_DIR}/references/x.md` in
# a template literal, `$SKILL_DIR/references/x.md` in the one prompt that runs before it is
# known. Both spellings mean this directory.
RUNTIME_ROOT_VAR = "SKILL_DIR"
RUNTIME_PREFIX = re.compile(r"^\$\{?" + RUNTIME_ROOT_VAR + r"\}?/")

HEADING = re.compile(r"^#{1,6}\s+(.+?)\s*$", re.MULTILINE)
# [label](path.md#anchor) — only links carrying an anchor.
ANCHOR_LINK = re.compile(r"\[[^\]]*\]\(([^)#\s]+\.md)#([A-Za-z0-9_-]+)\)")
# `"Section Name" in <something>.md`, the form the workflow prompts use.
PROSE_POINTER = re.compile(r"[\"“]([^\"”\n]{3,80})[\"”]\s+in\s+(\S+?\.md)\b")


def _doc_files() -> list[Path]:
    return sorted(
        p
        for p in PLUGIN_ROOT.rglob("*")
        if p.suffix in {".md", ".js"}
        and not SKIP_DIRS.intersection(p.parts)
        and not p.name.startswith("test_")
    )


def _slug(heading: str) -> str:
    """GitHub's anchor rule: lowercase, drop punctuation, spaces to hyphens."""
    text = heading.strip().lower()
    text = re.sub(r"[^\w\s-]", "", text)
    return re.sub(r"\s+", "-", text).strip("-")


def _headings(path: Path) -> list[str]:
    return HEADING.findall(path.read_text(encoding="utf-8"))


def _js_constants(source: str) -> dict[str, str]:
    """Resolve the path constants a workflow builds its pointers from.

    Read from the source rather than hardcoded, so renaming SKILL_DIR does not quietly
    turn every pointer in that file into an unresolvable string this check skips.
    """
    values: dict[str, str] = {}
    for name, raw in re.findall(r"const (\w+) = [`'\"]([^`'\"]+)[`'\"]", source):
        values[name] = re.sub(r"\$\{(\w+)\}", lambda m: values.get(m.group(1), m.group(0)), raw)
    return values


def _resolve(target: str, source_file: Path, constants: dict[str, str]) -> Path | None:
    """Turn a pointer's path into a real file, or None if it is not one we can check."""
    resolved = re.sub(r"\$\{(\w+)\}", lambda m: constants.get(m.group(1), m.group(0)), target)
    # `${SKILL_DIR}/references/x.md` is a real pointer whose root is only known at runtime.
    # Rewriting it onto this checkout keeps it checkable; without this it reads as an
    # unresolvable string and every pointer in the workflow is skipped in silence.
    resolved = RUNTIME_PREFIX.sub(f"{SKILL_ROOT}/", resolved)
    if "${" in resolved or "{baseDir}" in resolved:
        return None
    candidates = [Path(resolved), PLUGIN_ROOT.parent.parent / resolved]
    if not resolved.startswith("/"):
        candidates.insert(0, (source_file.parent / resolved).resolve())
    for candidate in candidates:
        if candidate.is_file():
            return candidate
    return None


def _collect() -> tuple[list[tuple[Path, str, Path, str]], list[tuple[Path, str, Path, str]]]:
    anchors: list[tuple[Path, str, Path, str]] = []
    prose: list[tuple[Path, str, Path, str]] = []
    for path in _doc_files():
        text = path.read_text(encoding="utf-8")
        constants = _js_constants(text) if path.suffix == ".js" else {}
        for target, anchor in ANCHOR_LINK.findall(text):
            resolved = _resolve(target, path, constants)
            if resolved is not None:
                anchors.append((path, anchor, resolved, target))
        for section, target in PROSE_POINTER.findall(text):
            resolved = _resolve(target, path, constants)
            if resolved is not None:
                prose.append((path, section, resolved, target))
    return anchors, prose


ANCHORS, PROSE = _collect()


def test_pointers_were_found() -> None:
    """Guard the guard: two regexes matching nothing would pass this file in silence."""
    assert ANCHORS, f"no anchored markdown links found under {PLUGIN_ROOT} — the scanner broke"
    assert PROSE, f"no prose section pointers found under {PLUGIN_ROOT} — the scanner broke"


def test_the_workflows_pointers_are_still_checked() -> None:
    """The global guard above is satisfied by markdown alone.

    Every pointer in `codeql-build.js` carries a `${SKILL_DIR}` prefix, so one change to how
    that prefix is spelled makes all of them unresolvable — and `_resolve` returning None
    reports that as "nothing to check here", indistinguishable from a file with no pointers.
    """
    from_workflow = [entry for entry in PROSE if entry[0] == WORKFLOW_JS]
    assert len(from_workflow) >= 4, (
        f"only {len(from_workflow)} prose pointers resolved out of {WORKFLOW_JS.name} — its "
        f"paths are no longer being checked against the sections they name"
    )


def test_the_workflow_does_not_hardcode_a_path_into_this_repo() -> None:
    """A repo-relative skill path resolves only in a checkout of the marketplace.

    `codeql-build.js` shipped `const SKILL_DIR = 'plugins/static-analysis/skills/codeql'`.
    Installed, the first build block sourced a `build_log.sh` that was not there, exited 127,
    and every rung of the ladder reported a build failure for a project that builds. The path
    is now resolved at runtime by the Detect phase; nothing else in this suite would notice it
    being written back down, because CI runs from the repo where the literal does resolve.
    """
    source = WORKFLOW_JS.read_text(encoding="utf-8")
    offenders = [
        line.strip()
        for line in source.splitlines()
        if "plugins/static-analysis" in line and not line.lstrip().startswith("//")
    ]
    assert not offenders, (
        "the workflow names its own path inside this repository, which only resolves when the "
        "run starts in a checkout of it:\n  " + "\n  ".join(offenders)
    )


@pytest.mark.parametrize(
    ("source", "anchor", "target", "raw"),
    ANCHORS,
    ids=[f"{s.relative_to(PLUGIN_ROOT)}->{raw}#{a}" for s, a, _, raw in ANCHORS],
)
def test_anchor_link_lands_on_a_heading(source: Path, anchor: str, target: Path, raw: str) -> None:
    slugs = {_slug(h) for h in _headings(target)}
    assert anchor in slugs, (
        f"{source.relative_to(PLUGIN_ROOT)} links to {raw}#{anchor}, but that file has no "
        f"heading with that anchor. Available: {', '.join(sorted(slugs))}"
    )


@pytest.mark.parametrize(
    ("source", "section", "target", "raw"),
    PROSE,
    ids=[f"{s.relative_to(PLUGIN_ROOT)}->{sec}" for s, sec, _, raw in PROSE],
)
def test_prose_pointer_names_a_real_section(
    source: Path, section: str, target: Path, raw: str
) -> None:
    """Prefix match: "Method 4: No-Build Fallback" may point at a heading that goes on
    to say "(Last Resort)". A pointer that is not even a prefix is aimed at nothing."""
    headings = _headings(target)
    assert any(h.startswith(section) for h in headings), (
        f'{source.relative_to(PLUGIN_ROOT)} points at "{section}" in {raw}, which has no '
        f"heading starting with that. Headings: {'; '.join(headings)}"
    )

"""Lint the shell that lives inside this skill's markdown.

`make shell` covers `*.sh` files; this skill ships none, so nothing checked its
commands. That gap hid one bug class in nine places: a pipeline's exit status used as
a success test, which without `pipefail` belongs to the formatter, not the command.
"""

from __future__ import annotations

import re
import subprocess
from pathlib import Path

import pytest

SKILL_ROOT = Path(__file__).resolve().parent.parent
BASH_BLOCK = re.compile(r"^```bash\n(.*?)^```", re.MULTILINE | re.DOTALL)

# Defined in build_log.sh alongside `set -o pipefail`, so it preserves exit status by
# construction; callers need not repeat the setting.
SAFE_PIPE_WRAPPER = "run_logged"

# A single `|`. `if [ -f a ] || [ -f b ]` is a logical or, not a pipeline, and matching it
# made two blocks look like offenders that the whole-block exemption then waved through.
PIPE = r"(?<!\|)\|(?!\|)"
PIPED_TO_TEE = re.compile(PIPE + r"\s*tee\b")
PIPELINE_IN_CONDITION = re.compile(r"if\s+!?\s*[^|]*" + PIPE)
# The line itself runs under the wrapper, wherever it sits in the block.
WRAPPED = re.compile(r"^(if\s+!?\s*)?" + SAFE_PIPE_WRAPPER + r"\b")
# Sourcing the helpers sets pipefail for the rest of the block, exactly as writing it out
# would. Matching the bare token `run_logged` anywhere in the block did not mean that.
SOURCES_LOG_HELPERS = re.compile(r"^\s*(\.|source)\s+.*build_log\.sh")


def _status_consuming_pipelines(source: str) -> list[str]:
    """Lines whose pipeline *exit status* is consumed.

    `VAR=$(cmd | wc -l)` consumes output and is fine. `cmd | tee "$LOG"` as a statement,
    or a pipeline inside an `if`, decides what happens next.
    """
    found: list[str] = []
    for raw in source.splitlines():
        stripped = raw.strip()
        if stripped.startswith("#"):
            continue
        # Output capture — status is not what is being used.
        if re.search(r"(\$\(|`|=\s*\$\()", stripped) and not stripped.startswith("if "):
            continue
        if PIPED_TO_TEE.search(stripped) or PIPELINE_IN_CONDITION.match(stripped):
            found.append(stripped)
    return found


def _sets_pipefail(source: str) -> bool:
    """pipefail is block-scoped, so set it directly or inherit it by sourcing the helpers."""
    return (
        "set -o pipefail" in source
        or "set -euo pipefail" in source
        or any(SOURCES_LOG_HELPERS.match(raw) for raw in source.splitlines())
    )


def _unpreserved_pipelines(source: str) -> list[str]:
    """Status-consuming pipelines in `source` that read the formatter's status instead.

    The wrapper exemption is applied per line. Applied per block — "does `run_logged`
    appear anywhere in this source" — a bare `cmd | tee "$LOG_FILE"` added later to a
    block that already used the wrapper elsewhere passed silently, which is the exact
    regression this check exists to catch.
    """
    if _sets_pipefail(source):
        return []
    return [line for line in _status_consuming_pipelines(source) if not WRAPPED.match(line)]


def _markdown_files() -> list[Path]:
    return sorted(p for p in SKILL_ROOT.rglob("*.md") if ".pytest_cache" not in p.parts)


def _blocks() -> list[tuple[Path, int, str]]:
    """Every bash block in the skill, as (file, line number, source)."""
    found: list[tuple[Path, int, str]] = []
    for path in _markdown_files():
        text = path.read_text(encoding="utf-8")
        for match in BASH_BLOCK.finditer(text):
            line = text.count("\n", 0, match.start()) + 1
            found.append((path, line, match.group(1)))
    return found


ALL_BLOCKS = _blocks()


def _ident(path: Path, line: int) -> str:
    return f"{path.relative_to(SKILL_ROOT)}:{line}"


def test_extraction_found_blocks() -> None:
    """Guard the guard: a broken regex would turn every test below into a no-op."""
    assert len(ALL_BLOCKS) >= 20, (
        f"only {len(ALL_BLOCKS)} bash blocks extracted from {SKILL_ROOT} — the extractor "
        f"is broken, not the skill"
    )


def test_every_block_is_syntactically_valid() -> None:
    """Every block must parse. Catches unterminated heredocs and quoting errors."""
    offenders = []
    for path, line, source in ALL_BLOCKS:
        # <BUILD_CMD>, <mode> etc. are documentation, not shell — neutralise before parsing.
        cleaned = re.sub(r"<[A-Za-z0-9_ .-]+>", "PLACEHOLDER", source)
        result = subprocess.run(["bash", "-n"], input=cleaned, capture_output=True, text=True)
        if result.returncode != 0:
            offenders.append(f"{_ident(path, line)}: {result.stderr.strip()}")

    assert not offenders, "a block is not valid bash:\n  " + "\n  ".join(offenders)


def test_no_block_consumes_an_unpreserved_pipeline_status() -> None:
    """A pipeline whose *exit status* is consumed must preserve the real one.

    Without pipefail that status is the formatter's, which is always 0. No block ships a
    bare pipeline today — every one goes through `run_logged` — so this is a tripwire for
    new markdown, and it scans every block in one test rather than parametrizing over all
    of them to skip the ones with no pipeline. What proves the detector still fires is
    `test_detector_flags_unsafe_pipelines` below.
    """
    offenders = [
        f"{_ident(path, line)}: {offender}"
        for path, line, source in ALL_BLOCKS
        for offender in _unpreserved_pipelines(source)
    ]
    assert not offenders, (
        f"a block decides control flow on a pipeline's exit status without "
        f"`set -o pipefail` or {SAFE_PIPE_WRAPPER}, so it reads the formatter's status "
        f"(always 0) rather than the command's:\n  " + "\n  ".join(offenders)
    )


# The detector matches no block in the skill today, so these fixtures are the only thing
# that would notice it silently breaking. Kept as tables inside two tests rather than
# parametrized: seven pytest cases for one detector was more ceremony than it earns.
_BARE_TEE = 'codeql database analyze "$DB_NAME" suite.qls 2>&1 | tee -a "$LOG_FILE"'

MUST_FLAG = (
    ('codeql database create "$DB_NAME" --language=cpp 2>&1 | tee -a "$LOG_FILE"', None),
    ('if codeql resolve queries "$SUITE_FILE" | grep -q "\\.ql"; then echo ok; fi', None),
    # The regression the per-line exemption exists for: under the old whole-block test
    # `run_logged` appearing anywhere waved through every pipeline in the block.
    (f'run_logged codeql database create "$DB_NAME" --language=cpp\n{_BARE_TEE}\n', [_BARE_TEE]),
)

MUST_PASS = (
    # Output capture: the pipeline's status is never read.
    'COUNT=$(codeql resolve queries "$SUITE_FILE" | wc -l)',
    # Logical or, not a pipeline.
    "if [ -f setup.py ] || [ -f pyproject.toml ]; then echo python; fi",
    # The wrapper sets pipefail itself.
    'run_logged codeql database create "$DB_NAME" --language=cpp',
    # `. build_log.sh` runs `set -o pipefail`, so the rest of the block is genuinely safe.
    f'. "{{baseDir}}/scripts/build_log.sh"\n{_BARE_TEE}\n',
)


def test_detector_flags_unsafe_pipelines() -> None:
    """Guard the guard: with no offending block left in the skill, nothing else would
    notice if this detector stopped matching."""
    for source, expected in MUST_FLAG:
        assert _unpreserved_pipelines(source) == (expected or [source]), (
            f"the detector stopped flagging:\n  {source}"
        )


def test_detector_passes_safe_pipelines() -> None:
    """False positives get silenced, and a silenced check catches nothing."""
    for source in MUST_PASS:
        assert _unpreserved_pipelines(source) == [], (
            f"the detector now fires on safe shell, which is how it gets disabled:\n  {source}"
        )


# Everything build_log.sh defines. Used in a block that never sourced the file, a helper is
# 127 and LOG_FILE is an append to "".
LOG_HELPERS = ("run_logged", "log_step", "log_cmd", "log_result", "LOG_FILE")
USES_LOG_HELPER = re.compile(r"\b(" + "|".join(LOG_HELPERS) + r")\b")


def _unsourced_helper_uses(source: str) -> list[str]:
    """Lines that use a build_log.sh helper in a block that never sources build_log.sh."""
    if any(SOURCES_LOG_HELPERS.match(raw) for raw in source.splitlines()):
        return []
    lines = (raw.strip() for raw in source.splitlines())
    return [line for line in lines if not line.startswith("#") and USES_LOG_HELPER.search(line)]


def test_helper_uses_are_sourced_in_the_same_block() -> None:
    """A helper is a function, and a function does not survive into the next Bash call.

    Same rule as the array check below, higher cost: 127 is a non-zero status, so the ladder
    in build-database.md records the method as failed and walks to the next one. Method 2m-a
    was the case that motivated this: the first rung an affected Mac lands on called
    `log_step` without sourcing the helpers.
    """
    offenders = [
        f"{_ident(path, line)}: {offender}"
        for path, line, source in ALL_BLOCKS
        for offender in _unsourced_helper_uses(source)
    ]
    assert not offenders, (
        "a block uses a build_log.sh helper without sourcing it. Each block is a separate "
        "Bash call, so the helper is undefined there and the line exits 127. Add `. "
        '"{baseDir}/scripts/build_log.sh" || exit 1` to the block:\n  ' + "\n  ".join(offenders)
    )


HELPERS_MUST_FLAG = (
    'log_step "METHOD 2m-a: macOS arm64 Homebrew compiler"',
    'run_logged codeql database create "$DB_NAME" --language=cpp',
    # Not a function, but build_log.sh is what defaults it. Unsourced, this appends to "".
    'echo "=== Build Complete ===" >> "$LOG_FILE"',
)

HELPERS_MUST_PASS = (
    '. "{baseDir}/scripts/build_log.sh" || exit 1\nlog_step "building"',
    "source {baseDir}/scripts/build_log.sh\nrun_logged make",
    # A comment about the helpers, not a call. Blocks carry those.
    "# run_logged returns the build's exit status. Check it before moving on.",
)


def test_detector_flags_unsourced_helpers() -> None:
    """Guard the guard: no block violates the rule now, so nothing else would notice this
    detector going quiet."""
    for source in HELPERS_MUST_FLAG:
        assert _unsourced_helper_uses(source) == [source], (
            f"the detector stopped flagging:\n  {source}"
        )


def test_detector_passes_sourced_helpers() -> None:
    """False positives get silenced, and a silenced check catches nothing."""
    for source in HELPERS_MUST_PASS:
        assert _unsourced_helper_uses(source) == [], (
            f"the detector now fires on a block that sources the helpers, which is how it "
            f"gets disabled:\n  {source}"
        )


def test_the_sourcing_rule_has_something_to_check() -> None:
    """Guard the guard: if the token list or the block extractor broke, every block would
    look helper-free and the check above would inspect nothing."""
    users = [
        _ident(path, line)
        for path, line, source in ALL_BLOCKS
        if any(USES_LOG_HELPER.search(raw) for raw in source.splitlines())
    ]
    assert len(users) >= 10, (
        f"only {len(users)} blocks use a build_log.sh helper across {len(ALL_BLOCKS)} blocks "
        f"in {SKILL_ROOT} — the detector is broken, not the skill"
    )


# `codeql database finalize` after a failed `trace-command` writes a database that resolves
# and holds nothing, so the build method reports success. Something has to gate it: a test on
# the same line, or an enclosing `if` the step already set, which shows as indentation.
FINALIZE = "codeql database finalize"
GATED_ON_THE_LINE = re.compile(r"^(if|elif)\s+!|&&")


def _ungated_finalizes(source: str) -> list[str]:
    """`codeql database finalize` calls that run whatever the step before them returned."""
    found: list[str] = []
    for raw in source.splitlines():
        stripped = raw.strip()
        if FINALIZE not in stripped or stripped.startswith("#"):
            continue
        # Indented means it sits inside an enclosing block; 2m-a's is under `if $TRACE_OK`.
        if raw != stripped or GATED_ON_THE_LINE.search(stripped):
            continue
        found.append(stripped)
    return found


def test_finalize_is_gated_on_the_step_before_it() -> None:
    """A build method that finalizes a database it failed to populate reports success.

    `build_log.sh` deliberately omits `set -e` so the ladder survives a failed method, which
    leaves each block responsible for its own sequencing. Method 3 in build-database.md ran
    four `run_logged` calls in a row, so a `trace-command` that failed on build step 2 still
    reached `finalize`, and `codeql resolve database` then succeeded.
    """
    offenders = [
        f"{_ident(path, line)}: {offender}"
        for path, line, source in ALL_BLOCKS
        for offender in _ungated_finalizes(source)
    ]
    assert not offenders, (
        "a block finalizes a database without checking that the step before it succeeded, so "
        "a failed trace-command still produces a database that resolves and holds nothing:\n  "
        + "\n  ".join(offenders)
    )


FINALIZE_MUST_FLAG = (
    'run_logged codeql database finalize "$DB_NAME"',
    'codeql database finalize "$DB_NAME"',
)

FINALIZE_MUST_PASS = (
    'elif ! run_logged codeql database finalize "$DB_NAME"; then',
    'if ! run_logged codeql database finalize "$DB_NAME"; then',
    # 2m-a's shape: the trace loop sets TRACE_OK, and the finalize sits inside that test.
    'if $TRACE_OK; then\n  run_logged codeql database finalize "$DB_NAME"\nfi',
)


def test_detector_flags_ungated_finalize() -> None:
    """Guard the guard: two call sites, both gated now, so nothing else would notice this
    detector going quiet."""
    for source in FINALIZE_MUST_FLAG:
        assert _ungated_finalizes(source) == [source], f"the detector stopped flagging:\n  {source}"


def test_detector_passes_gated_finalize() -> None:
    """False positives get silenced, and a silenced check catches nothing."""
    for source in FINALIZE_MUST_PASS:
        assert _ungated_finalizes(source) == [], (
            f"the detector now fires on a gated finalize, which is how it gets disabled:\n"
            f"  {source}"
        )


def test_the_finalize_rule_has_something_to_check() -> None:
    """Guard the guard: no `finalize` anywhere means the check above inspects nothing."""
    sites = [_ident(path, line) for path, line, source in ALL_BLOCKS if FINALIZE in source]
    assert len(sites) >= 2, (
        f"only {len(sites)} `{FINALIZE}` call sites found across {len(ALL_BLOCKS)} blocks in "
        f"{SKILL_ROOT} — the detector is broken, not the skill"
    )


def test_exit_status_is_not_captured_after_a_pipe() -> None:
    """`EXIT_CODE=$?` after a pipeline captures the wrong process.

    This is how the arm64e exit-137 check was neutered.
    """
    offenders = []
    for path, line, source in ALL_BLOCKS:
        if "set -o pipefail" in source or "set -euo pipefail" in source:
            continue
        lines = source.splitlines()
        for index, raw in enumerate(lines[:-1]):
            if raw.strip().startswith("#") or "|" not in raw:
                continue
            if re.search(r"\|\s*(tee|wc|head|tail)\b", raw) and re.match(
                r"\s*\w+=\$\?", lines[index + 1]
            ):
                offenders.append(
                    f"{_ident(path, line)}: {raw.strip()} / {lines[index + 1].strip()}"
                )

    assert not offenders, (
        "a block captures $? straight after a pipe without pipefail, so it records the "
        "formatter's status rather than the command's:\n  " + "\n  ".join(offenders)
    )


def test_no_unquoted_command_string_expansion() -> None:
    """`CMD="a b c"` then `$CMD` word-splits on any path containing a space."""
    offenders = [
        f"{_ident(path, line)}: {raw.strip()}"
        for path, line, source in ALL_BLOCKS
        for raw in source.splitlines()
        if re.match(r"\s*\$(CMD|BUILD_CMD)\b", raw) and not raw.strip().startswith("#")
    ]
    assert not offenders, (
        "a block runs a command built as a string, unquoted. Pass argv as a list (see "
        "run_logged) so paths with spaces survive:\n  " + "\n  ".join(offenders)
    )


# Variables that hold a filesystem path. An unquoted expansion of any of these splits on
# a space, so `$OUTPUT_DIR` under "~/My Scans" silently targets the wrong path.
PATH_VARS = (
    "DB_NAME",
    "OUTPUT_DIR",
    "SUITE_FILE",
    "RAW_DIR",
    "RESULTS_DIR",
    "LOG_FILE",
    "DIAG_DIR",
)
UNQUOTED_PATH_VAR = re.compile(r"\$\{?(" + "|".join(PATH_VARS) + r")\}?(?![\w\"])")


def _outside_heredoc(source: str):
    """Yield (line, is_shell) — heredoc bodies are literal text, not shell."""
    terminator: str | None = None
    for raw in source.splitlines():
        if terminator is not None:
            if raw.strip() == terminator:
                terminator = None
            yield raw, False
            continue
        opener = re.search(r"<<-?\s*'?([A-Za-z_][A-Za-z0-9_]*)'?", raw)
        yield raw, True
        if opener:
            terminator = opener.group(1)


def test_path_variables_are_quoted() -> None:
    """Every path variable must be quoted at the point of use.

    The narrower CMD/BUILD_CMD check above missed the production command:
    `codeql database analyze $DB_NAME` sits in run-analysis Step 4 and breaks on any
    output directory containing a space.
    """
    offenders = []
    for path, line, source in ALL_BLOCKS:
        for raw, is_shell in _outside_heredoc(source):
            stripped = raw.strip()
            if not is_shell or stripped.startswith("#"):
                continue
            for match in UNQUOTED_PATH_VAR.finditer(raw):
                # Inside double quotes is fine; count quotes before the match to tell.
                if raw[: match.start()].count('"') % 2 == 1:
                    continue
                offenders.append(f"{_ident(path, line)}: {stripped}")
                break

    assert not offenders, (
        "a block expands a path variable unquoted, so it word-splits on any path "
        "containing a space:\n  " + "\n  ".join(offenders)
    )


ARRAY_ASSIGN = re.compile(r"^\s*([A-Z_][A-Z0-9_]*)=\(")

# Array names assigned anywhere in the skill's bash blocks.
ARRAY_NAMES = frozenset(
    m.group(1)
    for _, _, blk in ALL_BLOCKS
    for raw in blk.splitlines()
    if (m := ARRAY_ASSIGN.match(raw))
)


def test_array_names_were_found() -> None:
    """Guard the guard: with no names collected, the scan below inspects nothing and
    passes. A skip here would report that as success."""
    assert ARRAY_NAMES, (
        f"no `NAME=()` assignment found in any of the {len(ALL_BLOCKS)} blocks in "
        f"{SKILL_ROOT} — the collector is broken, not the skill"
    )


def test_arrays_are_expanded_with_subscript() -> None:
    """`$ARR` on an array yields only element 0, silently.

    SKILL.md built FOUND_DBS as an array, counted it with ${#FOUND_DBS[@]}, then looped
    with `for db in $FOUND_DBS` — so multi-database discovery always offered exactly one
    database, contradicting three other sections of the same file.

    Keyed on assignment rather than a name list, so a new array is covered on the day it
    is written. Names are collected across every block and checked against every block, so
    a bad expansion is caught wherever it sits relative to the assignment.
    """
    offenders = []
    for path, line, source in ALL_BLOCKS:
        for raw in source.splitlines():
            stripped = raw.strip()
            if stripped.startswith("#"):
                continue
            for name in ARRAY_NAMES:
                # Bare $NAME or ${NAME}: no [@], [*], or [n] subscript, and not ${#NAME[@]}.
                if re.search(rf"(?<!#)\$\{{?{name}\}}?(?!\[|\w)", raw):
                    offenders.append(f"{_ident(path, line)}: {stripped}")
                    break

    assert not offenders, (
        "an array is expanded without a subscript, which yields only its first element. "
        'Use "${NAME[@]}":\n  ' + "\n  ".join(offenders)
    )


# A subscripted read: "${NAME[@]}", ${#NAME[@]}, ${NAME[0]}. Bare $NAME is the other bug,
# caught by the test above.
def _subscripted_reads(source: str, name: str) -> bool:
    return bool(re.search(rf"\$\{{#?{name}\[", source))


def _assigns(source: str, name: str) -> bool:
    return any((m := ARRAY_ASSIGN.match(raw)) and m.group(1) == name for raw in source.splitlines())


def test_arrays_are_consumed_in_the_block_that_builds_them() -> None:
    """An array does not survive into the next Bash call, so neither half works alone.

    Each fenced block is its own Bash invocation. SKILL.md built FOUND_DBS in the Database
    Discovery block and looped over it in the next one to print each database's language
    and creation time: the loop iterated zero times and printed nothing, and the selection
    prompt it feeds had no metadata to offer — one paragraph after the text stating the
    rule. Nothing caught it, because both halves are correct read on their own.

    Scoped to arrays because that is the case where the failure is silent: an unset array
    expands to nothing and the loop simply does not run. An unset scalar usually surfaces
    as a command that fails.
    """
    offenders = []
    for path, line, source in ALL_BLOCKS:
        for name in ARRAY_NAMES:
            if _subscripted_reads(source, name) and not _assigns(source, name):
                offenders.append(f"{_ident(path, line)}: reads ${{{name}[@]}}, never builds it")

    assert not offenders, (
        "a block reads an array it does not build. Each block is a separate Bash call, so "
        "the array is empty there and the loop over it silently does nothing — build it in "
        "the same block that reads it:\n  " + "\n  ".join(offenders)
    )


def test_the_same_block_rule_has_something_to_check() -> None:
    """Guard the guard: with no subscripted read anywhere, the check above inspects nothing.

    Also pins the detector itself, since no block in the skill violates the rule today and
    a detector that stopped matching would look identical to a clean skill.
    """
    reads = [
        (path, line, name)
        for path, line, source in ALL_BLOCKS
        for name in ARRAY_NAMES
        if _subscripted_reads(source, name)
    ]
    assert len(reads) >= 2, (
        f"only {len(reads)} subscripted array reads found across {len(ALL_BLOCKS)} blocks — "
        f"the detector is broken, not the skill"
    )
    split = 'FOUND_DBS=()\nFOUND_DBS+=("$db")\n'
    assert _subscripted_reads('for db in "${FOUND_DBS[@]}"; do :; done', "FOUND_DBS")
    assert not _assigns('for db in "${FOUND_DBS[@]}"; do :; done', "FOUND_DBS")
    assert _assigns(split, "FOUND_DBS"), "an append must not be mistaken for the declaration"


# find_databases.sh exits 2 when codeql is missing from the calling shell's PATH, precisely
# so that state cannot be mistaken for "this project has no databases". `done < <(script)`
# throws the status away: the loop's status is the loop's, and the substitution's is
# unobservable. Command substitution keeps it, so the caller can tell the two apart.
DISCOVERY_SCRIPT = "find_databases.sh"
PROCESS_SUBSTITUTION = re.compile(r"<\s*<\(")


def _discovery_without_status(source: str) -> list[str]:
    """Lines that run the discovery script through a process substitution."""
    offenders = []
    for raw in source.splitlines():
        stripped = raw.strip()
        if stripped.startswith("#") or DISCOVERY_SCRIPT not in stripped:
            continue
        if PROCESS_SUBSTITUTION.search(stripped):
            offenders.append(stripped)
    return offenders


def test_discovery_exit_status_is_observed() -> None:
    """All three callers read discovery through a process substitution and dropped its status.

    Failure scenario: codeql is not on this Bash call's PATH — a fresh shell each block, so
    the preflight that checked it ran in a different one. The script prints its ERROR to
    stderr and exits 2, the array comes back empty, and the caller reports "No CodeQL
    database found" while three good databases sit on disk. The user is then walked through
    a full rebuild that fails for the same reason at the same place.
    """
    offenders = [
        f"{_ident(path, line)}: {offender}"
        for path, line, source in ALL_BLOCKS
        for offender in _discovery_without_status(source)
    ]
    assert not offenders, (
        "a block reads find_databases.sh through a process substitution, whose exit status "
        "is unobservable, so exit 2 (no codeql on PATH) reads as an empty result. Use "
        '`if ! DB_LIST=$("{baseDir}/scripts/find_databases.sh" …); then … fi` and loop over '
        '`<<<"$DB_LIST"`:\n  ' + "\n  ".join(offenders)
    )


DISCOVERY_MUST_FLAG = (
    'done < <("{baseDir}/scripts/find_databases.sh" "${OUTPUT_DIR:-.}" .)',
    "done < <(find_databases.sh)",
    # Spacing between the redirect and the substitution is the caller's habit, not a signal.
    'done <  <("{baseDir}/scripts/find_databases.sh" .)',
)

DISCOVERY_MUST_PASS = (
    'if ! DB_LIST=$("{baseDir}/scripts/find_databases.sh" "${OUTPUT_DIR:-.}" .); then',
    'done <<<"$DB_LIST"',
    # Prose about the script, which the blocks carry above every call site.
    "# Command substitution, not `done < <(...)`: find_databases.sh exits 2 when codeql is "
    "missing.",
)


def test_detector_flags_discovery_process_substitution() -> None:
    """Guard the guard: no block violates the rule now, so nothing else would notice this
    detector going quiet."""
    for source in DISCOVERY_MUST_FLAG:
        assert _discovery_without_status(source) == [source], (
            f"the detector stopped flagging:\n  {source}"
        )


def test_detector_passes_status_checked_discovery() -> None:
    """False positives get silenced, and a silenced check catches nothing."""
    for source in DISCOVERY_MUST_PASS:
        assert _discovery_without_status(source) == [], (
            f"the detector now fires on a caller that does check the status, which is how it "
            f"gets disabled:\n  {source}"
        )


def test_the_discovery_rule_has_something_to_check() -> None:
    """Guard the guard: if the block extractor or the script name broke, every block would
    look discovery-free and the check above would inspect nothing."""
    callers = [
        _ident(path, line)
        for path, line, source in ALL_BLOCKS
        if any(
            DISCOVERY_SCRIPT in raw and not raw.strip().startswith("#")
            for raw in source.splitlines()
        )
    ]
    assert len(callers) >= 3, (
        f"only {len(callers)} blocks call {DISCOVERY_SCRIPT} across {len(ALL_BLOCKS)} blocks "
        f"in {SKILL_ROOT} — SKILL.md, run-analysis.md and create-data-extensions.md each do, "
        f"so the detector is broken, not the skill"
    )


# Bodies of `python3 -c '…'`, either quote style, on one line or many. Requiring a newline
# after the opening quote missed the two one-liners in create-data-extensions.md, and a
# missed block reports `skip` — indistinguishable from a block with no Python in it.
EMBEDDED_PYTHON = re.compile(r"""python3 -c (['"])(.*?)\1""", re.DOTALL)


def _embedded_python(source: str) -> list[str]:
    return [match.group(2) for match in EMBEDDED_PYTHON.finditer(source)]


def _embedded_python_bodies() -> list[tuple[str, str]]:
    """Every `python3 -c` body in the skill, as (where it came from, the source)."""
    return [
        (f"{_ident(path, line)}#{index}", body)
        for path, line, source in ALL_BLOCKS
        for index, body in enumerate(_embedded_python(source))
    ]


ALL_EMBEDDED_PYTHON = _embedded_python_bodies()


def test_embedded_python_extraction_still_matches() -> None:
    """Guard the guard: a regex that matched nothing would leave the compile check below
    running against an empty parameter set, which passes without inspecting anything.

    Two bodies today, both in create-data-extensions.md. Was three until
    quality-assessment.md stopped parsing baseline-info.json inline — check_db_quality.py
    reports baseline_loc, so the block was computing it twice.
    """
    assert len(ALL_EMBEDDED_PYTHON) >= 2, (
        f"only {len(ALL_EMBEDDED_PYTHON)} embedded python bodies extracted from "
        f"{SKILL_ROOT} — the extractor is broken, not the skill"
    )


@pytest.mark.parametrize(
    "snippet",
    (
        # create-data-extensions.md counts SARIF results this way.
        "BASELINE=$(python3 -c \"import json; print(len(json.load(open('x.sarif'))))\")",
        # The argv form, which the extractor must handle even though no doc uses it today.
        "LOC=$(python3 -c '\nimport json, sys\nprint(json.load(open(sys.argv[1])))\n' \"$DB\")",
    ),
)
def test_both_quote_styles_are_extracted(snippet: str) -> None:
    """Either form is real shell in this skill, and both must reach the compiler below."""
    assert _embedded_python(snippet)


@pytest.mark.parametrize(
    ("origin", "body"),
    ALL_EMBEDDED_PYTHON,
    ids=[origin for origin, _ in ALL_EMBEDDED_PYTHON],
)
def test_embedded_python_compiles(origin: str, body: str) -> None:
    """Python inside a bash block must parse.

    Quoting the shell string correctly and quoting the Python correctly are separate
    problems, and fixing one can break the other: moving a script from a double- to a
    single-quoted shell string leaves `\\"` escapes behind that are then literal
    backslashes to Python.
    """
    try:
        compile(body, "<embedded>", "exec")
    except SyntaxError as error:
        pytest.fail(f"{origin} embeds Python that does not parse: {error}\n{body}")

"""Structural checks on the `.qls` templates in the suite reference docs.

A malformed or over-filtered template resolves to few or no queries, and the resulting
empty SARIF reads as a clean codebase.

Text-based rather than YAML-parsed: `make python-tests` runs pytest with no extra
dependencies, so importing PyYAML would fail on a clean checkout.
"""

from __future__ import annotations

import re
from pathlib import Path

import pytest

SKILL_ROOT = Path(__file__).resolve().parent.parent
YAML_BLOCK = re.compile(r"^```yaml\n(.*?)^```", re.MULTILINE | re.DOTALL)

IMPORTANT_ONLY = SKILL_ROOT / "references" / "important-only-suite.md"
RUN_ALL = SKILL_ROOT / "references" / "run-all-suite.md"
DOCS = {"important-only": IMPORTANT_ONLY, "run-all": RUN_ALL}


def _template(doc: Path) -> str:
    blocks = YAML_BLOCK.findall(doc.read_text(encoding="utf-8"))
    if len(blocks) != 1:
        pytest.fail(
            f"expected exactly one yaml template in {doc.name}, found {len(blocks)}. "
            f"The doc changed shape — update this test rather than deleting it."
        )
    return blocks[0]


def _entry_keys(template: str) -> list[str]:
    """Top-level list-entry keys, e.g. queries / import / include / exclude."""
    return re.findall(r"^- (\w+):", template, re.MULTILINE)


@pytest.mark.parametrize("name", sorted(DOCS))
def test_template_exists_and_is_non_trivial(name: str) -> None:
    template = _template(DOCS[name])
    assert len(template.splitlines()) > 5, f"{name}: template is too short to be real"


@pytest.mark.parametrize("name", sorted(DOCS))
def test_template_has_at_least_one_query_source(name: str) -> None:
    """A suite with no `queries:` or `import:` resolves to nothing."""
    keys = _entry_keys(_template(DOCS[name]))
    sources = [k for k in keys if k in {"queries", "import"}]
    assert sources, (
        f"{name}: template declares no query source — it would resolve to zero queries "
        f"and the analysis would report no findings"
    )


@pytest.mark.parametrize("name", sorted(DOCS))
def test_template_declares_the_language_placeholder(name: str) -> None:
    template = _template(DOCS[name])
    assert "<CODEQL_LANG>" in template, (
        f"{name}: template hardcodes a language instead of using <CODEQL_LANG>"
    )


@pytest.mark.parametrize("name", sorted(DOCS))
def test_template_selects_alert_queries_only(name: str) -> None:
    """Both suites restrict to problem/path-problem — the query kinds that produce alerts."""
    template = _template(DOCS[name])
    assert "include" in _entry_keys(template), f"{name}: no include block"
    assert "problem" in template and "path-problem" in template, (
        f"{name}: does not restrict to alert-producing query kinds"
    )


@pytest.mark.parametrize("name", sorted(DOCS))
def test_template_excludes_deprecated_and_model_queries(name: str) -> None:
    template = _template(DOCS[name])
    assert "deprecated" in template, f"{name}: does not exclude deprecated queries"
    assert "modeleditor" in template and "modelgenerator" in template, (
        f"{name}: does not exclude model editor/generator queries, which are tooling "
        f"support rather than alerts"
    )


def test_important_only_covers_every_precision_it_claims() -> None:
    """The doc's Phase 1 table promises high, very-high, and medium. Pin all three.

    Dropping one narrows the scan silently: the suite still resolves and still reports
    findings, just fewer, with nothing indicating queries were excluded.
    """
    template = _template(IMPORTANT_ONLY)
    precisions = set(re.findall(r"^\s+- (high|very-high|medium|low)$", template, re.MULTILINE))
    assert precisions == {"high", "very-high", "medium"}, (
        f"important-only covers {sorted(precisions)}; the documented criteria are "
        f"high, very-high, and medium"
    )


def test_important_only_filters_on_security_tag() -> None:
    template = _template(IMPORTANT_ONLY)
    assert "tags contain" in template and "security" in template, (
        "important-only must restrict to security-tagged queries; without it the mode "
        "is just security-extended under another name"
    )


def test_run_all_imports_both_official_suites() -> None:
    """The two suites are complementary, not hierarchical.

    security-and-quality excludes experimental/ paths; security-experimental re-adds
    them. Importing only the first drops the delta run-all exists to capture.
    """
    template = _template(RUN_ALL)
    assert "security-and-quality" in template, "run-all: missing security-and-quality"
    assert "security-experimental" in template, (
        "run-all: missing security-experimental — experimental/ queries would be dropped "
        "with no signal that the scan narrowed"
    )


def test_run_all_does_not_filter_on_precision() -> None:
    """Filtering by precision here would make run-all a duplicate of important-only."""
    template = _template(RUN_ALL)
    assert not re.search(r"^\s+precision:", template, re.MULTILINE), (
        "run-all applies a precision filter, which is what distinguishes important-only"
    )

"""Regression tests for CodeQL query-suite validation."""

from __future__ import annotations

import subprocess
from pathlib import Path

import pytest
from verify_query_suite import SuiteResolutionFailure, main, resolve_queries


def _result(stdout: str, returncode: int = 0, stderr: str = "") -> subprocess.CompletedProcess[str]:
    return subprocess.CompletedProcess([], returncode, stdout, stderr)


def test_non_empty_suite_succeeds(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
    monkeypatch.setattr(
        subprocess,
        "run",
        lambda *args, **kwargs: _result('["one.ql", "two.ql"]'),
    )

    assert resolve_queries(tmp_path / "queries.qls") == ["one.ql", "two.ql"]


def test_empty_suite_fails(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
    monkeypatch.setattr(subprocess, "run", lambda *args, **kwargs: _result("[]"))

    with pytest.raises(SuiteResolutionFailure, match="resolved zero queries"):
        resolve_queries(tmp_path / "queries.qls")


def test_empty_suite_cli_exits_nonzero(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
    monkeypatch.setattr(subprocess, "run", lambda *args, **kwargs: _result("[]"))

    assert main([str(tmp_path / "queries.qls")]) == 1


def test_codeql_failure_fails(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
    monkeypatch.setattr(
        subprocess,
        "run",
        lambda *args, **kwargs: _result("", returncode=2, stderr="missing pack"),
    )

    with pytest.raises(SuiteResolutionFailure, match="missing pack"):
        resolve_queries(tmp_path / "queries.qls")


def test_malformed_output_fails(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None:
    monkeypatch.setattr(subprocess, "run", lambda *args, **kwargs: _result("not json"))

    with pytest.raises(SuiteResolutionFailure, match="invalid JSON"):
        resolve_queries(tmp_path / "queries.qls")

# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
"""Fail unless a CodeQL query suite resolves to one or more queries."""

from __future__ import annotations

import argparse
import json
import subprocess
import sys
from pathlib import Path


class SuiteResolutionFailure(Exception):
    """A query suite that must not be analysed with."""


def resolve_queries(suite_path: Path, codeql_executable: str = "codeql") -> list[str]:
    """Resolve a query suite and return its selected query paths."""
    command = [
        codeql_executable,
        "resolve",
        "queries",
        "--format=json",
        "--",
        str(suite_path),
    ]
    try:
        result = subprocess.run(command, capture_output=True, text=True)
    except OSError as error:
        raise SuiteResolutionFailure(
            f"Could not run CodeQL while validating {suite_path}: {error}. "
            "Check that the CodeQL CLI is installed and executable."
        ) from error

    if result.returncode != 0:
        diagnostic = result.stderr.strip() or "CodeQL produced no diagnostic"
        raise SuiteResolutionFailure(
            f"CodeQL failed to resolve {suite_path}: {diagnostic}. "
            "Check the suite imports and installed query packs."
        )

    try:
        query_paths = json.loads(result.stdout)
    except json.JSONDecodeError as error:
        raise SuiteResolutionFailure(
            f"CodeQL returned invalid JSON while resolving {suite_path}: {error}. "
            "Re-run the command manually with --format=json."
        ) from error

    if not isinstance(query_paths, list) or not all(
        isinstance(query_path, str) for query_path in query_paths
    ):
        raise SuiteResolutionFailure(
            f"CodeQL returned an unexpected query list while resolving {suite_path}. "
            "Check that this CodeQL version supports --format=json."
        )
    if not query_paths:
        raise SuiteResolutionFailure(
            f"CodeQL resolved zero queries from {suite_path}. "
            "Check suite filters, imports, and installed query packs."
        )
    return query_paths


def main(argv: list[str] | None = None) -> int:
    """Validate one suite and print its resolved query count."""
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("suite", type=Path)
    parser.add_argument("--codeql", default="codeql", help="CodeQL executable path")
    args = parser.parse_args(argv)

    try:
        query_paths = resolve_queries(args.suite, args.codeql)
    except SuiteResolutionFailure as error:
        print(f"ERROR: {error}", file=sys.stderr)
        return 1

    print(f"Resolved {len(query_paths)} queries from {args.suite}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

# Build Database Workflow

Create high-quality CodeQL databases by trying build methods in sequence until one produces good results.

## Overview

What matters is which build modes a language accepts, not whether it is interpreted.
Go is compiled but has no `none` mode; C# and Java are compiled and do. Confirm against
your own CLI with `codeql database create --help`, and note its `none` list is
incomplete — it omits C/C++ and Rust, both of which do support `none` (2.25.6).

### No build needed (Python, JavaScript/TypeScript, Ruby)
- CodeQL extracts source directly
- **Exclusion config supported** — use `--codescanning-config` to skip irrelevant files

### Build required, no fallback (Go, Swift)
- `--build-mode=none` is **rejected**: *"Go does not support the none build mode. Please
  try using one of the following build modes instead: autobuild, manual."*
- Autobuild usually suffices for Go when the toolchain is present and the module builds.
  If it fails there is no no-build escape — fix the build or stop.
- Skip Method 4 for these languages.

### Build required, `none` available as a fallback (C/C++, Java/Kotlin, C#, Rust)
- **Build required for complete extraction** — CodeQL must trace the compilation
- **Exclusion config NOT supported** — all traced code is extracted
- `--build-mode=none` works but produces partial analysis. Method 4, last resort.
- Try build methods in order until one succeeds:
  1. **Autobuild** — CodeQL auto-detects and runs the build
  2. **Custom Command** — Explicit build command for the detected build system
  2m. **macOS arm64 Toolchain** — Homebrew compiler + multi-step tracing (Apple Silicon workaround)
  3. **Multi-step** — Fine-grained control with init → trace-command → finalize
  4. **No-build fallback** — `--build-mode=none` (partial analysis, last resort)

> **macOS Apple Silicon:** On arm64 Macs, system tools (`/usr/bin/make`, `/usr/bin/clang`, `/usr/bin/ar`) are `arm64e` but CodeQL's `libtrace.dylib` only has `arm64`. macOS kills `arm64e` processes with a non-`arm64e` injected dylib (SIGKILL, exit 137). Step 2a detects this and routes to Method 2m.

---

## Build Log

`$OUTPUT_DIR` arrives from the parent skill, resolved once at invocation. Every file this
workflow writes goes inside it. Source the log helpers before any build step, and in any
reference doc that uses `run_logged`:

```bash
DB_NAME="$OUTPUT_DIR/codeql.db"
. "{baseDir}/scripts/build_log.sh" || exit 1
log_step "CodeQL database build — $DB_NAME"
```

That provides `log_step`, `log_cmd`, `log_result`, and `run_logged`; defaults `LOG_FILE` to
`$OUTPUT_DIR/build.log` and stops if it is not writable; and sets `pipefail` so a command's
exit status survives being piped to `tee`. It deliberately does not set `-e`: the method
ladder below has to survive each failed method to reach the next one.

> **Every block below that uses a helper repeats the source line.** A function defined in an
> earlier Bash call is gone by the next one, and `run_logged` then exits 127, which the ladder
> reads as a failed build method. Set `DB_NAME` and `CODEQL_LANG` in the block as well.
> See [Each Bash call is a fresh shell](../SKILL.md#each-bash-call-is-a-fresh-shell).

**What to log:** Detected language/build system, each build attempt with exact command, fix attempts and outcomes, quality assessment results, final successful command.

---

## Step 1: Detect Language and Configure

**Entry:** CodeQL CLI installed and on PATH (`codeql --version` succeeds)
**Exit:** `CODEQL_LANG` variable set to a valid CodeQL language identifier; exclusion config created (interpreted) or skipped (compiled)

### 1a. Detect Language

```bash
# fd is not in the Quick Start preflight. Without this fallback a machine that lacks it
# prints an empty histogram — fd's error goes to stderr and the rest of the pipeline
# succeeds — and the language gets picked by guess.
if command -v fd >/dev/null 2>&1; then
  fd -t f -e py -e js -e ts -e go -e rb -e java -e c -e cpp -e h -e hpp -e rs -e cs
else
  find . -type f \( -name '*.py' -o -name '*.js' -o -name '*.ts' -o -name '*.go' \
    -o -name '*.rb' -o -name '*.java' -o -name '*.c' -o -name '*.cpp' -o -name '*.h' \
    -o -name '*.hpp' -o -name '*.rs' -o -name '*.cs' \) -not -path './.git/*'
fi | sed 's/.*\.//' | sort | uniq -c | sort -rn | head -5

ls -la Makefile CMakeLists.txt build.gradle pom.xml Cargo.toml *.sln 2>/dev/null || true
```

| Language | `--language=` | Build needed | `--build-mode=none` |
|----------|---------------|-------------|---------------------|
| Python | `python` | No | Supported |
| JavaScript/TypeScript | `javascript` | No | Supported |
| Ruby | `ruby` | No | Supported |
| Go | `go` | **Yes** | **Rejected** — autobuild or manual only |
| Swift | `swift` | **Yes** (macOS) | **Rejected** |
| Java/Kotlin | `java` | Yes | Supported (partial analysis) |
| C# | `csharp` | Yes | Supported (partial analysis) |
| C/C++ | `cpp` | Yes | Supported (partial analysis) |
| Rust | `rust` | Yes | Supported (partial analysis) — omitted from `--help`, like C/C++ |

Verified against CodeQL 2.25.6. Re-check with `codeql database create --help` if your
version differs; the supported modes have changed between releases.

### 1b. Create Exclusion Config (Interpreted Languages Only)

> **Skip for compiled languages** — exclusion config is not supported when build tracing is required.

Scan for irrelevant directories and create `$OUTPUT_DIR/codeql-config.yml` with `paths-ignore` entries for `node_modules`, `vendor`, `venv`, third-party code, and generated/minified files.

---

## Step 2: Build Database

**Entry:** Step 1 complete (`CODEQL_LANG` set, `DB_NAME` assigned, log file initialized)
**Exit:** `codeql resolve database -- "$DB_NAME"` succeeds (database exists and is valid)

### For Interpreted Languages

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "Building database for interpreted language: <LANG>"
run_logged codeql database create "$DB_NAME" \
  --language="$CODEQL_LANG" \
  --source-root=. \
  --codescanning-config="$OUTPUT_DIR/codeql-config.yml" \
  --overwrite
```

**Skip to Step 4 after success.**

---

### For Compiled Languages

#### Step 2a: macOS arm64e Detection (C/C++ primarily)

```bash
IS_MACOS_ARM64E=false
if [[ "$(uname -s)" == "Darwin" ]] && [[ "$(uname -m)" == "arm64" ]]; then
  LIBTRACE=$(find "$(dirname "$(command -v codeql)")" -name libtrace.dylib 2>/dev/null | head -1)
  if [ -n "$LIBTRACE" ]; then
    LIBTRACE_ARCHS=$(lipo -archs "$LIBTRACE" 2>/dev/null)
    if [[ "$LIBTRACE_ARCHS" != *"arm64e"* ]]; then
      MAKE_ARCHS=$(lipo -archs /usr/bin/make 2>/dev/null)
      [[ "$MAKE_ARCHS" == *"arm64e"* ]] && IS_MACOS_ARM64E=true
    fi
  fi
fi
```

**If `IS_MACOS_ARM64E=true`:** Skip Methods 1 and 2 — go directly to Method 2m.

---

Try build methods in sequence until one succeeds:

#### Method 1: Autobuild

> **Skip if `IS_MACOS_ARM64E=true`.**

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "METHOD 1: Autobuild"
run_logged codeql database create "$DB_NAME" \
  --language="$CODEQL_LANG" --source-root=. --overwrite
```

`run_logged` returns the build's exit status. Check it before moving on. A non-zero
status means this method failed and the next one should be tried.

#### Method 2: Custom Command

> **Skip if `IS_MACOS_ARM64E=true`.**

Detect build system and use explicit command:

| Build System | Detection | Command |
|--------------|-----------|---------|
| Make | `Makefile` | `make clean && make -j"$(nproc 2>/dev/null || sysctl -n hw.ncpu)"` |
| CMake | `CMakeLists.txt` | `cmake -B build && cmake --build build` |
| Gradle | `build.gradle` | `./gradlew clean build -x test` |
| Maven | `pom.xml` | `mvn clean compile -DskipTests` |
| Cargo | `Cargo.toml` | `cargo clean && cargo build` |
| .NET | `*.sln` | `dotnet clean && dotnet build` |

Also check for project-specific build scripts (`build.sh`, `compile.sh`) and README instructions.

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "METHOD 2: Custom command"
run_logged codeql database create "$DB_NAME" \
  --language="$CODEQL_LANG" \
  --source-root=. \
  --command="$BUILD_CMD" \
  --overwrite
```

Use `--command="$BUILD_CMD"`, not `--command='$BUILD_CMD'`. Single quotes inside a
double-quoted string are literal characters, so the second form passes CodeQL a command
that starts with a `'`.

#### Method 2m: macOS arm64 Toolchain (Apple Silicon workaround)

> **Use when `IS_MACOS_ARM64E=true`.** Replaces Methods 1 and 2 on affected systems.

See [macos-arm64e-workaround.md](../references/macos-arm64e-workaround.md) for the full sub-method sequence (2m-a through 2m-d): Homebrew compiler with multi-step tracing → Rosetta x86_64 → system compiler verification → ask user.

#### Method 3: Multi-step Build

For complex builds needing fine-grained control:

> **On macOS with `IS_MACOS_ARM64E=true`:** Only trace arm64 Homebrew binaries. Do NOT trace system tools.

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "METHOD 3: Multi-step build"

# Each step gates the next. `finalize` after a failed `trace-command` produces a database
# that resolves correctly and contains nothing, so the rung reports success.
if ! run_logged codeql database init "$DB_NAME" \
  --language="$CODEQL_LANG" --source-root=. --overwrite; then
  log_result "FAILED (init)"
elif ! run_logged codeql database trace-command "$DB_NAME" -- <build step 1>; then
  log_result "FAILED (build step 1)"
elif ! run_logged codeql database trace-command "$DB_NAME" -- <build step 2>; then
  log_result "FAILED (build step 2)"
elif ! run_logged codeql database finalize "$DB_NAME"; then
  log_result "FAILED (finalize)"
else
  log_result "SUCCESS (multi-step)"
fi
```

Add one `elif` per build step. A method that stops early has failed: move to Method 4.

#### Method 4: No-Build Fallback (Last Resort)

> **WARNING:** Creates a database without build tracing. Only source-level patterns detected.
>
> **Not available for Go or Swift.** They reject `--build-mode=none` outright, so there is
> no fallback: fix the build, or report that a database could not be created. Do not burn
> a cycle attempting this for those languages.

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

log_step "METHOD 4: No-build fallback (partial analysis)"
run_logged codeql database create "$DB_NAME" \
  --language="$CODEQL_LANG" --source-root=. --build-mode=none --overwrite
```

Databases built this way often fail `check_db_quality.py` in Step 4. That is the expected
result: without build tracing there may be no analysable source, and the analysis would
report zero findings.

---

## Step 3: Apply Fixes (if build failed)

**Entry:** Step 2 build method failed (non-zero exit or `codeql resolve database` fails)
**Exit:** Fix applied and current build method retried; either succeeds (go to Step 4) or all fixes exhausted (try next build method in Step 2)

Try fixes in order, then retry current build method. See [build-fixes.md](../references/build-fixes.md) for the full fix catalog: clean state, clean build cache, install dependencies, handle private registries.

---

## Steps 4-5: Assess and Improve Quality

**Entry:** Database exists and `codeql resolve database` succeeds
**Exit (Step 4):** `check_db_quality.py` exits zero
**Exit (Step 5):** Quality improvements applied and the check re-run, OR user accepts a database that fails it

Run the gate first:

```bash
uv run {baseDir}/scripts/check_db_quality.py "$DB_NAME"
```

**A non-zero exit means do not proceed to analysis.** A database in that state analyses
without error and reports zero findings, which is the same output a clean codebase
produces. The two failure exits call for different responses:

Exit 1 is not a judgement call — go to Step 5 and fix the build. Exit 3 is, and
[quality-assessment.md](../references/quality-assessment.md) has the table for it and for
exits 2 and 4.

If it fails, go to [quality-assessment.md](../references/quality-assessment.md) for the
metric breakdown and the improvement steps, then re-run the gate. If it still fails after
those, present the metrics to the user and let them decide rather than continuing silently.

---

## Exit Conditions

**Success:** Quality assessment shows GOOD or user accepts current state.

**Failure (all methods exhausted):**
```
AskUserQuestion: "All build methods failed. Options:"
  1. "Accept current state" (if any database exists)
  2. "I'll fix the build manually and retry"
  3. "Abort"
```

---

## Final Report

```bash
. "{baseDir}/scripts/build_log.sh" || exit 1

echo "=== Build Complete ===" >> "$LOG_FILE"
echo "Finished: $(date -Iseconds)" >> "$LOG_FILE"
echo "Final database: $DB_NAME" >> "$LOG_FILE"
echo "Successful method: <METHOD>" >> "$LOG_FILE"
codeql resolve database -- "$DB_NAME" >> "$LOG_FILE" 2>&1
```

Report to user:
```
## Database Build Complete

**Output directory:** $OUTPUT_DIR
**Database:** $DB_NAME
**Language:** <LANG>
**Build method:** autobuild | custom | multi-step
**Files extracted:** <COUNT>

### Quality:
- Errors: <N>
- Coverage: <good/partial/poor>

### Build Log:
See `$OUTPUT_DIR/build.log` for complete details.

**Final command used:** <EXACT_COMMAND>
**Ready for analysis.**
```

# Create Data Extensions Workflow

Generate data extension YAML files to improve CodeQL's data flow coverage for project-specific APIs. Runs after database build and before analysis.

## Early Exit Points

| After Step | Condition | Action |
|------------|-----------|--------|
| Step 1 | Extensions already exist | Return found packs/files to run-analysis workflow, finish |
| Step 3 | No missing models identified | Report coverage is adequate, finish |

---

## Steps

### Step 1: Check for Existing Data Extensions

**Entry:** CodeQL database exists (`codeql resolve database` succeeds)
**Exit:** Either existing extensions found (report and finish) OR no extensions found (proceed to Step 2)

Search the project for existing data extensions and model packs.

```bash
# 1. In-repo model packs (exclude output dirs and legacy database dirs)
fd '(qlpack|codeql-pack)\.yml$' . --exclude 'static_analysis_codeql_*' --exclude 'codeql_*.db' | while read -r f; do
  if grep -q 'dataExtensions' "$f"; then
    echo "MODEL PACK: $(dirname "$f") - $(grep '^name:' "$f")"
  fi
done

# 2. Standalone data extension files
rg -l '^extensions:' --glob '*.yml' --glob '!static_analysis_codeql_*/**' --glob '!codeql_*.db/**' | head -20

# 3. Installed model packs
codeql resolve qlpacks 2>/dev/null | grep -iE 'model|extension'
```

**If any found:** Report to user and finish. These will be picked up by the run-analysis workflow.

**If none found:** Proceed to Step 2.

---

### Step 2: Query Known Sources and Sinks

**Entry:** Step 1 found no existing extensions; database and language identified
**Exit:** `sources.csv` and `sinks.csv` exist in `$DIAG_DIR` with enumerated source/sink locations

Run custom QL queries against the database to enumerate all sources and sinks CodeQL currently recognizes.

#### 2a: Select Database and Language

`$DB_NAME` may already be set by the parent skill. If not, discover with `find_databases.sh`,
which filters candidates through `codeql resolve database`. Do not use a bare `find` for the
marker file: `codeql database create` writes `codeql-database.yml` before the build finishes, so
a run killed mid-build leaves one behind. Selecting it here makes `list-sources.ql` and
`list-sinks.ql` return nothing, `sources.csv` and `sinks.csv` come back empty, and Step 3's
"no missing models identified" early exit reports coverage as adequate for a database that was
never finalized.

```bash
if [ -z "$DB_NAME" ]; then
  # Discovery and selection share a block: an array built in an earlier Bash call is gone by
  # this one, and an empty FOUND_DBS reads as "no database".
  # Command substitution, not `done < <(...)`: a process substitution discards the script's
  # exit status, so exit 2 ("codeql not on this shell's PATH") would arrive as an empty list
  # and be reported below as "No CodeQL database found".
  if ! DB_LIST=$("{baseDir}/scripts/find_databases.sh" "${OUTPUT_DIR:-.}" .); then
    echo "ERROR: database discovery failed — see the message above" >&2
    exit 1
  fi

  FOUND_DBS=()
  while IFS= read -r db; do
    [ -n "$db" ] || continue
    FOUND_DBS+=("$db")
  done <<<"$DB_LIST"

  if [ ${#FOUND_DBS[@]} -eq 0 ]; then
    echo "ERROR: No CodeQL database found in $OUTPUT_DIR"; exit 1
  elif [ ${#FOUND_DBS[@]} -eq 1 ]; then
    DB_NAME="${FOUND_DBS[0]}"
  else
    # Multiple databases. Use AskUserQuestion to select, or skip the prompt if the user
    # already named one. The `:` is required: an else branch containing only comments is
    # a bash syntax error and the block will not parse.
    :
  fi
fi

CODEQL_LANG=$(codeql resolve database --format=json -- "$DB_NAME" | jq -r '.languages[0]')
DIAG_DIR="$OUTPUT_DIR/diagnostics"
mkdir -p "$DIAG_DIR"
```

#### 2b: Write Source Enumeration Query

Use the `Write` tool to create `$DIAG_DIR/list-sources.ql` using the source template from [diagnostic-query-templates.md](../references/diagnostic-query-templates.md#source-enumeration-query). Pick the correct import block for `$CODEQL_LANG`.

#### 2c: Write Sink Enumeration Query

Use the `Write` tool to create `$DIAG_DIR/list-sinks.ql` using the language-specific sink template from [diagnostic-query-templates.md](../references/diagnostic-query-templates.md#sink-enumeration-queries).

**For Java:** Also create `$DIAG_DIR/qlpack.yml` with a `codeql/java-all` dependency and run `codeql pack install` before executing queries.

#### 2d: Run Queries

```bash
codeql query run --database="$DB_NAME" --output="$DIAG_DIR/sources.bqrs" -- "$DIAG_DIR/list-sources.ql"
codeql bqrs decode --format=csv --output="$DIAG_DIR/sources.csv" -- "$DIAG_DIR/sources.bqrs"

codeql query run --database="$DB_NAME" --output="$DIAG_DIR/sinks.bqrs" -- "$DIAG_DIR/list-sinks.ql"
codeql bqrs decode --format=csv --output="$DIAG_DIR/sinks.csv" -- "$DIAG_DIR/sinks.bqrs"
```

#### 2e: Summarize Results

Read both CSV files and present a summary showing source types and sink kinds with counts.

---

### Step 3: Identify Missing Sources and Sinks

**Entry:** Step 2 complete (`sources.csv` and `sinks.csv` available)
**Exit:** Either no gaps found (report adequate coverage and finish) OR user confirms which gaps to model (proceed to Step 4)

Cross-reference the project's API surface against CodeQL's known models.

#### 3a: Map the Project's API Surface

Read source code to identify security-relevant patterns:

| Pattern | What To Find | Likely Model Type |
|---------|-------------|-------------------|
| HTTP/request handlers | Custom request parsing | `sourceModel` (kind: `remote`) |
| Database layers | Custom ORM, raw query wrappers | `sinkModel` (kind: `sql-injection`) |
| Command execution | Shell wrappers, process spawners | `sinkModel` (kind: `command-injection`) |
| File operations | Custom file read/write | `sinkModel` (kind: `path-injection`) |
| Template rendering | HTML output, response builders | `sinkModel` (kind: `xss`) |
| Deserialization | Custom deserializers | `sinkModel` (kind: `unsafe-deserialization`) |
| HTTP clients | URL construction | `sinkModel` (kind: `ssrf`) |
| Sanitizers | Input validation, escaping | `neutralModel` |
| Pass-through wrappers | Logging, caching, encoding | `summaryModel` (kind: `taint`) |

Use `Grep` to search for these patterns in source code (adapt per language).

#### 3b: Cross-Reference Against Known Sources and Sinks

For each API pattern found, check if it appears in `sources.csv` or `sinks.csv` from Step 2.

**An API is "missing" if:**
- It handles user input but does not appear in `sources.csv`
- It performs a dangerous operation but does not appear in `sinks.csv`
- It wraps tainted data but has no summary model

#### 3c: Report Gaps

Present findings and use `AskUserQuestion`:

```
header: "Extensions"
question: "Create data extension files for the identified gaps?"
options:
  - label: "Create all (Recommended)"
    description: "Generate extensions for all identified gaps"
  - label: "Select individually"
    description: "Choose which gaps to model"
  - label: "Skip"
    description: "No extensions needed, proceed to analysis"
```

---

### Step 4: Create Data Extension Files

**Entry:** Step 3 identified gaps and user confirmed which to model
**Exit:** YAML extension files created in `$OUTPUT_DIR/extensions/` and deployed to `<lang>-all` ext/ directory

Generate YAML data extension files for the gaps confirmed by the user.

#### File Structure

Create files in `$OUTPUT_DIR/extensions/`:

```
$OUTPUT_DIR/extensions/
  sources.yml       # sourceModel entries
  sinks.yml         # sinkModel entries
  summaries.yml     # summaryModel and neutralModel entries
```

#### YAML Format and Deployment

See [extension-yaml-format.md](../references/extension-yaml-format.md) for column definitions, per-language examples (Python, Java, JS, Go, C/C++), and the deployment workaround for pre-compiled query packs.

Use the `Write` tool to create each file. Only create files that have entries — skip empty categories.

---

### Step 5: Validate with Re-Analysis

**Entry:** Step 4 complete (extension files deployed)
**Exit:** Finding delta measured (with-extensions count >= baseline count); extensions validated as loading correctly

Run a full security analysis with and without extensions to measure the finding delta.

#### 5a: Run Baseline Analysis (without extensions)

Validation artifacts go in `$DIAG_DIR` (not `results/`) since these are intermediate comparisons, not the final analysis output.

```bash
codeql database analyze "$DB_NAME" \
  --format=sarif-latest --output="$DIAG_DIR/baseline.sarif" --threads=0 \
  -- codeql/<lang>-queries:codeql-suites/<lang>-security-extended.qls
```

#### 5b: Run Analysis with Extensions

```bash
codeql database cleanup "$DB_NAME"
codeql database analyze "$DB_NAME" \
  --format=sarif-latest --output="$DIAG_DIR/with-extensions.sarif" --threads=0 --rerun \
  -- codeql/<lang>-queries:codeql-suites/<lang>-security-extended.qls
```

Use `-vvv` flag to verify extensions are being loaded.

#### 5c: Compare Findings

```bash
BASELINE=$(python3 -c "import json; print(sum(len(r.get('results',[])) for r in json.load(open('$DIAG_DIR/baseline.sarif')).get('runs',[])))")
WITH_EXT=$(python3 -c "import json; print(sum(len(r.get('results',[])) for r in json.load(open('$DIAG_DIR/with-extensions.sarif')).get('runs',[])))")
echo "Findings: $BASELINE → $WITH_EXT (+$((WITH_EXT - BASELINE)))"
```

**If counts did not increase:** Check extension loading (`-vvv`), pre-compiled pack workaround, Java `True`/`False` capitalization, column value accuracy.

---

## Final Output

```
## Data Extensions Created

**Output directory:** $OUTPUT_DIR
**Database:** $DB_NAME
**Language:** <LANG>

### Files Created:
- $OUTPUT_DIR/extensions/sources.yml — <N> source models
- $OUTPUT_DIR/extensions/sinks.yml — <N> sink models
- $OUTPUT_DIR/extensions/summaries.yml — <N> summary/neutral models

### Model Coverage:
- Sources: <BEFORE> → <AFTER> (+<DELTA>)
- Sinks: <BEFORE> → <AFTER> (+<DELTA>)

### Usage:
Extensions deployed to `<lang>-all` ext/ directory (auto-loaded).
Source files in `$OUTPUT_DIR/extensions/` for version control.
Run the run-analysis workflow to use them.
```

## References

- [Threat models reference](../references/threat-models.md) — control which source categories are active during analysis
- [CodeQL data extensions](https://codeql.github.com/docs/codeql-cli/using-custom-queries-with-the-codeql-cli/#using-extension-packs)
- [Customizing library models](https://codeql.github.com/docs/codeql-language-guides/customizing-library-models-for-python/)

# Run Analysis Workflow

Execute CodeQL security queries on an existing database with ruleset selection and result formatting.

## Scan Modes

Two modes control analysis scope. Both use all installed packs — the difference is filtering.

| Mode | Description | Suite Reference |
|------|-------------|-----------------|
| **Run all** | The `security-and-quality` + `security-experimental` suites from every installed pack. Not literally every query in the packs — see [run-all-suite.md](../references/run-all-suite.md) | [run-all-suite.md](../references/run-all-suite.md) |
| **Important only** | Security queries filtered by precision and security-severity threshold | [important-only-suite.md](../references/important-only-suite.md) |

> **WARNING:** Do NOT pass pack names directly to `codeql database analyze` (e.g., `-- codeql/cpp-queries`). Each pack's `defaultSuiteFile` silently applies strict filters and can produce zero results. Always use an explicit suite reference.

---

## The one gate

Ask once, in Step 3, and present everything the run depends on together: scan mode, query
packs, model packs, and threat model.

Let the user change any part, then proceed. If the user already specified something in
their prompt, show it as chosen rather than asking again.

---

## Steps

### Step 1: Select Database and Detect Language

**Entry:** `$OUTPUT_DIR` is set (from parent skill). `$DB_NAME` may already be set if the parent skill resolved database selection.
**Exit:** `DB_NAME` and `CODEQL_LANG` variables set; database resolves successfully.

**If `$DB_NAME` is already set** (parent skill handled database selection): validate it and proceed.

**If `$DB_NAME` is not set:** discover databases with `find_databases.sh`, which filters
candidates through `codeql resolve database` so a marker file left behind by a failed
build cannot be selected as though it were a database.

```bash
# Discovery and selection must share a block: an array built in an earlier Bash call is
# gone by this one, and an empty FOUND_DBS reads as "no database" for a project that has
# one. The script is what SKILL.md's Database Discovery section calls too, so the search
# depth and the validity filter are defined once.
if [ -z "${DB_NAME:-}" ]; then
  # Command substitution, not `done < <(...)`: a process substitution discards the script's
  # exit status, so exit 2 ("codeql not on this shell's PATH" — a fresh shell each block,
  # so the preflight's PATH does not carry) would arrive here as an empty list and be
  # reported as "No CodeQL database found" for a project that has several.
  if ! DB_LIST=$("{baseDir}/scripts/find_databases.sh" "${OUTPUT_DIR:-.}" .); then
    echo "ERROR: database discovery failed — see the message above" >&2
    exit 1
  fi

  FOUND_DBS=()
  while IFS= read -r db; do
    [ -n "$db" ] || continue
    FOUND_DBS+=("$db")
  done <<<"$DB_LIST"

  if [ "${#FOUND_DBS[@]}" -eq 0 ]; then
    echo "ERROR: No CodeQL database found in $OUTPUT_DIR or project root" >&2
    exit 1
  elif [ "${#FOUND_DBS[@]}" -eq 1 ]; then
    DB_NAME="${FOUND_DBS[0]}"
  else
    # More than one: select with AskUserQuestion, at most four options — the three most
    # recent plus "Build a new database", the rest named in the prompt text. Skip the
    # prompt when the user already said which database to use.
    #
    # DB_NAME stays unset here on purpose, and the check below turns that into an error.
    # Falling through to FOUND_DBS[0] would analyse whichever database `find` happened to
    # return first — a different language or a stale build, chosen without the user ever
    # being told there was a choice. The `:` is required: an else branch of only comments
    # is a bash syntax error.
    :
  fi
fi

if [ -z "${DB_NAME:-}" ]; then
  echo "ERROR: more than one database found. Ask which one, then re-run this block with DB_NAME set." >&2
  exit 1
fi

CODEQL_LANG=$(codeql resolve database --format=json -- "$DB_NAME" | jq -r '.languages[0]')
echo "Using: $DB_NAME (language: $CODEQL_LANG)"
```

If the database holds more than one language, ask which to analyze.

---

### Step 2: Gather What the Run Depends On

**Entry:** Step 1 complete (`DB_NAME` and `CODEQL_LANG` set)
**Exit:** Scan mode, installed packs, and model packs determined. Nothing presented to the user yet.

Collect everything here and present it as one plan in Step 3. Default the scan mode to
**run all** unless the user's prompt says otherwise.

#### 2a: Query Packs

For each pack available for the detected language (see [ruleset-catalog.md](../references/ruleset-catalog.md)):

| Language | Trail of Bits | Community Pack |
|----------|---------------|----------------|
| C/C++ | `trailofbits/cpp-queries` | `GitHubSecurityLab/CodeQL-Community-Packs-CPP` |
| Go | `trailofbits/go-queries` | `GitHubSecurityLab/CodeQL-Community-Packs-Go` |
| Java | `trailofbits/java-queries` | `GitHubSecurityLab/CodeQL-Community-Packs-Java` |
| JavaScript | — | `GitHubSecurityLab/CodeQL-Community-Packs-JavaScript` |
| Python | — | `GitHubSecurityLab/CodeQL-Community-Packs-Python` |
| C# | — | `GitHubSecurityLab/CodeQL-Community-Packs-CSharp` |
| Ruby | — | `GitHubSecurityLab/CodeQL-Community-Packs-Ruby` |

Check if installed (`codeql resolve qlpacks | grep -i "<PACK_NAME>"`). If not, ask user to install or ignore.

#### 2b: Detect Model Packs

Search three locations for data extension model packs:
1. **In-repo model packs** — `qlpack.yml`/`codeql-pack.yml` with `dataExtensions`
2. **In-repo standalone data extensions** — `.yml` files with `extensions:` key
3. **Installed model packs** — resolved by CodeQL

Record all detected packs for Step 3.

---

### Step 3: Confirm the Plan

**Entry:** Step 2 complete (mode, pack availability, and model packs all determined)
**Exit:** User confirmed; flag arrays built (`THREAT_MODEL_FLAGS`, `MODEL_PACK_FLAGS`, `ADDITIONAL_PACK_FLAGS`)

Present the whole plan in one `AskUserQuestion`, defaults filled in, and let the user
change any part before proceeding:

```
## CodeQL Analysis Plan

**Database:** $DB_NAME (language: $CODEQL_LANG)
**Scan mode:** Run all | Important only
**Query packs:** <installed packs — official, Trail of Bits, Community>
**Model packs:** <detected packs, or "None">
**Threat model:** Remote only (default) | + Local | All sources

Change anything, or say proceed.
```

Defaults, all overridable: **run all**, every installed pack, every detected model pack,
and **remote-only** threat models. Remote-only matches CodeQL's default. Widen it for CLI
tools, file parsers, and config readers, where the sources are `local` rather than
`remote`. See [threat-models.md](../references/threat-models.md).

Build the flags from the answer as arrays: `THREAT_MODEL_FLAGS=()` for remote-only,
`THREAT_MODEL_FLAGS=(--threat-model local)`, and so on. See Step 4 for why arrays rather
than strings.

**Model pack flags:**
- In-repo standalone extensions (`.yml`) are auto-discovered — pass source directory via `--additional-packs`
- In-repo model packs (with `qlpack.yml`) need parent directory via `--additional-packs`
- Installed model packs use `--model-packs`

---

### Step 4: Execute Analysis

**Entry:** Step 3 complete (all flags and pack selections finalized)
**Exit:** `$RAW_DIR/results.sarif` exists and contains valid SARIF output

#### Log selected query packs

Write the selected query packs, model packs, and threat models to `$OUTPUT_DIR/rulesets.txt`:

```bash
cat > "$OUTPUT_DIR/rulesets.txt" << RULESETS
# CodeQL Analysis — Selected Query Packs
# Generated: $(date -Iseconds)
# Scan mode: <run-all|important-only>
# Database: $DB_NAME
# Language: $CODEQL_LANG

## Query packs:
<one pack per line>

## Model packs:
<one pack per line, or "None">

## Threat models:
<threat model selection, or "default (remote)">
RULESETS
```

#### Generate custom suite

**Important-only mode:** Generate the custom `.qls` suite using the template and script in [important-only-suite.md](../references/important-only-suite.md).

**Run-all mode:** Generate the custom `.qls` suite using the template in [run-all-suite.md](../references/run-all-suite.md).

```bash
set -euo pipefail

RAW_DIR="$OUTPUT_DIR/raw"
RESULTS_DIR="$OUTPUT_DIR/results"
mkdir -p "$RAW_DIR" "$RESULTS_DIR"
# SCAN_MODE is "run-all" or "important-only", chosen in Step 3.
SUITE_FILE="$RAW_DIR/${SCAN_MODE}.qls"
```

The generation scripts above end by running `verify_query_suite.py`, so a suite produced
here is already checked. **Run it explicitly only if the suite came from somewhere else** —
reused from a previous run, or hand-edited:

```bash
uv run {baseDir}/scripts/verify_query_suite.py "$SUITE_FILE"
```

#### Run analysis

Output goes to `$RAW_DIR/results.sarif` (unfiltered). The final results are produced in Step 5.

Build the optional flags as **arrays**, not strings. A quoted empty string becomes an
empty argument that CodeQL rejects, and leaving a string unquoted so it can be empty also
lets `$DB_NAME` split on spaces. An array expands to nothing when empty and to each
element intact otherwise.

Expand them as `"${ARRAY[@]+"${ARRAY[@]}"}"`, not `"${ARRAY[@]}"`. Before bash 4.4 an
empty array under `set -u` is an unbound variable, so on macOS's `/bin/bash` 3.2 the plain
form aborts with `THREAT_MODEL_FLAGS[@]: unbound variable` before CodeQL runs — for the
documented default, a user who selected no threat models and no model packs.

**Declare the three arrays and the scalars in this block, filled in with the choices from
Step 3.** Nothing survives from Step 3 — see
[Each Bash call is a fresh shell](../SKILL.md#each-bash-call-is-a-fresh-shell). Leaving an
array empty is correct only when Step 3 selected nothing for it.

```bash
set -euo pipefail

DB_NAME="${DB_NAME:?set this to the database selected in Step 1}"
RAW_DIR="${RAW_DIR:-$OUTPUT_DIR/raw}"
SUITE_FILE="${SUITE_FILE:?set this to the .qls written in Step 2}"
mkdir -p "$RAW_DIR"

# Fill these from the Step 3 answers. Empty means "the user chose none".
THREAT_MODEL_FLAGS=()        # e.g. (--threat-model local --threat-model environment)
MODEL_PACK_FLAGS=()          # e.g. (--model-packs myorg/java-models)
ADDITIONAL_PACK_FLAGS=()     # e.g. (--additional-packs ./codeql-extensions)

codeql database analyze "$DB_NAME" \
  --format=sarif-latest \
  --output="$RAW_DIR/results.sarif" \
  --threads=0 \
  ${THREAT_MODEL_FLAGS[@]+"${THREAT_MODEL_FLAGS[@]}"} \
  ${MODEL_PACK_FLAGS[@]+"${MODEL_PACK_FLAGS[@]}"} \
  ${ADDITIONAL_PACK_FLAGS[@]+"${ADDITIONAL_PACK_FLAGS[@]}"} \
  -- "$SUITE_FILE"
```

`set -e` matters here: a failed analysis (out of memory, an unresolvable model pack) leaves
`raw/results.sarif` truncated or absent, and without it Step 5 copies that file forward and
the report prints "Total findings: 0" for a scan that never completed.

**Flag reference for model packs:**

| Source | Flag | Example |
|--------|------|---------|
| Installed model packs | `--model-packs` | `--model-packs=myorg/java-models` |
| In-repo model packs | `--additional-packs` | `--additional-packs=./lib/codeql-models` |
| In-repo standalone extensions | `--additional-packs` | `--additional-packs=.` |

### Performance

If codebase is large, read [performance-tuning.md](../references/performance-tuning.md) and apply relevant optimizations.

---

### Step 5: Process and Report Results

**Entry:** Step 4 complete (`$RAW_DIR/results.sarif` exists)
**Exit:** `$RESULTS_DIR/results.sarif` contains final results; findings summarized by severity, rule, and location; zero-finding results investigated; final report presented to user

#### Produce final results

- **Run-all mode:** Copy unfiltered results to the final location:
  ```bash
  cp "$RAW_DIR/results.sarif" "$RESULTS_DIR/results.sarif"
  ```

- **Important-only mode:** Apply the post-analysis filter from [sarif-processing.md](../references/sarif-processing.md#important-only-post-filter) to remove medium-precision results with `security-severity` < 6.0. The filter reads from `$RAW_DIR/results.sarif` and writes to `$RESULTS_DIR/results.sarif`, preserving the unfiltered original.

Process the final SARIF output (`$RESULTS_DIR/results.sarif`) using the jq commands in [sarif-processing.md](../references/sarif-processing.md): count findings, summarize by level, summarize by security severity, summarize by rule.

---

## Final Output

Report to user:

```
## CodeQL Analysis Complete

**Output directory:** $OUTPUT_DIR
**Database:** $DB_NAME
**Language:** <LANG>
**Scan mode:** Run all | Important only
**Query packs:** <list of query packs used>
**Model packs:** <list of model packs used, or "None">
**Threat models:** <list of threat models, or "default (remote)">

### Results Summary:
- Total findings: <N>
- Error: <N>
- Warning: <N>
- Note: <N>

### Output Files:
- SARIF (final): $OUTPUT_DIR/results/results.sarif
- SARIF (unfiltered): $OUTPUT_DIR/raw/results.sarif
- Rulesets: $OUTPUT_DIR/rulesets.txt
```