sota-privacy-compliance
State-of-the-art privacy and compliance engineering guidance for building privacy-respecting systems and auditing existing code for privacy/compliance gaps. Use when work involves privacy, GDPR, PII, personal data, consent, data retention, deletion, DSAR (data subject access requ
Install
npx skills add https://github.com/martinholovsky/SOTA-skills/tree/main/skills/sota-privacy-compliance
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install martinholovsky-sota-skills@llmmart
git clone https://github.com/martinholovsky/SOTA-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole martinholovsky/sota-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
SOTA Privacy & Compliance Engineering
Engineering-facing privacy and compliance architecture: how to design, build, and audit systems so that data protection is a property of the code and infrastructure, not a binder of policy documents. Compliance that lives in schemas, TTLs, IAM policies, and CI checks survives staff turnover and audit scrutiny; compliance that lives in wiki pages does not.
This is engineering guidance, not legal advice. Regulations change, vary by jurisdiction, and turn on facts about your business that code review cannot see. Regulatory facts below were verified against primary sources as of June 2026 — re-verify deadlines and statuses before relying on them, and route legal interpretation (lawful basis selection, contract terms, breach reportability decisions) to qualified counsel or your DPO. This skill tells you how to build the machinery those decisions require.
Related skills — reference, don't duplicate:
sota-databasesrules/06 — DB-level PII mechanics (column encryption, row-level security, GDPR-friendly schema design)sota-secrets-management— credentials, KMS, key rotation (crypto-shredding depends on it)sota-observability— log redaction and PII-safe telemetry pipelinessota-threat-modeling— LINDDUN privacy threat modeling methodologysota-code-security/sota-devsecops— vulnerability management, supply chain (SOC 2/ISO control overlap)
BUILD mode
When designing or implementing systems that touch personal data:
- Inventory first. Before writing a schema or integrating a vendor, classify every field you intend to collect and record where it flows (rules/01). The cheapest control is the field you never store.
- Annotate at the source. Schemas, structs, and API contracts carry classification and purpose annotations; tooling derives the data map from code, not the other way around (rules/01, rules/02).
- Build user rights as features, not afterthoughts. Export, deletion, and consent are product capabilities with APIs, state machines, and tests — design them with the first table, because retrofitting deletion into a 200-table schema with denormalized copies is a quarter-long project (rules/03).
- Automate retention. Every datastore gets a TTL, lifecycle rule, or partition-drop schedule at creation time. "We'll clean it up later" is how seven-year-old PII ends up in a breach disclosure (rules/03).
- Emit evidence as a byproduct. Access reviews, change approvals, and config baselines should fall out of normal engineering workflow (PRs, IaC, IdP logs) so audits are queries, not scrambles (rules/05).
- Know your regimes. Check rules/04 for which regulations the system triggers (data types × subjects' locations × sector) before architecture freezes — data residency and breach-clock requirements shape topology.
AUDIT mode
When reviewing an existing codebase/infrastructure for privacy and compliance gaps:
Process: (1) Build or obtain the data inventory — grep schemas, API payloads, log statements, analytics events, object storage for personal data (rules/01 has discovery patterns). (2) Trace lifecycle per data category: collection → purpose → storage → sharing → retention → deletion. (3) Test user rights paths end-to-end (does deletion actually propagate?). (4) Check evidence trails for auditable controls. (5) Map findings to applicable regimes (rules/04).
Severity conventions:
| Severity | Meaning | Examples |
|---|---|---|
| CRITICAL | Active violation with regulatory/breach exposure; fix now | PII in world-readable bucket; deletion endpoint that doesn't delete; special-category data collected without any consent record; cardholder PANs stored unencrypted in app DB |
| HIGH | Violation likely under normal operation or on first DSAR/audit/breach | No deletion propagation to backups/analytics; consent not versioned or not propagated to processors; no retention enforcement anywhere; PII in logs shipped to third party without DPA |
| MEDIUM | Gap that degrades posture or audit readiness | Classification annotations missing; data map stale/manual; soft-delete only with no purge job; access reviews manual and undocumented |
| LOW | Hardening/hygiene | Cookie banner lacks granular toggles; export format not machine-readable; missing purpose comments on schema fields |
Finding format:
[SEVERITY] <title>
Location: <file:line / table / bucket / service>
Data: <what personal data, what classification tier>
Regimes: <GDPR Art. X / CCPA / HIPAA / PCI DSS req N / SOC 2 CC-N — as applicable>
Issue: <what is wrong, lifecycle stage affected>
Impact: <realistic consequence: fine exposure, breach scope, audit failure, DSAR failure>
Fix: <concrete engineering remediation>
Evidence: <how you verified — query, code path, test>
Report findings grouped by data lifecycle stage (collection / storage / sharing / retention / deletion), not by file — that is how regulators and auditors think.
Rules index
| File | Read this when... |
|---|---|
| rules/01-data-inventory-classification.md | Starting any privacy work; building/auditing a data map; defining classification tiers; hunting PII in schemas, logs, buckets, backups, analytics; mapping flows to processors |
| rules/02-privacy-by-design.md | Designing schemas/APIs that touch personal data; minimization and purpose limitation in code; choosing pseudonymization vs anonymization vs tokenization; exposing aggregate stats; evaluating re-identification risk |
| rules/03-consent-and-user-rights.md | Building consent management, cookie/tracker governance, DSAR export, deletion (hard/soft/crypto-shred + propagation), or retention automation; auditing whether user rights actually work |
| rules/04-regulatory-landscape.md | Determining which regimes apply; GDPR engineering mechanics (lawful basis, transfers, DPIA, 72h); US state laws; HIPAA; PCI DSS 4.x scoping; EU AI Act timeline; DORA/NIS2; data residency architecture |
| rules/05-audit-ready-engineering.md | Preparing for SOC 2 / ISO 27001; automating evidence; mapping controls to engineering practice; vendor/subprocessor management; policy-as-code; avoiding common audit findings |
| rules/06-incident-breach-readiness.md | Building breach response capability; classification (is it reportable?); notification clocks per regime; forensics-friendly logging without privacy violations; post-incident obligations |
Top 10 non-negotiables
- No unmapped personal data. Every field of personal data has a recorded classification, purpose, owner, retention period, and list of systems it flows to. Unmapped data is unprotectable data.
- Collect the minimum. Each field collected must trace to a specific, documented purpose. A field without a purpose is deleted, not "kept just in case" — it is pure liability with zero value.
- Deletion must actually delete. A deletion request propagates to primary stores, replicas, caches, search indexes, analytics, ML training sets, and is handled for backups (expiry or crypto-shred). A soft-delete flag alone is a finding, not a deletion architecture.
- Retention is enforced by machines. TTLs, object lifecycle rules, partition drops — running and monitored. A retention policy with no automated enforcement is fiction.
- Consent is versioned, granular, revocable state — recorded with timestamp,
policy version, and scope; checked at point of use; revocation propagates to
processors. Never inferred, never a boolean column named
gdpr_ok. - No PII in logs, URLs, or analytics events unless explicitly classified, redaction-tested, and retention-bounded (see sota-observability for pipeline mechanics). Logs are the most common shadow copy of personal data.
- Pseudonymized ≠ anonymous. Data that can be re-linked (hashed emails, "anonymized" user IDs, quasi-identifier combinations) is still personal data. Treat claimed anonymization as a re-identification risk to verify, not a label to trust.
- Encrypt personal data at rest and in transit, with keys you can destroy. Key-per-user or key-per-tenant where deletion/residency demands it (crypto-shredding); keys managed per sota-secrets-management.
- Cross-border flows are deliberate. Know which regions data lives in and transits; region-pin where required; every processor/subprocessor has a DPA and appears in the data map before the first byte flows.
- Evidence or it didn't happen. Access reviews, consent records, deletion proofs, DPIAs, breach timelines — generated and retained automatically. If you cannot produce the artifact in minutes, the control will fail its audit.
Files (sota-skills)
-
rules
-
01-data-inventory-classification.md 14.2 KB
# 01 — Data Inventory & Classification You cannot protect, minimize, export, or delete what you have not mapped. Every other rule in this skill assumes an inventory exists. In AUDIT mode, building the inventory IS step one — most findings fall out of the gap between what the org thinks it stores and what it actually stores. ## 1. The data inventory is a living artifact, generated from code **Rule:** Maintain a machine-readable data inventory (catalog of personal-data fields with classification, purpose, owner, retention, and flow targets) that is derived from schemas and code, validated in CI, and versioned in git. A spreadsheet updated annually before the audit is not an inventory; it is a memorial to last year's architecture. **Rationale:** GDPR Art. 30 requires records of processing activities (RoPA); SOC 2 and ISO 27001 (A.5.9, A.5.12 in the 2022 control set) require asset inventories and classification. But the engineering reason is stronger: every DSAR, deletion request, breach-scope assessment, and residency question starts with "where is this person's data?" — and you need the answer in minutes, not weeks. **Implementation pattern — annotate at the schema, derive the map:** ```sql -- GOOD: classification and purpose live with the column (PostgreSQL COMMENT, -- harvested by a CI job into the catalog) CREATE TABLE customers ( id uuid PRIMARY KEY, email text NOT NULL, -- @pii:contact @purpose:account,transactional-email @retention:account-life+30d full_name text, -- @pii:identity @purpose:account @retention:account-life+30d date_of_birth date, -- @pii:identity @purpose:age-verification @retention:account-life health_notes text -- @pii:special-category @purpose:?? <- FAILS CI: special-category needs explicit purpose + lawful basis ref ); ``` ```yaml # GOOD: equivalent for typed schemas — protobuf/avro/JSON-schema custom options # message Customer { # string email = 1 [(privacy.class) = PII_CONTACT, # (privacy.purpose) = "account,transactional_email", # (privacy.retention) = "account_life_plus_30d"]; # } ``` ```text BAD: a Confluence page titled "Data Map v3 FINAL (2)" last edited 14 months ago, listing tables that were renamed two migrations back and missing the Kafka topics, the Mixpanel events, and the support-ticket attachments entirely. ``` **CI enforcement:** a check that (a) every new column/field in a migration or schema diff carries a classification annotation, (b) `special-category` and `financial` tiers also carry purpose + retention, (c) the generated catalog diff is attached to the PR. New unannotated field = failing build. This converts the inventory from a documentation chore into a property of the codebase. ## 2. Classification tiers — pick few, define sharply, bind controls to tiers **Rule:** Use a small fixed tier set; every tier maps to mandatory minimum controls. A classification scheme without attached controls is taxonomy theater. | Tier | Definition | Examples | Minimum controls | |---|---|---|---| | `public` | Intended for publication | Docs, marketing pages | Integrity only | | `internal` | Business data, no personal data | Configs, non-PII metrics | Access control, standard retention | | `confidential` | Business-sensitive, no personal data | Contracts, source code, pricing | Need-to-know access, encryption at rest | | `pii` | Relates to an identified/identifiable person | Name, email, IP, device ID, user ID, location, behavioral events | Everything above + purpose binding, retention limit, DSAR/deletion coverage, processor mapping | | `pii:special-category` | GDPR Art. 9 / sensitive under US state laws | Health, biometrics for ID, race/ethnicity, religion, sexual orientation, precise geolocation (CPRA), genetic data, children's data | Everything above + explicit lawful basis/consent record, strictest access (break-glass audited), DPIA, encryption with dedicated keys | | `financial/regulated` | Payment card data (PCI), bank details, regulated sector data | PAN, CVV, account numbers, PHI under HIPAA | Tier controls of the governing standard (PCI DSS scoping — rules/04; HIPAA safeguards — rules/04) | **Sharp edges to encode in the definitions:** - **Identifiers are PII.** IP addresses, cookie IDs, device fingerprints, internal user UUIDs — anything linkable to a person is personal data under GDPR and most state laws. Classifying `user_id` as `internal` is the single most common inventory error. - **Derived data inherits the tier.** An ML embedding of a support ticket, a propensity score, a session replay — derived from PII = PII. - **Aggregates can drop tier only after a documented anonymization review** (rules/02 §4) — not by assertion. - **Inference creates special-category data.** Purchase history that reveals health conditions or religion is special-category by inference; regulators have treated it as such. ## 3. PII discovery — schemas are the start, not the whole hunt **Rule:** When building the inventory (or auditing), sweep every place data comes to rest, not just the primary database. Personal data accumulates in six shadow locations that schemas don't show: 1. **Logs** — request/response bodies, exception messages with user objects, debug logging of auth tokens and emails. Grep patterns: log statements interpolating `user`, `email`, `req.body`, `params`, `headers`; structured-logging fields without an allowlist. (Redaction mechanics: sota-observability.) 2. **Object storage** — uploads, exports, report dumps, CSVs from "one-off" scripts. Audit bucket-by-bucket: what writes here, does it contain PII, is there a lifecycle rule, who can read it? Use cloud-native scanners (e.g., DLP services that sample objects) for backfill discovery; use path/prefix conventions + IaC review for prevention. 3. **Backups & snapshots** — full copies of everything above, often with longer retention than the source and weaker access control. Inventory must record backup retention per datastore (deletion implications: rules/03 §5). 4. **Analytics & event pipelines** — every `track()` call is a collection event. Audit the event schema registry (if none exists: HIGH finding); look for free-text properties, URLs with query params (tokens, emails in `?email=`), and identify-calls shipping whole user objects to third parties. 5. **Search indexes & caches** — Elasticsearch/OpenSearch documents, Redis session blobs, CDN-cached API responses. Re-indexed and re-cached copies outlive source-row deletion unless explicitly propagated. 6. **ML training data & feature stores** — training snapshots, fine-tuned model artifacts, vector stores of user content. Record dataset lineage: which training sets contain which user cohorts, so deletion/consent changes can be honored (rules/03 §5). **Audit greps that pay for themselves** (adapt per stack): ```bash # Email/phone/SSN-shaped literals & column names in schemas and code rg -i '(ssn|social_security|date_of_birth|dob|passport|tax_id|national_id)' --type sql rg -i '(email|phone|address|geo|lat.?lon|ip_addr)' migrations/ schemas/ # PII flowing into logs rg 'log(ger)?\.(info|warn|error|debug)\(.*\b(user|email|req\.body|password|token)' src/ # Analytics calls with raw payloads rg '(track|identify|capture)\(.*\b(email|name|phone)' src/ # Query params carrying identity rg '[?&](email|token|user_id|ssn)=' src/ logs/ ``` Automated scanners (regex + ML classifiers over column names, sampled values, and object stores) are good for breadth; they miss context (an `id` column that is a person) and produce false positives (a `name` column for products). Always pair scanner output with schema-owner review. ## 4. Data flow mapping — processors, subprocessors, and egress **Rule:** For every PII category, the inventory records every external destination: processor name, what data, what purpose, what region, DPA status, and the code path or integration that sends it. Generate the candidate list from code and config, then reconcile with contracts: - Dependency manifests + SDK initializations (Stripe, Segment, Sentry, Intercom, OpenAI, ...): each analytics/error/support/AI SDK is a processor. - Egress audit: outbound HTTP destinations from production (service-mesh or NAT/proxy logs) vs. the approved processor list. Unknown destination receiving request bodies = investigate immediately. - Webhooks, ETL jobs, reverse-ETL, data-share agreements (Snowflake shares, S3 cross-account replication). ```yaml # GOOD: flow record (one per processor, in the versioned catalog) processor: sentry data: [pii:contact(email), pii:identity(username), ip_address] # via error events purpose: error monitoring region: us # check against residency requirements (rules/04 §7) dpa: signed-2025-03 # link to contract record code_paths: [src/instrument.ts] controls: [beforeSend PII scrubber — tested in test/sentry_scrub.test.ts] ``` A processor in production without a DPA and inventory entry is a HIGH finding (CRITICAL if special-category data flows to it). Subprocessors of your processors matter too: your vendor's subprocessor list is part of your transfer analysis (rules/04 §2) and your customers' subprocessor disclosures. **Internal flows count as flows.** Map cross-boundary internal movement as well: service A's PII landing in team B's warehouse, the data-science sandbox with a prod replica, the staging environment seeded from production. Each is an access expansion that the inventory must show. Production data in staging/dev is a standing HIGH finding — seed non-prod from synthetic or irreversibly masked data (masking mechanics: sota-databases), and treat any "temporary" prod copy as having a TTL and an owner. ## 4b. Ownership — every data category has a name attached **Rule:** Each PII category and each datastore has a designated owner (a team, with an individual steward) recorded in the catalog. Owners answer DSAR-scoping questions, approve purpose changes, sign retention justifications, and receive the violation alerts from retention monitoring (rules/03 §6). Unowned data is unmaintained data: when the team that built the feature dissolved, its tables became the audit findings of two years later. CI can enforce this too — a catalog entry whose `owner` no longer matches an active team in the org chart goes stale loudly, not silently. ## 5. System-of-record designation — one writer, everyone else derives **Rule:** For every personal-data entity (user profile, consent state, contact info), designate exactly one system of record (SoR). All other copies are declared derivatives with a recorded sync mechanism and staleness bound. Deletion, rectification, and export execute against the SoR and propagate outward along the recorded derivative edges. **Rationale:** When three services each hold an editable copy of the user's email, rectification (GDPR Art. 16) is unimplementable and exports self-contradict. The deletion graph (rules/03 §5) is exactly the derivative graph — if it isn't recorded, deletion is guesswork. ```text GOOD: users-service = SoR for profile. CRM, analytics, search index are listed derivatives fed by CDC stream; deletion tombstone flows the same path; max staleness 5 min, monitored. BAD: email stored in users DB, billing DB, and marketing platform; support staff edit the CRM copy directly; nobody syncs back. Rectification request updates one of three copies. Export returns the stale two. ``` ## 6. Backfill strategy for the unmapped legacy estate When auditing (or inheriting) a system with no inventory, sequence the build — don't attempt a perfect map before extracting value: 1. **Schema sweep first** (hours): dump all schemas, flag candidate PII columns by name patterns + sampled values; classify with owners in a working session. This catches 70% and produces the catalog skeleton. 2. **Egress second** (days): SDK/dependency scan + outbound traffic review → processor map (§4). Third-party flows carry the highest regulatory risk per unit effort. 3. **Shadow locations third** (weeks): logs, buckets, analytics, backups (§3), prioritized by access breadth — a world-readable bucket outranks a locked backup vault. 4. **Wire the CI gate immediately** (§1) — even before backfill completes — so the unmapped estate stops growing while you drain it. 5. **Unstructured/SaaS sprawl last**, and accept sampling: ticket systems, CRMs, spreadsheets, chat exports. Inventory the *systems* and their retention/access even where field-level mapping is impractical; field-level fidelity there is rarely worth the cost. Record confidence levels per entry (`verified-by-code` / `declared-by-owner` / `scanner-detected`) so consumers of the map (DSAR jobs, breach scoping) know what's load-bearing. ## Audit checklist - [ ] Machine-readable data inventory exists, is versioned, and was updated within the current quarter (or is CI-generated) - [ ] Every schema field holding personal data carries classification + purpose + retention annotations; CI rejects unannotated additions - [ ] Classification tiers have documented, bound minimum controls; spot-check 5 `pii` fields for control compliance - [ ] Internal identifiers (user IDs, device IDs, IPs) are classified as PII, not `internal` - [ ] Special-category data is explicitly tiered with lawful-basis reference and DPIA link - [ ] Logs, object storage, backups, analytics events, search indexes, caches, and ML datasets are covered by the inventory (sample each) - [ ] Analytics event schema registry exists; no free-text/whole-object payloads to third parties - [ ] Processor list is reconciled against actual production egress and SDK usage; every processor has a DPA reference and region - [ ] Each personal-data entity has a designated system of record; derivative copies and sync paths are recorded - [ ] Backup retention per datastore is recorded and consistent with deletion obligations - [ ] Every data category and datastore has a current owner; staleness is detected (owner no longer exists → alert) - [ ] No production personal data in staging/dev/sandbox environments (or masked irreversibly); "temporary" copies have TTL + owner - [ ] Catalog entries carry confidence levels; load-bearing consumers (DSAR, breach scoping) rely only on verified entries - [ ] RoPA (where GDPR applies) is derivable from the inventory, not maintained as a separate divergent document -
02-privacy-by-design.md 12.9 KB
# 02 — Privacy by Design Patterns Privacy by design means the default behavior of the system — schemas, API contracts, access paths, settings — protects the data subject without anyone remembering to. For systematic privacy threat elicitation (linkability, identifiability, non-repudiation, detectability, disclosure, unawareness, non-compliance), use LINDDUN via `sota-threat-modeling`; this file gives the build patterns that those threats demand. ## 1. Data minimization — the field you don't store can't leak **Rule:** Collect a field only when a named feature needs it now. Reject "might be useful later," "the form library includes it," and "marketing asked." For every proposed field, require: purpose, consumer (which code reads it), retention, and deletion path. No answer = no field. **Rationale:** Every stored PII field has permanent carrying cost: breach blast radius, DSAR/export surface, deletion graph edge, transfer analysis, audit scope. Minimization is the only privacy control with negative cost. ```text GOOD: Signup form: email + password. Date of birth requested only when the user enables the age-gated feature, with the gate as documented purpose. BAD: Signup form: full name, DOB, phone, gender, company, job title — "for the profile" — where the product uses only email. Six liabilities, zero features. ``` **Minimize in dimensions, not just fields:** - **Precision:** store city, not GPS coordinates; year of birth, not full DOB; truncate IPs (/24, /48) when only geo/abuse-tier matters. - **Duration:** event-level data → aggregate after N days, drop raw (rules/03 §6). - **Population:** sample analytics (1-in-N users) instead of recording everyone. - **Payloads in transit:** internal services pass user IDs, not embedded user objects; each consumer fetches only the fields it is purpose-scoped to read. **API design corollary:** responses return the minimal projection per consumer. A `GET /users/{id}` that returns 40 fields to every internal caller makes every caller part of the PII surface. Use field masks / per-audience DTOs / GraphQL with field-level authorization. ```typescript // GOOD: explicit per-audience projections — adding a column to the table // changes NO response until someone deliberately adds it to a DTO const PublicProfile = pick(User, ['id', 'displayName', 'avatarUrl']); const SupportView = pick(User, ['id', 'displayName', 'email', 'createdAt']); const BillingView = pick(User, ['id', 'email', 'countryCode']); // BAD: res.json(user) — serializes the ORM entity; the day someone adds // `internal_risk_score` or `date_of_birth` to the model, every consumer // (and every consumer's logs) receives it. Disclosure by default. ``` ## 2. Purpose limitation in code **Rule:** Purpose is machine-readable metadata bound to data (rules/01 §1) AND an access-control dimension. A service or job declares the purposes it serves; access to purpose-tagged data outside a declared purpose is denied or at minimum audited. ```python # GOOD: purpose is an explicit, logged dimension of data access @requires_purpose("fraud_detection") # declared in service manifest, def get_payment_history(user_id, ctx): # checked against field purpose tags, ... # recorded in the access log # BAD: any authenticated internal service reads any column; "purpose" exists only # in the privacy policy. Marketing job quietly reads fraud-purpose device data — # nothing stops it, nothing records it. ``` Pragmatic ladder (adopt the highest rung you can sustain): 1. Purpose tags in catalog + code review enforcement (minimum). 2. Separate credentials/roles per purpose-bound consumer; DB grants per view/column group (see sota-databases for column/row-level mechanics). 3. Purpose claim in service-to-service auth tokens, enforced at the data API, producing per-purpose access logs (gold standard; also makes DSAR "who accessed my data and why" answerable). **Repurposing is a change event:** using existing data for a new purpose (e.g., training an ML model on support tickets) requires a catalog purpose update, compatibility/lawful-basis review, and possibly consent re-prompt — encode this as a required review on purpose-tag diffs. ## 3. Pseudonymization, tokenization — and why most "anonymization" fails **Definitions that matter legally and technically:** - **Pseudonymized:** identifiers replaced, but a mapping (key, table, or feasible linkage) exists. Still personal data under GDPR (Recital 26). Valuable as a security measure; changes nothing about DSAR/deletion obligations. - **Anonymized:** re-identification not reasonably likely by any party with any auxiliary data — only then does data leave GDPR scope. This bar is far higher than engineers assume. **Common failures to flag on sight (each is a finding, not a nitpick):** | Claim | Why it fails | |---|---| | "We hash the email (SHA-256), so it's anonymous" | Deterministic hash of a low-entropy identifier = trivially reversible by dictionary; also still linkable across datasets — it IS a pseudonymous ID | | "We removed names and emails" | Quasi-identifiers remain: ZIP + birthdate + gender uniquely identifies most of a population; timestamps + location traces are fingerprints | | "It's just internal user IDs" | Linkable to identity via any join; pseudonymous, not anonymous | | "We aggregated it" | Small cells (count < k), max/min values, or repeated overlapping queries re-identify; differencing two aggregates isolates one person | **k-anonymity basics (use as a floor, not a guarantee):** a release is k-anonymous if every quasi-identifier combination matches ≥ k records. Enforce via generalization (age → 5-year bands), suppression (drop rare combinations), and minimum cell-size thresholds on any exported aggregate (k ≥ 10 is a common floor; pick per risk). Know the limits: homogeneity (all k records share the sensitive value) and auxiliary-data attacks still break it — which is why "anonymized" requires a documented review with attack assumptions, not a checkbox. **Tokenization:** replace the sensitive value with a random token; the vault holding the mapping becomes the high-security zone, and downstream systems fall out of sensitive scope. This is THE PCI DSS descoping move (rules/04 §5) and works for SSNs/bank details too: ```text GOOD: checkout posts card → tokenization service/PSP; app DB stores tok_8f3a...; fraud, refunds, reporting all operate on tokens. PCI scope = the vault + capture path, not your whole platform. BAD: orders table stores PAN "for refund convenience" → entire DB, its backups, its replicas, every service with DB access, and every admin laptop with a SQL client are now in PCI scope. ``` Format-preserving encryption is a variant when downstream systems require the original format; treat keys per sota-secrets-management. **If you need a linkable pseudonym, build it honestly** — keyed, salted, rotatable, scoped — not a bare hash: ```python # GOOD: keyed pseudonymization — per-context key from KMS; unlinkable across # contexts; rotating the key severs all linkage (a deletion lever) def pseudonym(user_id: str, context: str) -> str: key = kms.get_key(f"pseudo/{context}") # per-purpose key, access-logged return hmac_sha256(key, user_id).hex()[:32] # BAD: sha256(email) — dictionary-reversible (emails are low-entropy), # globally linkable (same input → same output everywhere, forever), # irrevocable (no key to rotate or destroy). This is a tracking ID, not privacy. ``` Per-context keys give you compartmentalization (analytics pseudonyms can't join against support pseudonyms) and a kill switch (destroy the context key → pseudonyms in that context become unlinkable noise — crypto-shredding's cousin, rules/03 §4). ## 4. Differential privacy & privacy-preserving analytics — when aggregates are exposed **Rule:** If you expose statistics computed over personal data (public dashboards, customer-facing benchmarks, partner data shares, published research), per-record suppression is not enough — use differential privacy (calibrated noise with a privacy budget) or strict k-thresholding with query auditing. DP is the only approach with a formal guarantee against differencing attacks; mature libraries (OpenDP, Google DP building blocks) make it practical for counts/sums/means. Don't hand-roll noise mechanisms; budget composition is where amateurs leak. ```sql -- GOOD: aggregate endpoint with k-threshold enforced in the view itself, -- so no caller can extract small cells CREATE VIEW public_usage_stats AS SELECT region, plan, count(*) AS users, avg(events_30d) AS avg_activity FROM accounts GROUP BY region, plan HAVING count(*) >= 10; -- cells under k are suppressed, structurally -- BAD: the dashboard API accepts arbitrary filters over raw events; an analyst -- (or a partner with API access) filters to zip=94110 AND age=87 AND plan=pro -- → cohort of one. Re-identification by query, no breach required. ``` **Decision ladder for analytics generally (prefer the earliest viable rung):** 1. Don't collect (feature flags + server metrics may answer the question). 2. Collect aggregates only, client-side or at ingestion (no event-level PII at rest). 3. Event-level, first-party, pseudonymous, short raw retention (e.g., 90 days → aggregate), no third-party sharing. 4. Third-party analytics: processor with DPA, EU-hostable if needed, IP truncation on, identify-calls minimal (rules/01 §4). Session replay and heatmap tools deserve special suspicion: they exfiltrate form contents and screens wholesale — require explicit masking allowlists, not blocklists. ## 5. Default-private settings **Rule:** Every setting affecting personal-data exposure defaults to the most private option (GDPR Art. 25 "data protection by default"). Visibility = private; optional analytics/marketing = off until consent; discoverability (search by email/phone) = opt-in; new fields excluded from existing exports/shares until reviewed. ```text GOOD: new "activity status" feature ships default-off with a consent-gated toggle. BAD: profile photos default public "for engagement"; telemetry opt-out buried in settings; new column auto-included in the partner data feed because the export does SELECT *. ``` That last one is an architectural rule: **no `SELECT *` across trust boundaries.** Exports, feeds, webhooks, and API responses enumerate fields explicitly so that adding a column never silently widens disclosure. ## 6. LINDDUN threats → build patterns (quick map) When a LINDDUN session (sota-threat-modeling) raises a threat, these are the default mitigations in this skill: | LINDDUN threat | Primary mitigation here | |---|---| | Linkability | Per-context pseudonyms (§3), purpose-scoped access (§2), compartmentalized keys | | Identifiability | Minimization/precision reduction (§1), k-threshold & DP on releases (§4) | | Non-repudiation (user can't deny) | Minimize identity binding in records; aggregate where intent allows | | Detectability (existence leaks) | Uniform responses/timing on lookup APIs; don't confirm account existence | | Disclosure of information | Projections (§1), tier-bound encryption & access (rules/01 §2) | | Unawareness | Notices + consent records at collection (rules/03 §1), default-private (§5) | | Non-compliance | Catalog + CI gates (rules/01), retention automation (rules/03 §6) | ## 7. Retrofit order for existing systems (AUDIT → remediation) When auditing a system that was not built this way, recommend remediation in this order (highest risk-reduction per effort first): 1. Stop the bleeding: kill unpurposed collection, PII logging, `SELECT *` exports. 2. Tokenize/segregate the highest-tier data (cards, special-category) to shrink scope. 3. Add retention enforcement (rules/03 §6) — drains the historical pool. 4. Bind purpose/classification annotations + CI gate (rules/01) — stops regression. 5. Tighten access to purpose-scoped roles. 6. Re-evaluate claimed anonymization with §3's failure table. ## Audit checklist - [ ] Every collected field traces to a named feature/purpose; sample 10 fields and demand the consumer code path - [ ] Precision minimized where full precision is unused (GPS, full DOB, full IP) - [ ] Internal APIs return minimal projections; no whole-user-object passing or `SELECT *` across trust boundaries - [ ] Purpose tags exist and are enforced by at least code review; access roles segmented by purpose for high tiers - [ ] New-purpose reuse of existing data triggers a documented review (check git history of purpose tags) - [ ] No "anonymized" claim rests on hashing identifiers or removing direct identifiers only; quasi-identifier risk assessed, k-threshold on aggregate releases - [ ] Cardholder/SSN-class data is tokenized or vaulted; verify no raw values in app DB, logs, or backups - [ ] Public/partner-facing aggregates use DP or k ≥ threshold with documented rationale - [ ] All privacy-affecting settings default private/off; check the three most recent features - [ ] Session replay / heatmap tooling absent or mask-allowlisted and consent-gated -
03-consent-and-user-rights.md 16.7 KB
# 03 — Consent & User Rights Engineering User rights (access, export, rectification, deletion, objection, consent withdrawal) are product features with APIs, SLAs, state machines, and tests. Most regimes give you ~30–45 days to fulfill them (GDPR: one month, extensible; CCPA/CPRA: 45 days) — verify the active SLA for your regimes (rules/04). The hard part is never the endpoint; it is propagation through every copy of the data (which is why rules/01 §5's derivative graph must exist first). ## 1. Consent is a state machine, not a boolean **Rule:** Model consent as an append-only, versioned record per (subject, purpose), with explicit states and transitions. Never a `marketing_ok boolean` column — that loses when, to what text, how granular, and whether it was withdrawal or absence. ```sql -- GOOD: append-only consent ledger; current state = latest event per (subject, purpose) CREATE TABLE consent_events ( id uuid PRIMARY KEY, subject_id uuid NOT NULL, purpose text NOT NULL, -- 'marketing_email', 'analytics', 'ml_training' action text NOT NULL CHECK (action IN ('granted','withdrawn','expired','superseded')), policy_version text NOT NULL, -- exact notice text version shown mechanism text NOT NULL, -- 'signup_checkbox','preference_center','cmp_banner','api' occurred_at timestamptz NOT NULL, evidence jsonb -- UI context, locale, IP-derived region (minimized) ); -- BAD: ALTER TABLE users ADD COLUMN gdpr_ok boolean DEFAULT true; -- default-on consent is not consent ``` **Required properties:** - **Granular:** one purpose per consent; no bundling ("agree to terms + marketing + analytics" in one checkbox fails GDPR validity and several state laws). - **Versioned:** record the policy/notice version. When the notice materially changes, prior consents are `superseded` and re-prompt is required for the new scope — never silently widened. - **Revocable as easily as granted:** withdrawal path ≤ effort of grant path (one click from any marketing email; preference center, not a support ticket). - **Checked at point of use:** the email sender, the analytics loader, the ML training-set builder each query current consent state at execution time — not at enrollment time. Stale snapshots of consent are a classic audit finding. - **Propagated to processors:** withdrawal triggers suppression downstream (marketing platform suppression list, ad-platform audience removal, CMP signal to tags). Track propagation lag; minutes-to-hours, not "next quarterly sync." - **Auditable:** for any (subject, purpose, time) you can answer "what was the consent state and how do we know" — this is your defense artifact. ```python # GOOD: point-of-use check — the campaign sender asks the consent service NOW def send_campaign(campaign, recipients): allowed = consent.filter_granted(recipients, purpose="marketing_email") suppressed = consent.suppression_list("marketing_email") # tombstoned/deleted users for user in allowed - suppressed: ... # BAD: the campaign audience was exported to a CSV three weeks ago, when # consent was checked once. 214 users withdrew since. The CSV doesn't know. ``` **Don't use consent as the lawful basis for everything.** Necessary-for-service processing (auth, billing, fraud) typically rests on contract/legitimate-interest bases (rules/04 §1); flooding users with consent prompts for essential processing trains them to click yes and makes real consent meaningless. Map basis per purpose in the catalog; reserve consent for genuinely optional processing. ## 2. Cookie & tracker governance **Rule:** No non-essential tag, SDK, or cookie executes before consent for its category, in jurisdictions requiring opt-in (EU/UK ePrivacy). Technical enforcement, not banner theater: - Tag manager / CMP integration gates script injection by consent category; verify by loading the site fresh in an EU context and diffing network requests pre/post consent. Trackers firing pre-consent = HIGH finding. - Server-side tagging doesn't exempt you — consent state must gate server-side forwarding too. - Maintain a tracker inventory (it's part of rules/01 §4's processor map): each tag → category, vendor, consent purpose, data sent. - **Honor Global Privacy Control:** CCPA/CPRA regulations require treating GPC as a valid opt-out of sale/share; wire the header/JS signal into the same consent state machine as the banner. - Dark-pattern check: reject UIs where "Accept all" is one click and "Reject" is three; EU regulators have fined exactly this asymmetry. ## 3. DSAR — access & export engineering **Rule:** Build one export pipeline that walks the data inventory (rules/01), not N hand-written queries that drift from the schema. **Identity verification first.** A DSAR is an attack vector: an attacker who can "export" someone else's data has industrialized doxxing. Verify to the assurance level of the data: session re-auth for logged-in users; for email-only requests, verify control of the account identifier (signed link + step-up where data is sensitive). Never return data to an unverified third-party request; agent requests (CCPA authorized agents) need documented verification flow. Log the verification evidence with the request. **Completeness = inventory-driven.** The export job iterates the catalog's per-entity field list across SoR + derivatives that hold unique data (support tickets, billing, comms logs). A hand-rolled export that misses tables added since it was written is a silent compliance failure — test it: create a fixture user touching every PII table, export, assert coverage against the catalog (CI test). **Machine-readable:** JSON/CSV with documented schema (GDPR Art. 20 portability requires "structured, commonly used, machine-readable"). PDF-only = finding. **Exclude others' data:** exports must not leak other subjects (e.g., a thread includes the counterparty's messages — redact/scope). Also exclude your secrets: internal fraud scores or moderation notes may be exemptable depending on regime — flag for counsel, don't decide ad hoc in the export code. **Operational:** ticket every request with a deadline clock; deliver via authenticated channel (in-app download with expiring link — not an email attachment); record fulfillment proof. ## 4. Deletion architecture — hard, soft, crypto-shred **Rule:** Choose the mechanism per datastore deliberately; combinations are normal. | Mechanism | What it is | Use when | Watch out | |---|---|---|---| | **Hard delete** | Row/object physically removed | Primary stores, object storage | FK cascades; ensure replicas/indexes follow; DB may retain in WAL/vacuum lag (acceptable, bounded) | | **Soft delete → purge** | Flag now, hard-delete via scheduled purge after grace period | Need short undo/fraud window | Soft delete WITHOUT a running purge job is retention-forever wearing a costume — always pair flag with purge automation and monitoring | | **Crypto-shredding** | Per-subject (or per-tenant) encryption key; destroy the key, all ciphertext becomes noise | Immutable/append-only stores (event logs, Kafka with long retention), backups, archives where physical deletion is impractical | Key management is the whole game (sota-secrets-management); every copy must actually be encrypted under that key — one plaintext side-channel defeats it; key destruction must be provable and itself logged | | **Anonymization in place** | Strip/overwrite identifying fields, keep skeleton row | Referential integrity or aggregate stats require the row (orders for accounting) | Must pass rules/02 §3's bar — overwriting name with 'DELETED' while keeping address and device ID is not anonymization | **Legal holds and retention duties override deletion** (tax records, fraud evidence, litigation hold): model as explicit, reviewable exemptions per data category in the deletion engine — not as "we keep everything just in case." ## 5. Deletion propagation — the graph is the architecture **Rule:** Deletion executes against the system of record and propagates along the recorded derivative graph (rules/01 §5) to ALL of: 1. **Primary DB + replicas** (replication handles it; verify lag-bounded). 2. **Caches** — Redis/memcached keys, CDN-cached responses keyed to the user: explicit invalidation or TTL ≤ your propagation SLA. 3. **Search indexes** — delete-by-query/doc-id; verify index rebuild jobs don't resurrect from a stale source snapshot. 4. **Analytics/warehouse** — delete or null user rows in warehouse + downstream marts; modern lakehouse formats (Iceberg/Delta) support targeted deletes; schedule compaction so deleted data physically leaves storage. 5. **Logs** — bounded retention (e.g., 30–90 d) usually suffices instead of per-user log scrubbing — acceptable and defensible IF retention is short, enforced, and documented; long-retention logs containing PII make per-user deletion intractable (that's the finding: fix the logs, per sota-observability). 6. **Backups** — industry-standard approach: backups expire on schedule; deleted data dies with the backup cycle; document the max window (e.g., "fully purged within retention period + cycle = 35 days") AND ensure restore procedure re-applies deletions (replay tombstones after restore) so restoring a backup doesn't resurrect deleted users. Crypto-shredding makes backups immediate instead. 7. **Third-party processors** — call their deletion APIs (most major processors have them) or ticket per DPA; record confirmation. 8. **ML training data & derived models** — remove from training sets and feature stores; for models already trained, document the position (retrain cadence within which the data washes out; vector-store entries deleted immediately). An ML pipeline with no answer here is a HIGH finding as of 2026. ```yaml # GOOD: deletion fan-out is declared, not hardcoded — generated from the # catalog's derivative graph (rules/01 §5); adding a store without a deletion # handler fails CI deletion_targets: - system: users-db # SoR — executes first mechanism: hard_delete - system: redis-sessions mechanism: key_pattern_invalidate # sess:{user_id}:* - system: opensearch-users mechanism: delete_by_id - system: warehouse.events mechanism: iceberg_delete # + weekly compaction verified - system: mailchimp mechanism: vendor_api_delete # confirmation stored - system: kafka.audit-topic mechanism: crypto_shred # per-user key in KMS - system: feature-store mechanism: row_delete + retrain_window: 30d each_emits: completion_record(system, request_id, count, ts, job_version) ``` **Tombstones:** keep a minimal deletion record (subject ID hash, request ID, timestamp, scope) so you can (a) prove deletion happened, (b) suppress re-ingestion (CDC replays, partner re-syncs, backup restores re-inserting the user), (c) honor "don't recontact" without holding the contact data. The tombstone itself is minimized — never store the deleted data in it. **Proof of deletion:** the deletion job emits a signed/immutable completion record per target system (system, object count, timestamp, job version). This is your DSAR response artifact and your audit evidence. "We ran a script" is not proof. ```text GOOD: DELETE /me → request row (state machine: received → verified → executing → completed) → fan-out workers per target system from the catalog → per-system completion records → tombstone → user notified. End-to-end test in CI creates a full-footprint fixture user, deletes, then asserts zero residue via catalog scan. BAD: UPDATE users SET deleted = true. Email remains in: warehouse, Algolia, Mailchimp, Redis sessions, S3 exports, and the model fine-tuned last month. The DSAR response claimed completion 40 days ago. ``` ## 6. Retention enforced by automation, not policy docs **Rule:** Every datastore/table/bucket/topic holding personal data has a retention period in the catalog AND a running enforcement mechanism. Policy that no machine executes is fiction; auditors and regulators increasingly ask to see the mechanism. Mechanisms by store (set at creation time — IaC templates should require them): - **Relational:** time-based partitioning + automated partition drop (cheap, instant, vacuum-free) > row-level DELETE jobs (verify they actually run and keep up). - **Object storage:** lifecycle rules (transition → expire) on every bucket; bucket without lifecycle rule and without documented "indefinite + why" = finding. - **Streams:** topic retention.ms / compaction; long-retention topics with PII prefer crypto-shred keys. - **Search/cache:** index ILM policies / TTLs. - **Warehouse:** retention-partitioned raw layer; aggregate-then-drop pattern (rules/02 §4 ladder). - **Backups/logs:** retention policies per rules above. **Monitor enforcement:** a scheduled check that samples each store for oldest-record age vs. cataloged retention and alerts on violation. Retention jobs fail silently for months otherwise — the audit query "show me the oldest PII row" should never surprise you. **Pick periods by purpose, not comfort:** retention derives from the purpose's lifetime (account-life + grace; legal minimums for tax/AML; days-to-weeks for raw telemetry). "Indefinite" requires a written justification per category and an owner's signature; default answer is a number. ```hcl # GOOD: retention is a required, validated attribute in the storage module — # you literally cannot create a PII bucket without choosing a number module "bucket" { source = "modules/s3-classified" name = "support-attachments" data_classification = "pii" # required tag, policy-checked (rules/05 §4) retention_days = 730 # creates the lifecycle rule # omit retention_days on a pii bucket -> terraform validation error } # BAD: aws_s3_bucket with no lifecycle block, created in 2021 by a script, # 14 TB of exports, classification unknown, owner left in 2023. ``` ## 7. Rectification & objection — the forgotten rights - **Rectification (GDPR Art. 16; correction under state laws):** user-editable fields are self-serve; non-editable ones (KYC'd identity) get a support flow. The fix executes at the SoR and propagates along the same derivative graph as deletion — a corrected email that still bounces from the marketing platform's stale copy is a failed rectification. - **Objection / opt-out of specific processing (Art. 21; state-law profiling opt-outs):** implement as a per-purpose processing block in the consent state machine (`action: 'objected'`), honored at point of use exactly like withdrawal. Objection to direct marketing is absolute — no balancing test, suppress immediately. - **Automated-decision review (Art. 22; state profiling rules):** where fully automated decisions have significant effects (credit, hiring, pricing), build the human-review path and decision-input logging now — it is also an EU AI Act high-risk overlap (rules/04 §5). ## Audit checklist - [ ] Consent stored as versioned, append-only, per-purpose records with policy version and mechanism; no boolean consent columns; no default-on consent - [ ] Consent checked at point of use (read the sender/tracker/training-job code), withdrawal ≤ grant effort, propagation to processors automated and lag-monitored - [ ] No non-essential trackers fire pre-consent (verify with fresh-session network diff); GPC honored where CCPA/CPRA applies - [ ] DSAR export is inventory-driven with CI completeness test; identity verification precedes disclosure and is logged; output machine-readable; other subjects' data excluded - [ ] Deletion request flow exists as a state machine with deadline tracking; end-to-end deletion test (fixture user → delete → catalog residue scan) in CI - [ ] Soft deletes paired with running, monitored purge jobs - [ ] Propagation covers caches, search, warehouse, logs (or bounded log retention), third-party processors, ML datasets; backup window documented; restore replays tombstones - [ ] Crypto-shredding used (or justified absent) for immutable/long-retention stores; key destruction provable - [ ] Tombstones prevent re-ingestion and contain no deleted data; per-system proof-of-deletion records produced - [ ] Every PII store has cataloged retention + running enforcement (partition drops, lifecycle rules, TTLs) + oldest-record monitoring; "indefinite" only with signed justification - [ ] Legal holds modeled as explicit exemptions, not blanket non-deletion - [ ] Rectification propagates along the derivative graph (test: correct a fixture user's email, verify downstream copies) - [ ] Objection/profiling opt-outs implemented per purpose; direct-marketing objection suppresses unconditionally - [ ] Storage modules require retention + classification parameters at creation (IaC-level enforcement) -
04-regulatory-landscape.md 17 KB
# 04 — Regulatory Landscape: Engineering Obligations What each major regime actually demands from the codebase and infrastructure. Statuses verified against primary/secondary sources as of **June 2026** — laws move; re-verify any date you are about to rely on, and route interpretation to counsel (this is engineering guidance, not legal advice). ## 0. Applicability triage (run this first) Determine regimes from three axes — the answers shape architecture before any control work starts: | Axis | Question | Triggers | |---|---|---| | Data subjects | Where are the people whose data you process? | EU/UK → GDPR/UK GDPR; US states → state patchwork; minors → COPPA/state minor laws (stricter everywhere) | | Data types | What categories? | Payment cards → PCI DSS; US health data as covered entity/BA → HIPAA; health-adjacent outside HIPAA → state health-data laws (e.g., Washington My Health My Data); special-category → GDPR Art. 9 | | Sector/role | What are you? | EU financial entity → DORA; EU essential/important entity → NIS2; AI system provider/deployer in EU → AI Act; processor for enterprises → SOC 2/ISO 27001 expectations (rules/05) | ## 1. GDPR / UK GDPR — the mechanics that hit engineering **Lawful basis (Art. 6).** Every processing purpose in the catalog (rules/01) carries exactly one lawful basis. Engineering consequences: - **Consent** → build the consent machinery of rules/03 §1; withdrawal stops processing, so every consumer must check state. - **Contract** → scope limited to what the service actually requires; doesn't cover marketing or analytics bolted onto a contract basis. - **Legitimate interest** → requires a documented balancing test (LIA) AND an objection (opt-out) path users can actually exercise — that's a feature to build. Basis choice is a counsel/DPO call; representing it in the catalog and enforcing its consequences in code is yours. **International transfers (Ch. V).** Personal data leaving the EU/EEA requires a transfer mechanism: adequacy decision (e.g., EU–US Data Privacy Framework for certified US companies — verify the receiving entity's certification and the framework's current validity before relying on it: a CJEU appeal against the adequacy decision is pending (C-703/25 P), and the June 2026 US Supreme Court ruling ending FTC commissioners' removal protections has raised fresh independent-oversight questions), Standard Contractual Clauses (SCCs) + transfer impact assessment, or BCRs. Engineering consequences: - Know region per processor (rules/01 §4) — including support access and admin consoles, not just storage ("data at rest in EU, support in US" is a transfer). - Be ready to region-pin (§7) if a mechanism collapses (it has happened twice — Safe Harbor, Privacy Shield). **DPIA triggers (Art. 35).** A Data Protection Impact Assessment is required for likely-high-risk processing: systematic large-scale monitoring, large-scale special-category processing, automated decisions with legal/significant effects, new tracking tech, matching/combining datasets. Engineering hook: make "needs DPIA?" a checkbox on design docs for features touching PII; the DPIA references the data map and threat model (LINDDUN via sota-threat-modeling) — don't write it from scratch. **Breach notification (Art. 33/34).** Current law: notify the supervisory authority **within 72 hours** of becoming aware, unless the breach is unlikely to result in risk; notify affected individuals without undue delay when high risk. *Pending change (NOT yet law as of June 2026):* the EU **Digital Omnibus** proposal (published 19 Nov 2025, still in Parliament/Council) would extend the deadline to 96 hours and raise the threshold to high-risk breaches, with a single EU entry point across GDPR/NIS2/DORA. **Build to 72h until the amendment is adopted and applicable.** Mechanics: rules/06. **Other engineering-facing articles:** Art. 25 (privacy by design/default — rules/02), Art. 30 (RoPA — rules/01), Art. 32 (security of processing — encryption, resilience, testing), Arts. 15–22 (user rights — rules/03), Art. 28 (processor DPAs — rules/05 §5). ## 2. US state privacy laws — the patchwork As of 2026, **twenty states** have comprehensive consumer privacy laws in effect; Indiana, Kentucky, and Rhode Island took effect 1 Jan 2026 (verified via IAPP/ MultiState trackers — re-check the tracker for new states before scoping). Applicability thresholds vary (typically 100k consumers, or 25k + revenue share from data sales; Rhode Island as low as 35k). **Engineer to the strictest common denominator instead of per-state logic:** - Rights: access, deletion, correction, portability, opt-out of sale/share/ targeted advertising — one rights pipeline (rules/03) serving all states. - **Honor Global Privacy Control** as a binding opt-out signal (required under CCPA/CPRA regulations and several other states). - Sensitive data: opt-in consent before processing (most states) — same consent machinery as GDPR Art. 9. - CCPA/CPRA specifics: "sale/share" is defined broadly enough to cover ad-tech data flows; "Do Not Sell or Share" link; service-provider contracts restricting use (the DPA analog); California also enforces data-broker registration and (effective 2026) location/health-adjacent restrictions. - CPPA's finalized regulations (effective 1 Jan 2026): **risk assessments** (DPIA analog) for high-risk processing — pre-2026 processing assessed by 31 Dec 2027, first submissions to the CPPA by 1 Apr 2028; **ADMT** significant-decision obligations (pre-use notice, opt-out, access right) from **1 Jan 2027** — same profiling opt-out machinery as rules/03 §7; annual **cybersecurity audits** phase in with certifications due 1 Apr 2028–2030 by revenue tier. - Universal quirk to encode: deletion/access SLAs cluster at 45 days; cure periods vary and are disappearing — don't architect around "we'll fix it if warned." No federal comprehensive law as of June 2026; FTC Act §5 (unfair/deceptive) still applies everywhere — your privacy policy is a promise your code must keep. ## 2b. Children's & minors' data — strictest tier everywhere If your service is child-directed or you have actual knowledge of users under 13 (US COPPA) — or under 16/18 for various state minor laws and GDPR Art. 8 (member-state ages 13–16) — obligations jump sharply: verifiable parental consent, hard minimization, no targeted advertising to minors (several state laws ban it outright for under-18s), default-private settings mandated, age assurance proportional to risk. Engineering consequences: an age signal in the data model that gates features/processing (not a birthday collected for fun — rules/02 §1), separate consent flows, and treating "we don't know users' ages" as a risk position to document, not an exemption. UK Age Appropriate Design Code and similar codes apply the same defaults-private logic. This area is enforcement- heavy and fast-moving — counsel review is non-optional. ## 2c. One capability set serves all regimes Most regimes demand the same engineering machinery; build once, map many: | Capability (built per this skill) | GDPR | US states | HIPAA | PCI | AI Act | DORA/NIS2 | |---|---|---|---|---|---|---| | Data inventory / RoPA (rules/01) | Art. 30 | scoping | risk analysis | scope def. | data governance | asset mgmt | | Rights pipeline (rules/03) | Arts. 15–22 | all rights | access/amend | — | — | — | | Retention automation (rules/03 §6) | Art. 5(1)(e) | required | required | log windows | log retention | — | | Breach clocks (rules/06) | 72h | state matrix | 60d | brand rules | serious-incident | 24h/4h | | Access control + audit logs | Art. 32 | safeguards | §164.312 | req. 7/8/10 | logging | ICT risk | | Vendor/DPA management (rules/05 §5) | Art. 28 | service-provider K | BAAs | TPSP | value chain | third-party register | ## 3. HIPAA — technical safeguards (when you're a covered entity or business associate) Security Rule technical safeguards (45 CFR §164.312), engineering summary: access control (unique user IDs, emergency access, auto-logoff, encryption), audit controls (record and examine activity in systems with ePHI), integrity controls, person/entity authentication, transmission security. **Status note (verified June 2026):** HHS published a proposed Security Rule overhaul (NPRM, Jan 2025) that would make **encryption at rest/in transit and MFA mandatory** (removing "addressable" flexibility), require asset inventories, network segmentation, and tighter BA verification. It is **not final as of June 2026** — but build to it anyway: it codifies what competent engineering already does, and the compliance window after finalization is short (~180–240 days). Breach notification: HHS + individuals within **60 days** of discovery (media for 500+ records); BAs notify the covered entity. BAAs (business associate agreements) are the DPA analog — required before PHI flows to any vendor. ## 4. PCI DSS 4.x — scope is the whole game **Status (verified June 2026):** PCI DSS **v4.0.1** is the only active version (v4.0 retired 31 Dec 2024). All future-dated v4 requirements became **mandatory 31 March 2025** — including authenticated internal vulnerability scans, expanded MFA, automated log review, payment-page script integrity and tamper-detection (requirements 6.4.3/11.6.1 targeting e-skimming), and phishing controls. **The engineering strategy is descoping, not compliance-spreading:** 1. **Never touch PANs if possible:** hosted payment fields/redirect (PSP-hosted iframe) keep card data out of your servers entirely → SAQ-A-class scope. 2. **If you must touch, tokenize at the edge** (rules/02 §3) and segment: the CDE (cardholder data environment) is an isolated network/account with its own access control; everything that can reach it is in scope — flat networks make the whole company in scope. 3. **Never store:** CVV (prohibited post-auth, full stop), track data; PAN only if a documented business need survives challenge, then encrypted with PCI-grade key management. 4. E-commerce pages: inventory and integrity-check all scripts on payment pages (CSP + SRI + tamper monitoring) — mandatory since the 2025 date, and the top gap found in 2025–26 assessments. ## 5. EU AI Act — engineering obligations & current timeline **Timeline (verified June 2026 — moving target, re-verify):** - In force 1 Aug 2024. Prohibited practices + AI literacy: applicable since **2 Feb 2025**. GPAI (general-purpose model) obligations: since **2 Aug 2025**. - General applicability: **2 Aug 2026**. - **Digital Omnibus on AI (adopted: Parliament 16 Jun 2026, Council 29 Jun 2026; OJ publication pending as of early Jul 2026):** postpones high-risk obligations — Annex III (use-case high-risk: employment, credit, education, essential services...) from Aug 2026 to **2 Dec 2027**; Annex I (AI in regulated products) to **2 Aug 2028**. These are now the operative dates; do not treat the delay as a reason to defer architecture — the obligations are data/logging/docs-shaped and cheap at design time, brutal at retrofit time. The omnibus also adds a prohibited practice (AI generation of CSAM / non-consensual intimate content) and postpones Art. 50 synthetic-content marking to **2 Dec 2026** for systems on the market before 2 Aug 2026. **Engineering obligations if your system is high-risk (provider side):** risk management system across lifecycle; data governance (training-data relevance, representativeness, error/bias examination — your data catalog extends to training datasets); technical documentation; **automatic event logging** through the system's life; human-oversight affordances; accuracy/robustness/cybersecurity testing; post-market monitoring. Deployers: use per instructions, human oversight, input-data quality, log retention (≥ 6 months), worker notification. GPAI providers: training-content summary, copyright policy, downstream documentation (systemic-risk tier adds evals and incident reporting). Transparency (Art. 50): disclose AI interaction (chatbots), machine-readable marking of synthetic content / deepfakes — applicability aligned with the 2026–27 dates above. ## 6. DORA & NIS2 — sector awareness - **DORA** (EU financial entities + their critical ICT providers): applicable since **17 Jan 2025**, first enforcement cycle underway in 2026. Engineering surface: ICT risk management, **major-incident reporting (initial notification within 4h of classifying as major / max 24h from awareness; intermediate 72h; final ≤ 1 month)**, digital operational resilience testing (TLPT for significant entities), ICT third-party register and contract clauses. If you sell to EU banks/insurers, expect DORA clauses in contracts even as a vendor. - **NIS2** (EU essential/important entities — energy, transport, health, digital infra, cloud/DNS, managed services...): transposition deadline was Oct 2024; many member states transposed late (infringement procedures ran through 2025–26) — check the national law that applies to you. Incident reporting: **early warning 24h, incident notification 72h, final report ≤ 1 month**. Management liability and supply-chain security requirements flow down to vendors. ## 7. Data residency & sovereignty patterns When law or contract requires data to stay in-region (GDPR transfers, sector rules, public-sector procurement, China/Russia-style localization, customer contractual residency): - **Region pinning:** per-tenant home region decided at onboarding; all primary storage, backups, and replicas constrained by IaC policy (deny-by-default on cross-region replication; policy-as-code checks — rules/05 §4). Tenant→region routing at the edge. - **In-region processing, not just storage:** compute, queues, caches, telemetry, and supportability paths stay in-region; centralized logging that ships EU user data to a US SIEM defeats the design — keep regional sinks with cross-region access to aggregates/metadata only. - **Key residency / hold-your-own-key:** keys in in-region KMS (or customer-controlled external KMS) so out-of-region copies are useless without in-region keys; combine with crypto-shredding (rules/03 §4). This is the strongest mitigation against extraterritorial-access concerns, though residual legal exposure of a foreign-controlled provider remains a counsel question. - **The hard parts to design for explicitly:** global uniqueness (email→tenant lookup) needs a minimized global directory (identifier + region only); support staff access becomes a transfer — gate cross-region admin reads; analytics prefer per-region aggregation with only anonymous/aggregate roll-up to global. ```rego # GOOD: residency enforced as policy-as-code in CI (rules/05 §4) — # an EU-pinned dataset physically cannot be declared outside allowed regions deny[msg] { r := input.resource_changes[_] r.type == "aws_s3_bucket" r.change.after.tags.data_residency == "eu" not startswith(r.change.after.region, "eu-") msg := sprintf("%s: residency=eu but region=%s", [r.address, r.change.after.region]) } # BAD: residency promised in the enterprise contract, enforced by "the team # knows"; a replication rule added for latency quietly mirrors the bucket # to us-east-1 for eight months. ``` ## Audit checklist - [ ] Applicability triage documented: regimes per data type × subject location × sector; reviewed on entering new markets - [ ] Every cataloged purpose has a recorded lawful basis; legitimate-interest purposes have an LIA and a working objection path - [ ] Cross-border flows enumerated with transfer mechanism per processor (incl. support/admin access paths); DPF certifications verified where relied on - [ ] DPIA trigger checklist embedded in design-review process; existing high-risk processing has DPIAs on file - [ ] Breach capability meets 72h GDPR clock today (rules/06); pending Digital Omnibus changes tracked, not assumed - [ ] US: single rights pipeline covers strictest state requirements; GPC honored; sensitive-data opt-in implemented; state-tracker re-checked this quarter; California risk-assessment/ADMT/cyber-audit deadlines (2026–2030 phase-in) tracked - [ ] Minors: age signal modeled and gating where service could reach under-18s; no targeted ads to known minors; parental-consent flow where required - [ ] HIPAA (if applicable): §164.312 safeguards mapped to controls; encryption + MFA universal (NPRM-proof); BAAs precede every PHI vendor flow - [ ] PCI: scope minimized (hosted fields/tokenization); no CVV at rest anywhere (grep + scanner); CDE segmented; payment-page script integrity controls live - [ ] AI Act: systems classified (prohibited/high-risk/transparency/GPAI); for high-risk candidates, logging + data-governance + documentation designed now against the post-omnibus dates - [ ] DORA/NIS2 (if in sector/supply chain): incident-reporting clocks wired into IR runbooks; third-party register current - [ ] Residency requirements per tenant/market recorded; region pinning enforced by policy-as-code; key residency where required - [ ] All dates/statuses in this file re-verified within the last 6 months against primary sources -
05-audit-ready-engineering.md 13.6 KB
# 05 — Audit-Ready Engineering (SOC 2 / ISO 27001) The goal: evidence as a byproduct of normal engineering, not a quarterly scramble. An audit-ready org can produce any control artifact in minutes because the artifact is generated by the system that implements the control. If audit prep takes a "compliance sprint," the controls don't actually run — and a competent auditor will notice. **Framework status (verified June 2026):** SOC 2 uses the AICPA 2017 Trust Services Criteria with the 2022 revised points of focus — still current; points of focus are illustrative, not mandatory checkboxes. ISO/IEC 27001:**2022** is the only valid certification baseline — all 2013 certificates expired at the transition deadline (31 Oct 2025); the 2022 Annex A has 93 controls in 4 themes, including the newer ones auditors probe (threat intelligence A.5.7, cloud security A.5.23, data leakage prevention A.8.12, data masking A.8.11, secure coding A.8.28, configuration management A.8.9). ## 1. Evidence as a byproduct — design principle **Rule:** For every control, identify the system that already enforces it and make that system's records the evidence. Add tooling only where no system produces a record. Screenshots pasted into spreadsheets are the smell of controls that exist only at screenshot time. | Control area | Enforcing system | Evidence (a query, not a project) | |---|---|---| | Change management | PR-based workflow: branch protection, required reviews, CI gates | Git/PR API export: every prod change → linked PR → approver ≠ author → passing checks | | Access provisioning | IdP + groups-as-code (Terraform/SCIM), joiner-mover-leaver via HR-system trigger | IdP audit log + IaC history: who got access, who approved (the PR), when revoked | | Access reviews | Automated quarterly job: dump entitlements per system → diff vs role matrix → route exceptions to owners → record disposition | Review job output + tickets; deprovision latency report from offboarding events | | Config/hardening baseline | IaC + policy-as-code (§4) + drift detection | Policy CI results, drift alerts, exception register | | Vulnerability mgmt | Scanners in CI + runtime, SLA-tracked tickets (see sota-devsecops) | SLA report by severity from the tracker | | Backups/DR | Scheduled backups + automated restore tests | Job logs + restore-test results with RTO/RPO measurements | | Encryption | KMS + TLS termination config in IaC (sota-secrets-management) | IaC + cloud-config export; CI check that flags unencrypted resources | | Logging/monitoring | Centralized pipeline + alert rules as code (sota-observability) | Alert-rule repo + incident records | | Security awareness/endpoint | MDM + training platform | MDM compliance export, completion reports | **The 90% insight:** if you practice the rest of this skill library (sota-devsecops, sota-secrets-management, sota-observability, sota-databases, sota-cloud-infrastructure, sota-testing), you already operate most SOC 2 Common Criteria and ISO Annex A controls. Audit readiness is mostly **mapping and recording**, not new engineering. Build the control matrix as code: a YAML/CSV in-repo mapping `control ID → implementing mechanism → evidence source → owner`, reviewed like code. ```yaml # GOOD: controls/cc6.1.yaml (excerpt) control: CC6.1 # SOC 2 — logical access iso_27001: [A.5.15, A.8.2] mechanism: "SSO-only access via IdP; groups managed in terraform/iam/; MFA enforced tenant-wide" evidence: - source: idp_audit_log query: "auth events, MFA challenges, last 12 months" - source: github query: "PRs touching terraform/iam/ with approvals" owner: platform-team ``` ```text BAD: a shared drive folder "SOC2 Evidence Q3" full of screenshots taken the week before fieldwork, half of them showing settings changed the same week. ``` ## 2. Scoping and audit mechanics engineers should know - **SOC 2 Type I** = design at a point in time; **Type II** = operating effectiveness over a period (6–12 months). Type II means controls must run all year — you cannot retro-create operating evidence, which is the entire argument for automation. Security (Common Criteria) is mandatory; add Availability / Confidentiality / Processing Integrity / Privacy categories only when customers demand them (the Privacy category overlaps heavily with rules/01–03 machinery). - **ISO 27001** certifies the ISMS (management system): risk assessment → Statement of Applicability (justify inclusion/exclusion of all 93 Annex A controls) → internal audit → management review → surveillance audits annually, recertification at 3 years. Engineers feel it as: documented risk register, control ownership, and the internal-audit findings loop. - **Scope deliberately:** a narrow, honest scope (the production platform and the teams operating it) beats a vague company-wide scope full of exceptions. Scope follows the data flows from rules/01 — auditors trace data, so should you. ## 2b. Access reviews that are real (and cheap) The control auditors test hardest, and the one most often faked. Make it a pipeline, not a meeting: ```text GOOD pipeline (quarterly, ~1 engineer-day total): 1. Collector dumps entitlements: IdP groups, cloud IAM, DB roles, GitHub teams, prod-SSH, admin panels — into one normalized table (system, principal, grant). 2. Diff against (a) last quarter, (b) the role matrix (role → allowed grants), (c) HR roster (catches offboarding leaks mechanically). 3. Only NEW grants + matrix violations + roster mismatches go to humans — each system owner gets a short list with approve/revoke buttons. 4. Revocations execute via the same IaC path that granted them; the run's inputs, decisions, and actions are archived. That archive IS the evidence. BAD: a quarterly calendar event where a manager scrolls a 400-row spreadsheet and replies "all good". Rubber-stamp reviews are a named audit finding (§3.4), and worse: they certify the orphaned admin account you'll meet again in rules/06. ``` ## 3. Common audit findings catalog (fix before the auditor finds them) 1. Offboarded users with live access (orphaned accounts, lingering API keys, VPN certs) — automate deprovisioning from the HR event; measure latency. 2. Shared/service accounts with no owner, no rotation, interactive use — see sota-secrets-management. 3. Direct-to-prod changes bypassing review (admin pushes, console hotfixes, infra clickops) — branch protection + drift detection close the gap; emergency-change path documented and logged, not an excuse. 4. Access reviews that approve everything ("rubber-stamp reviews") — require diff-based review (only changes + exceptions need human eyes) so attention is spendable. 5. Missing or stale risk assessment / vendor reviews (§5). 6. Untested backups and unexercised incident response (a tabletop with notes counts; "we have a runbook" alone does not — rules/06). 7. No evidence of log review/alert triage — alert-to-ticket linkage solves this. 8. Policy/practice divergence: the policy says 90-day access reviews, reality is ad hoc. **Never let policy promise more than automation delivers** — write policies to match the enforced mechanism, then ratchet. 9. Crypto/TLS exceptions without an exception register and expiry. 10. MFA gaps on admin planes (cloud root, CI/CD, DB consoles) — enforce org-wide, verify with config export. ## 4. Policy-as-code **Rule:** Encode compliance-relevant infrastructure rules as machine-enforced policy in CI and at the platform boundary (admission controllers, org policies, SCPs): encryption-at-rest required, public-bucket prohibition, region allowlists (residency — rules/04 §7), mandatory tags incl. data classification, no `0.0.0.0/0` ingress to data stores, logging enabled on data services. Tools: OPA/Rego, Sentinel, cloud-native policy engines. Two products fall out: prevention (the control) and the CI/audit log of evaluations (the evidence). Exceptions live in a register with owner + expiry, never as silent policy carve-outs. ```rego # GOOD: classification tag is mandatory; untagged data stores cannot ship — # this single policy keeps the rules/01 inventory honest at the infra layer deny[msg] { r := input.resource_changes[_] r.type in {"aws_s3_bucket", "aws_rds_cluster", "aws_dynamodb_table"} not r.change.after.tags.data_classification msg := sprintf("%s: missing data_classification tag", [r.address]) } deny[msg] { r := input.resource_changes[_] r.change.after.tags.data_classification in {"pii", "pii-special", "financial"} not r.change.after.storage_encrypted msg := sprintf("%s: classified data store without encryption at rest", [r.address]) } ``` ```text BAD: a 40-page "Infrastructure Security Standard" PDF requiring encryption and tagging, last opened during onboarding; 60% of buckets untagged; the auditor samples three and finds one public. ``` **Exception register shape:** `policy, resource, justification, compensating control, owner, approved-by, expires`. CI re-fails the build the day an exception expires — exceptions that can't expire are policy changes and should be argued as such. ## 5. Vendor & subprocessor management **Rule:** Vendor onboarding is a gated workflow, tiered by data classification (rules/01) the vendor will touch: | Tier | Vendor touches | Gate | |---|---|---| | 1 | Special-category/regulated PII or prod infrastructure access | DPA + security review (SOC 2/ISO report review, pen-test summary, subprocessor list), residency check, contractual breach-notice clause, annual re-review | | 2 | Ordinary PII | DPA, questionnaire or attestation review, listed in processor map | | 3 | No personal data, no prod access | Procurement basics | Engineering enforcement: a new SDK/integration PR must reference a vendor-registry entry (CI check on dependency/egress allowlists) — this is how the rules/01 §4 processor map stays true. Your own subprocessor list (the one your customers see) is generated from the same registry; customer-facing change notifications (required by most DPAs) trigger from registry diffs. **Continuous-compliance platforms** (the Vanta/Drata/Secureframe category) are useful evidence aggregators and check engines — but they attest to what their integrations can see. They do not replace the control design above, they cannot see your data flows, and "the dashboard is green" is not a substitute for knowing which mechanism enforces which control. Evaluate the current ecosystem at adoption time; treat the platform itself as a Tier-1 vendor (it holds your org's security posture data). ## 6. Working with auditors without derailing engineering - **One interface:** a single owner (security/compliance eng) brokers requests; auditors get read access to the evidence store, not standing meetings with every team. Sample requests ("show 25 random changes from Q2 with approvals") should be answerable by query — if a request requires a human archaeology project, that's a §1 gap to fix, not a heroic effort to repeat annually. - **Population integrity matters:** auditors select samples from populations YOU provide (all changes, all hires, all incidents). Generate populations from systems of record (git, HRIS, tracker) with reproducible queries — a hand-curated population is both an integrity finding and an invitation to re-audit. - **Don't gold-plate mid-period:** changing a control's mechanism mid-Type-II window creates explanation burden. Schedule control upgrades at period boundaries; document the transition. - **Findings are backlog, not shame:** map each finding to an engineering ticket with owner/date; partial remediation with a credible plan reads far better than disputed findings. ## 7. Privacy-specific audit artifacts Beyond SOC 2/ISO, keep these generated and current (they're what a regulator or enterprise customer asks for): RoPA (from the catalog, rules/01), DPIAs (rules/04 §1), consent records (rules/03 §1), DSAR/deletion logs with timing stats (rules/03 §3–5), retention-enforcement monitoring output (rules/03 §6), processor list + DPAs (§5), breach register incl. non-reportable incidents (rules/06), training-data lineage where AI Act applies (rules/04 §5). ## Audit checklist - [ ] Control matrix exists as code: control → mechanism → evidence source → owner; covers SOC 2 CC and/or ISO 27001:2022 Annex A as applicable - [ ] Certification target is ISO 27001:2022 (2013 is dead); SoA current and justified - [ ] Every prod change traces to a reviewed PR; emergency path logged; drift detection on infra - [ ] Access provisioning is IaC/IdP-driven; offboarding automated with measured latency; quarterly diff-based access reviews with recorded dispositions - [ ] Policy-as-code gates: encryption, public exposure, regions, classification tags; exception register with expiries - [ ] Backup restores tested on schedule with recorded results; IR exercised (rules/06) - [ ] Policies promise no more than mechanisms enforce (read policy, then find the mechanism — divergence is a finding) - [ ] Vendor registry tiered by data touched; Tier 1/2 vendors have DPAs + reviews; CI ties new integrations to registry entries; customer-facing subprocessor list generated from it - [ ] Continuous-compliance tooling (if any) treated as evidence aggregator, not control owner; itself risk-assessed as Tier 1 - [ ] Privacy artifacts (RoPA, DPIAs, consent/DSAR/deletion logs, breach register) generated from running systems, producible in minutes - [ ] Access-review pipeline is diff-based with archived runs; sample the latest run for actual revocations (zero revocations ever = rubber stamp) - [ ] Audit populations generated by reproducible queries from systems of record - [ ] Top-10 findings list above swept explicitly: orphaned access, shared accounts, rubber-stamp reviews, MFA gaps on admin planes -
06-incident-breach-readiness.md 13.3 KB
# 06 — Incident & Breach Readiness A personal-data breach is an incident response problem with a regulatory clock attached. The clocks are short (hours, not weeks), start at "awareness," and assume capabilities — scope determination, affected-user enumeration, regulator contact paths, communication drafts — that cannot be built mid-incident. Build them now; exercise them on a schedule. Whether a specific incident is legally reportable is a counsel/DPO decision — your job is to make that decision possible within the clock, with facts. ## 1. Breach classification — the triage questions **Rule:** Encode breach triage as a decision flow your IR process runs in the first hours, producing a written record even when the answer is "not reportable." 1. **Is personal data involved?** Check the data inventory (rules/01) for the affected systems — this is why the inventory must answer "what data lives here" in minutes. No personal data → security incident path only (still recorded; DORA/NIS2 may still apply to significant operational incidents). 2. **What kind of breach?** Confidentiality (disclosure/access), integrity (unauthorized alteration), availability (loss/destruction — yes, ransomware encrypting personal data without exfiltration is still a personal-data breach under GDPR). 3. **Whose data, which regimes?** Subjects' locations + data categories → rules/04 triage → which clocks start (see §2). 4. **Scope:** which subjects, which fields, what period? Enumerate from logs and the affected store — this drives risk assessment and individual notification. "We can't tell who was affected" defaults to "everyone in the store" — strong incentive for the access logging in §3. 5. **Risk to individuals:** identity-theft potential (SSNs, financials), special-category exposure, credential reuse, physical safety. Mitigating factors that genuinely matter: data encrypted with uncompromised keys (this is why encryption + key custody, sota-secrets-management, can render a stolen disk a non-notifiable event under several regimes), tokenized values without vault access, provable non-access. 6. **Processor or controller?** If you're a processor, your duty is typically notifying the controller without undue delay (GDPR Art. 33(2)) and per contract — check your DPAs' notice clauses (often 24–72h contractual). ## 2. Notification clocks per regime (verify before relying — rules/04 caveats apply) | Regime | Notify whom | Clock (as of June 2026) | |---|---|---| | GDPR (current law) | Supervisory authority | **72h** from awareness unless unlikely to result in risk; document even if not notified (Art. 33(5)) | | GDPR | Individuals | Without undue delay if high risk (Art. 34) | | GDPR — pending Digital Omnibus (NOT in force) | Authority | Proposed 96h + high-risk threshold + single EU entry point — track adoption, build to 72h today | | HIPAA | HHS + individuals | ≤ **60 days** from discovery (≥500 records: media + HHS without unreasonable delay; <500: annual log to HHS) | | US state breach laws (all 50 states, separate from privacy acts) | State AGs + residents | Varies: "most expedient time" to fixed 30/45/60-day windows per state — maintain a per-state matrix via counsel | | NIS2 (in-scope entities) | National CSIRT/authority | Early warning **24h**, notification **72h**, final report ≤ 1 month | | DORA (EU financial) | Competent authority | Initial ≤ **4h from classifying as major** (≤ 24h from awareness), intermediate 72h, final ≤ 1 month | | PCI DSS / card brands | Acquirer/brands | Per contract — "immediately"/24h customary; PFI investigation likely | | Contracts (B2B DPAs) | Your customers (you as processor) | Whatever you signed — inventory these clauses; they're often your shortest clock | **Engineering consequence:** the IR runbook embeds the clock table, regulator submission URLs/forms, and DPA notice clauses, and the incident tracker computes deadlines from the awareness timestamp automatically. Discovering the reporting portal during the incident is a self-inflicted wound. ```yaml # GOOD: clocks as config in the IR tooling — opening a privacy-class incident # instantiates deadline tasks with owners incident_class: personal_data_breach awareness_at: "{{ trigger.detected_at }}" deadlines: - { at: +4h, task: "Triage complete: data categories, regimes, prelim scope", owner: ir-lead } - { at: +24h, task: "Counsel/DPO reportability decision recorded", owner: dpo } - { at: +48h, task: "Draft regulator notification from template", owner: dpo } - { at: +72h, task: "GDPR Art.33 submitted OR documented decision not to", owner: dpo, hard: true } - { at: +72h, task: "Customer DPA notices sent per contract matrix", owner: account-mgmt } escalation: page exec sponsor at T-12h for any hard deadline not in_progress ``` ```text BAD: "Legal will tell us when to notify." Legal learns of the incident on day 4, because the engineering bridge never paged them — the 72h clock started at the on-call engineer's first Slack message acknowledging unusual access ("awareness" is when the organization could reasonably know, not when the lawyer reads email). ``` ## 3. Forensics-friendly logging without privacy violations **Rule:** You need logs good enough to scope a breach (who accessed what, when, from where) without the logs themselves becoming a PII lake (rules/01 §3 shadow copy #1). Resolve the tension deliberately: - **Log access events about data, not the data:** `actor, action, object type, object ID, purpose claim, timestamp, origin` — never field values. `read user=8f3a fields=[email,address] by=support-agent-12 case=4411` scopes a breach precisely and contains no PII payload beyond pseudonymous IDs. - **Data-access audit logs on high-tier stores** (DB audit logging, object-store access logs, admin-console logs) are the difference between "3 records accessed" and "assume all 4M" — enable them on every `pii:special-category` and `financial` store; they are also a HIPAA/PCI/SOC 2 requirement. - **Integrity & retention:** audit logs write-once (object lock / append-only sink), clock-synced, retained longer than app logs (1 year+ common; PCI: 12 months, 3 months immediately available) — a *deliberate* exception in the retention catalog, with the legal-obligation justification recorded. - **Pseudonymize in the pipeline:** user identifiers in security telemetry as stable internal IDs, never emails; redaction at the edge per sota-observability. Resolution ID→identity happens at investigation time, access-controlled and itself logged. - **Don't log yourself into a breach:** secrets, session tokens, auth headers, and request bodies in logs both violate minimization and turn log access into account takeover. Treat log stores at the classification tier of the most sensitive thing they contain — which is the argument for keeping that tier low. ```json // GOOD: data-access audit event — scopes a breach, leaks nothing { "ts": "2026-06-12T09:14:03Z", "actor": "svc:support-portal/agent:a8c1", "action": "read", "object": "customer/7f3e22", "fields": ["email","address"], "purpose": "support_ticket:4411", "origin": "10.4.2.17", "decision": "allow" } // BAD: app log that IS the breach when the log store is popped: // "INFO fetched user {email: 'ana@example.com', dob: '1991-02-14', // card_last4: '4242', session: 'eyJhbGciOi...'}" ``` ## 4. Communication readiness — templates before you need them **Rule:** Pre-draft and counsel-review skeletons for (stored with the runbook, parameterized, in version control): 1. **Regulator notification** (per primary regime): nature of breach, categories and approximate counts of subjects/records, DPO contact, likely consequences, measures taken/proposed. GDPR explicitly allows **phased notification** when you don't have everything at 72h — notify with what you have, supplement; never blow the deadline waiting for perfect information. 2. **Individual notification:** plain language; what happened, what data, what you've done, what they should do (password reset, credit monitoring where appropriate), contact channel. No minimizing weasel-words — regulators read these, and "we take your privacy seriously" without facts reads as evasion. 3. **Customer/DPA notification** (processor role): facts, scope, your IR status, their data specifically — enterprise customers will ask for affected-record lists; the scoping capability in §1.4 feeds this. 4. **Status page / press holding statement.** Decision rights matrix (who may declare a breach, who approves external comms — counsel always in the loop) lives with the templates. Exercise the whole path in tabletops at least annually: a scenario, real clocks, drafting from templates, mock regulator submission. Record the exercise — it is itself audit evidence (rules/05 §3.6). ## 4b. Preservation vs. privacy during the investigation Forensics wants to copy everything; privacy law still applies during an incident. Reconcile explicitly in the IR runbook: - **Preserve narrowly:** snapshot the affected systems and relevant log windows under legal hold (an explicit, scoped exemption per rules/03 §4 — not "pause all deletion globally"). Scheduled retention jobs continue everywhere else; blanket deletion freezes that linger for months are themselves findings. - **Forensic copies are classified data:** images and exports containing personal data inherit the highest tier present — encrypted, access-limited to the IR team, inventoried (yes, in the rules/01 catalog, marked `legal_hold:IR-2026-014`), and destroyed on case closure with a recorded disposition. The forensic S3 bucket that outlives the incident by three years is a recurring real-world breach amplifier. - **External responders are processors:** the DFIR firm gets a DPA/appropriate contract before receiving data, appears in the processor map, and returns or destroys data at engagement end. - **Investigation access is itself logged** (§3's audit events) — regulators ask who accessed subject data during the response. ## 5. Post-incident data subject & data obligations **Rule:** Closing the incident is not closing the obligations: - **Individual follow-through:** credit-monitoring offers honored, support channel staffed beyond the announcement week, DSAR spike absorbed (breach announcements reliably trigger access/deletion waves — your rules/03 pipeline takes the load). - **Compromised-data hygiene:** force credential resets, rotate tokens/keys that touched the breach path (sota-secrets-management), invalidate exposed session artifacts; if exfiltrated data included consent or contact records, ensure suppression lists still hold. - **Breach register:** every personal-data incident — including non-reportable ones — recorded with facts, risk assessment, decision and rationale (GDPR Art. 33(5) requires documenting non-notified breaches; auditors ask for the register). - **Corrective-action loop:** findings become tracked engineering work with owners and dates; the regulator's follow-up (and your next audit) will check. If the breach revealed inventory gaps ("we didn't know that bucket had PII"), the remediation includes the rules/01 discovery fix, not just the patched vulnerability. - **Re-verify deletion/retention posture:** breaches disproportionately expose data that should already have been deleted (rules/03 §6). "Why did we still have 2019 records?" is the question every post-mortem must ask — over-retention converts directly into breach scope and fine multipliers. ## Audit checklist - [ ] Breach triage decision flow documented; first-hours questions (§1) answerable — test: pick a datastore, demand "what personal data, whose, which regimes" in under 30 minutes - [ ] Clock table with regulator submission paths and DPA notice clauses embedded in IR runbook; incident tracker auto-computes deadlines from awareness time - [ ] Processor-role obligations mapped: every customer DPA's breach-notice clause inventoried with its clock - [ ] Data-access audit logging enabled on all special-category/financial/regulated stores; write-once, clock-synced, retention ≥ regime minimum and cataloged as a justified exception - [ ] Security telemetry pseudonymized; no secrets/tokens/bodies in logs (sample); log stores classified and access-controlled accordingly - [ ] Affected-subject enumeration capability demonstrated (query from access logs + store snapshot), not asserted - [ ] Encryption/tokenization state per store recorded so "data was protected" claims are provable in a notification decision - [ ] Communication templates (regulator, individual, customer, public) exist, counsel-reviewed, version-controlled; decision-rights matrix current - [ ] Tabletop exercised within 12 months with real clocks and template drafting; record retained - [ ] Breach register includes non-reportable incidents with documented rationale - [ ] Legal-hold mechanism is scoped per case, not a global deletion freeze; forensic copies inventoried, access-controlled, destroyed at closure with disposition record - [ ] External DFIR/forensics vendors covered by DPA and present in the processor map before any data flows - [ ] Post-incident loop verified on last incident: corrective actions tracked to closure, credential rotation done, retention question asked in the post-mortem
-
-
SKILL.md 9.6 KB
--- name: sota-privacy-compliance description: State-of-the-art privacy and compliance engineering guidance for building privacy-respecting systems and auditing existing code for privacy/compliance gaps. Use when work involves privacy, GDPR, PII, personal data, consent, data retention, deletion, DSAR (data subject access requests), SOC 2, ISO 27001, HIPAA, PCI DSS, compliance evidence, data residency/sovereignty, data classification, anonymization/pseudonymization, breach notification, or EU AI Act obligations — whether designing new data flows, implementing user-rights features (export/delete), preparing for an audit, or reviewing a codebase for places personal data is over-collected, under-protected, retained forever, or impossible to delete. --- # SOTA Privacy & Compliance Engineering Engineering-facing privacy and compliance architecture: how to design, build, and audit systems so that data protection is a property of the code and infrastructure, not a binder of policy documents. Compliance that lives in schemas, TTLs, IAM policies, and CI checks survives staff turnover and audit scrutiny; compliance that lives in wiki pages does not. > **This is engineering guidance, not legal advice.** Regulations change, vary by > jurisdiction, and turn on facts about your business that code review cannot see. > Regulatory facts below were verified against primary sources as of June 2026 — > re-verify deadlines and statuses before relying on them, and route legal > interpretation (lawful basis selection, contract terms, breach reportability > decisions) to qualified counsel or your DPO. This skill tells you how to build > the machinery those decisions require. **Related skills — reference, don't duplicate:** - `sota-databases` rules/06 — DB-level PII mechanics (column encryption, row-level security, GDPR-friendly schema design) - `sota-secrets-management` — credentials, KMS, key rotation (crypto-shredding depends on it) - `sota-observability` — log redaction and PII-safe telemetry pipelines - `sota-threat-modeling` — LINDDUN privacy threat modeling methodology - `sota-code-security` / `sota-devsecops` — vulnerability management, supply chain (SOC 2/ISO control overlap) ## BUILD mode When designing or implementing systems that touch personal data: 1. **Inventory first.** Before writing a schema or integrating a vendor, classify every field you intend to collect and record where it flows (rules/01). The cheapest control is the field you never store. 2. **Annotate at the source.** Schemas, structs, and API contracts carry classification and purpose annotations; tooling derives the data map from code, not the other way around (rules/01, rules/02). 3. **Build user rights as features, not afterthoughts.** Export, deletion, and consent are product capabilities with APIs, state machines, and tests — design them with the first table, because retrofitting deletion into a 200-table schema with denormalized copies is a quarter-long project (rules/03). 4. **Automate retention.** Every datastore gets a TTL, lifecycle rule, or partition-drop schedule at creation time. "We'll clean it up later" is how seven-year-old PII ends up in a breach disclosure (rules/03). 5. **Emit evidence as a byproduct.** Access reviews, change approvals, and config baselines should fall out of normal engineering workflow (PRs, IaC, IdP logs) so audits are queries, not scrambles (rules/05). 6. **Know your regimes.** Check rules/04 for which regulations the system triggers (data types × subjects' locations × sector) before architecture freezes — data residency and breach-clock requirements shape topology. ## AUDIT mode When reviewing an existing codebase/infrastructure for privacy and compliance gaps: **Process:** (1) Build or obtain the data inventory — grep schemas, API payloads, log statements, analytics events, object storage for personal data (rules/01 has discovery patterns). (2) Trace lifecycle per data category: collection → purpose → storage → sharing → retention → deletion. (3) Test user rights paths end-to-end (does deletion actually propagate?). (4) Check evidence trails for auditable controls. (5) Map findings to applicable regimes (rules/04). **Severity conventions:** | Severity | Meaning | Examples | |---|---|---| | CRITICAL | Active violation with regulatory/breach exposure; fix now | PII in world-readable bucket; deletion endpoint that doesn't delete; special-category data collected without any consent record; cardholder PANs stored unencrypted in app DB | | HIGH | Violation likely under normal operation or on first DSAR/audit/breach | No deletion propagation to backups/analytics; consent not versioned or not propagated to processors; no retention enforcement anywhere; PII in logs shipped to third party without DPA | | MEDIUM | Gap that degrades posture or audit readiness | Classification annotations missing; data map stale/manual; soft-delete only with no purge job; access reviews manual and undocumented | | LOW | Hardening/hygiene | Cookie banner lacks granular toggles; export format not machine-readable; missing purpose comments on schema fields | **Finding format:** ``` [SEVERITY] <title> Location: <file:line / table / bucket / service> Data: <what personal data, what classification tier> Regimes: <GDPR Art. X / CCPA / HIPAA / PCI DSS req N / SOC 2 CC-N — as applicable> Issue: <what is wrong, lifecycle stage affected> Impact: <realistic consequence: fine exposure, breach scope, audit failure, DSAR failure> Fix: <concrete engineering remediation> Evidence: <how you verified — query, code path, test> ``` Report findings grouped by data lifecycle stage (collection / storage / sharing / retention / deletion), not by file — that is how regulators and auditors think. ## Rules index | File | Read this when... | |---|---| | [rules/01-data-inventory-classification.md](rules/01-data-inventory-classification.md) | Starting any privacy work; building/auditing a data map; defining classification tiers; hunting PII in schemas, logs, buckets, backups, analytics; mapping flows to processors | | [rules/02-privacy-by-design.md](rules/02-privacy-by-design.md) | Designing schemas/APIs that touch personal data; minimization and purpose limitation in code; choosing pseudonymization vs anonymization vs tokenization; exposing aggregate stats; evaluating re-identification risk | | [rules/03-consent-and-user-rights.md](rules/03-consent-and-user-rights.md) | Building consent management, cookie/tracker governance, DSAR export, deletion (hard/soft/crypto-shred + propagation), or retention automation; auditing whether user rights actually work | | [rules/04-regulatory-landscape.md](rules/04-regulatory-landscape.md) | Determining which regimes apply; GDPR engineering mechanics (lawful basis, transfers, DPIA, 72h); US state laws; HIPAA; PCI DSS 4.x scoping; EU AI Act timeline; DORA/NIS2; data residency architecture | | [rules/05-audit-ready-engineering.md](rules/05-audit-ready-engineering.md) | Preparing for SOC 2 / ISO 27001; automating evidence; mapping controls to engineering practice; vendor/subprocessor management; policy-as-code; avoiding common audit findings | | [rules/06-incident-breach-readiness.md](rules/06-incident-breach-readiness.md) | Building breach response capability; classification (is it reportable?); notification clocks per regime; forensics-friendly logging without privacy violations; post-incident obligations | ## Top 10 non-negotiables 1. **No unmapped personal data.** Every field of personal data has a recorded classification, purpose, owner, retention period, and list of systems it flows to. Unmapped data is unprotectable data. 2. **Collect the minimum.** Each field collected must trace to a specific, documented purpose. A field without a purpose is deleted, not "kept just in case" — it is pure liability with zero value. 3. **Deletion must actually delete.** A deletion request propagates to primary stores, replicas, caches, search indexes, analytics, ML training sets, and is handled for backups (expiry or crypto-shred). A soft-delete flag alone is a finding, not a deletion architecture. 4. **Retention is enforced by machines.** TTLs, object lifecycle rules, partition drops — running and monitored. A retention policy with no automated enforcement is fiction. 5. **Consent is versioned, granular, revocable state** — recorded with timestamp, policy version, and scope; checked at point of use; revocation propagates to processors. Never inferred, never a boolean column named `gdpr_ok`. 6. **No PII in logs, URLs, or analytics events** unless explicitly classified, redaction-tested, and retention-bounded (see sota-observability for pipeline mechanics). Logs are the most common shadow copy of personal data. 7. **Pseudonymized ≠ anonymous.** Data that can be re-linked (hashed emails, "anonymized" user IDs, quasi-identifier combinations) is still personal data. Treat claimed anonymization as a re-identification risk to verify, not a label to trust. 8. **Encrypt personal data at rest and in transit, with keys you can destroy.** Key-per-user or key-per-tenant where deletion/residency demands it (crypto-shredding); keys managed per sota-secrets-management. 9. **Cross-border flows are deliberate.** Know which regions data lives in and transits; region-pin where required; every processor/subprocessor has a DPA and appears in the data map before the first byte flows. 10. **Evidence or it didn't happen.** Access reviews, consent records, deletion proofs, DPIAs, breach timelines — generated and retained automatically. If you cannot produce the artifact in minutes, the control will fail its audit.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.