# Official sources

Use this reference only when you need source grounding for Databricks Lakehouse engineering or Azure service behavior, or the detailed source list.

## Databricks and Azure documentation

Use these as starting points, not as proof of the user's live workspace state:
- https://docs.databricks.com/en/lakehouse/index.html
- https://docs.databricks.com/en/connect/storage/azure-storage.html
- https://learn.microsoft.com/en-us/azure/databricks/connect/storage/tutorial-azure-storage
- https://docs.databricks.com/en/clusters/cluster-config-best-practices.html

## Grounding rule

Official documentation explains Databricks and Azure service behavior. It does not prove the user's current workspace, cluster configuration, storage credential state, or operational environment. Prefer read-only workspace MCP or CLI evidence, repository evidence (Terraform/IaC), or sanitized user-provided evidence for current-state claims.

## Current documentation refresh (2026-06-17)

Service facts from official docs:

**Medallion architecture:** Bronze (raw ingestion), Silver (cleaned/conformed), Gold (business-level aggregations). Delta Lake underpins all layers with ACID transactions, schema enforcement, and time travel.

**ADLS Gen2 access:** Unity Catalog manages storage access via external locations and storage credentials. Storage credentials reference an Access Connector (Microsoft.Databricks/accessConnectors) with a system-assigned or user-assigned managed identity. ADLS Gen2 accounts must have hierarchical namespace (HNS) enabled; flat namespace accounts are not supported for Unity Catalog external locations.

**Access Connector:** The preferred managed identity model. The Access Connector's managed identity is granted Storage Blob Data Contributor (or Reader) on the ADLS Gen2 container. Direct service principal credential injection into clusters is discouraged.

**Credential passthrough DEPRECATED:** Azure Active Directory (now Microsoft Entra ID) credential passthrough is deprecated as of Databricks Runtime 15.0 and will be removed. Migrate to Unity Catalog storage credentials and external locations.

**Cluster access modes:** Dedicated (formerly Single User) clusters are Unity Catalog-compatible. Standard (formerly Shared) clusters support Unity Catalog in DBR 13.3 LTS+. No Isolation Shared clusters do not support Unity Catalog. Cluster policies (Premium plan) enforce access mode compliance.

**AKV-backed secret scopes:** Secret values stored in Azure Key Vault are read-only from Databricks; Databricks cannot write back to AKV. Use Vault access policy model (not RBAC model) when the workspace requires AKV-backed scopes.

**Network isolation:** VNet injection places the Databricks control plane and data plane in the customer VNet. Private Link further isolates the workspace front-end and back-end. Both are required for high-compliance environments.

**Certification reference:** DP-750 (Azure Databricks Data Engineer Associate) covers Lakehouse architecture and Unity Catalog integration fundamentals.

Review implications:
- Do not approve credential passthrough patterns; require migration timeline and Unity Catalog external location design.
- Require cluster access mode evidence and policy enforcement before approving production cluster configuration.
- Documentation cannot prove the user's actual cluster state, storage credentials, or VNet configuration.

# Safety checklist

Use this reference before privileged, destructive, compliance-impacting, or production-impacting recommendations.

## Non-negotiables

- Never ask users to paste access tokens, service principal secrets, client secrets, connection strings, storage account keys, SAS tokens, or customer identifiers into chat.
- Use read-only workspace MCP or IaC repository evidence for live state when available; otherwise use sanitized user evidence or official documentation and label the evidence level.
- Do not invent cluster IDs, workspace URLs, storage account names, Access Connector resource IDs, external location paths, or live configuration state.
- Require explicit user approval before privileged, destructive, compliance-impacting, or production-impacting recommendations.
- Use current official Databricks and Microsoft Learn documentation for service behavior when the answer depends on platform details.
- Keep remediation least-privilege, reversible, and scoped to the requested workspace or storage boundary.
- Static review only: never execute cluster create/edit, storage credential create, external location create, or ADLS Gen2 configuration changes against live infrastructure. Production infrastructure changes are live-guard gated (escalate).

## Stress checks

- What can expose ADLS Gen2 data to unintended identities (misconfigured managed identity, open container ACL)?
- What deprecated credential passthrough pattern creates a compliance or security gap?
- What cluster access mode violation bypasses Unity Catalog enforcement?
- What AKV-backed secret scope misconfiguration leaks secrets or creates read-path failures?
- What VNet or Private Link gap allows data exfiltration or unintended public access?
- What rollback or validation path is unproven?

## Evidence labels

Use `live evidence`, `repo evidence`, `user-provided evidence`, `documentation-based`, or `inference`. Documentation alone never proves the user's live cluster state, storage credential configuration, or network topology.

# Workflow and output contract

Use this reference only when performing the full engineering review, incident triage, implementation guidance, or production-readiness pass.

## Review domains

Check these areas before giving a verdict:

- Architecture: medallion layer design (bronze/silver/gold), Delta Lake table properties, schema enforcement
- Storage access: ADLS Gen2 HNS enabled, external location design, storage credential (Access Connector managed identity)
- Credential passthrough: detect deprecated pattern; require migration plan to Unity Catalog access controls
- Cluster access mode: Dedicated vs Standard vs No Isolation Shared; cluster policy enforcement via Premium plan
- Secret management: AKV-backed secret scope read-only semantics, Vault access policy model
- Network isolation: VNet injection configuration, Private Link workspace and data plane endpoints
- Production posture: service principal identity, Access Connector managed identity, no interactive-user storage access

## Safe workflow

1. **Frame scope**
   - Workspace/metastore/cluster/environment:
   - Business criticality and owner:
   - Data classification and compliance driver:
   - Required outcome:
   - Explicit non-goals:
2. **Collect evidence**
   - Prefer read-only workspace MCP evidence, repository IaC (Terraform), notebook/SQL source, or sanitized user-provided evidence for current-state claims.
   - Otherwise inspect official documentation.
   - Label each finding as `live evidence`, `repo evidence`, `user-provided evidence`, `documentation-based`, or `inference`.
3. **Stress-test risk**
   - What can expose storage data to unintended identities?
   - What deprecated patterns create a compliance or security gap?
   - What cluster mode violations break Unity Catalog enforcement?
   - What network paths bypass Private Link isolation?
   - What evidence is missing?
4. **Recommend the smallest safe action**
   - Prefer Unity Catalog managed access, managed identity, least-privilege cluster policies, and validated rollback.
   - If the safest action is to stop and gather evidence, say that plainly.
   - Production cluster/storage/network changes are live-guard gated (escalate).

## Output contract

Return this structure:

```markdown
# Databricks Lakehouse Engineering Review: <scope>
## Executive verdict
- Status: READY / READY WITH RISKS / NOT READY / NEEDS EVIDENCE
- Biggest risk:
- Evidence level:
## Scope and assumptions
- Confirmed:
- Unknown:
- Out of scope:
## Findings
| Severity | Finding | Evidence | Why it matters | Minimum safe action |
|---|---|---|---|---|
## Recommended actions
1. <action> — owner: <owner>, validation: <check>, rollback: <rollback>
## Validation
- Commands or checks:
- Expected result:
## Residual risk
- <risk or explicit none>
```