Claude Skill

lance-format

Deep reference for Lance v11 - the open columnar lakehouse format for multimodal AI - and its Rust crate workspace plus pylance. Covers the 2.x file format and structural encodings, the table format (manifests, fragments, transactions, OCC), vector / scalar / full-text indexes, M

LLM Mart · 0 points · 28 0 0

Virus-scanned Reviewed automatically before listing.

Full trust report

Download tenequm-skills-skills_lance-format-1cf72df.zip · 381 KB

tenequm/skills 37 1 forks MIT Updated 7d ago

Part of tenequm/skills — 25 skills

Install

skills CLI npx skills add https://github.com/tenequm/skills/tree/main/skills/lance-format

Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install tenequm-skills@llmmart

Git git clone https://github.com/tenequm/skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole tenequm/skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Lance v13 reference

Lance is an open columnar format for multimodal AI - "a columnar data format that is 100x faster than Parquet for random access." It is not one format but a stack of interoperating specs: a file format, a table format, index formats, catalog specs, and a namespace client spec. The Rust workspace at lance-format/lance implements all of them plus Python (pylance) and Java bindings.

This skill tracks v13.0.0-beta.4 (the lance-format/lance git tag), the current development frontier; v12.0.0 is the stable pin, released 2026-09-17. Pin against tags, not main - Lance ships beta tags every few days and next-format encodings can change. Version landscape below.

Three layers of reference, load what the task needs:

The deep reference - any concrete schema, parameter, proto, or constraint. Split by topic:

File in `references/`	Covers	Sections
`format-file.md`	What Lance is, the 26 crates, file format, data types	1-4
`format-table.md`	Dataset layout, manifests, fragments, schema evolution, versioning/tags/branches, row IDs, transactions + OCC, MemWAL	5-10
`indexes.md`	Vector / scalar / FTS / geo indexes, distributed builds	11-12
`ops.md`	Object store, capability matrix, source map	13, 15, 16
`changelog-v7-v13.md`	The full v7 -> v13 delta	14

Cross-references written as "section N" resolve through references/lance-reference.md.

references/performance.md - ALL performance guidance. Part A routes to the official text and adds the source-derived changes upstream has not documented; Part B is field-verified remote-storage practice. Load for any performance, tuning, maintenance-cost, or "why is this slow" question.
references/docs/ - a verbatim mirror of the official docs (docs/src at the tracked tag): every guide, quickstart, and format spec, unedited. Load when you need the full official text. Directory map below.

references/maintenance.md covers refreshing this skill against a new upstream tag.

Lance vs LanceDB

These are two different things and conflating them produces wrong answers.

Lance - the format and engine. The lance-format/lance repo; the lance /lance-* Rust crates; pylance. It gives you datasets, the file/table format, indexes, commits, scans. Consumed directly by DuckDB, Polars, Ray, Spark, PyTorch, DataFusion, or your own Rust/Python code. This skill is about Lance.
LanceDB - a separate database product (lancedb/lancedb) built on top of Lance. It adds a query-builder API, an embedding registry, rerankers-as-API, multi-language SDK parity, and managed Cloud / Enterprise tiers. Not covered here.

The wider ecosystem (separate repos, own version lines, none covered here): Flink streaming writes (lance-flink), PostgreSQL reads via pglance, a Cypher graph engine (lance-graph), a dataset browser (lance-data-viewer), agentic context management (lance-context), and namespace catalogs for Hive, Polaris, Gravitino, Unity Catalog, and AWS Glue. The canonical docs site is lance.org. Generated per-language SDK docs live at lance-format.github.io/lance-python-doc for Python and javadoc.io for Java - the matching lance-format.github.io/lance-java-doc path 404s.

Linking the lance crate in Cargo.toml means you are using Lance directly - use this skill. For LanceDB internals, the storage layer underneath is still Lance, so this skill remains the authority for the format itself.

The wrapper can hide format features. LanceDB's create_table cannot enable stable row IDs; only pylance's write_dataset(enable_stable_row_ids=True) can. If a format-level capability matters to your design, check whether the wrapper exposes it before assuming the underlying format settles the question - and reach for pylance directly when it does not.

The crate workspace

26 crate directories under rust/. lance is the public entry point - Dataset, scanner, indexes, commits; everything else (lance-table, lance-file, lance-encoding, lance-index, lance-io, lance-core, lance-datafusion, lance-linalg, lance-namespace*, ...) is a layer beneath it. Edition 2024, MSRV 1.91.0, arrow 58, datafusion 54; Python bindings need 3.10+. Full table with roles, versions, and every workspace dep in references/format-file.md section 2.

If you depend on anything below lance, v11 will break you - PRs #8020-#8026 deleted lance-encoding::version with no re-export (LanceFileVersion and ConcreteFileVersion both live in lance-file::version now), removed lance_io::encodings and the previous namespaces, and gave each current format its own versions/v2_{0,1,2,3} module. Section 2.1.

The transaction code moved too (#8053/#8054/#8056): rust/lance/src/dataset/transaction.rs is deleted, replaced by a rust/lance-table/src/transaction/ module tree (builder, conflicts, operation, proto, manifest_build, validate, index_maintenance, row_version, update_map). A lance::dataset::transaction shim still re-exports Operation, Transaction, TransactionBuilder, RewriteGroup, UpdateMap and friends, so the common surface is unbroken - but a symbol the shim omits, or a citation of the old path, needs retargeting.

File format versions

The file format carries a single major.minor version. data_storage_version is set per dataset at creation - but as of v12.0.0 it is no longer fixed once the dataset exists. It is the default for writes that omit a target, not a summary of what the dataset holds: "Create and overwrite establish this default; append, update, merge-insert, and compaction do not change it." An existing V2 dataset can take "2.0", "2.1", "2.2" or "2.3" per operation without rewriting its other files (#8582-#8585), so one dataset can hold data files at several exact V2 versions. V1 and V2 still cannot be mixed. Section 3.

Version	Status	Notes
`0.1` (`legacy`)	read-only	Original format; no longer writable
`2.0`	stable	Removed row groups; null support for lists/FSL/primitives
`2.1`	previous default	Adaptive structural encodings; better integer/string compression; nulls in struct fields; better nested random access. Was the default from Lance 5.0.0 until `v12.0.0-beta.15`
`2.2`	current default (`stable`)	Map type, Blob v2, `VariablePackedStruct`, larger mini-blocks. Required for Map and Blob v2
`2.3`	unstable (`next`)	The current `next` alias target (`V2_3` in the enum). Ships sparse structural pages, which the 2.3 writer now auto-selects under a rep/def budget heuristic

stable now resolves to 2.2, not 2.1 (#8657, beta.15), and 2.2 is the enum #[default], so a dataset created without an explicit data_storage_version is written as 2.2. The change reaches new-dataset creation through DataStorageFormat::default() -> stable_file_version(), and Python's write_dataset inherits it because its default routes through stable. The docs were not updated with it - format/file/versioning.md still only says stable is an "alias for the default version", so the code is the authority here. next resolves to 2.3. Pin an explicit number for deterministic behavior across builds.

2.3 is the only version the code flags unstable; 2.2 never was, and is now what you get by default. The release selectors (LanceFileVersion) are a type distinct from the persisted identity (ConcreteFileVersion). Details, plus the sparse auto-selection rules, in references/format-file.md sections 3.1 and 3.6.

Version landscape

The major is bumped by a bot, not a human: ci/publish_beta.sh re-roots at MAJOR+1 whenever any PR since the release root carries the GitHub breaking-change label - the marker is the label, not a conventional-commit !. A major bump therefore means "some labeled breaking change landed", not a redesign, and a ! without the label bumps nothing. It has now fired on four consecutive lines, which is why none of v9.1.0, v10.1.0, v11.1.0, or v12.1.0 was ever released. The 12.1 line is the clearest case: main took a chore: bump main to 12.1.0-beta.0 commit, and four commits later the bot re-rooted to 13, so release-root/12.1.0-beta.N and release-root/13.0.0-beta.N are the same base commit (c3c9632a2) and no v12.1.0-beta.* tag exists.

Three recent lines did ship a final: v10.0.0 (2026-08-08), v11.0.0 (2026-08-30) and v12.0.0 (2026-09-17). Each sits on a stabilization branch that is not an ancestor of main

normal for a Lance final, not a sign the release is unofficial.

Major	Its breaking theme
v13 (current, `v13.0.0-beta.4`)	`WriteParams` gained `file_writer_options` (#9192 - the one labeled PR that re-rooted the major); lazy page-metadata init changed the `StructuralFieldScheduler` signature and the metadata cache key shape (#7465); `json_extract`/`json_get` no longer route to JSON indices (#9101). Delta below
v12 (`v12.0.0`, 2026-09-17)	`WrappingObjectStore` implementors must add `wrap_paginated` (no default); MemWAL `ShardManifestStore` renamed and narrowed; `lance-namespace` returns response objects; external stores gained predecessor-conditioned publication; namespace merge-insert keys became a list; the caller-provided Writer / `open_part` flow was removed (#9072). Unlabeled but bigger: `stable` -> 2.2 and the IVF_RQ 5-bit default. Net-new format capability: mixed data-file versions. Delta below
v11 (`v11.0.0`, 2026-08-30)	Fragment ids became a dataset-lifetime high-water mark; large internal reorganization of `lance-file` / `lance-encoding`; the first new manifest feature flag since v7 - which was then reallocated before the final. Net-new: covering indexes, `merge_insert` `write_mode`, row-address prefilter. Delta below
v10	Blob APIs preserve null selections; cache keys became opaque BLAKE3 digests (every warm or persisted cache cold-misses, no legacy fallback); async `create_remapper`; MemWAL renamed generation -> SSTable, merge -> compaction (wire-compatible, symbol-breaking)
v9.1 (never released; renamed into v10)	FTS/inverted creation took a `block_size` param. Net-new: Data Overlay Files (cell-level updates without base-file rewrite, unstable + env-gated), sparse structural pages, `lance-index-core`
v9	Python 3.9 dropped; `alter_columns` fails fast when casting an indexed column; FM-Index proto rename made existing FM indexes unreadable; FTS/inverted defaults to on-disk format v2
v8	All index builds unified onto one segment-based lifecycle. Net-new: `lance-derive`, FM-Index, multi-bit IVF_RQ, public `approx_mode`, TOS + GooseFS object stores
v7	MemWAL, branches, the geo/RTree index, the `lance-select` crate, ICU FTS

v12.0.0 is the stable pin and what GitHub Releases marks Latest. crates.io carries finals only (newest lance 12.0.0, no 13.x); PyPI pylance is likewise at 12.0.0. So a beta pin means a git dependency - beta wheels publish to fury.io instead, under the renamed org (https://pypi.fury.io/lance-format), which currently carries pylance-13.0.0b1 through b4.

Full per-tag deltas with every PR citation: references/changelog-v7-v13.md.

The v11 delta

357 commits from v10.0.0-beta.7 to the v11.0.0 final, with 16 breaking-change-labeled PRs (14 through beta.16, plus #8407 and #8535 in the final). Most structural invariants held: 26 crates, 16 transaction ops, CommitConfig.num_retries 20, arrow 58 / datafusion 54, MSRV 1.91.0, Edition 2024, Python 3.10+ - and all of them still hold at v13.0.0-beta.4.

references/changelog-v7-v13.md has the full delta - every PR citation, the per-tag breakdown from v7 forward, the Python/Java surface, and each correctness fix with its trigger condition. Load it for any "what changed / will this break me" question. What follows is only what bites hardest.

Five things that break you at v11:

Fragment ids are a dataset-lifetime high-water mark (#8206) - a format invariant, not just an API. Overwrite no longer restarts ids at 0, an overwrite fragment carrying a deletion file is rejected, and any commit producing duplicate ids is rejected - so datasets written by Lance 0.16 and earlier may still read but no longer commit. dataset.get_fragment(0) after an overwrite must read ids from the manifest. Section 5 - which also covers a resolution hazard on pre-0.10 unsorted manifests that can make a fragment-filtered index cover the wrong fragments.
The file-version types and reader/writer composition moved (#8020-#8026) - lance-encoding::version deleted with no re-export; LanceFileVersion lost PartialOrd/Ord (#8027, #8028), so v >= LanceFileVersion::Next no longer compiles. FileWriter is now an enum with all constructors removed. Most of these break silently at compile time. Section 3.6.
Transaction code moved to lance-table (#8053/#8054/#8056) - see the crate-workspace note above; the lance::dataset::transaction shim covers the common surface.
Operation::Project / Merge gained preserves_nullability (#8347) - a nullability tightening must not set it, and such a projection now conflicts with any concurrent value-write. This closed a real hole where alter_columns could let a racing write land nulls unreadable under the tightened schema. Section 9.2.
The external-manifest protocol changed (#8499) - object storage is authoritative, the external store's put-if-not-exists is a reservation, and a stored ETag must be ignored; a retained one makes readers reject a good manifest with Manifest e_tag mismatch. Section 9.

The manifest feature flags changed - and bit 128 was reallocated before the final. v11 added the first new bit since v7 and moved FLAG_UNKNOWN 128 -> 256. But the bit it added, FLAG_MEM_WAL_INDEX_CATCHUP, was retired again (#8680) and the reclaimed bit handed to FLAG_COVERED_INDEX_METADATA = 128 (#8535) before v11.0.0 shipped. At the final and at v12 there is no index-catchup flag and no require_index_catchup proto field; a shard absent from index_catchup now unconditionally means unknown. Both reader and writer must hold bit 128 or refuse the table. Section 7.

Do not pin anywhere in v11.0.0-beta.4 through beta.17. Those builds treat bit 128 as a MemWAL flag they support, so they open a covering-index dataset instead of refusing it - wrong neighbours, no error. The exposure is inherited by whichever flag takes the bit.

Covering indexes are the v11 net-new format feature (#8535), redefined at v13 (#8856). IndexMetadata.covering_fields (proto field 11) names the columns an index carries values for, so a query projecting only those columns is answered without a base-table take. It is no longer a trailing suffix of fields: it "must be a subset of fields, in the order the index emits them. A column is carried if and only if it is named here", including a column the index is also keyed on - and fields[0] remains a keyed column. Index invalidation stays wide: any index whose fields include the updated column, "whether the index is keyed on it or merely carries it".

The old "no index builder writes carried values yet" no longer holds. V3 IVF auxiliary files can physically carry columns, and "a reader discovers carried columns by exclusion, not by position: any column in the auxiliary file's schema that is not one of the quantizer's internal columns is a carried column", bound to source fields by a new covering_field_ids metadata key. Coverage is now per-segment, not per-index: "one logical index may hold values for some of its segments and not others". VectorQueryProto.covering_projection (field 15) reserves the query-side tag, where absent / present-and-empty / present-and-non-empty are three distinct meanings. Section 11.

Bit 8 was spent in the v12.0.0 final. FLAG_MIXED_DATA_FILE_VERSIONS = 1 << 8 (256) is no longer a reservation pinned equal to FLAG_UNKNOWN: the assert relaxed to FLAG_MIXED_DATA_FILE_VERSIONS < FLAG_UNKNOWN, FLAG_UNKNOWN moved 1 << 8 -> 1 << 9 (512), and the build now both reads and writes mixed-version datasets. It is still carried by STICKY_PAIRED_FLAGS, and a half-set manifest is now a hard error: "Manifest has only one of the mixed data-file-version reader and writer feature bits set, so its semantics are undefined". Section 7.

Bit 1024 is where the docs and the code disagree - trust the code. FLAG_FRAGMENT_REUSE_INDEX = 1 << 10 is declared at rust/lance-table/src/feature_flags.rs:69 and, at v13.0.0-beta.4, that declaration is its only occurrence in the entire tree. It sits above FLAG_UNKNOWN (512), and the supported set is computed as FLAG_UNKNOWN - 1, so a manifest setting it is refused. The spec page meanwhile lists it as reader Yes / writer Yes and puts the unknown boundary at 2048. The docs describe the intended end state; the code has only reserved the constant. Anything you build against tagged FRI today is building against prose, not behavior.

Two LANCE_* env vars landed (from the AMX work, #8540): LANCE_DISABLE_AMX (runtime kill switch) and LANCE_AMX_FP16_CC (build-time compiler override). Grep trap: LANCE_AMX_CFG_* and LANCE_AMX_TILE_COUNT are C macros in amx_fp16.c, not env vars, and LANCE_FACTOR is a substring of BALANCE_FACTOR - a plain LANCE_* grep reports all four as if they were real.

Worth knowing without reading the full delta: FTS gained a document-boundary axis (DocumentGranularity, #7788) whose list_element mode is a third trigger requiring FTS on-disk format v3; transactions above 20 MiB spill out of the manifest entirely (#7881); MemWAL catch-up became derived rather than declared (#8481); transaction proto field 9 is deprecated for field 10 (#7432); compaction gained row/byte budgets plus fragment exclusion (#8235, #8532); merge_insert gained write_mode (#8423); and Python commit conflicts became lance.commit.CommitConflictError, a subclass of OSError, so existing handlers keep working (#8563). Full list with citations in references/changelog-v7-v13.md.

Address-domain indexes stopped falsely claiming compacted fragments (v11, beta.16 or earlier). A rewrite used to advance every index's fragment_bitmap onto the new fragment ids - including ZoneMap, whose stored addresses point into the fragments the rewrite dropped. The Rewrite path now branches on results_are_row_addrs(): an address-domain index gets drop_rewritten_fragments and a full-scan fallback, correct-but-slower instead of stale addresses. Heals only for new compactions: an index already damaged under v10 or earlier must be recreated, and the damage does not self-heal through routine maintenance, because the refreshed fragment_bitmap also makes incremental folds a no-op. Section 11.

Correctness fixes split by whether upgrading is enough. Most are read-path only and heal on upgrade. These do not - they need data rewritten or repaired: #8382, #8669, #8509, #7703, #8539, #8459, #8378, #8482, #8834 (rebuild HNSW - a persisted graph can hold edges to ids it does not contain; lost recall stays lost), #8101 (nullable primary keys silently duplicated rows on every repeat merge_insert; existing duplicates must be removed by hand), #8511, #8427, #8513, #8839, #8904. Conditions for each in references/changelog-v7-v13.md.

The v12 delta

225 commits from release-root/12.0.0-beta.N to the v12.0.0 final, with 7 breaking-change-labeled PRs - the five visible at beta.15 plus #9072 and #9101 in the run-up to the final. No new index types and no new crates; every structural invariant above still holds. The label is a floor, not a ceiling - the two biggest behavior changes in the line carry a conventional-commit ! but no label, so the bot never counted them: the stable -> 2.2 move (#8657, above) and the IVF_RQ 5-bit default (below).

WrappingObjectStore implementors must add wrap_paginated (#8606) - "There is deliberately no default: getting this wrong is either a silent loss of speed or a silent loss of the wrapper, and neither announces itself." Return Some to keep listing pushdown through the wrapper, None to give it up and fall back through inner. One wrapper giving it up gives it up for the whole chain. Anything wrapping the object store fails to compile until updated.
New paged listing: ObjectStore::read_dir_page (#8606) - one page of a prefix's immediate children plus an opaque resume token. The trap: "One page is one request, so a page can hold fewer children than limit asked for and still be followed by more" - walk until the token is None, never until a page comes back short.
MemWAL ShardManifestStore renamed and narrowed (#8640) - read_latest -> latest, read_latest_uncached -> refresh_latest, and write is now crate-private (reach it through commit_update, claim_epoch, or initialize_shard). Existing commit_update closures need no change. Section 10.
lance-namespace 0.8.5 -> 0.11.1 (#8903) - four LanceNamespace methods now return response objects instead of bare values: count_table_rows -> CountTableRowsResponse, query_table -> QueryTableResponse, namespace_exists / table_exists -> their own response types. Callers unwrap; anyone implementing the trait needs the same signature updates.
External manifest stores gained predecessor-conditioned publication (#8800) - put_if_predecessor reserves a version only while the predecessor still carries the identity the writer observed, and commit_after refuses with PrerequisiteFailed, "never a conflict". The hard compile break is the new ManifestLocation.identity field, not the trait methods (all default-implemented). No built-in store implements it. Section 9.
Namespace merge-insert keys became a list (#8915) - on moves from Option<String> to Option<Vec<String>>, with arity-dependent NULL semantics: a single-column key treats NULL as equal to NULL, a composite key uses SQL equality, "under which a NULL key matches nothing - not even a byte-identical NULL".

The lance-namespace pin is no longer one number. #8915 moved the Rust client to 0.12.0 and #8979 moved Java to 0.12.0 as well; only Python still holds >=0.11.1,<0.12, because its generated models still send on as a bare string. Quote a language-specific pin, never one number for all three - and note this split moved once already, so re-check it rather than carrying the pairing forward.

IVF_RQ now defaults to 5 bits per dimension, not 1 (#8936) - roughly a 4.4x index-size increase at the default (upstream's 100M x 768d example: ~10.8 GiB -> ~47.3 GiB). Fast search mode "uses only the 1-bit sign code even when the index stores additional bits", so it pays the storage without using it; set num_bits=1 explicitly to opt out, at the cost of the multi-bit distance estimate and some recall. Sizing formulas in references/indexes.md.

Column slice stitching (#8660) was reverted at beta.9 (#8926) - it "should not ship while the caller-managed replacement in #8923 is being developed". rust/lance-file/src/concat.rs exists again at beta.15, but holds #8923's caller-managed data file parts, not the reverted stitching.

Two proto additions. MemWAL SsTable gained in_memory_bytes, physical_rows and primary_key_bytes (fields 3-5, #8981); all optional, and absence must not be read as zero. FilteredReadOptions gained materialization_readahead_bytes and batch_size_bytes (13, 14) - not a format change under a new rule in protos/AGENTS.md: execution-plan schemas "are wire contracts, not persisted Lance formats". transaction.proto / ann.proto / index.proto are untouched.

Net-new, non-breaking: provider-native bulk copy and a deep-clone concurrency bound (section 13); Python ObjectStoreProvider registration (#8522); BinaryView in the packed blob writer (#8700); caller-managed data file parts (#8923); cleanup of specific versions (#8617); LanceDataset.slice() (#8059) and six more LanceFragment.scanner options (#8429); restored Python index retraining (#8786); namespace-managed clone deprecated to a shim (#8964). Namespace latest-version resolution no longer lists the whole _versions/ prefix (#8679) - on a ~340k-version table that was ~344 list pages, "~25s of pure I/O wait", paid by every open.

Fixes needing a rebuild or rewrite, not just an upgrade: #8779 (rebuild NGRAM indexes), #8510 (rewrite data compacted from uniformly reordered fragments), #8984 (re-drop a resurrected index), #8837 (repair a MemWAL shard below ~2.7KB/row - it cannot be reopened). Full per-PR conditions, plus the much longer list that does heal on upgrade, in references/changelog-v7-v13.md.

Mixed data-file versions LANDED - it is no longer "1 of 6". #8581-#8584 shipped in v12.0.0 (validation, per-operation V2 write targets, propagation across dataset operations, compaction targeting) and #8585 exposed it in the bindings in the v13 line. The proto changed with it: DataStorageFormat.version is now "the default format version used when writing data files", and "each DataFile's version is authoritative for decoding" once the capability is set.

In flight, not landed - do not treat as shipped: generic block v5 compression is still 1 of 10 PRs merged (#8324; #8325-#8333 all remain open). Next big dependency break in the queue: #8997, "upgrade to arrow 59, DataFusion 55, and pyo3 0.29" - still open at v13.0.0-beta.4, so arrow 58 / datafusion 54 still hold. It also gates two outstanding PyO3 advisories (RUSTSEC-2026-0176/0177); rustls was separately patched to 0.23.45 for RUSTSEC-2026-0285 (#9212).

The v13 delta

66 commits from release-root/13.0.0-beta.N to v13.0.0-beta.4, with 2 breaking-change-labeled PRs. No new crates and no new index types; 26 crates, 16 transaction ops, CommitConfig.num_retries 20, arrow 58 / datafusion 54, MSRV 1.91.0, Edition 2024 and Python 3.10+ all still hold.

The !-vs-label rule inverted in this window. All three conventional-commit ! commits (#7465, #9192, #9101) do carry the breaking-change label. Keep treating the label as a floor rather than a ceiling - but this window is the counter-example, not more evidence for the gap.

WriteParams gained file_writer_options (#9192) - the single labeled PR that re-rooted the major. FileWriterOptions { data_cache_bytes, max_page_bytes, keep_original_array } is now reachable from the dataset write APIs in Rust, Python and Java. A zero max_page_bytes is rejected before encoder construction rather than misbehaving later. Anything constructing WriteParams by struct literal fails to compile.
Page metadata is initialized lazily, and the metadata cache key changed shape (#7465). StructuralFieldScheduler::initialize now takes requested_ranges, and the page-scheduler initialize splits into init_ranges() and init_from_buffers(buffers, io) - any external implementor fails to compile. The public DecodeBatchScheduler::try_new kept its signature; the range-aware entry point is the crate-private try_new_with_ranges. The part that bites without a compile error: caching moved from a per-column FieldDataCacheKey to a per-page PageDataCacheKey { column_index, page_index, view_tag }, so every warm or persisted metadata cache cold-misses across this upgrade. The payoff is real - "a cold point/range read's metadata IO is invariant to the column's total page count".
json_extract and json_get no longer route to JSON indices (#9101). Only the four typed accessors (json_get_int / _float / _bool / _string) reach the index; everything else falls back to a full scan. This fixes three real wrong-answer bugs - a quoted-key mismatch that "searched for a quoted key and matched nothing", a Utf8 literal driving an Int64 btree into a panic, and an unsound range because "quoting is not order-preserving (ab < ab! but "ab" > "ab!")". The cost is silent: a json_extract workload that used to hit an index now scans, with no error and no plan warning. Rewrite those predicates onto the typed accessors.

The Fragment Reuse Index gained a versioned on-disk contract (#9136). InlineContent field 1 was renamed versions -> legacy_versions and a tagged transitions list added at field 2, gated on index_version >= 1; mappings are now a oneof of OrderedCompaction or StablePartition. A stable partition "assigns source rows to destination fragments while preserving their relative source order within each destination", which lets FRI reuse existing indices after reclustering - a second use case the v0 model had no concept of. Its physical form is an immutable row-map Lance file with uint16 labels and an LSPC-magic counts matrix. Two hard rules: stable row IDs and tagged FRI are mutually exclusive ("writers must not publish index_version >= 1 on them"), and cleanup "must retain intermediate transitions still needed to translate old addresses". Upstream also softened the old claim - "FRI does not remove conflicts between overlapping rewrites". Section 11.

Net-new, non-breaking: Dataset::frag_reuse_index() is public (#9112) and documented in the performance guide; FileFragment::write_overlay returns a real OverlayWriter (#8761, still env-gated); an hf:// object store with hf_enable_resolve_cache (#9236); Python lance.bitmap.Bitmap, deep_clone() (#9181), base_paths() (#9191) and update_columns(with_offsets=True) (#8891); Java DataStorageVersion, FileWriteOptions and ScanOptions.indexSegments; and namespace table listing finally bounded by read_dir_page (#9165). inline_optimization_enabled flipped true -> false (#9180), which upstream justifies with a -49% write-p50 measurement at 1M entries.

Performance questions

For anything performance-shaped - slow scans or searches, remote/object-storage cost, index maintenance cost, memory sizing, version bloat, benchmarking - load references/performance.md first. Part A routes to the official guidance plus the undocumented source-derived changes; Part B is field-verified practice against S3-compatible storage. The governing rule stays minimize remote calls - fewer commits, fewer scans, fewer round trips - because that is where the order-of-magnitude wins are. The official "Tuning remote scans" section (v11, unchanged at v12) gives a starting point for cross-region or public-internet access, where the cloud default of 64 concurrent requests is too aggressive: LANCE_IO_THREADS=8, fragment_readahead=1, batch_readahead=2, io_buffer_size=64MB. It is a legitimate second move once call volume is already minimized.

AMX-FP16 (#8540, beta.16) is the one v11 performance change that alters results, not just speed: where it engages, IVF partition assignment becomes exact instead of approximate, so recall improves and assignments differ from an older build. It is shape-gated (float16 + dot, dimension >= 32, num_centroids >= 32); everything else keeps the previous path. LANCE_DISABLE_AMX=1 disables it, but reverts assignment to the approximate path too - so an index built with it set is not equivalent to one built without it.

Two cache facts to know before tuning anything remote: Lance has no resident data cache (a Session holds only index and metadata caches, never decoded values, so repeated point reads re-pay object-store IO), and one Arc<Session> shared via DatasetBuilder::with_session lets datasets share it. Cold first search is dominated by paging indexes in - prewarm_index is the remedy. Note that #7465 changes the metadata cache key shape, so the first run after a v13 upgrade re-pays that paging even against a warm or persisted cache. Details and build-time requirements in references/performance.md.

Time travel is not an archive mechanism. Versions look like free history, but the default cleanup reclaims anything older than 7 days and cleanup is part of routine optimize - so a design that treats old versions as the durable record loses it on the first maintenance pass. Keep an explicit archive if you need one.

Official docs mirror

references/docs/ mirrors docs/src of lance-format/lance at the tracked tag, verbatim - 45 markdown files plus 4 diagrams, all directly readable.

Directory	Files	Covers
`guide/`	14	CRUD, performance, object store, distributed write + indexing, JSON, tokenizers, data types, data evolution, blob, arrays, tags/branches, migration, observability
`quickstart/`	4	First dataset, vector search, full-text search, versioning
`format/`	1	Spec-stack overview
`format/file/`	3	Container spec, structural encodings + compression, format versions
`format/table/`	9	Layout, schema, transactions (conflict-resolution matrix), versioning, row-id lineage, branch/tag, MemWAL, data overlay files
`format/index/`	1 + 4 svg	Index lifecycle, fragment coverage, compaction interplay
`format/index/scalar/`	9	fts, fmindex, ngram, btree, bitmap, bloom_filter, label_list (`array_has_any/all`), zonemap, rtree
`format/index/vector/`	1	IVF / PQ / SQ / RQ / HNSW concepts and storage layout
`format/index/system/`	2	Fragment reuse index, MemWAL system index
`integrations/`	1	DataFusion SQL over Lance, incl. JSON functions

Not mirrored: docs/src/images/ (PNG/GIF assets), so image links in the mirrored pages do not resolve - the prose is self-contained, and the four .drawio.svg diagrams are mirrored. Also out by design: community/, examples/, integrations/{index,pytorch,tensorflow}.md; and the landing stubs and contributor files (format/AGENTS.md, format/CLAUDE.md).

A whole tier of docs is not in this repo at all, so it cannot be mirrored and cannot be enumerated from a clone. docs/make-full-website.sh assembles format/catalog, format/namespace, and the integrations/{duckdb,huggingface,spark,ray,trino,context} sections at build time from six sibling repos with their own version lines - the checked-in integrations/index.md links spark/, duckdb and trino as if they were local, but those paths do not exist in the tree. Lance Context and the HuggingFace integration docs are whole nav sections that exist only on the built site. For any of those, read lance.org rather than this mirror. Protobuf message bodies are likewise expanded at build time from protos/ by mkdocs_protobuf, so the mirrored spec pages show %%% proto.message.X %%% placeholders where the site shows a rendered schema.

Files (skills)

references

docs

format

file

encoding.md 46 KB

# Lance Encoding Strategy

The encoding strategy determines how array data is encoded into a disk page. The encoding strategy tends to evolve
more quickly than the file format itself.

## Older Encoding Strategies

The 0.1 and 2.0 encoding strategies are no longer documented. They were significantly different from future encoding
strategies and describing them in detail would be a distraction.

## Terminology

An array is a sequence of values. An array has a data type which describes the semantic interpretation of the values.
A layout is a way to encode an array into a set of buffers and child arrays. A buffer is a contiguous sequence of
bytes. An encoding describes how the semantic interpretation of data is mapped to the layout. An encoder converts
data from one layout to another.

Data types and layouts are orthogonal concepts. An integer array might be encoded into two completely different
layouts which represent the same data.

![Multiple Encodings](../../images/encoding_v_array.png)

### Data Types

Lance uses a subset of Arrow's type system for data types. An Arrow data type is both a data type and an encoding.
When writing data Lance will often normalize Arrow data types. For example, a string array and a large string array
might end up traveling down the same path (variable width data). In fact, most types fall into two general paths. One
for fixed-width data and one for variable-width data (where we recognize both 32-bit and 64-bit offsets).

At read time, the Arrow data type is used to determine the target encoding. For example, a string array and large
string array might both be stored in the same layout but, at read time, we will use the Arrow data type to determine
the size of the offsets returned to the user. There is no requirement the output Arrow type matches the input Arrow
type. For example, it is acceptable to write an array as "large string" and then read it back as "string".

## Search Cache

The search cache is a key component of the Lance file reader. Random access requires that we locate the physical
location of the data in the file. To do so we need to know information such as the encoding used for a column,
the location of the page, and potentially other information. This information is collectively known as the "search
cache" and is implemented as a basic LRU cache. We define a "initialization phase" which is when we load the various indexing information into the search cache. The cost of initialization is assumed to be amortized over the lifetime
of the reader.

When performing full scans (i.e. not random access), we should be able to ignore the search cache and sometimes
can avoid loading it entirely. We _do_ want to optimize for cold scans as the initialization phase is often not
amortized over the lifetime of the reader.

## Structural Encoding

The first step in encoding an array is to determine the structural encoding of the array. A structural encoding
breaks the data into smaller units which can be independently decoded. Structural encodings are also responsible
for encoding the "structure" (struct validity, list validity, list offsets, etc.) typically utilizing repetition
levels and definition levels.

Structural encoding is fairly complicated! However, the goal is to suck out all the details related to I/O
scheduling so that compression libraries can focus on compression. This keeps our compression traits simple
without sacrificing our ability to perform random access.

There are only a few structural encodings. The structural encoding is described by the `PageLayout` message and
is the top-level message for the encoding.

```protobuf
%%% proto.message.PageLayout %%%
```

### Repetition and Definition Levels

Repetition and definition levels are an alternative to validity bitmaps and offset arrays for expressing struct
and list information. They have a significant advantage in that they combine all of these buffers into a single
buffer which allows us to avoid multiple IOPS.

A more extensive explanation of repetition and definition levels can be found in the code. One particular note
is that we use 0 to represent the "inner-most" item and Parquet uses 0 to represent the "outer-most" item. Here
is an example:

#### Definition Levels

Consider the following array:

```text
[{"middle": {"inner": 1]}}, NULL, {"middle": NULL}, {"middle": {"inner": NULL}}]
```

In Arrow we would have the following validity arrays:

```text
Outer validity : 1, 0, 1, 1
Middle validity: 1, ?, 0, 1
Inner validity : 1, ?, ?, 0
Values         : 1, ?, ?, ?
```

The ? values are undefined in the Arrow format. We can convert these into definition levels as follows:

| Values | Definition | Notes                |
| ------ | ---------- | -------------------- |
| 1      | 0          | Valid at all levels  |
| ?      | 3          | Null at outer level  |
| ?      | 2          | Null at middle level |
| ?      | 1          | Null at inner level  |

#### Repetition Levels

Consider the following list array with 3 rows

```text
[{<0,1>, <>, <2>}, {<3>}, {}], [], [{<4>}]
```

We would have three offsets arrays in Arrow:

```text
Outer-most ([]): [0, 3, 3, 4]
Middle     ({}): [0, 3, 4, 4, 5]
Inner      (<>): [0, 2, 2, 3, 4, 5]
Values         : [0, 1, 2, 3, 4]
```

We can convert these into repetition levels as follows:

| Values | Repetition | Notes                                     |
| ------ | ---------- | ----------------------------------------- |
| 0      | 3          | Start of outer-most list                  |
| 1      | 0          | Continues inner-most list (no new lists)  |
| ?      | 1          | Start of new inner-most list (empty list) |
| 2      | 1          | Start of new inner-most list              |
| 3      | 2          | Start of new middle list                  |
| ?      | 2          | Start of new inner-most list (empty list) |
| ?      | 3          | Start of new outer-most list (empty list) |
| 4      | 3          | Start of new outer-most list              |

### Mini Block Page Layout

The mini block page layout is the default layout for smallish types. This fits most of the classical data types
(integers, floats, booleans, small strings, etc.) that Parquet and related formats already handle well. As is no
surprise, the approach used is pretty similar to those formats.

![Mini Block Layout](../../images/miniblock.png)

The data is divided into small mini-blocks. Each mini-block should contain a power-of-two number of values (except
for the last mini-block) and should be less than 32KiB of compressed data. We have to read an entire mini-block to
get a single value so we want to keep the mini-block size small. Mini blocks are padded to 8 byte boundaries. This
helps to avoid alignment issues. Each mini-block starts with a small header which helps us figure out how much
padding has been applied.

The repetition and definition levels are sliced up and stored in the mini-blocks along with the compressed buffers.
Since we need to read an entire mini-block there is no need to zip up the various buffers and they are stored
one after the other (repetition, definition, values, ...).

#### Buffer 1 (Mini Blocks)

| Bytes | Meaning                             |
| ----- | ----------------------------------- |
| 1     | Number of buffers in the mini-block |
| 2     | Size of buffer 0                    |
| 2     | Size of buffer 1                    |
| ...   | ...                                 |
| 2     | Size of buffer N                    |
| 0-7   | Padding to ensure 8 byte alignment  |
| \*    | Buffer 0                            |
| 0-7   | Padding to ensure 8 byte alignment  |
| \*    | Buffer 1                            |
| ...   | ...                                 |
| 0-7   | Padding to ensure 8 byte alignment  |
| \*    | Buffer N                            |
| 0-7   | Padding to ensure 8 byte alignment  |

Note: It is natural to explain this buffer first but it is actually the second buffer in the page.

#### Buffer 0 (Mini Block Metadata)

![Mini Block Layout](../../images/miniblock_meta.png)

To enable random access we have a small metadata lookup which contains two bytes per mini-block. This lookup
tells us how many bytes are in each mini block and how many items are in the mini block. This metadata lookup
must be loaded at initialization time and placed in the search cache.

| Bits (not bytes) | Meaning                             |
| ---------------- | ----------------------------------- |
| 12               | Number of 8-byte words in block 0   |
| 4                | Log2 of number of values in block 0 |
| 12               | Number of 8-byte words in block 1   |
| 4                | Log2 of number of values in block 1 |
| ...              | ...                                 |
| 12               | Number of 8-byte words in block N   |
| 4                | Log2 of number of values in block N |

For all chunks except the last, the lower 4 bits store `log2(num_values)` and `num_values` must be a power of two.
For the last chunk, these bits are set to `0`. The protobuf stores the total number of values in the page, so readers
can derive the final chunk size by subtracting the values from earlier chunks.

#### Buffer 2 (Dictionary, optional)

Dictionary encoding is an encoding that can be applied at many different levels throughout a file. For example,
it could be used as a compressive encoding or it could even be entirely external to the file. We've found the
most convenient simple place to apply dictionary encoding is at the structural level. Since dictionary indices are
small we always use the mini block layout for dictionary encoding. When we use dictionary encoding we store the
dictionary in the buffer at index 2. We require the dictionary to be full loaded and decoded at initialization time.
This means we don't have to load the dictionary during random access but it does require the dictionary be placed
in the search cache.

Dictionary values are stored as a single buffer and compressed through the block compression path. The compression
scheme for dictionary values can be configured separately (see `lance-encoding:dict-values-compression` below).

#### Buffer 2 (or 3) (Repetition Index, optional)

If there is repetition (list levels) then we need some way to translate row offsets into item offsets. The mini
blocks always store items. During a full scan the list offsets are restored when we decode the repetition levels.
However, to support random access, we don't have the repetition levels available. Instead we store a repetition
index in the next available buffer (index 2 or 3 depending on whether the dictionary is present).

The repetition index is a flat buffer of u64 values. We have N \* D values where N is the number of mini blocks
and D Is the desired depth of random access plus one. For example, to support 1-dimensional lookups (random access
by rows) then D is 2. To support two-dimensional lookups (e.g. rows\[50\]\[17\]) then we could set D to 3.

Currently we only support 1-dimensional random access.
Currently we do not compress the repetition index.

This may change in future versions.

| Bytes | Meaning                            |
| ----- | ---------------------------------- |
| 8     | Number of rows in block 0          |
| 8     | Number of partial items in block 0 |
| 8     | Number of rows in block 1          |
| 8     | Number of partial items in block 1 |
| ...   | ...                                |
| 8     | Number of rows in block N          |
| 8     | Number of partial items in block N |

The last 8 bytes of each block stores the number of "partial" items. These are items leftover after the last
complete row. We don't require rows to be bounded by mini-blocks so we need to keep track of this. For example,
if we have 10,000 items per row then we might have several mini-blocks with only partial items and 0 rows.

At read time we can use this repetition index to translate row offsets into item offsets.

#### Mini Block Compression

The mini block layout relies on the compression algorithm to handle the splitting of data into mini-blocks. This
is because the number of values per block will depend on the compressibility of the data. As a result, there is
a special trait for mini block compression.

The data compression algorithm is the algorithm that decides chunk boundaries. The repetition and definition levels
are then sliced appropriately and sent to a block compressor. This means there are no constraints on how the repetition
and definition levels are compressed.

Beyond splitting the data into mini-blocks, there are no additional constraints. We expect to fully decode mini
blocks as opaque chunks. This means we can use any compression algorithm that we deem suitable.

#### Protobuf

```protobuf
%%% proto.message.MiniBlockLayout %%%
```

The protobuf for the mini block layout describes the compression of the various buffers. It also tells us
some information about the dictionary (if present) and the repetition index (if present).

### Full Zip Page Layout

The full zip page layout is a layout for larger values (e.g. vector embeddings) which are large but not so large
that we can justify a single IOP per value. In this case we are trying to avoid storing a large amount of "chunk
overhead" (both in terms of buffer space and the RAM space in the search cache that we would need to store the
repetition index). As a tradeoff, we are introducing a second IOP per-range for random access reads (unless the
data is fixed-width such as vector embeddings).

We currently use 256 bytes as the cutoff for the full zip layout. At this point we would only be fitting 16 values
in a 4KiB disk sector and so creating a mini-block descriptor for every 16 values would be too much overhead.

As a further consequence, we must ensure that the compression algorithm is "transparent" so that we can index
individual values after compression has been applied. This prevents us from using compression algorithms such
as delta encoding. If we want to apply general compression we have to apply them on a per-value basis. The way
we enforce this is by requiring the compression to return either a flat fixed-width or variable-width layout
so that we know the location of each element.

The repetition and definition levels, along with all compressed buffers, are all zipped together into a single
buffer.

#### Data Buffer (Buffer 0)

![Full Zip Layout](../../images/fullzip.png)

The data buffer is a single buffer that contains the repetition, definition, and value data, all zipped into a
single buffer. The repetition and definition information are combined and byte packed. This is referred to as
a control word. If the value is null or an empty list, then the control word is all that is serialized. If there
is no validity or repetition information then control words are not serialized. If the value is variable-width
then we encode the size of the value. This is either a 4-byte or 8-byte integer depending on the width used in
the offsets returned by the compression (in future versions this will likely be encoded with some kind of
variable-width integer encoding). Finally the value buffers themselves are appended.

| Bytes | Meaning        |
| ----- | -------------- |
| 0-4   | Control word 0 |
| 0/4/8 | Value 0 size   |
| \*    | Value 0 data   |
| ...   | ...            |
| 0-4   | Control word N |
| 0/4/8 | Value N size   |
| \*    | Value N data   |

Note: a fixed-width data type that has no validity information (e.g. non-nullable vector embeddings) is simply a
flat buffer of data.

#### Repetition Index (Buffer 1)

![Full Zip Layout](../../images/fullzip_rep.png)

If there is repetition information or the values are variable width then we need additional help to locate values
in the disk page. The repetition index is an array of u64 values. There is one value per row and the value is an
offset to the start of that row in the data buffer. To perform random access we require two IOPS. First we issue
an IOP into the repetition index to determine the location and then a second IOP into the data buffer to load the
data. Alternatively, the entire repetition index can be loaded into memory in the initialization phase though this
can lead to high RAM usage by the search cache.

The repetition index must have a fixed width (or else we would need a repetition index to read the repetition
index!) and be transparent. As a result the compression options are limited. That being said, there is little
value (in terms of performance) in compressing the repetition index. It is never read in its entirety as it is
not needed for full scans. Currently the repetition index is always compressed with simple (non-chunked) byte
packing into 1,2,4, or 8 byte values.

#### Protobuf

```protobuf
%%% proto.message.FullZipLayout %%%
```

The protobuf for the full zip layout describes the compression of the data buffer. It also tells us the
size of the control words and how many bits we have per value (for fixed-width data) or how many bits we
have per offset (for variable-width data).

### Sparse Page Layout

Sparse pages require Lance 2.3. They represent flat or nested Arrow structure directly as slot-domain mappings instead
of dense repetition and definition events. Writers emit this layout only in files declared as 2.3 or above. The layout is
identified only by `PageLayout`; field metadata does not identify the layout of an existing page.

A domain is a layer-local integer coordinate space `[0, num_slots)`, and a slot is one element in that space. The
outer-most domain contains the page's top-level rows. Each layer maps its parent domain to the next layer's parent
domain, and the terminal child domain contains the leaf value slots stored in value chunks.

Structural layers are ordered from outer-most to inner-most:

- validity maps a nullable item or struct slot to valid or null
- list maps non-empty parent slots to variable-size child ranges
- fixed-size-list maps each parent slot to a child range of a fixed dimension

The layer list may be empty for a flat, non-nullable leaf page. In that case the scheduling domain and
`num_visible_items` must be equal. The explicit writer currently emits its normalized all-valid layer even when a flat
page could use this shorter wire representation.

A list slot that is valid and absent from `non_empty_positions` is an empty list. Maps use the same structural contract
as lists. The terminal child-domain size equals `SparseLayout.num_visible_items`.

`SparseLayout.num_visible_items` is the number of leaf value slots encoded in value chunks. Null leaf slots count
because they still occupy positions in Arrow's leaf value buffer; a nullable primitive with 100 slots, including 30
nulls, has 100 visible items. `SparseLayout.num_items` is the number of entries in the equivalent dense repetition and
definition stream. It equals `num_visible_items` plus one structural placeholder for every list slot without children.
The first layer's `num_slots` is the logical top-level row count used for projection.

Position sets have four semantic representations: `empty`, `all`, one non-empty `range`, or an `explicit`
delta-compressed `u64` buffer. Count sets are `empty`, one positive `constant` value, or an `explicit` compressed `u64`
buffer. Every layer has a `SparseValiditySet` whose meaning is explicit:

- `SPARSE_VALIDITY_NULL_POSITIONS`: stored positions are null and all other positions are valid
- `SPARSE_VALIDITY_VALID_POSITIONS`: stored positions are valid and all other positions are null

The unspecified validity meaning is invalid. Both polarities are part of the wire contract and have identical Arrow
semantics after normalization.

#### Writer Selection

Writers may emit this layout only for Lance 2.3+ fields. A field can request it explicitly with
`lance-encoding:structural-encoding=sparse`; the same request is an input error for earlier file versions. Without an
explicit structural encoding, the Lance 2.3 writer selects sparse only when the dense mini-block repetition/definition
budget would split the page or one top-level row exceeds that budget, and only when the value path is supported by the
sparse writer. Explicit `miniblock`, `fullzip`, and `sparse` requests are not changed by this automatic policy. Lance
2.2 and earlier writers never select sparse.

Unsupported sparse value paths, including dictionary values and variable-width packed structs, retain their dense
behavior. Writers normalize Arrow validity and list structure once. Within-budget dense pages do not build sparse
position/count plans. All-valid layers use null positions plus `empty`; all-null layers use valid positions plus
`empty`. Other layers choose the validity polarity with the lower semantic encoded cost, with ties using null
positions. Field metadata controls writer selection only: readers always use `PageLayout` to determine the layout of
an encoded page and must not use field metadata for that decision.

Pages without a value payload keep the existing canonical `ConstantLayout`: structural-only types such as an empty
struct, and leaf pages whose visible values are all null, do not emit `SparseLayout`. An explicitly sparse page with
at least one non-null visible value does emit `SparseLayout`, even when all non-null values are equal. This boundary
avoids introducing a second structural-only representation without evidence that it improves the existing constant
encoding.

#### Buffers and Selective Reads

A sparse page contains the following physical buffers:

| Buffer | Contents |
| ------ | -------- |
| 0 | Value chunk metadata, one 8-byte entry per chunk |
| 1 | Mini-block compressed value chunks without repetition or definition levels |
| 2+ | One buffer for each explicit position or count set, in structural-layer field order |

Each value chunk metadata entry stores `(chunk_size / 8) - 1` as little-endian `u32`, followed by its visible value
count as little-endian `u32`. Chunk sizes must be positive multiples of 8 and fit this representation. The sum of
chunk sizes must equal buffer 1 exactly and the sum of chunk value counts must equal `num_visible_items`. A value chunk
contains at most 32,768 visible values. `num_buffers` describes the number of value buffers inside every chunk and
excludes the structural buffers.

General-compressed sparse buffers use the existing length-prefixed LZ4 or Zstd representation and must not contain
another general-compression wrapper. SparseLayout does not impose additional size or descriptor-complexity limits on
otherwise representable buffers.

Readers normalize structural metadata once, project requested top-level ranges through each layer, and read only value
chunks that intersect the resulting leaf ranges. When no leaf range remains, readers rebuild offsets and validity from
the structural plan without reading buffer 1.

#### Caching and Point Reads

Reader initialization loads buffer 0 and every explicit structural buffer, then validates and normalizes them into a
cached page plan. The cached state contains parsed value-chunk descriptors and prefix offsets, decoded semantic
position/count sets, validity, and the ordered structural layers. It does not contain value payload bytes from buffer
1. The plan is cached per field and page and reused by later scans, range reads, and takes.

After that plan is cached, reading one primitive leaf value reads only the value chunk that contains it. A cold read
first loads the structural metadata and then the intersecting value chunk. Reading one top-level list or
fixed-size-list value may intersect multiple leaf chunks and reads each intersecting chunk. A selection whose projected
structure contains no leaf slots reads no value chunk.

#### Validation

Readers must reject malformed sparse metadata instead of inferring or repairing it. Required checks include:

- physical buffer count, chunk-count bounds, and every checked offset/size range
- first-layer row domain, adjacent parent/child domain chaining, and terminal visible-value domain
- semantic set cardinality, explicit position ordering and bounds, and validity meaning
- exact `num_items`
- list non-empty positions being valid, count cardinality, positive counts, and child-count sum
- fixed-size-list dimension and checked child-domain multiplication
- value chunk byte/value sums, size representation and alignment, general-compression headers, descriptor buffer
  count, and complete chunk consumption

```protobuf
%%% proto.message.SparseLayout %%%
```

```protobuf
%%% proto.message.SparseStructuralLayer %%%
```

```protobuf
%%% proto.message.SparseValidityLayer %%%
```

```protobuf
%%% proto.message.SparseListLayer %%%
```

```protobuf
%%% proto.message.SparseFixedSizeListLayer %%%
```

```protobuf
%%% proto.message.SparseValiditySet %%%
```

```protobuf
%%% proto.message.SparsePositionSet %%%
```

```protobuf
%%% proto.message.SparseCountSet %%%
```

### Constant Page Layout

This layout is used when all (visible) values in the page are the same scalar value.

The all-null case is represented by a constant page without an inline scalar value. Surprisingly, this does not
mean there is no data. If there are any levels of struct or list then we need to store the rep/def levels so that
we can distinguish between null structs, null lists, empty lists, and null values.

#### Repetition and Definition Levels (Buffers 0 and 1)

Note: We currently store rep levels in the first buffer with a flat layout of 16-bit values and def levels
in the second buffer with a flat layout of 16-bit values. This will likely change in future versions.

#### Protobuf

```protobuf
%%% proto.message.ConstantLayout %%%
```

All we need to know is the meaning of each rep/def level and (when present) the inline scalar value bytes.

### Blob Page Layout

The blob page layout is a layout for large binary values where we would only have a few values per disk page.
The actual data is stored out-of-line in external buffers. The disk page stores a "description" which is a
struct array of two fields: `position` and `size`. The `position` is the absolute file offset of the blob and
the `size` is the size (in bytes) of the blob. The inner page layout describes how the descriptions are encoded.

The validity information (definition levels) is smuggled into the descriptions. If the size and position are
both zero then the value is empty. Otherwise, if the size is zero and the position is non-zero then the
value is null and the position is the definition level.

This layout is only recommended when you can justify a single IOP per value. For example, when values are 1MiB
or larger.

This layout has no buffers of its own and merely wraps an inner layout.

#### Protobuf

```protobuf
%%% proto.message.BlobLayout %%%
```

Since we smuggle the validity into the descriptions we don't need to store it in the inner layout and so the
rep/def meaning is stored in the blob layout and the rep/def meaning in the inner layout will be 1 all valid item
layer.

## Semi-Structural Transformations

There are some data transformations that are applied to the data before (or during) the structural encoding process.
These are described here.

### Dictionary Encoding

Dictionary encoding is a technique that can be applied to any kind of array. It is useful when there are not very
many unique values in the array. First, a "dictionary" of unique values is created. Then we create a second array
of indices into the dictionary.

Dictionary encoding is also known as "categorical encoding" in other contexts.

Dictionary encoding could be treated as simply another compression technique but, when applied, it would be an
opaque compression technique which would limit its usability (e.g. in a full zip context). As a result, we apply
it before any structural encoding takes place. This allows us to place the dictionary in the search cache for
random access.

### Struct Packing

Struct packing is an alternative representation to apply to struct values. Instead of storing that struct in a
columnar fashion it will be stored in a row-major fashion. This will reduce the number of IOPS needed for random
access but will prevent the ability to read a single field at a time. This is useful when all fields in the struct
are always accessed together.

Packed struct is always opt-in (see section on configuration below).

In Lance 2.1, packed struct is limited to fixed-width children (`PackedStruct`).
Starting with Lance 2.2, variable-width children are also supported via `VariablePackedStruct`.

### Fixed Size List

Fixed size lists are an Arrow data type that needs specialized handling at the structural level. If the underlying
data type is primitive then the fixed size list will be primitive (e.g. a tensor). If the underlying data type
is structural (struct/list) then the fixed size list is structural and should be treated the same as a
variable-size list.

We don't want compression libraries to need to worry about the intricacies of fixed-size lists. As a result we
flatten the list as part of structural encoding. This complicates random access as we must translate between
rows (an entire fixed size list) and items (a single item in the list).

If the items in a fixed size list are nullable then we do not treat that validity array as a repetition or
definition level. Instead, we store the validity as a separate buffer. For example, when encoding nullable fixed
size lists with mini-block encoding the validity buffer is another buffer in the mini-block. When encoding
nullable fixed size lists with full-zip encoding the validity buffer is zipped together with the values.

The good news is that fixed size lists are entirely a structural encoding concern. Compression techniques are
free to pretend that the fixed-size list data type does not exist.

## Compression

Once a structural encoding is chosen we must determine how to compress the data. There are various buffers that
might be compressed (e.g. data, repetition, definition, dictionary, etc.). The available compression algorithms
are also constrained by the structural encoding chosen. For example, when using the full zip layout we require
transparent compression. As a result, each encoding technique may or may not be usable in a given scenario. In
addition, the same technique may be applied in a different way depending on the encoding chosen.

In implementation terms we have a trait for each compression constraint. The techniques then implement the traits
that they can be applied to. To start with, here is a summary of compression techniques which are implemented in
at least one scenario and a list of which traits the technique implements. A ❓ is used to indicate that the
technique should be usable in that context but we do not yet do so while a ❌ indicates that the technique is
not usable because it is not transparent. Note, even though a technique is not transparent it can still be applied
on a per-value basis. We use ☑️ to mark a technique that is applied on a per-value basis:

| Compression     | Used in Block Context | Used in Full Zip Context | Used in Mini-Block Context |
| --------------- | --------------------- | ------------------------ | -------------------------- |
| Flat            | ✅ (2.1)              | ✅ (2.1)                 | ✅ (2.1)                   |
| Variable        | ✅ (2.1)              | ✅ (2.1)                 | ✅ (2.1)                   |
| Constant        | ✅ (2.1)              | ❓                       | ❓                         |
| Bitpacking      | ✅ (2.1)              | ❓                       | ✅ (2.1)                   |
| Fsst            | ❓                    | ✅ (2.1)                 | ✅ (2.1)                   |
| Rle             | ✅ (2.2)              | ❌                       | ✅ (2.1)                   |
| ByteStreamSplit | ❓                    | ❌                       | ✅ (2.1)                   |
| General         | ✅ (2.2)              | ☑️ (2.1)                 | ✅ (2.1)                   |

In the following sections we will describe each technique in a bit more detail and explain how it is utilized
in various contexts.

### Flat

Flat compression is the uncompressed representation of fixed-width data. There is a single buffer of data
with a fixed number of bits per value.

When applied in a mini-block context we find the largest power of 2 number of values that will be less than
8,186 bytes and use that as the block size.

### Variable

Variable compression is the uncompressed representation of variable-width data. There is a buffer of values and
a buffer of offsets.

When applied in a mini-block context each block may have a different number of values. We walk through the values
until we find the point that would exceed 4,096 bytes and then use the most recent power of 2 number of values that
we have passed.

### Constant

Constant compression is currently only utilized in a few specialized scenarios such as all-null arrays.

This will likely change in future versions.

### Bitpacking

Bitpacking is a compression technique that removes the unused bits from a set of values. For example, if we have
a u32 array and the maximum value is 5000 then we only need 13 bits to store each value.

When used in a mini-block context we always use 1024 values per block. In addition, we store the compressed bit
width inline in the block itself.

Bitpacking is, in theory, usable in a full zip context. However, values in this context are so large that shaving
off a few bits is unlikely to have any meaningful impact. Also, the full-zip context keeps things byte-aligned and
so we would have to remove at least 8 bits per value.

### Fsst

Fsst is a fast and transparent compression algorithm for variable-width data. It is the primary compression
algorithm that we apply to variable-width data.

Currently we use a single FSST symbol table per disk page and store that symbol table in the protobuf description.
This is for historical reasons and is not ideal and will likely change in future versions.

When FSST is applied in a mini-block context we simply compress the data and let the underlying compressor (always
`Variable` at the moment) handle the chunking.

### Run Length Encoding (RLE)

Run length encoding is a compression technique that compresses large runs of identical values into an array
of values and an array of run lengths. This is currently used in the mini-block context. To determine if we
should apply run-length encoding we look at the number of runs divided by the number of values. If the ratio is
below a threshold (by default 0.5) then we apply run-length encoding.

### Byte Stream Split (BSS)

Byte stream split is a compression technique that splits multi-byte values by byte position, creating separate streams
for each byte position across all values. This is a rudimentary and simple form of translating floating point values
into a more compressible format because it tends to cluster the mantissa bits together which are often consistent
across a column of floating point values. It does not actually make the data smaller by itself. As a result, BSS is
only applied if general compression is also applied on the column.

We currently determine whether or not to apply BSS by looking at an entropy statistics. There is a configurable
sensitivity parameter. A sensitivity of 0.0 means never apply BSS and a sensitivity of 1.0 means always apply BSS.

### General

General compression is a catch-all term for classical opaque compression techniques such as LZ4, ZStandard, Snappy,
etc. These techniques are typically back-referencing compressors which replace values with a "back reference" to
a spot where we already saw the value.

When applied in a mini-block context we run general compression after all other compression and compress the entire
mini-block.

When applied in a full zip context we run general compression on each value.

The only time general compression is automatically applied is in a full-zip context when we have values that are at
least 32KiB large. This is because general compression can be CPU intensive.

However, general compression is highly effective and we allow it to be opted into in other contexts via configuration.

## Compression Configuration

The following section lists the available configuration options. These can be set programmatically through writer
options. However, they can also be set in the field metadata in the schema.

| Key                                  | Values                               | Default          | Description                                                                             |
| ------------------------------------ | ------------------------------------ | ---------------- | --------------------------------------------------------------------------------------- |
| `lance-encoding:compression`         | `lz4`, `zstd`, `none`, ...           | `none`           | Opt-in to general compression. The value indicates the scheme.                          |
| `lance-encoding:compression-level`   | Integers (range is scheme dependent) | Varies by scheme | Higher indicates more work should be done to compress the data.                         |
| `lance-encoding:rle-threshold`       | `0.0-1.0`                            | `0.5`            | See below                                                                               |
| `lance-encoding:bss`                 | `off`, `on`, `auto`                  | `auto`           | See below                                                                               |
| `lance-encoding:dict-divisor`        | Integers greater than 1              | `2`              | See below                                                                               |
| `lance-encoding:dict-size-ratio`     | `0.0-1.0`                            | `0.8`            | See below                                                                               |
| `lance-encoding:dict-values-compression` | `lz4`, `zstd`, `none`             | `lz4`            | Select general compression scheme for dictionary values                                 |
| `lance-encoding:dict-values-compression-level` | Integers (scheme dependent) | Varies by scheme | Compression level for dictionary values general compression                             |
| `lance-encoding:general`             | `off`, `on`                          | `off`            | Whether to apply general compression.                                                   |
| `lance-encoding:packed`              | Any string                           | Not set          | Whether to apply packed struct encoding (see above).                                    |
| `lance-encoding:structural-encoding` | `miniblock`, `fullzip`, `sparse`     | Not set          | Force a structural encoding; `sparse` requires Lance 2.3.                                |

### Configuration Details

#### Compression Scheme

The `lance-encoding:compression` setting enables general-purpose compression algorithms to be applied. Available schemes:

- **`lz4`**: Fast compression with good compression ratios. Default compression level is fast mode.
- **`zstd`**: High compression ratios with configurable levels (0-22). Better compression than LZ4 but slower.
- **`none`**: No general compression applied (default).
- **`fsst`**: Fast Static Symbol Table compression for string data.

General compression is applied on top of other encoding techniques (RLE, BSS, bitpacking, etc.) to further reduce
data size. For mini-block layouts, compression is applied to entire mini-blocks. For full-zip layouts with large values
(≥32KiB), compression is automatically applied per-value.

#### Compression Level

The compression level is scheme dependent. Currently the following schemes support the following levels:

| Scheme | Crate Used                              | Levels | Default                                                                                                                                                                                                                           |
| ------ | --------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `zstd` | [`zstd`](https://crates.io/crates/zstd) | `0-22` | `crate dependent` (3 as of this writing)                                                                                                                                                                                          |
| `lz4`  | [`lz4`](https://crates.io/crates/lz4)   | N/A    | The LZ4 crate has two modes (fast and high compression) and currently this is not exposed to configuration. The LZ4 crate wraps a C library and the default is dependent on the C library. The default as of this writing is fast |

Higher compression levels generally provide better compression at the cost of slower encoding speed. Decoding speed
is typically less affected by the compression level.

#### Run Length Encoding (RLE) Threshold

The RLE threshold is used to determine whether or not to apply run-length encoding. The threshold is a ratio
calculated by dividing the number of runs by the number of values. If the ratio is less than the threshold then
we apply run-length encoding. The default is 0.5 which means we apply run-length encoding if the number of runs
is less than half the number of values.

**Key points:**
- RLE is automatically selected when data has sufficient repetition (run_count / num_values < threshold)
- Supported types: All fixed-width primitives (u8, i8, u16, i16, u32, i32, f32, u64, i64, f64)
- Maximum chunk size: 2048 values per mini-block
- Setting threshold to `0.0` effectively disables RLE
- Setting threshold to `1.0` makes RLE very aggressive (used whenever any runs exist)

RLE is particularly effective for:
- Sorted or partially sorted data
- Columns with many repeated values (status codes, categories, etc.)
- Low-cardinality columns

#### Byte Stream Split (BSS)

The configuration variable for BSS is a simple enum. A value of `off` means to never apply BSS, a value of `on`
means to always apply BSS, and a value of `auto` means to apply BSS based on an entropy calculation (see code for
details).

**Important:** BSS is only applied when the `lance-encoding:compression` variable is also set (to a non-`none` value).
BSS is a data transformation that makes floating-point data more compressible; it does not reduce size on its own.

**Key points:**
- Supported types: Only 32-bit and 64-bit data (f32, f64, timestamps)
- Maximum chunk sizes: 1024 values (f32), 512 values (f64)
- `auto` mode: Uses entropy analysis with 0.5 sensitivity threshold
- `on` mode: Always applies BSS for supported types
- `off` mode: Never applies BSS

BSS works by splitting multi-byte values by byte position, creating separate byte streams. This clusters similar
bits together (especially mantissa bits in floating-point numbers), which general compression algorithms can then
compress more effectively.

BSS is particularly effective for:
- Floating-point measurements with similar ranges
- Time-series data with consistent precision
- Scientific data with correlated mantissa patterns

#### Dictionary Encoding Controls

Dictionary encoding is gated by a few heuristics.
The decision is made on the leaf value page, so nested types can still benefit.
For example, `List<u32>` can use dictionary encoding for its `u32` values.

Two field-level metadata keys control when dictionary encoding is attempted:

- `lance-encoding:dict-divisor` (default `2`): the encoder computes a unique-value budget as `num_values / divisor`
- `lance-encoding:dict-size-ratio` (default `0.8`): the estimated dictionary-encoded representation must stay below this ratio of the raw page size

There are additional global guards available as environment variables:

- `LANCE_ENCODING_DICT_TOO_SMALL` (minimum page size before trying dictionary encoding, default `100` values)
- `LANCE_ENCODING_DICT_DIVISOR` (fallback divisor when field metadata is not set, default `2`)
- `LANCE_ENCODING_DICT_MAX_CARDINALITY` (upper cap for dictionary entries, default `100000`)
- `LANCE_ENCODING_DICT_SIZE_RATIO` (fallback ratio when field metadata is not set, default `0.8`)

Dictionary encoding is effective when values repeat frequently and the number of distinct values stays low.

#### Dictionary Values Compression

Dictionary values are compressed through the block-compression path and have their own configuration:

- `lance-encoding:dict-values-compression`: `lz4`, `zstd`, `none`
- `lance-encoding:dict-values-compression-level`: optional scheme-specific level

Environment-variable fallbacks:

- `LANCE_ENCODING_DICT_VALUES_COMPRESSION`
- `LANCE_ENCODING_DICT_VALUES_COMPRESSION_LEVEL`

Priority order is:

1. Field metadata (`dict-values-*`)
2. Environment variables (`LANCE_ENCODING_DICT_VALUES_*`)
3. Default (`lz4`)

`none` disables general (opaque) compression for dictionary values. For fixed-width dictionary values, structural
encodings such as RLE or bitpacking may still be selected when beneficial.

#### Packed Struct Encoding

Packed struct encoding is a semi-structural transformation described above. When enabled, struct values are stored
in row-major format rather than the default columnar format. This reduces the number of I/O operations needed for
random access but prevents reading individual fields independently.

This is always opt-in and should only be used when all struct fields are typically accessed together.

#### Mini-Block Size Tuning

Each mini-block contains at most 4096 values by default. Because an entire mini-block must be fetched to
read any value within it, workloads that read only a small contiguous slice of each mini-block may experience
read amplification.

The default is appropriate for the vast majority of deployments. Local disks and typical cloud object storage
(where the client and bucket are in the same region) have more than enough bandwidth that the overhead from
the default mini-block size is negligible. You should only consider changing this setting if you have
confirmed — through profiling — that mini-block read amplification is saturating your available bandwidth
(for example, accessing a remote object store over a constrained network link).

The maximum number of values per mini-block can be tuned via an environment variable:

- `LANCE_MINIBLOCK_MAX_VALUES` (default `4096`, maximum `32768`): upper bound on the number of values in a single mini-block chunk.

Reducing this value produces smaller mini-blocks, which reduces the amount of data fetched per read at the
cost of more mini-blocks and slightly more metadata overhead. Increasing it can reduce metadata overhead and
improve throughput for highly compressible data, but it may increase random-read amplification.

index.md 9.4 KB

# Lance File Format

The Lance file format is a columnar container optimized for cloud object stores, random access, and Arrow-native processing. It deliberately focuses on page layout and encoding mechanics, while leaving table semantics and search structures to higher layers.

## Design Goals

### No Row Groups

Lance does not use Parquet-style row groups. Each column may have its own number of pages, which keeps column data in large storage-friendly chunks regardless of schema width and avoids coupling scanner partitioning to physical file layout.

### Random-Access-Friendly Encoding

Pages are designed so readers can fetch contiguous row ranges with a small and predictable number of I/O operations. This is important for selective filters, point lookups, vector-search follow-up reads, and ML training workloads that sample rows non-sequentially.

### Functional Decomposition

The file layer does not bundle table-level statistics or query-side indices into the base file structure. Those capabilities are defined as separate index formats so they can evolve independently of the core file container.

## File Structure

A Lance file is a container for tabular data. The data is stored in "disk pages". Each disk page contains some rows
for a single column. There may be one or more disk pages per column. Different columns may have different numbers of
disk pages. Metadata at the end of the file describes where the pages are located and how the data is encoded.

![Format Overview](../../images/file_high_level_overview.png)

!!! Note

    This page describes the container specification. We also have a set of default encodings that are used to encode
    data into disk pages. See the [Encoding Strategy](encoding.md) page for more details.

### Disk Pages

Disk pages are designed to be large enough to justify a dedicated I/O operation, even on cloud storage, typically several megabytes. Using a larger page size may reduce the number of I/O operations required to read a file, but it also increases the amount of memory required to write the file. In practice, very large page sizes are not useful when high speed reads are required because large contiguous reads need to be broken into smaller reads for performance (particularly on cloud storage). As a result, a default of 8MB is recommended for the page size and should yield ideal performance on all storage systems.

Disk pages should not generally be opaque. It is possible to read a portion of a disk page when a subset of the rows are
required. However, the specifics of this process depend on the column encoding which is described in a later section.

### No Row Groups

Unlike similar formats, there is no "row group" concept, only pages. We believe the concept of row groups to be
fundamentally harmful to performance. If the row group size is too small then columns will be split into "runt pages" which yield poor read performance on cloud storage. If the row group size is too large then a file writer will need
a large amount of RAM since an entire row group must be buffered in memory before it can be written. Instead, to split
a file amongst multiple readers we rely on the fact that partial page reads are possible and have minimal read
amplification. As a result, you can split the file at whatever row boundary you want.

### Buffer Alignment

The file format does not require that buffers be contiguous as buffers are referenced by absolute offsets. In practice,
we always align buffers to 64 byte boundaries.

### External Buffers

Every page in the file is referenced by an absolute offset. This means that non-page data may be inserted amongst the
pages. This can be useful for storing extremely large data types which might only fit a few rows per page otherwise. We
can instead store the data out-of-line and store the locations in a page.

In addition, the file format supports "global buffers" which can be used for auxiliary data. This may be used to
store a file schema, file indexes, column statistics, or other metadata. References to the global buffers are stored
in a special spot in the footer.

### Column Descriptors

At the tail of the file is metadata that describes each page in the file, particularly the encoding strategy used.
This metadata consists of a series of "column descriptors", which are standalone protobuf messages for each column
in the file. Since each column has its own message there is no need to read all file metadata if you are only interested
in a subset of the columns. However, in many cases, the column descriptors are small enough that it is cheaper to read
the entire footer in a single read than split it into multiple reads.

### Offsets & Footer

After the column descriptors there are offset arrays for the column descriptors and global buffers. These simply
point to the locations of each item. Finally, there is a fixed-size footer which describes the position of the
offset arrays and start of the metadata section.

### Identifiers and Type Systems

This basic container format has no concept of types. These are added later by the encoding layer. All columns are
referenced by an integer "column index". All global buffers are referenced by an integer "global buffer index".
The schema is typically stored in the global buffers, but the file format is unaware of this.

## Reading Strategy

The file metadata will need to be known before reading the data. A simple approach for loading the footer is to
read one sector from the end (sector depends on the filesystem, 4KiB for local disk, larger for cloud storage). Then
parse the footer and read the rest of the metadata (at this point the size will be known). This requires 1-2 IOPS. By
storing the metadata size in some other location (e.g. table manifest) it is possible to always read the footer in
a single IOP. If there are _many_ columns in the file and only some are desired then it may be better to read
individual columns instead of reading all column metadata, increasing the number of IOPS but decreasing the amount
of data read.

Next, to read the data, scan through the pages for each column to determine which pages are needed. Each page stores
the row offset of the first row in the page. This makes it easy to quickly determine the required pages. The encoding
information for the page can then be used to determine exactly which byte ranges are needed from the page.

Disk pages should be large enough that there should no significant benefit to sequentially reading the file. However,
if such a use case is desired then the file can be read sequentially once the metadata is known, assuming you want to
read all columns in the file.

## Detailed Overview

![Format Overview](../../images/file_overview.png)

A detailed description of the file layout follows:

```protobuf
// Note: the number of buffers (BN) is independent of the number of columns (CN)
//       and pages.
//
//       Buffers often need to be aligned.  64-byte alignment is common when
//       working with SIMD operations.  4096-byte alignment is common when
//       working with direct I/O.  In order to ensure these buffers are aligned
//       writers may need to insert padding before the buffers.
//
//       If direct I/O is required then most (but not all) fields described
//       below must be sector aligned.  We have marked these fields with an
//       asterisk for clarity.  Readers should assume there will be optional
//       padding inserted before these fields.
//
//       All footer fields are unsigned integers written with little endian
//       byte order.
//
// ├──────────────────────────────────┤
// | Data Pages                       |
// |   Data Buffer 0*                 |
// |   ...                            |
// |   Data Buffer BN*                |
// ├──────────────────────────────────┤
// | Column Metadatas                 |
// | |A| Column 0 Metadata*           |
// |     Column 1 Metadata*           |
// |     ...                          |
// |     Column CN Metadata*          |
// ├──────────────────────────────────┤
// | Column Metadata Offset Table     |
// | |B| Column 0 Metadata Position*  |
// |     Column 0 Metadata Size       |
// |     ...                          |
// |     Column CN Metadata Position  |
// |     Column CN Metadata Size      |
// ├──────────────────────────────────┤
// | Global Buffers Offset Table      |
// | |C| Global Buffer 0 Position*    |
// |     Global Buffer 0 Size         |
// |     ...                          |
// |     Global Buffer GN Position    |
// |     Global Buffer GN Size        |
// ├──────────────────────────────────┤
// | Footer                           |
// | A u64: Offset to column meta 0   |
// | B u64: Offset to CMO table       |
// | C u64: Offset to GBO table       |
// |   u32: Number of global bufs     |
// |   u32: Number of columns         |
// |   u16: Major version             |
// |   u16: Minor version             |
// |   "LANC"                         |
// ├──────────────────────────────────┤
//
// File Layout-End
```

### Column Metadata

The protobuf messages for the column metadata are as follows:

```protobuf
%%% proto.message.ColumnMetadata %%%
```

versioning.md 3.9 KB

# Versioning

The Lance file format has a single version number for both the overall file format and the encoding strategy. The
major number is changed when the file format itself is modified while the minor number is changed when only the encoding
strategy is modified. Newer versions will typically have better performance and compression but may not be readable
by older versions of Lance.

Any version explicitly labeled unstable, including the current 2.3 format and the `next` alias, should not be used for
production use cases. Unstable formats have no compatibility guarantee: breaking encoding changes may make files
written by one Lance build unreadable by later builds. They should only be used for experimentation and benchmarking
upcoming features.

The `stable` and `next` aliases are resolved by the specific Lance release you are using. During a format rollout
(for example, 2.3), prefer explicit version pinning for deterministic behavior across environments.

The following values are supported:

| Version        | Minimal Lance Version | Maximum Lance Version | Description |
| -------------- | --------------------- | --------------------- | ----------- |
| 0.1            | Any                   | 0.34 (write)          | This is the initial Lance format. It is no longer writable. |
| 2.0            | 0.16.0                | Any                   | Rework of the Lance file format that removed row groups and introduced null support for lists, fixed size lists, and primitives |
| 2.1            | 0.38.1                | Any                   | Enhances integer and string compression, adds support for nulls in struct fields, and improves random access performance with nested fields. |
| 2.2            | None                  | Any                   | Adds support for newer nested type/encoding capabilities (including map support) and 2.2-era storage features. |
| 2.3 (unstable) | None                  | Unspecified           | Adds sparse structural pages and other experimental encodings. |
| legacy         | N/A                   | N/A                   | Alias for 0.1 |
| stable         | N/A                   | N/A                   | Alias for the default version for new datasets in the Lance release you are running. |
| next           | N/A                   | N/A                   | Alias for the latest unstable version in the Lance release you are running.|

## Compatibility Caveats

Stable formats carry a compatibility guarantee, but certain data patterns exposed encoder bugs
that required encoding changes to fix.  Files containing those patterns written by the fixed
encoder are not readable by readers predating the fix.  The affected scenarios are listed here
so operators running mixed-version deployments know the minimum reader version required.

### FixedSizeList with all-null inner values (Lance 11.1.0)

**Affected format**: 2.1 and later.

**Scenario**: A `FixedSizeList` column where every inner value (not the outer list item itself)
is null — for example, `FixedSizeList<nullable Float32, dim=4>` where all eight Float32 values
across two outer rows are null.

**Buggy writer (Lance < 11.1.0)**: The encoder wrote `bits_per_value=0` into the FullZip page
layout.  Readers of any version rejected these pages with an error, so the data was unreadable
regardless of reader version.

**Fixed writer (Lance ≥ 11.1.0)**: The encoder stores per-row validity bytes for the null inner
values, producing `bits_per_value > 0`.  The fixed reader (Lance ≥ 11.1.0) can also decode the
old buggy pages, so old files written before 11.1.0 become readable after upgrading.

**Forward compatibility**: Files containing this pattern written by Lance ≥ 11.1.0 are **not
readable by Lance < 11.1.0**.  Old readers encounter the `Compression::Constant` inner encoding
in the FSL descriptor and panic rather than returning an error.

**Minimum reader version for new files**: Lance 11.1.0.

index

scalar

bitmap.md 1.4 KB

# Bitmap Index

Bitmap indices use bit arrays to represent the presence or absence of values,
providing extremely fast query performance for low-cardinality columns.

## Index Details

```protobuf
%%% proto.message.BitmapIndexDetails %%%
```

## Storage Layout

The bitmap index consists of a single file `bitmap_page_lookup.lance` that stores the mapping from values to their bitmaps.

### File Schema

| Column    | Type       | Nullable | Description                                                             |
|-----------|------------|----------|-------------------------------------------------------------------------|
| `keys`    | {DataType} | true     | The unique value from the indexed column                                |
| `bitmaps` | Binary     | true     | Serialized RowAddrTreeMap containing row addrs where this value appears |

## Accelerated Queries

| Query Type | Description               | Operation                                  |
|------------|---------------------------|--------------------------------------------|
| **Equals** | `column = value`          | Returns the bitmap for the specific value  |
| **Range**  | `column BETWEEN a AND b`  | Unions all bitmaps for values in the range |
| **IsIn**   | `column IN (v1, v2, ...)` | Unions bitmaps for all specified values    |
| **IsNull** | `column IS NULL`          | Returns the pre-computed null bitmap       |

bloom_filter.md 5.3 KB

# Bloom Filter Index

Bloom filters are probabilistic data structures that allow for fast membership testing.
They are space-efficient and can test whether an element is a member of a set.
It's an inexact filter - they may include false positives but never false negatives.

In addition, since finding NULLs is a common query pattern, the index also maintains a
bitmap of null rows which allows it to return exact results for IS NULL queries.

## Index Details

```protobuf
%%% proto.message.BloomFilterIndexDetails %%%
```

## Storage Layout

The bloom filter index stores zone-based bloom filters in a single file:

1. `bloomfilter.lance` - Bloom filter statistics and data for each zone

### Bloom Filter File Schema

| Column              | Type    | Nullable | Description                                     |
|---------------------|---------|----------|-------------------------------------------------|
| `fragment_id`       | UInt64  | false    | Fragment containing this zone                   |
| `zone_start`        | UInt64  | false    | Starting row offset within the fragment         |
| `zone_length`       | UInt64  | false    | Number of rows in this zone                     |
| `has_null`          | Boolean | false    | Whether this zone contains any null values      |
| `bloom_filter_data` | Binary  | false    | Serialized SBBF (Split Block Bloom Filter) data |

### Schema Metadata

| Key                       | Type   | Description                                                 |
|---------------------------|--------|-------------------------------------------------------------|
| `bloomfilter_item`        | String | Expected number of items per zone (default: "8192")         |
| `bloomfilter_probability` | String | False positive probability (default: "0.00057", ~1 in 1754) |
| `null_bitmap`             | UInt32 | Index of null bitmap global buffer                          |

### Global Buffers

| Metadata Key        | Description                                                |
|---------------------|------------------------------------------------------------|
| `null_bitmap`       | A serialized RowAddrTreeMap specifying which rows are null |

## Bloom Filter Spec

The bloom filter index uses a Split Block Bloom Filter (SBBF) implementation,
which is optimized for SIMD operations.

### SBBF Structure

The SBBF divides the bit array into blocks of 256 bits, where each block consists of 8 contiguous 32-bit words.
This structure enables efficient SIMD operations and cache-friendly memory access patterns.
The block layout is the following:

- **Block size**: 256 bits (32 bytes)
- **Words per block**: 8 × 32-bit integers
- **Minimum filter size**: 32 bytes (1 block)
- **Maximum filter size**: 128 MiB

### Hashing Mechanism

The SBBF uses xxHash64 with seed=0 for primary hashing, combined with a salt-based secondary hashing scheme:

1. **Primary hash**: xxHash64(value) → 64-bit hash
2. **Block selection**: Upper 32 bits determine which block to use
3. **Bit selection**: Lower 32 bits combined with 8 salt values set 8 bits in the block

#### Salt Values

```
0x47b6137b
0x44974d91
0x8824ad5b
0xa2b7289d
0x705495c7
0x2df1424b
0x9efc4947
0x5c6bfb31
```

Each salt value generates one bit position within the block, ensuring uniform distribution.

### Filter Sizing Algorithm

The SBBF automatically determines optimal filter size based on:
- **NDV** (Number of Distinct Values): Expected unique items
- **FPP** (False Positive Probability): Target error rate

The implementation uses binary search to find the minimum log₂(bytes) that achieves the desired FPP,
using Putze et al.'s cache-efficient bloom filter formula.

#### FPP Convergence

The implementation uses up to 750 iterations of Poisson distribution calculations to ensure accurate FPP estimation,
particularly for dense filters where NDV approaches filter capacity.

### Serialization

The SBBF is serialized as a contiguous byte array stored in the `bloom_filter_data` column:

```
[Block 0][Block 1]...[Block N-1]
```

Where each block is 32 bytes:

```
[Word 0][Word 1][Word 2][Word 3][Word 4][Word 5][Word 6][Word 7]
```

Each word is a 32-bit little-endian integer (4 bytes), with:

- **Total size**: Must be a multiple of 32 bytes
- **Byte order**: Little-endian for all 32-bit words
- **Block alignment**: Each block starts at offset `i * 32`
- **Word offset**: Word `j` in block `i` is at byte offset `i * 32 + j * 4`

#### Example

For a filter with 2 blocks (64 bytes total):
```
Offset  0-3:   Block 0, Word 0 (32-bit LE)
Offset  4-7:   Block 0, Word 1 (32-bit LE)
...
Offset 28-31:  Block 0, Word 7 (32-bit LE)
Offset 32-35:  Block 1, Word 0 (32-bit LE)
...
Offset 60-63:  Block 1, Word 7 (32-bit LE)
```

## Accelerated Queries

The bloom filter index provides inexact results for the following query types (nullability queries
return exact results):

| Query Type | Description               | Operation                                 | Result Type |
|------------|---------------------------|-------------------------------------------|-------------|
| **Equals** | `column = value`          | Tests if value exists in bloom filter     | AtMost      |
| **IsIn**   | `column IN (v1, v2, ...)` | Tests if any value exists in bloom filter | AtMost      |
| **IsNull** | `column IS NULL`          | Returns zones where has_null is true      | Exact       |

btree.md 3 KB

# BTree Index

The BTree index is a two-level structure that provides efficient range queries and sorted access.
It strikes a balance between an expensive memory structure containing all values
and an expensive disk structure that can't be efficiently searched.

The upper layers of the BTree are designed to be cached in memory and stored in a
BTree structure (`page_lookup.lance`), while the leaves are searched using sub-indices
(`page_data.lance`, currently just a flat file).
This design enables efficient memory usage - for example, with 1 billion values,
the index can store 256K leaves of size 4K each, requiring only a few MiB of memory
(depending on data type) for the BTree metadata while narrowing any search to just 4K values.

## Index Details

```protobuf
%%% proto.message.BTreeIndexDetails %%%
```

## Storage Layout

The BTree index consists of two files:

1. `page_lookup.lance` - The BTree structure mapping value ranges to page numbers
2. `page_data.lance` - The actual sub-indices (flat file) containing sorted values and row IDs

### Page Lookup File Schema (BTree Structure)

| Column       | Type       | Nullable | Description                                              |
|--------------|------------|----------|----------------------------------------------------------|
| `min`        | {DataType} | true     | Minimum value in the page (forms BTree keys)             |
| `max`        | {DataType} | true     | Maximum value in the page (for range pruning)            |
| `null_count` | UInt32     | false    | Number of null values in the page                        |
| `page_idx`   | UInt32     | false    | Page number pointing to the sub-index in page_data.lance |

### Schema Metadata

| Key | Type | Description |
|-----|------|-------------|
| `batch_size` | String | Number of rows per page (default: "4096") |

### Page Data File Schema (Sub-indices)

| Column   | Type       | Nullable | Description                                       |
|----------|------------|----------|---------------------------------------------------|
| `values` | {DataType} | true     | Sorted values from the indexed column (flat file) |
| `ids`    | UInt64     | false    | Row IDs corresponding to each value               |

## Accelerated Queries

The BTree index provides exact results for the following query types:

| Query Type | Description               | Operation                                                                   |
|------------|---------------------------|-----------------------------------------------------------------------------|
| **Equals** | `column = value`          | BTree lookup to find relevant pages, then search within sub-indices         |
| **Range**  | `column BETWEEN a AND b`  | BTree traversal for pages overlapping the range, then search each sub-index |
| **IsIn**   | `column IN (v1, v2, ...)` | Multiple BTree lookups, union results from all matching sub-indices         |
| **IsNull** | `column IS NULL`          | Returns rows from all pages where null_count > 0                            |

fmindex.md 4.5 KB

# FM-Index (Full-text / Substring / Regex Search)

The FM-Index (Ferragina-Manzini Index) is a compressed substring index based on the Burrows-Wheeler Transform (BWT). Unlike traditional inverted indexes (Full-Text Search) which index distinct words, the FM-Index enables efficient **arbitrary substring search**, **prefix match**, and **suffix/regular-expression search** directly on raw bytes.

In Lance, the FM-Index is designed to scale dynamically across millions of documents or large-scale datasets, and is partitioned using Lance's **Segmented Index** architecture to support incremental appends, disjoint fragment tracking, and segment merging.

## High-Level Architecture

The FM-Index indexes raw text by treating columns of strings or binary payloads as raw byte arrays.

```
                     +----------------------------------------+
                     |            Lance Dataset               |
                     |   (Disjoint groups of Fragments 0..N)   |
                     +----------------------------------------+
                                         |
                       Divide fragments into num_segments
                                         |
                                         v
                     +----------------------------------------+
                     |            Segmented Index             |
                     |  +-----------+ +-----------+ +-------+ |
                     |  | Segment 1 | | Segment 2 | | ...   | |
                     |  |  (FM-Idx) | |  (FM-Idx) | |       | |
                     |  +-----------+ +-----------+ +-------+ |
                     +----------------------------------------+
```

Each segment contains its own self-contained physical FM-Index mapping byte sub-sequences to Lance global row IDs.

## Data Normalization & Sanitization

The FM-Index is **normalization-independent by design** because it operates entirely on raw bytes.

### Byte Sanitization vs. Text Normalization

1. **Byte Sanitization (Core Index Layer)**:
   The physical FM-Index uses specific sentinel bytes internally to mark boundaries:
   - `\x00` is reserved as the global Burrows-Wheeler Transform (BWT) terminator character.
   - `\xFF` is reserved as the document/row separator character.

   To avoid breaking the indexing structures, any incoming occurrences of `\x00` or `\xFF` are sanitized by remapping them to space (`\x20`) characters at index-build time. No other bytes are changed in this layer.

2. **Text Normalization (User/Application Layer)**:
   Because the index faithfully maps raw bytes, any semantic normalization (such as case folding `Hello` -> `hello`, Unicode NFKC normalization, stemming, or whitespace collapsing) is fully decoupled from the core index engine:
   - To build a case-insensitive search index, users apply a lowercase transform to the column *prior* to indexing.
   - When querying, the user's query text must undergo the exact same normalization pipeline.

## Configurable Segment Partitioning

Merging or appending to BWT-based indexes cannot be done via simple concatenation; the BWT suffix array must be reconstructed by re-reading the text and rebuilding. To balance build cost and search performance, Lance allows configuring how fragments map to index segments.

- **`num_segments` parameter**: Configured at index-creation time. If `num_segments` is specified (e.g. `num_segments = 4`), Lance splits the target dataset fragments into disjoint subsets and builds independent FM-Index segments over each chunk.
- **Unindexed Appends**: When new fragments are appended to the dataset, a subsequent `create_index` execution with unindexed fragment coverage will construct a new separate segment representing only those new fragments, keeping existing segments fully intact.
- **Segment Merging**: Multiple existing index segments can be merged into a single segment under Lance's `merge_segments` protocol. Lance unions the fragment coverage bitmaps of the selected segments, re-reads the raw text from those covered fragments, and constructs a fresh unified FM-Index.

## Query Evaluation

When a substring query is submitted (e.g., `CONTAINS(column, "query_string")`):
1. The search string is sanitized (remapping any `\x00` or `\xFF` to spaces) and optionally normalized if the target index is normalized.
2. The query is dispatched across all active segments in the logical index in parallel.
3. Each segment performs a BWT backward-search to locate occurrences of the pattern.
4. Matching offsets are mapped back to absolute dataset Row IDs.
5. Results from all segments are unioned to produce the final selection.

fts.md 18.6 KB

# Full Text Search Index

The full text search (FTS) index (a.k.a. inverted index) provides efficient text search by mapping terms to the documents containing them.
It's designed for high-performance text search with support for various scoring algorithms and phrase queries.

## Index Details

```protobuf
%%% proto.message.InvertedIndexDetails %%%
```

## Storage Layout

The FTS index consists of multiple files storing the token dictionary, document information, and posting lists:

1. `tokens.lance` - Token dictionary mapping tokens to token IDs
2. `docs.lance` - Document metadata including token counts
3. `invert.lance` - Compressed posting lists for each token
4. `metadata.lance` - Index metadata and configuration

An FTS index may contain multiple partitions. Each partition has its own set of token, document, and posting list files, prefixed with the partition ID (e.g. `part_0_tokens.lance`, `part_0_docs.lance`, `part_0_invert.lance`). The `metadata.lance` file lists all partition IDs in the index. At query time, every partition must be searched and the results combined to produce the final ranked output. Fewer partitions generally means better query performance, since each partition requires its own token dictionary lookup and posting list scan. The number of partitions is controlled by the training configuration -- specifically `LANCE_FTS_TARGET_SIZE` determines how large each merged partition can grow (see [Training Process](#training-process) for details).

### Token Dictionary File Schema

| Column      | Type   | Nullable | Description                     |
|-------------|--------|----------|---------------------------------|
| `_token`    | Utf8   | false    | The token string                |
| `_token_id` | UInt32 | false    | Unique identifier for the token |

### Document File Schema

| Column        | Type   | Nullable | Description                      |
|---------------|--------|----------|----------------------------------|
| `_rowid`      | UInt64 | false    | Document row ID                  |
| `_num_tokens` | UInt32 | false    | Number of tokens in the document |

Partitioned `docs.lance` files may include the optional schema metadata key
`total_tokens`. Its decimal `UInt64` value is the sum of `_num_tokens` in that
file. `_num_tokens` remains the canonical per-document data. Readers use the
metadata value to construct exact corpus statistics without scanning the column;
when the key is absent, they compute the sum from `_num_tokens`. Writers produce
the key from the same document table in the same file commit. A present value
that cannot be parsed, or that differs when `_num_tokens` is subsequently
loaded, is file corruption.

### FTS List File Schema

| Column                 | Type                    | Nullable | Description                                                      |
|------------------------|-------------------------|----------|------------------------------------------------------------------|
| `_posting`             | List<LargeBinary>       | false    | Compressed posting lists (delta-encoded row IDs and frequencies) |
| `_max_score`           | Float32                 | false    | Maximum score for the token (for query optimization)             |
| `_length`              | UInt32                  | false    | Number of documents containing the token                         |
| `_compressed_position` | List<List<LargeBinary>> | true     | Optional compressed position lists for phrase queries            |

The posting-list file schema metadata includes `posting_block_size`, the number of documents encoded per compressed posting block. Older indexes that do not have this metadata use the legacy block size `128`.

### Metadata File Schema

The metadata file contains JSON-serialized configuration and partition information:

| Key          | Type          | Description                                              |
|--------------|---------------|----------------------------------------------------------|
| `partitions` | Array<UInt64> | List of partition IDs for distributed index organization |
| `params`     | JSON Object   | Serialized InvertedIndexParams with tokenizer config     |

#### InvertedIndexParams Structure

| Field               | Type    | Default   | Description                                                    |
|---------------------|---------|-----------|----------------------------------------------------------------|
| `base_tokenizer`    | String  | "simple"  | Base tokenizer type (see Tokenizers section)                   |
| `language`          | String  | "English" | Language for stemming and stop words                           |
| `with_position`     | Boolean | false     | Store term positions for phrase queries (increases index size) |
| `max_token_length`  | UInt32? | None      | Maximum token length (tokens longer than this are removed)     |
| `lower_case`        | Boolean | true      | Convert tokens to lowercase                                    |
| `stem`              | Boolean | false     | Apply language-specific stemming                               |
| `remove_stop_words` | Boolean | false     | Remove common stop words for the specified language            |
| `ascii_folding`     | Boolean | true      | Convert accented characters to ASCII equivalents               |
| `min_gram`          | UInt32  | 2         | Minimum n-gram length (only for ngram tokenizer)               |
| `max_gram`          | UInt32  | 15        | Maximum n-gram length (only for ngram tokenizer)               |
| `prefix_only`       | Boolean | false     | Generate only prefix n-grams (only for ngram tokenizer)        |
| `block_size`        | UInt32  | 128       | Documents per compressed posting block. Must be 128 or 256. Missing values from older indexes read as 128. `256` is experimental and may introduce breaking changes. |

## Tokenizers

The full text search index supports multiple tokenizer types for different text processing needs:

### Base Tokenizers

| Tokenizer      | Description                                                               | Use Case               |
|----------------|---------------------------------------------------------------------------|------------------------|
| **simple**     | Splits on whitespace and punctuation, removes non-alphanumeric characters | General text (default) |
| **whitespace** | Splits only on whitespace characters                                      | Preserve punctuation   |
| **raw**        | No tokenization, treats entire text as single token                       | Exact matching         |
| **ngram**      | Breaks text into overlapping character sequences                          | Substring/fuzzy search |
| **icu**        | ICU dictionary-based Unicode word segmentation                            | Mixed-language text    |
| **icu/split**  | ICU segmentation with simple-style delimiter splitting                    | Mixed-language identifiers |
| **jieba/***    | Chinese text tokenizer with word segmentation                             | Chinese text           |
| **lindera/***  | Japanese text tokenizer with morphological analysis                       | Japanese text          |

#### ICU Tokenizer (Mixed-language text)

The ICU tokenizer uses Unicode word boundary rules and dictionary-based segmentation for complex scripts. It is useful for mixed-language text where the default `simple` tokenizer would keep an unspaced CJK span as one large token.

By default, Lance preserves ICU word segments as returned by ICU. Use `base_tokenizer: "icu/split"` to split ICU word segments again on non-alphanumeric delimiters such as underscores and punctuation. For example, `hello_world こんにちは世界` is tokenized as `hello`, `world`, `こんにちは`, and `世界`.

- **Models**: Uses compiled ICU4X segmenter data bundled with Lance
- **Usage**: Specify as `icu`, or `icu/split` to split punctuation-delimited identifiers
- **Features**:
  - Unicode-aware word boundary detection
  - Dictionary-based segmentation for Chinese, Japanese, Khmer, Lao, Myanmar, and Thai
  - No external language model download required

#### Jieba Tokenizer (Chinese)

Jieba is a popular Chinese text segmentation library that uses a dictionary-based approach with statistical methods for word segmentation.

- **Configuration**: Uses a `config.json` file in the model directory
- **Models**: Must be downloaded and placed in the Lance home directory under `jieba/`
- **Usage**: Specify as `jieba/<model_name>` or just `jieba` for the default model
- **Config Structure**:
  ```json
  {
    "main": "path/to/main/dictionary",
    "users": ["path/to/user/dict1", "path/to/user/dict2"]
  }
  ```
- **Features**:
  - Accurate word segmentation for Simplified and Traditional Chinese
  - Support for custom user dictionaries
  - Multiple segmentation modes (precise, full, search engine)

#### Lindera Tokenizer (Japanese)

Lindera is a morphological analysis tokenizer specifically designed for Japanese text. It provides proper word segmentation for Japanese, which doesn't use spaces between words.

- **Configuration**: Uses a `config.yml` file in the model directory
- **Models**: Must be downloaded and placed in the Lance home directory under `lindera/`
- **Usage**: Specify as `lindera/<model_name>` where `<model_name>` is the subdirectory containing the model files
- **Features**:
  - Morphological analysis with part-of-speech tagging
  - Dictionary-based tokenization
  - Support for custom user dictionaries

### Token Filters

Token filters are applied in sequence after the base tokenizer:

| Filter           | Description                                 | Configuration                   |
|------------------|---------------------------------------------|---------------------------------|
| **RemoveLong**   | Removes tokens exceeding max_token_length   | `max_token_length`              |
| **LowerCase**    | Converts tokens to lowercase                | `lower_case` (default: true)    |
| **Stemmer**      | Reduces words to their root form            | `stem`, `language`              |
| **StopWords**    | Removes common words like "the", "is", "at" | `remove_stop_words`, `language` |
| **AsciiFolding** | Converts accented characters to ASCII       | `ascii_folding` (default: true) |

### Supported Languages

For stemming and stop word removal, the following languages are supported:
Arabic, Danish, Dutch, English, Finnish, French, German, Greek, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Tamil, Turkish

## Document Type
Lance supports 2 kinds of documents: text and json. Different document types have different tokenization rules, and
parse tokens in different format.

### Text Type
Text type includes text and list of text. Tokens are generated by base_tokenizer.

The example below shows how text document is parsed into tokens.
```text
Tom lives in San Francisco.
```

The tokens are below.
```text
Tom
lives
in
San
Francisco
```

### Json Type
Json is a nested structure, lance breaks down json document into tokens in triplet format `path,type,value`. The valid
types are: str, number, bool, null.

In scenarios where the triplet value is a str, the text value will be further tokenized using the base_tokenizer,
resulting in multiple triplet tokens.

During querying, the Json Tokenizer uses the triplet format instead of the json format, which simplifies the query
syntax.

The example below shows how the json document is tokenized. Assume we have the following json document:
```json
{
  "name": "Lance",
  "legal.age": 30,
  "address": {
    "city": "San Francisco",
    "zip:us": 94102
  }
}
```

After parsing, the document will be tokenized into the following tokens:
```
name,str,Lance
legal.age,number,30
address.city,str,San
address.city,str,Francisco
address.zip:us,number,94102
```

Then we do full text search in triplet format. To search for "San Francisco," we can search with one of the triplets
below:
```
address.city:San Francisco
address.city:San
address.city:Francisco
```

## Training Process

Building an FTS index is a multi-phase pipeline: the source column is scanned, documents are tokenized in parallel, intermediate results are spilled to part files on disk, and the part files are merged into final output partitions.

### Phase 1: Tokenization

The input column is read as a stream of record batches and dispatched to a pool of tokenizer worker tasks. Each worker tokenizes documents independently, accumulating tokens, posting lists, and document metadata in memory.

When a worker's accumulated data reaches the partition size limit or the document count hits `u32::MAX`, it flushes the data to disk as a set of part files (`part_<id>_tokens.lance`, `part_<id>_invert.lance`, `part_<id>_docs.lance`). A single worker may produce multiple part files if it processes enough data.

### Phase 2: Merge

After all workers finish, the part files are merged into output partitions. Part files are streamed with bounded buffering so that not all data needs to be loaded into memory at once. For each part file, the token dictionaries are unified, document sets are concatenated, and posting lists are rewritten with adjusted IDs.

When a merged partition reaches the target size, it is written to the destination store and a new one is started. After all part files are consumed the final partition is flushed, and a `metadata.lance` file is written listing the partition IDs and index parameters.

### Configuration

| Environment Variable       | Default                          | Description                                                                                                           |
|----------------------------|----------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| `LANCE_FTS_NUM_SHARDS`     | Number of compute-intensive CPUs | Number of parallel tokenizer worker tasks. Higher values increase indexing throughput but use more memory.             |
| `LANCE_FTS_PARTITION_SIZE` | 256 (MiB)                        | Maximum uncompressed size of a worker's in-memory buffer before it is spilled to a part file.                         |
| `LANCE_FTS_TARGET_SIZE`    | 4096 (MiB)                       | Target uncompressed size for merged output partitions. Fewer, larger partitions improve query performance.             |

### Memory and Performance Considerations

Memory usage is primarily determined by two factors:

- **`LANCE_FTS_NUM_SHARDS`** -- Each worker holds an independent in-memory buffer. Peak memory is roughly `NUM_SHARDS * PARTITION_SIZE` plus the overhead of token dictionaries and posting list structures.
- **`LANCE_FTS_PARTITION_SIZE`** -- Larger values reduce the number of part files and make the merge phase cheaper. Smaller values reduce per-worker memory at the cost of more part files.

Merge phase memory is bounded by the streaming approach: part files are loaded one at a time with a small concurrency buffer. The merged partition's in-memory size is bounded by `LANCE_FTS_TARGET_SIZE`.

Building an FTS index requires temporary disk space to store the part files generated during tokenization. The amount of temporary space depends heavily on whether position information is enabled. An index with `with_position: true` stores the position of every token occurrence in every document, which can easily require 10x the size of the original column or more in temporary disk space. An index without positions tends to be smaller than the original column and will typically need less than 2x the size of the column in total disk space.

Performance tips:

- Larger `LANCE_FTS_TARGET_SIZE` produces fewer output partitions, which is beneficial for query performance because queries must scan every partition's token dictionary. When memory allows, prefer fewer, larger partitions.
- `with_position: true` significantly increases index size because term positions are stored for every occurrence. Only enable it when phrase queries are needed.
- The ngram tokenizer generates many more tokens per document than word-level tokenizers, so expect larger index sizes and higher memory usage.

### Distributed Training

The FTS index supports distributed training where different worker nodes each index a subset of the data and the results are assembled afterward.

1. Each distributed worker is assigned a **fragment mask** (`(fragment_id as u64) << 32`) that is OR'd into the partition IDs it generates, ensuring globally unique IDs across workers.
2. Workers set `skip_merge: true` so they write their part files directly without running the merge phase.
3. Instead of a single `metadata.lance`, each worker writes per-partition metadata files named `part_<id>_metadata.lance`.
4. After all workers finish, a coordinator merges the metadata files: it collects all partition IDs, remaps them to a sequential range starting from 0 (renaming the corresponding data files), and writes the final unified `metadata.lance`.

This allows each worker to operate independently during the tokenization phase. Only the final metadata merge requires a single-node step, and it is lightweight since it only renames files and writes a small metadata file.

## Accelerated Queries

Lance SDKs provide dedicated full text search APIs to leverage the FTS index capabilities.
These APIs support complex query types beyond simple token matching,
enabling sophisticated text search operations.
Here are the query types enabled by the FTS index:

| Query Type          | Description                                                                              | Example Usage                                        | Result Type |
|---------------------|------------------------------------------------------------------------------------------|------------------------------------------------------|-------------|
| **contains_tokens** | Basic token-based search (UDF) with BM25 scoring and automatic result ranking            | SQL: `contains_tokens(column, 'search terms')`       | AtMost      |
| **match**           | Match query with configurable AND/OR operators and relevance scoring                     | `{"match": {"query": "text", "operator": "and/or"}}` | AtMost      |
| **phrase**          | Exact phrase matching with position information (requires `with_position: true`)         | `{"phrase": {"query": "exact phrase"}}`              | AtMost      |
| **boolean**         | Complex boolean queries with must/should/must_not clauses for sophisticated search logic | `{"boolean": {"must": [...], "should": [...]}}`      | AtMost      |
| **multi_match**     | Search across multiple fields simultaneously with unified scoring                        | `{"multi_match": [{"field1": "query"}, ...]}`        | AtMost      |
| **boost**           | Boost relevance scores for specific terms or queries by a configurable factor            | `{"boost": {"query": {...}, "factor": 2.0}}`         | AtMost      |

label_list.md 1.7 KB

# Label List Index

Label list indices are optimized for columns containing multiple labels or tags per row.
They provide efficient set-based queries on multi-value columns using an underlying bitmap index.

## Index Details

```protobuf
%%% proto.message.LabelListIndexDetails %%%
```

## Storage Layout

The label list index uses a bitmap index internally and stores its data in:

1. `bitmap_page_lookup.lance` - Bitmap index mapping unique labels to row IDs

### File Schema

| Column    | Type       | Nullable | Description                                                            |
|-----------|------------|----------|------------------------------------------------------------------------|
| `keys`    | {DataType} | true     | The unique label value from the indexed column                         |
| `bitmaps` | Binary     | true     | Serialized RowAddrTreeMap containing row addr where this label appears |

## Accelerated Queries

The label list index provides exact results for the following query types:

| Query Type                          | Description                            | Operation                                   | Result Type |
|-------------------------------------|----------------------------------------|---------------------------------------------|-------------|
| **array_has / array_contains**      | Array contains the specified value     | Bitmap lookup for a single label            | Exact       |
| **array_has_all**                   | Array contains all specified values    | Intersects bitmaps for all specified labels | Exact       |
| **array_has_any**                   | Array contains any of specified values | Unions bitmaps for all specified labels     | Exact       |

ngram.md 1.8 KB

# N-gram Index

N-gram indices break text into overlapping sequences (trigrams) for efficient substring matching.
They provide fast text search by indexing all 3-character sequences in the text after
applying ASCII folding and lowercasing.

## Index Details

```protobuf
%%% proto.message.NGramIndexDetails %%%
```

## Storage Layout

The N-gram index stores tokenized text as trigrams with their posting lists:

1. `ngram_postings.lance` - Trigram tokens and their posting lists

### File Schema

| Column         | Type   | Nullable | Description                                       |
|----------------|--------|----------|---------------------------------------------------|
| `tokens`       | UInt32 | true     | Hashed trigram token                              |
| `posting_list` | Binary | false    | Compressed bitmap of row IDs containing the token |

## Accelerated Queries

The N-gram index provides inexact results for the following query types:

| Query Type     | Description              | Operation                                             | Result Type |
|----------------|--------------------------|-------------------------------------------------------|-------------|
| **contains**   | Substring search in text | Finds all trigrams in query, intersects posting lists | AtMost      |
| **regexp_like** / **regexp_match** | Regular-expression match | Derives a necessary trigram condition from the pattern (AND of intersections, OR of unions), then rechecks the true regex | AtMost |
| **LIKE** (infix) | Wildcard match such as `%foo%bar%` | Uses the literal segments of the pattern as a trigram condition, then rechecks the LIKE | AtMost |

Patterns from which no trigram can be derived - for example `a.b`, `.*`,
case-insensitive matches, or literal runs shorter than three characters - fall
back to rechecking every row. This is always correct, just not accelerated.

rtree.md 8.2 KB

# R-Tree Index

The R-Tree index is a static, immutable 2D spatial index. It is built on bounding boxes to organize the data. This index is intended to accelerate rectangle-based pruning.

It is designed as a multi-level hierarchical structure: leaf pages store tuples `(bbox, id=rowid)` for indexed geometries; branch pages aggregate child bounding boxes and store `id=pageid` pointing to child pages; a single root page encloses the entire tree. Conceptually, it can be thought of as an extension of the B+-tree to multidimensional objects, where bounding boxes act as keys for spatial pruning.

The index uses a packed-build strategy where items are first sorted and then grouped into fixed-size leaf pages.

This packed-build flow is:
- Sort items (bboxes) according to the sorting algorithm.
- Pack consecutive items into leaf pages of `page_size` entries; then build parent pages bottom-up by aggregating child page bboxes.

## Sorting

Sorting does not change the R-Tree data structure, but it is critical to performance. Currently, Hilbert sorting is implemented, but the design is extensible to other spatial sorting algorithms.

### Hilbert Curve Sorting

Hilbert sorting imposes a linear order on 2D items using a space-filling Hilbert curve to maximize locality in both axes. This improves leaf clustering, which benefits query pruning.

Hilbert sorting is performed in three steps:

1. **Global bounding box**: compute the global bbox `[xmin_g, ymin_g, xmax_g, ymax_g]` over all items for training index.
2. **Normalize and compute Hilbert value**:
    - For each item bbox `[xmin_i, ymin_i, xmax_i, ymax_i]`, compute its center:
        - `cx = (xmin_i + xmax_i) / 2`
        - `cy = (ymin_i + ymax_i) / 2`
    - Map the center to a 16‑bit grid per axis using the global bbox. Let `W = xmax_g - xmin-g` and `H = ymax_g - ymin_g`. The normalized integer coordinates are:
        - `xi = round(((cx - xmin_g) / W) * (2^16 - 1))`
        - `yi = round(((cy - ymin_g) / H) * (2^16 - 1))`
    - If the global width or height is effectively zero, the corresponding axis is treated as degenerate and set to `0` for all items (the ordering then degenerates to 1D on the other axis).
    - For each `(xi, yi)` in `[0 .. 2^16-1] × [0 .. 2^16-1]`, compute a 32‑bit Hilbert value using a standard 2D Hilbert algorithm. In pseudocode (with `bits = 16`):
      ```
      fn hilbert_value(x, y, bits):
          # x, y: integers in [0 .. 2^bits - 1]
          h = 0
          mask = (1 << bits) - 1

          for s from bits-1 down to 0:
              rx = (x >> s) & 1
              ry = (y >> s) & 1
              d  = ((3 * rx) XOR ry) << (2 * s)
              h  = h | d

              if ry == 0:
                  if rx == 1:
                      x = (~x) & mask
                      y = (~y) & mask
                  swap(x, y)

          return h
      ```
      - The resulting `h` is stored as the item’s Hilbert value (type `u32` with `bits = 16`).
3. **Sort**: sort items by Hilbert value.

## Index Details

```protobuf
%%% proto.message.RTreeIndexDetails %%%
```

## Storage Layout

The R-Tree index consists of two files:

1. `page_data.lance` - Stores all pages (leaf, branch) as repeated `(bbox, id)` tuples, written bottom-up (leaves first, then branch levels)
2. `nulls.lance` - Stores a serialized RowAddrTreeMap of rows with null

### Page File Schema

| Column | Type     | Nullable | Description                                                                                                                                                                                                                                                     |
|:-------|:---------|:---------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `bbox` | RectType | false    | Type is Rect defined by [geoarrow-rs](https://github.com/geoarrow/geoarrow-rs) RectType; physical storage is Struct<xmin: Float64, ymin: Float64, xmax: Float64, ymax: Float64>. Represents the node bounding box (leaf: item bbox; branch: child aggregation). |
| `id`   | UInt64   | false    | Reuse the `id` column to store `rowid` in leaf pages and `pageid` in branch pages                                                                                                                                                                               |

### Nulls File Schema

| Column  | Type   | Nullable | Description                                                  |
|:--------|:-------|:---------|:-------------------------------------------------------------|
| `nulls` | Binary | false    | Serialized RowAddrTreeMap of rows with null/invalid geometry |

### Schema Metadata

The following optional keys can be used by implementations and are stored in the schema metadata:

| Key         | Type   | Description                                       |
|:------------|:-------|:--------------------------------------------------|
| `page_size` | String | Page size per page                                |
| `num_pages` | String | Total number of pages written                     |
| `num_items` | String | Number of non-null leaf items in the index        |
| `bbox`      | String | JSON-serialized global BoundingBox of the dataset |

### Query Traversal

This index serializes the multi-level hierarchical RTree structure into a single page file following the schema above. At lookup time, the reader computes each page offset using the algorithm below and reconstructs the hierarchy for traversal.

Offsets are derived from `num_items` and `page_size` of metadata as follows:

- Leaf: `leaf_pages = ceil(num_items / page_size)`; leaf `i` has `page_offset = i * page_size`.
- Branch: let `level_offset` be the starting offset for current level, which actually represents total items from all lower levels; let `prev_pages` be pages in the level below; `level_pages = ceil(prev_pages / page_size)`. For branch `j`, `page_offset = j * page_size + level_offset`.
- Iterate levels until one page remains; the root is the last page and has `pageid = num_pages - 1`.
- Page lengths: once all page offsets are collected, compute each `page_len` by the next offset difference; for the final page (root), `page_len = page_file_total_rows - page_offset` (where `page_file_total_rows` is total rows in `page_data.lance`).

Traversal starts from the root (`pageid = num_pages - 1`):

- If `page_offset < num_items` (leaf), read items `[page_offset .. page_offset + page_len)` and emit candidate `rowid`s matching the query bbox.
- Otherwise (branch), descend into children whose bounding boxes match the query bbox.
- Continue until there are no more pages to visit; the union of emitted `rowid`s forms the candidate set for evaluation.

## Accelerated Queries

The R-Tree index accelerates the following query types by returning a candidate set of matching bounding boxes. Exact geometry verification must be performed by the execution engine.

| Query Type     | Description                | Operation                                     | Result Type |
|:---------------|:---------------------------|:----------------------------------------------|:------------|
| **Intersects** | `St_Intersects(col, geom)` | Prunes candidates by bbox intersection        | AtMost      |
| **Contains**   | `St_Contains(col, geom)`   | Prunes candidates by bbox containment         | AtMost      |
| **Within**     | `St_Within(col, geom)`     | Prunes candidates by bbox within relation     | AtMost      |
| **Touches**    | `St_Touches(col, geom)`    | Prunes candidates by bbox touch relation      | AtMost      |
| **Crosses**    | `St_Crosses(col, geom)`    | Prunes candidates by bbox crossing relation   | AtMost      |
| **Overlaps**   | `St_Overlaps(col, geom)`   | Prunes candidates by bbox overlap relation    | AtMost      |
| **Covers**     | `St_Covers(col, geom)`     | Prunes candidates by bbox cover relation      | AtMost      |
| **CoveredBy**  | `St_Coveredby(col, geom)`  | Prunes candidates by bbox covered-by relation | AtMost      |
| **IsNull**     | `col IS NULL`              | Returns rows recorded in the nulls file       | Exact       |

zonemap.md 2.9 KB

# Zone Map Index

Zone maps are a columnar database technique for predicate pushdown and scan pruning.
They break data into fixed-size chunks called "zones" and maintain summary statistics
(min, max, null count) for each zone, enabling efficient filtering by eliminating
zones that cannot contain matching values.

Zone maps are "inexact" filters - they can definitively exclude zones but may include
false positives that require rechecking.

In addition, since finding NULLs is a common query pattern, the index also maintains a
bitmap of null rows which allows it to return exact results for IS NULL queries.

## Index Details

```protobuf
%%% proto.message.ZoneMapIndexDetails %%%
```

## Storage Layout

The zone map index stores zone statistics in a single file:

1. `zonemap.lance` - Zone statistics for query pruning

### Zone Statistics File Schema

| Column        | Type       | Nullable | Description                             |
|---------------|------------|----------|-----------------------------------------|
| `min`         | {DataType} | true     | Minimum value in the zone               |
| `max`         | {DataType} | true     | Maximum value in the zone               |
| `null_count`  | UInt32     | false    | Number of null values in the zone       |
| `nan_count`   | UInt32     | false    | Number of NaN values (for float types)  |
| `fragment_id` | UInt64     | false    | Fragment containing this zone           |
| `zone_start`  | UInt64     | false    | Starting row offset within the fragment |
| `zone_length` | UInt32     | false    | Number of rows in this zone             |

### Schema Metadata

| Key                 | Type   | Description                               |
|---------------------|--------|-------------------------------------------|
| `rows_per_zone`     | String | Number of rows per zone (default: "8192") |
| `null_bitmap`       | UInt32 | Index of null bitmap global buffer        |

### Global Buffers

| Metadata Key        | Description                                                |
|---------------------|------------------------------------------------------------|
| `null_bitmap`       | A serialized RowAddrTreeMap specifying which rows are null |

## Accelerated Queries

The zone map index provides inexact results for the following query types (nullability queries
return exact results):

| Query Type | Description               | Operation                                   | Result Type |
|------------|---------------------------|---------------------------------------------|-------------|
| **Equals** | `column = value`          | Includes zones where min ≤ value ≤ max      | AtMost      |
| **Range**  | `column BETWEEN a AND b`  | Includes zones where ranges overlap         | AtMost      |
| **IsIn**   | `column IN (v1, v2, ...)` | Includes zones that could contain any value | AtMost      |
| **IsNull** | `column IS NULL`          | Includes zones where null_count > 0         | Exact       |

system

frag_reuse.md 11 KB

# Fragment Reuse Index

The Fragment Reuse Index (FRI) is an internal index that keeps existing indices
usable while fragments are compacted or reclustered. It records how old physical
row addresses map to new addresses, without changing surviving row values.

## Use Case 1: Compact Fragments

When data modifications happen against a Lance table, they can trigger compaction
and index optimization at the same time to improve data layout and index coverage.
By default, compaction remaps all indices to prevent read regression.
This means both compaction and index optimization can modify the same index and
cause one process to fail. Typically, compaction fails because it has to modify
all indices and takes longer, resulting in table layout degrading over time.

Fragment Reuse Index allows compaction to defer the index remap process.
Suppose a compaction removes fragments A and B and produces C. At query runtime,
the existing indices are reused by translating their addresses in A and B to
addresses in C. Because indices are typically cached in memory after initial
load, the translated index can be reused by subsequent queries.

## Use Case 2: Recluster Fragments

Fragments are often organized by data arrival order. Queries filtering by fields
such as day or user UUID may therefore need to search many fragments or index
segments. An external engine, such as Spark, can reorganize rows into destination
fragments grouped by those fields.

A **stable partition** assigns source rows to destination fragments while
preserving their relative source order within each destination. This lets FRI
reuse existing indices after reclustering. Sorting or arbitrarily shuffling rows
within a destination requires a different mapping; it is not represented by the
stable-partition format defined here.

## Index Details

FRI uses one system-index entry named `__lance_frag_reuse`. Its
`IndexMetadata.index_details` contains `FragmentReuseIndexDetails`:

```protobuf
%%% proto.message.FragmentReuseIndexDetails %%%
```

The outer `InlineContent` / `ExternalFile` choice applies to the entire history.
Small histories are stored inline; larger histories store the serialized
`InlineContent` in `_indices/<FRI UUID>/details.binpb`. Updating the history
replaces the FRI metadata. Stable-partition row maps are separate immutable files
and are not rewritten with that history.

### FRI Index Versions

`IndexMetadata.index_version` identifies the format required to interpret FRI;
it is not a dataset version or the sequence number of a rewrite.

- **Version 0** retains the existing compaction format and read/write behavior.
  `InlineContent.legacy_versions` keeps the original field number, 1, and wire
  representation of `versions`.
- **Version 1** adds `InlineContent.transitions` at field 2. It supports both
  ordered compaction and stable partition. Each legacy group can be interpreted
  as an ordered-compaction transition, forming one history with the new records.

Adding a transition or a new mapping type does not itself require increasing
`index_version`. Readers may skip unsupported mappings and fall back to scanning
where a complete translation path is unavailable. Writers must reject operations
that require interpreting or maintaining unsupported mappings; operations that
carry the existing history unchanged need not interpret them. Changes to the
shared metadata contract that require reader upgrades must increase `index_version`.

### Shared Transition Metadata

Each transition records ordered `sources` and `destinations`, plus exactly one
mapping. A fragment digest contains its ID, physical row count, and deleted row
count. Source counts describe the rewrite input; destination counts describe the
newly written fragments, with zero deletions.

Sources define scan order; destinations define output order for compaction and
label order for stable partition. The surviving source row count must equal the
total destination row count. Each fragment has at most one producer and one
consumer in the retained history, and the graph must be acyclic. A fragment's
physical row count stays constant across transitions; its deletion count may
increase. Translation follows these dependencies, not the serialized list order.

## Mappings

### Ordered Compaction

Ordered compaction concatenates surviving rows in source-fragment order, with
ascending physical row offsets within each source, then splits that stream into
the ordered destination fragments.

`changed_row_addrs` stores a serialized RoaringTreemap of surviving source
addresses. A physical address uses the upper 32 bits for the fragment ID and the
lower 32 bits for the row offset. A valid source row absent from the bitmap was
deleted.

To translate a surviving row, count the surviving rows in preceding source
fragments and the surviving rows before its offset in its own fragment. Their
sum is its zero-based position in the output stream. Find the destination whose
cumulative physical-row range contains that position and subtract the start of
that range to obtain the destination offset.

The legacy `Group` and the new `OrderedCompaction` mapping use this same ordering
and bitmap representation. No per-row destination labels are required.

### Stable Partition

A stable partition processes source fragments sequentially in their recorded
order and assigns each surviving row a label: the zero-based position of its
destination in `destinations`. Rows with the same label retain their relative
source order. Deleted source rows carry a null label.

The mapping payload is one immutable Lance file,
`_fri/<map_id>/stable_partition.lance`. `map_id` is a UUID independent of the FRI
index UUID, so the file survives FRI metadata rewrites. `map_size_bytes` records
its exact size. An unset `base_id` selects the dataset base; otherwise it selects
the corresponding `Manifest.base_paths` entry. Relocating a dataset requires
copying these files or updating their references.

#### Row Map File Schema

The row map is a Lance file with one nullable `uint16` label per physical source
row. Rows follow source-fragment order, then physical row offset within each
fragment, including deleted rows.

```python
import pyarrow as pa

row_map_schema = pa.schema([pa.field("label", pa.uint16(), nullable=True)])
```

A label tells us which destination receives the row: `0` means
`destinations[0]`, `1` means `destinations[1]`, and `null` means the row was
deleted at rewrite time. It stores the destination's position in the list, not
its fragment ID or row offset. The number of destinations must be between 1
and 65,536, and each non-null label must be smaller than that number.

#### Counts Matrix

Labels tell us the destination fragment. To find the row offset inside that
fragment, we count earlier source rows with the same label.

To avoid reading all earlier labels, source rows are divided into blocks.
For each block and destination, the counts matrix stores the cumulative number
of rows sent to that destination through the end of the block. Null labels are
not counted. `block_rows` must be positive; the current writer uses 65,536.
Only the final block may be shorter.

The matrix is stored in a Lance global buffer. The schema metadata key
`lance:stable_partition:counts_buffer_index` contains its buffer index as a
decimal string. Readers locate the buffer through the normal Lance file metadata.

The buffer starts with a 28-byte header, in this order:

- Magic: four bytes, `LSPC`.
- Version: `u32`, value 1.
- Representation: `u32`, value 0 for the dense grid.
- Number of destinations: `u32`.
- Rows per block: `u32`.
- Total physical source rows: `u64`.

All integers are unsigned and little-endian. The header is followed by
`ceil(total_rows / block_rows) * num_destinations` cumulative `u32` counts,
ordered by block, then destination. No extra bytes are allowed. Unsupported
versions or representations must be rejected.

Counts must never decrease, and a block cannot contribute more live rows than
its length. Final destination counts must match the destination digests. The
label-file row count and header total must match the sum of source physical row
counts. An empty file has no grid rows and zero destination totals.

#### Address Translation

For source fragment `s` and row offset `o`:

1. Find its label-file position: the physical row counts of all preceding source
   fragments, plus `o`.
2. Read the block containing that position. A null label means the row was deleted;
   otherwise label `d` selects `destinations[d]`.
3. The destination row offset is the count for `d` before this block, plus the
   number of occurrences of `d` strictly before this row within the block.
   The count before the first block is zero.

For batch translation, read each requested block once. Initialize destination
counters from the preceding counts row, then scan the block in order. Each
non-null label takes the current counter as its offset and increments it.
Opening FRI history does not require reading labels; translation only needs the
counts matrix and the requested label blocks.

## Expected Use Pattern

When indexing or index remapping cannot keep up with compaction or reclustering,
FRI allows fragment rewrites to proceed while retaining existing indices.
Each rewrite that defers index remapping records its mapping: a reuse version
in FRI index version 0, or transitions in index version 1. A rewrite publishes
its mapping atomically, replacing the FRI entry in the same commit. A
transition must reference only fragments committed no later than itself.
Fragments not covered by any index are served by scanning, so correctness
never depends on a mapping being present.

Once all dependent indices have caught up, the corresponding history can be
trimmed. Cleanup must retain intermediate transitions still needed to translate
old addresses. External mapping files can be deleted only when no retained
dataset version references them.

## Impacts

### Conflict Resolution

Deferring index remapping avoids replacing existing indices during a fragment
rewrite, reducing conflicts with concurrent index building or optimization.
FRI does not remove conflicts between overlapping rewrites. See
[conflict resolution](../../table/transaction.md#conflict-resolution).

### Index Load Cost

Loading affected indices requires translating their stored row addresses.
Ordered compaction uses its bitmap and fragment layouts; stable partition also
reads the required row-map blocks. Translated indices can be cached and reused.
Longer mapping chains add translation work; trimming unused history reduces it.

### Reader and Writer Compatibility

The first commit publishing FRI index version 1 sets
`FLAG_FRAGMENT_REUSE_INDEX` (1024) in both manifest flag fields. Subsequent
manifests retain both bits. FRI index version 0 does not require this flag.

Tables using stable row IDs do not support tagged histories; writers must not
publish `index_version >= 1` on them.

The reader flag prevents older clients from partially interpreting the new
history. The writer flag prevents them from dropping mappings when rewriting
FRI metadata. Clients that do not support the corresponding flag must reject
the read or write and require an upgrade. See
[feature flags](../../table/versioning.md).

mem_wal.md 840 B

# MemWAL Index

The MemWAL Index is a system index that serves as the centralized structure for all MemWAL metadata.
It stores configuration (shard specs, indexes to maintain), SSTable compaction progress, and shard state snapshots.

A table has at most one MemWAL index.
The table may be a primary-key table or an append-only table without primary-key metadata.
Primary-key-dependent lookup and deduplication semantics only apply when a primary key is defined.

For the complete specification, see:

- [MemWAL Index Overview](../../table/mem_wal.md#memwal-index) - Purpose and high-level description
- [MemWAL Index Details](../../table/mem_wal.md#memwal-index-details) - Storage format, schemas, and staleness handling
- [MemWAL Implementation](../../table/mem_wal.md#implementation-expectation) - Implementation details and expectations

vector

index.md 21.2 KB

# Vector Indices

Lance provides a powerful and extensible secondary index system for efficient vector similarity search.
All vector indices are stored as regular Lance files, making them portable and easy to manage.
It is designed for efficient similarity search across large-scale vector datasets.

## Concepts

Lance splits each vector index into 3 parts - clustering, sub-index and quantization.

### Clustering

Clustering divides all the vectors into different disjoint clusters (a.k.a. partitions).
Lance currently supports using Inverted File (IVF) as the primary clustering mechanism.
IVF partitions the vectors into clusters using the k-means clustering algorithm.
Each cluster contains vectors that are similar to the cluster centroid.
During search, only the most relevant clusters are examined, dramatically reducing search time.
IVF can be combined with any sub-index type and quantization method.

### Sub-Index

The sub-index determines how vectors are organized for search. Lance currently supports:

- **FLAT**: Exact search with no approximation - scans all vectors
- **HNSW**: Hierarchical Navigable Small World graphs for fast approximate search

### Quantization

The quantization method determines how vectors are stored and compressed. Lance currently supports:

- **Product Quantization (PQ)**: Compresses vectors by splitting them into smaller sub-vectors and quantizing each independently
- **Scalar Quantization (SQ)**: Applies scalar quantization to each dimension of the vector independently
- **RabitQ (RQ)**: Uses random rotation and binary quantization for extreme compression
- **FLAT**: No quantization, keeps original vectors for exact search

### Common Combinations

When we refer to an index type, it is typically `{clustering}_{sub_index}_{quantization}`.
If sub-index is just `FLAT`, we usually omit it and just refer to it by `{clustering}_{quantization}`.
Here are the commonly used combinations:

| Index Type      | Name                                            | Description                                                                              |
| --------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------- |
| **IVF_PQ**      | Inverted File with Product Quantization         | Combines IVF clustering with PQ compression for efficient storage and search             |
| **IVF_HNSW_SQ** | Inverted File with HNSW and Scalar Quantization | Uses IVF for coarse clustering and HNSW for fine-grained search with scalar quantization |
| **IVF_SQ**      | Inverted File with Scalar Quantization          | Combines IVF clustering with scalar quantization for balanced compression                |
| **IVF_RQ**      | Inverted File with RabitQ                       | Combines IVF clustering with RabitQ for extreme compression using binary quantization    |
| **IVF_FLAT**    | Inverted File without quantization              | Uses IVF clustering with exact vector storage for precise search within clusters         |

### Versioning

The Lance vector index format has gone through 3 versions so far.
This document currently only records version 3 which is the latest version.
The specific version of the vector index is recorded in the `index_version` field of the generic [index metadata](../index.md#loading-an-index).

## Storage Layout (V3)

Each vector index is stored as 2 regular Lance files - index file and auxiliary file.

### Index File

The index structure file containing the search graph/structure with index-specific schema.
It is stored as a Lance file with name `index.idx` within the index directory.

#### Arrow Schema

The index file stores the search structure with graph or flat organization.
The Arrow schema of the Lance file varies depending on the sub-index type used.

!!! note
    All partitions are stored in the same file, and partitions must be written in order.

##### FLAT

FLAT indices perform exact search with no approximation. This is essentially an empty file with a minimal schema:

| Column          | Type   | Nullable | Description                                  |
| --------------- | ------ | -------- | -------------------------------------------- |
| `__flat_marker` | uint64 | false    | Marker field for FLAT index (no actual data) |

##### HNSW

HNSW (Hierarchical Navigable Small World) indices provide fast approximate search through a multi-level graph structure. This stores the HNSW graph with the following schema:

| Column        | Type          | Nullable | Description            |
| ------------- | ------------- | -------- | ---------------------- |
| `__vector_id` | uint32        | true     | Vector identifier      |
| `__neighbors` | list<uint32>  | true     | Neighbor node IDs      |
| `_distance`   | list<float32> | true     | Distances to neighbors |

!!! note
    HNSW consists of multiple levels, and all levels must be written in order starting from level 0.

#### Arrow Schema Metadata

The index file contains metadata in its Arrow schema metadata to describe the index configuration and structure.
Here are the metadata keys and their corresponding values:

##### "lance:index"

Contains basic index configuration information in JSON:

| JSON Key        | Type   | Expected Values                                           |
| --------------- | ------ | --------------------------------------------------------- |
| `type`          | String | Index type (e.g., "IVF_PQ", "IVF_RQ", "IVF_HNSW", "FLAT") |
| `distance_type` | String | Distance metric (e.g., "l2", "cosine", "dot")             |

##### "lance:ivf"

References the IVF metadata stored in the Lance file global buffer.
This value records the global buffer index, currently this is always "1".

!!! note
    Global buffer indices in Lance files are 1-based,
    so you need to subtract 1 when accessing them through code.

##### "lance:flat"

Contains partition-specific metadata for the `FLAT` sub-index structure.
This is an empty string since FLAT indices don't require additional metadata at this moment.

##### "lance:hnsw"

Contains the HNSW-specific JSON metadata for each partition, including graph structure information:

| JSON Key        | Type         | Expected Values                          |
| --------------- | ------------ | ---------------------------------------- |
| `entry_point`   | u32          | Starting node for graph traversal        |
| `params`        | Object       | HNSW construction parameters (see below) |
| `level_offsets` | Array<usize> | Offset for each level in the graph       |

The `params` object contains the following HNSW construction parameters:

| JSON Key            | Type          | Description                                                    | Default |
| ------------------- | ------------- | -------------------------------------------------------------- | ------- |
| `max_level`         | u16           | Maximum level of the HNSW graph                                | 7       |
| `m`                 | usize         | Number of connections to establish while inserting new element | 20      |
| `ef_construction`   | usize         | Size of the dynamic list for candidates                        | 150     |
| `prefetch_distance` | Option<usize> | Number of vectors ahead to prefetch while building             | Some(2) |

#### Lance File Global Buffer

##### IVF Metadata

For efficiency, Lance serializes IVF metadata to protobuf format and stores it in the Lance file global buffer:

```protobuf
%%% proto.message.IVF %%%
```

### Auxiliary File

The auxiliary file is a vector storage for quantized vectors.
It is stored as a Lance file named `auxiliary.idx` within the index directory.

#### Arrow Schema

Since the auxiliary file stores the actual (quantized) vectors,
the Arrow schema of the Lance file varies depending on the quantization method used.

!!! note
    All partitions are stored in the same file, and partitions must be written in order.

Every quantization format below lists only its internal columns. When a V3 IVF
writer materializes carried values, it appends one trailing column per carried
field after them, named and typed exactly as in the dataset schema. This physical
payload may be a subset of the manifest's `covering_fields` declaration (see
[Index Metadata](../index.md)). A reader returns only columns whose physical
schema and dataset field ids it verifies across every selected segment; all other
projected columns come from a base-table take.

A reader discovers carried columns by exclusion, not by position: any column in the
auxiliary file's schema that is not one of the quantizer's internal columns is a
carried column. Writers append them in trailing order, but a reader must not depend
on that ordering to identify them.

##### FLAT

No quantization applied - stores original vectors in their full precision:

| Column   | Type                     | Nullable | Description                                           |
| -------- | ------------------------ | -------- | ----------------------------------------------------- |
| `_rowid` | uint64                   | true     | Row identifier                                        |
| `flat`   | list<float32>[dimension] | true     | Original vector values (list_size = vector dimension) |

##### PQ

Compresses vectors using product quantization for significant memory savings:

| Column      | Type                                            | Nullable | Description                                   |
| ----------- | ----------------------------------------------- | -------- | --------------------------------------------- |
| `_rowid`    | uint64                                          | true     | Row identifier                                |
| `__pq_code` | list<uint8>[num_sub_vectors * num_bits / 8]     | true     | PQ codes, packed to `num_bits` per subvector  |

##### SQ

Compresses vectors using scalar quantization for moderate memory savings:

| Column      | Type                   | Nullable | Description                             |
| ----------- | ---------------------- | -------- | --------------------------------------- |
| `_rowid`    | uint64                 | true     | Row identifier                          |
| `__sq_code` | list<uint8>[dimension] | true     | SQ codes (list_size = vector dimension) |

##### RQ

Compresses vectors using RabitQ with random rotation and binary quantization for extreme compression:

| Column               | Type                                             | Nullable | Present when                | Description                                                     |
| -------------------- | ------------------------------------------------ | -------- | --------------------------- | --------------------------------------------------------------- |
| `_rowid`             | uint64                                           | true     | always                      | Row identifier                                                  |
| `_rabit_codes`       | list<uint8>[ceil(code_dim / 8)]                  | true     | always                      | Binary quantized codes (1 bit per dimension, packed into bytes) |
| `__add_factors`      | float32                                          | true     | always                      | Additive correction factors for distance computation            |
| `__scale_factors`    | float32                                          | true     | always                      | Scale correction factors for distance computation               |
| `__error_factors`    | float32                                          | true     | `raw_query` estimator       | Error factors for raw-query lower-bound pruning                 |
| `__blocked_ex_codes` | list<uint8>[next_multiple_of(code_dim, 64) * (num_bits - 1) / 8] | true | `num_bits > 1` | Extra RabitQ code bits for multi-bit RQ, in the blocked layout  |
| `__add_factors_ex`   | float32                                          | true     | `num_bits > 1`              | Additive correction factors for ex-code distance computation    |
| `__scale_factors_ex` | float32                                          | true     | `num_bits > 1`              | Scale correction factors for ex-code distance computation       |

!!! note
    Indexes written before the blocked ex-code layout store the same bits in
    `__ex_codes`, sized `ceil(dimension * (num_bits - 1) / 8)`. Readers still
    accept that column and repack it at load time; writers no longer emit it.

#### Arrow Schema Metadata

The auxiliary file also contains metadata in its Arrow schema metadata for vector storage configuration.
Here are the metadata keys and their corresponding values:

##### "distance_type"

The distance metric used to compute similarity between vectors (e.g., "l2", "cosine", "dot").

##### "lance:ivf"

Similar to the index file's "lance:ivf" but focused on vector storage layout.
This doesn't contain the partitions' centroids.
It's only used for tracking each partition's offset and length in the auxiliary file.

##### "lance:rabit"

Contains RabitQ-specific metadata in JSON format (only present for RQ quantization).
This includes the rotation matrix position, number of bits, and packing information.
See the RQ metadata specification in the "storage_metadata" section below.

##### "covering_field_ids"

The dataset field ids of the storage file's physical carried columns,
comma separated in physical schema order (only present when the storage carries
values). Arrow fields carry no Lance field id, so names and types alone cannot
prove which logical column a payload came from. Readers use these ids to bind
physical values to the segment's `covering_fields` declaration, and treat missing,
malformed, ambiguous, or mismatched metadata as no servable carried capability.
A merge must not combine shards whose carried columns disagree on these ids, even
when those columns match by name and type.

##### "storage_metadata"

Contains quantizer-specific metadata as a list of JSON strings.
Currently, the list always contains exactly 1 element with the quantizer metadata.

For **Product Quantization (PQ)**:

| JSON Key            | Type  | Description                                                      |
| ------------------- | ----- | ---------------------------------------------------------------- |
| `codebook_position` | usize | Position of the codebook in the global buffer                    |
| `nbits`             | u32   | Number of bits per subvector code (e.g., 8 bits = 256 codewords) |
| `num_sub_vectors`   | usize | Number of subvectors (m)                                         |
| `dimension`         | usize | Original vector dimension                                        |
| `transposed`        | bool  | Whether the codebook is stored in transposed layout              |

For **Scalar Quantization (SQ)**:

| JSON Key   | Type       | Description                            |
| ---------- | ---------- | -------------------------------------- |
| `dim`      | usize      | Vector dimension                       |
| `num_bits` | u16        | Number of bits for quantization        |
| `bounds`   | Range<f64> | Min/max bounds for scalar quantization |

For **RabitQ (RQ)**:

| JSON Key              | Type | Description                                          |
| --------------------- | ---- | ---------------------------------------------------- |
| `rotate_mat_position` | u32  | Position of the rotation matrix in the global buffer |
| `num_bits`            | u8   | Number of bits per dimension, in the range 1..=9     |
| `code_dim`            | u32  | Rotated vector dimension for the 1-bit binary code   |
| `packed`              | bool | Whether codes are packed for optimized computation   |
| `query_estimator`     | string | Distance estimator layout: `residual_query` or `raw_query`. Missing values are read as `residual_query` for compatibility with released 1-bit IVF_RQ indexes. |

#### Lance File Global Buffer

##### Quantization Codebook

For product quantization, the codebook is stored in `Tensor` format
in the auxiliary file's global buffer for efficient access:

```protobuf
%%% proto.message.Tensor %%%
```

##### Rotation Matrix

For RabitQ, the rotation matrix is stored in `Tensor` format
in the auxiliary file's global buffer. The rotation matrix is an orthogonal matrix used
to rotate vectors before binary quantization:

```protobuf
%%% proto.message.Tensor %%%
```

The rotation matrix has shape `[code_dim, code_dim]` where `code_dim` is the rotated vector
dimension. IVF_RQ always stores the 1-bit binary sign code in `_rabit_codes`; for `num_bits > 1`,
the remaining `num_bits - 1` ex-code bits are stored in `__blocked_ex_codes` instead of widening the
binary code path. New IVF_RQ indexes store raw-query estimator factors. `num_bits=1` indexes only
store the binary-code factor columns; multi-bit indexes also store separate ex-code additive and
scale factors.

## Appendices

### Appendix 1: Example IVF_PQ Format

This example shows how an `IVF_PQ` index is physically laid out. Assume vectors have dimension 128,
PQ uses 16 num_sub_vectors (m=16) with 8 num_bits per subvector, and distance type is "l2".

#### Index File

- Arrow Schema Metadata:
  - `"lance:index"` → `{ "type": "IVF_PQ", "distance_type": "l2" }`
  - `"lance:ivf"` → "1" (references IVF metadata in the global buffer)
  - `"lance:flat"` → `["", "", ...]` (one empty string per partition; IVF_PQ uses a FLAT sub-index inside each partition)

- Lance File Global buffer (Protobuf):
  - `Ivf` message containing:
    - `centroids_tensor`: shape `[num_partitions, 128]` (float32)
    - `offsets`: start offset (row) of each partition in `auxiliary.idx`
    - `lengths`: number of vectors in each partition
    - `loss`: k-means loss (optional)

#### Auxiliary File

- Arrow Schema Metadata:
  - `"distance_type"` → `"l2"`
  - `"lance:ivf"` → tracks per-partition `offsets` and `lengths` (no centroids here)
  - `"storage_metadata"` → `[ "{"pq":{"num_sub_vectors":16,"nbits":8,"dimension":128,"transposed":true}}" ]`
- Lance File Global buffer:
  - `Tensor` codebook with shape `[256, num_sub_vectors, dim/num_sub_vectors]` = `[256, 16, 8]` (float32)
- Rows with Arrow schema:

```python
pa.schema([
    pa.field("_rowid", pa.uint64()),
    pa.field("__pq_code", pa.list_(pa.uint8(), list_size=16)), # num_sub_vectors * num_bits / 8 = 16 * 8 / 8
])
```

### Appendix 2: Example IVF_RQ Format

This example shows how an `IVF_RQ` index is physically laid out. Assume vectors have dimension 128,
RQ uses 1 bit per dimension (`num_bits=1`), and distance type is "l2". For `num_bits > 1`, the
auxiliary schema also includes `__blocked_ex_codes`, `__add_factors_ex`, and `__scale_factors_ex`.

#### Index File

- Arrow Schema Metadata:
  - `"lance:index"` → `{ "type": "IVF_RQ", "distance_type": "l2" }`
  - `"lance:ivf"` → "1" (references IVF metadata in the global buffer)
  - `"lance:flat"` → `["", "", ...]` (one empty string per partition; IVF_RQ uses a FLAT sub-index inside each partition)

- Lance File Global buffer (Protobuf):
  - `Ivf` message containing:
    - `centroids_tensor`: shape `[num_partitions, 128]` (float32)
    - `offsets`: start offset (row) of each partition in `auxiliary.idx`
    - `lengths`: number of vectors in each partition
    - `loss`: k-means loss (optional)

#### Auxiliary File

- Arrow Schema Metadata:
  - `"distance_type"` → `"l2"`
  - `"lance:ivf"` → tracks per-partition `offsets` and `lengths` (no centroids here)
  - `"lance:rabit"` → `"{"rotate_mat_position":1,"num_bits":1,"packed":true,"query_estimator":"raw_query"}"`
- Lance File Global buffer:
  - `Tensor` rotation matrix with shape `[code_dim, code_dim]` = `[128, 128]` (float32)
- Rows with Arrow schema:

```python
pa.schema([
    pa.field("_rowid", pa.uint64()),
    pa.field("_rabit_codes", pa.list_(pa.uint8(), list_size=16)), # ceil(code_dim / 8) = ceil(128 / 8)
    pa.field("__add_factors", pa.float32()),
    pa.field("__scale_factors", pa.float32()),
    pa.field("__error_factors", pa.float32()),
])
```

### Appendix 3: Accessing Index File with Python

The following example demonstrates how to read and parse different components in the Lance index files using Python:

```python
import pyarrow as pa
import lance

# Open the index file
index_reader = lance.LanceFileReader.read_file("path/to/index.idx")

# Access schema metadata
schema_metadata = index_reader.metadata().schema.metadata

# Get the IVF metadata reference from schema
ivf_ref = schema_metadata.get(b"lance:ivf")  # Returns b"1" for global buffer index

# Read the global buffer containing IVF metadata
if ivf_ref:
    buffer_index = int(ivf_ref) - 1  # Global buffer indices are 1-based
    ivf_buffer = index_reader.global_buffer(buffer_index)

    # Parse the protobuf message (requires lance protobuf definitions)
    # ivf_metadata = parse_ivf_protobuf(ivf_buffer)

# For auxiliary file with PQ codebook
aux_reader = lance.LanceFileReader.read_file("path/to/auxiliary.idx")

# Get storage metadata
storage_metadata = aux_reader.metadata().schema.metadata.get(b"storage_metadata")
if storage_metadata:
    import json
    pq_metadata = json.loads(storage_metadata.decode())[0]  # First element of the list
    pq_params = json.loads(pq_metadata)

    # Access the codebook from global buffer
    codebook_position = pq_params.get("codebook_position", 1)
    if codebook_position > 0:
        codebook_buffer = aux_reader.global_buffer(codebook_position - 1)
        # Parse the tensor protobuf
        # codebook_tensor = parse_tensor_protobuf(codebook_buffer)
```

index.md 15.7 KB

# Indices in Lance

Lance treats indices as independent, redundant data structures layered on top of table row identifiers. This keeps the file format free of built-in search structures and lets index formats evolve independently from the table layout.

Lance supports three main categories of indices to accelerate data access: scalar
indices, vector indices, and system indices.

**Scalar indices** accelerate queries on scalar data types such as integers, timestamps,
and strings. This includes primary skipping structures such as [zone maps](scalar/zonemap.md)
as well as secondary structures such as [B-trees](scalar/btree.md), [bitmap indices](scalar/bitmap.md),
and [full-text search indices](scalar/fts.md). They typically accept predicates such as equality,
range, set-membership, or token matches and return matching row identifiers.

<figure markdown="span">
  ![](./scalar_index.drawio.svg)
</figure>

**[Vector indices](./vector/index.md)** are specialized for approximate nearest neighbor search on
high-dimensional embeddings. Examples include IVF-based layouts and HNSW graphs. Instead of scalar
predicates, vector indices receive a query vector and return row identifiers plus distance scores.

**System indices** are auxiliary structures that support internal table maintenance and row-identifier
resolution. They are not queried directly by end users. Examples include the [Fragment Reuse Index](system/frag_reuse.md),
which supports efficient remapping after compaction.

## Design

Lance indices are designed with the following design choices in mind:

1. **Indices are loaded on demand**: A dataset can be loaded and read without loading any indices.
   Indices are only loaded when a query can benefit from them.
   This design minimizes memory usage and speeds up dataset opening time.
2. **Indices can be loaded progressively**: indices are designed so that only the necessary parts
   are loaded into memory during query execution. For example, when querying a B-tree index,
   it loads a small page table to figure out which pages of the index to load for the given query,
   and then only loads those pages to perform the indexed search. This amortizes the cost of
   cold index queries, since each query only needs to load a small portion of the index.
3. **Indices can be coalesced to larger units than fragments.** Indices are much smaller than
   data files, so it is efficient to coalesce index segments to cover multiple fragments.
   This reduces the number of index files that need to be opened during query execution and
   then number of unique index data structures that need to be queried.
4. **Index files are immutable once written, similar to data files.** They can be modified only
   by creating new files. This means they can be safely cached in memory or on disk without
   worrying about consistency issues.

## Basic Concepts

An index in Lance is defined over a specific column (or multiple columns) of a dataset.
It is identified by its name.

An index is made up of multiple **index segments**, identified by their unique UUIDs.
Each segment is an independent, self-contained index covering a subset of the data.

Each index segment covers a disjoint subset of fragments in the dataset. The segments must cover
all rows in the fragments they cover, with one exception: if a fragment has delete markers at the time
of index creation, the index segment is allowed to not contain the deleted rows. The fragments an index
covers are those recorded in the `fragment_bitmap` field.

Index segments together **do not** need to cover all fragments. This means an index isn't required to
be fully up-to-date. When this happens, engines can split their queries into indexed and unindexed
subplans and merge the results.

<figure markdown="span">
  ![](./starter-example.drawio.svg)
  <figcaption>Abstract layout of a typical dataset, with three fragments and two indices.
  </figcaption>
</figure>

Consider the example dataset in the figure above:

- The dataset contains three fragments with ids 0, 1, 2. Fragment 1 has 10 deleted rows, indicated
  by the deletion file.
- There is an index called "id_idx", which has two segments: one covering fragments 0 and another covering
  fragment 1. Fragment 2 is not covered by the index. Queries using this index will need to query both
  segments and then scan fragment 2 directly. Additionally, when querying the segment covering fragment 1,
  the engine will need to filter out the 10 deleted rows.
- There is another index called "vec_idx", which has a single segment covering all three fragments.
  Because it covers all fragments, queries using this index do not need to scan any fragments directly.
  They do, however, need to filter out the 10 deleted rows from fragment 1.

## Index Storage

The content of each index is stored at the `_indices/{UUID}` directory under the [base path](../table/layout.md#base-path-system).
We call this location the **index directory**.
The actual content stored in the index directory depends on the index type. These can be
arbitrary files defined by the index implementation. However, often they are made up of
Lance files containing the index data structures. This allows reuse of the existing Lance
file format code for reading and writing index data.

## Creating and Updating Index Segments

Index segments are created and updated through a transactional process:

1. **Build the index data**: Read the relevant column data from the fragments to be indexed
   and construct the index data structures. Write these to files in a new `_indices/{UUID}`
   directory, where `{UUID}` is a newly generated unique identifier.

2. **Prepare the metadata**: Create an `IndexMetadata` message with:
   - `uuid`: The newly generated UUID
   - `name`: The index name (must match existing segments if adding to an existing index)
   - `fields`: The columns the index depends on: the column(s) it is keyed on, plus any it
     merely carries, as named in `covering_fields`. No id is repeated, and `fields[0]` is always
     a column the index is keyed on.
   - `covering_fields`: The subset of `fields` whose values the index carries, in the order it
     emits them, letting a query that only projects those columns be answered without a fragment
     take. A column is carried if and only if it is named here, including a column the index is
     also keyed on. Empty for an index that carries no extra columns. Declaring a column here
     does not by itself make it servable --
     see [Serving carried columns](#serving-carried-columns).
   - `fragment_bitmap`: The set of fragment IDs covered by this segment
   - `index_details`: Index-specific configuration and parameters
   - `version`: The format version of this index type
   - See the full protobuf definition in [table.proto](https://github.com/lance-format/lance/blob/main/protos/table.proto).

3. **Commit the transaction**: Write a new manifest that includes the new index segment
   in its `IndexSection`. This is done atomically using the same transaction mechanism
   as data writes.

When updating a column in place (without deleting the row), the engine must remove the
affected fragment IDs from the `fragment_bitmap` field of any index segment whose `fields`
include that column — whether the index is keyed on it or merely carries it. This marks
those fragments as needing re-indexing without invalidating the entire segment and prevents
invalid data from being read from the index.

## Index Compatibility

Before using an index segment, engines must verify they support it:

1. **Check the index type**: The `index_details` field contains a protobuf `Any` message
   whose type URL identifies the index type (e.g., B-tree, IVF, HNSW). If the engine
   does not recognize the type, it should skip this index segment.

2. **Check the version**: The `version` field in `IndexMetadata` indicates the format
   version of the index segment. If the engine does not support this version, it should
   skip this index segment. This allows index formats to evolve over time while
   maintaining backwards compatibility.

When an engine cannot use an index segment, it should fall back to scanning the
fragments that would have been covered by that segment.

### Serving carried columns

`IndexMetadata.covering_fields` records the columns an index segment *declares* it
carries. It does not establish that the segment's storage holds their values.

**The segment's storage schema is authoritative.** Before answering a query from a
carried column, an engine must confirm that column is present and bound to the declared
logical field in the storage it opened, and fall back to a take against the base table
when it cannot. A segment whose metadata identifies a column its storage does not hold is
a legal state, not corruption: a maintenance operation that cannot carry the payload
through a rebuild is permitted to withdraw it and leave the `covering_fields` listed in
the metadata.

!!! note "Capability varies by segment"

    Whether a segment's storage holds a declared column depends on the index type, on the
    writer that produced the segment, and on what later maintenance did to it, so one
    logical index may hold values for some of its segments and not others. An engine
    therefore verifies each selected segment rather than inferring capability from the
    index type, the writer version, or the `covering_fields` metadata alone, and serves
    from the base table every column it cannot verify.

## Loading an index

When loading an index:

1. Get the offset to the index section from the `index_section` field in the [manifest](../table/index.md#manifest).
2. Read the index section from the manifest file. This is a protobuf message of type `IndexSection`, which
   contains a list of `IndexMetadata` messages, each describing an index segment.
3. Read the index files from the `_indices/{UUID}` directory under the dataset directory,
   where `{UUID}` is the UUID of the index segment.

!!! tip "Optimizing manifest loading"

    When the manifest file is small, you can read and cache the index section eagerly. This avoids
    an extra file read when loading indices.

The `IndexMetadata` message contains important information about the index segment:

- `uuid`: the unique identifier of the index segment.
- `fields`: the columns the index depends on: the column(s) the index is keyed on, plus any it
  merely carries, as named in `covering_fields`. No id is repeated, and `fields[0]` is always a
  column the index is keyed on.
- `covering_fields`: the subset of `fields` whose values the index carries alongside its own data,
  in the order it emits them. A column is carried if and only if it is named here, including a
  column the index is also keyed on. Empty for an index that carries no extra columns. Every id in
  `covering_fields` names a top-level field. Covering a struct column carries the whole struct,
  its children included, as one column. This metadata is not authoritative for what the segment
  can serve -- see [Serving carried columns](#serving-carried-columns).
- `fragment_bitmap`: the set of fragment IDs covered by this index segment.
- `index_details`: a protobuf `Any` message that contains index-specific details, such as index type,
  parameters, and storage format. This allows different index types to store their own metadata.

<details>
  <summary>Full protobuf definitions</summary>

There are both part of the `table.proto` file in the Lance source code.

```protobuf
%%% proto.message.IndexSection %%%

%%% proto.message.IndexMetadata %%%
```

</details>

## Handling deleted and invalidated rows

Since index segments are immutable, they may contain references to rows that have been deleted
or updated. These should be filtered out during query execution.

<figure markdown="span">
  ![](./indices-fragment handling.drawio.svg)
  <figcaption>Representation of index segment covering fragments that have deleted rows,
  completely deleted fragments, and updated fragments.
  </figcaption>
</figure>

There are four situations to consider:

1. **A fragment has some deleted rows.** A few of the rows in the fragment have been marked
   as deleted, but some of the rows are still present. The row addresses from the deletion
   file should be used to filter out results from the index.
2. **A fragment has been completely deleted.** This can be detected by checking if a
   fragment ID present in the fragment bitmap is missing from the dataset.
   Any row addresses from this fragment should be filtered out.
3. **A fragment has had one of the index's columns updated in place.** This cannot be detected
   just by examining metadata. To prevent reading invalid data, the engine should filter out any
   row addresses that are not in the index's current `fragment_bitmap`.
   The column need not be one the index is keyed on: every column in `fields` counts, including
   the merely-carried ones named in `covering_fields`. A carried column can be updated while the
   keyed column is untouched, and a segment left covering that fragment would answer from an
   obsolete carried value.
4. **A fragment has an updated value in an [overlay file](../table/data_overlay_file.md).**
   This can be detected by checking if any of the fragments in the index's `fragment_bitmap`
   have overlay files. For each overlay whose `committed_version` is greater than the index
   segment's `dataset_version`, the overlay carries updated values not reflected in the index,
   so its covered rows must be excluded from index results. Excluded rows are re-evaluated
   against their current (overlaid) values on the flat path — dropping them without
   re-evaluation would silently lose rows that match under the new value. Exclusion is
   field-aware: only overlays covering a column in the index's `fields` matter — keyed or
   merely carried. Restricting this to the keyed column would leave a fragment covered after
   an overlay updated a carried one, and the index would then serve a stale carried value.
   You may exclude just the affected rows or the whole fragment; the latter is simpler and
   safer but re-evaluates more rows than necessary.
   See [Data Overlay Files](../table/data_overlay_file.md#index-integration)
   for the exclusion set, re-evaluation, and correctness invariant.

## Compaction and remapping

When fragments are compacted, the row addresses of the rows in the fragments change.
This means that any index segments referencing those fragments will no longer point
to existing row addresses. There are three ways to handle this:

<figure markdown="span">
![](./indices-compaction.drawio.svg)
</figure>

1. Do nothing and let the index segment not cover those fragments anymore. This approach is
   simple and valid, but it means compaction can immediately make an index out-of-date. This
   is the worst options for query performance.

2. Immediately rewrite the index segments with the row addresses remapped. This approach
   ensures the index is kept up-to-date, but it incurs significant write amplification
   during compaction.

3. Create a [Fragment Reuse Index](system/frag_reuse.md) that maps old row addresses to new
   row addresses. This allows readers to remap the row addresses in memory upon reading
   the index segments. This approach adds some IO and computation overhead during query
   execution, but avoids write amplification during compaction.

## Stable Row ID for Index

Indices can optionally use stable row IDs instead of row addresses. A stable row ID is a
logical identifier that remains constant even when rows are moved during compaction.

**Benefits:**

- No remapping needed after compaction
- Updates only invalidate the index if data in one of its `fields` changes — the keyed
  column(s) or any column named in `covering_fields`

**Tradeoffs:**

- Requires an additional lookup to translate stable row IDs to physical row addresses
  at query time

This feature is currently experimental. Performance evaluation is ongoing to determine
when the tradeoff is worthwhile.

indices-compaction.drawio.svg 50.2 KB · in bundle
indices-fragment handling.drawio.svg 21.3 KB · in bundle
scalar_index.drawio.svg 8 KB · in bundle
starter-example.drawio.svg 14.3 KB · in bundle

table

branch_tag.md 4.6 KB

# Branch and Tag Specification

## Overview

Lance supports branching and tagging for managing multiple independent version histories and creating named references to specific versions.
Branches enable parallel development workflows, while tags provide stable named references for important versions.

## Branching

### Branch Name

Branch names must follow these validation rules:

1. Cannot be empty
2. Cannot start or end with `/`
3. Cannot contain consecutive `//`
4. Cannot contain `..` or `\`
5. Segments must contain only alphanumeric characters, `.`, `-`, `_`
6. Cannot end with `.lock`
7. Cannot be named `main` (reserved for main branch)

### Branch Metadata Path

Branch metadata is stored at `_refs/branches/{branch-name}.json` in the dataset root.
Since branch names support hierarchical naming with `/` characters, the `/` is URL-encoded as `%2F` in the filename to distinguish it from directory separators (e.g., `bugfix/issue-123` becomes `bugfix%2Fissue-123.json`):

```
{dataset_root}/
    _refs/
        branches/
            feature-a.json
            bugfix%2Fissue-123.json  # Note: '/' encoded as '%2F'
```

### Branch Metadata File Format

Each branch metadata file is a JSON file with the following fields:

| JSON Key         | Type   | Optional | Description                                                                    |
|------------------|--------|----------|--------------------------------------------------------------------------------|
| `parentBranch`   | string | Yes      | Name of the branch this was created from. `null` indicates branched from main. |
| `parentVersion`  | number |          | Version number of the parent branch at the time this branch was created.       |
| `createAt`       | number |          | Unix timestamp (seconds since epoch) when the branch was created.              |
| `manifestSize`   | number |          | Size of the initial manifest file in bytes.                                    |
| `metadata`       | object | Yes      | String key/value metadata map. If absent, it is treated as an empty object.    |

### Branch Dataset Layout

Each branch dataset is technically a [shallow clone](layout.md#shallow-clone) of the source dataset.
Branch datasets are organized using the `tree/` directory at the dataset root:

```
{dataset_root}/
    tree/
        {branch_name}/
            _versions/
                *.manifest
            _transactions/
                *.txn
            _deletions/
                *.arrow
                *.bin
            _indices/
                {UUID}/
                    index.idx
```

Named branches store their version-specific files under `tree/{branch_name}/`, resembling the GitHub branch path convention.
It uses the branch name as is to form the path,
which means `/` would create a logical subdirectory (e.g., `bugfix/issue-123`, `feature/user-auth`):

```
{dataset_root}/
    tree/
        feature-a/
            _versions/
                1.manifest
                2.manifest
        bugfix/
            issue-123/
                _versions/
                    1.manifest
```

## Tagging

### Tag Name

Tag names must follow these validation rules:

1. Cannot be empty
2. Must contain only alphanumeric characters, `.`, `-`, `_`
3. Cannot start or end with `.`
4. Cannot end with `.lock`
5. Cannot contain consecutive `..`

Note that tag names do not support `/` characters, unlike branch names.

### Tag Storage

Tags are stored as JSON files under `_refs/tags/` at the dataset root:

```
{dataset_root}/
    _refs/
        tags/
            v1.0.0.json
            v1.1.0.json
            production.json
```

Tags are always stored at the root dataset level, regardless of which branch they reference.

### Tag File Format

Each tag file is a JSON file with the following fields:

| JSON Key        | Type   | Optional | Description                                                              |
|-----------------|--------|----------|--------------------------------------------------------------------------|
| `branch`        | string | Yes      | Branch name being tagged. `null` or absent indicates main branch.        |
| `version`       | number |          | Version number being tagged within that branch.                          |
| `createdAt`     | string | Yes      | RFC 3339 timestamp for when the tag was first created.                  |
| `updatedAt`     | string | Yes      | RFC 3339 timestamp for the latest tag reference update.                 |
| `manifestSize`  | number |          | Size of the manifest file in bytes. Used for efficient manifest loading. |
| `metadata`      | object | Yes      | String key/value metadata map. If absent, it is treated as an empty object. |

data_overlay_file.md 17.5 KB

# Data Overlay Files

!!! warning "Experimental"

    This feature is currently experimental and not yet supported in any library.

<!-- TODO: When overlay file support is implemented, update this note to state
     the released version that first supports the feature (and drop the
     "experimental" framing once it is stable). -->

!!! note "Overlay files require feature flag 64 (data overlay files)"

    A reader or writer that does not understand overlay files must refuse a
    dataset that uses them. Silently ignoring an overlay would return stale base
    values, which is a correctness bug rather than a degraded experience.

Overlay files supply new values for a subset of `(row offset, field)` cells
within a fragment **without rewriting the fragment's base data files**. They make
updates cheap when only a small fraction of rows and/or columns change: instead
of rewriting whole columns or moving rows to a new fragment, a writer appends a
small file carrying just the changed cells.

This is Lance's third mechanism for changing data in place, alongside
[deletion files](index.md#deletion-files) (which remove rows) and
[data evolution](index.md#data-evolution) (which adds or rewrites whole columns).
An overlay changes individual cells.

## Concepts

### Coverage and resolution

Each overlay declares which cells it provides through a **coverage** bitmap (or,
for sparse overlays, one bitmap per field). The bitmaps index **physical row
offsets**. They include deleted rows and are stable even as deletion vectors change.

To resolve a cell `(offset, field)` on read, walk the fragment's overlays from
**newest to oldest**. The first overlay that covers `(offset, field)` wins; its
value is used. If no overlay covers the cell, the value falls through to the base
data file (or is `NULL` if no base data file holds that field).

Precedence among overlays is determined by:

1. `committed_version` — higher wins (see [Versioning](#versioning-and-ordering)).
2. Position in `DataFragment.overlays` as a tiebreaker — a later entry is newer.

A covered offset whose value is `NULL` overrides the cell **to** `NULL`. This is
distinct from an offset that is simply absent from the bitmap, which falls
through to the base. Coverage, not value-nullness, decides whether an overlay
applies.

### Interaction with deletions

Deletions take precedence over overlays. If a row offset is marked deleted in the
fragment's deletion file, any overlay value for that offset is dead and is
ignored, regardless of commit order.

### Physical layout

An overlay's data file stores **one value column per field**, in the order of
`data_file.fields`. It does **not** store a row-offset key column. The position of
a covered offset's value within its column is the **rank** of that offset in the
field's coverage bitmap — the number of set bits below it. Resolving a cell is a
rank lookup plus one value fetch, with no separate offset column to store or
search.

Because different fields may cover different offset sets, the value columns of a
single sparse overlay may have **different lengths**. The Lance file format
permits columns of differing item counts within one file, so a sparse overlay is
representable as a single file. (See [Writer support](#writer-support) for the
current implementation status.)

### Dense vs. sparse overlays

A single overlay is one of two shapes:

- **Dense (rectangular).** One `shared_offset_bitmap` applies to every field. Every
  covered offset has a value for every field. This is the common case for a plain
  `UPDATE`, where one `SET` list is applied to one set of rows.
- **Sparse.** A `FieldCoverage` carries one bitmap per field, used when different
  fields cover different offset sets — for example a `MERGE` with multiple
  `WHEN MATCHED` branches, where different rows update different columns. A dense
  overlay would have to widen to the bounding rectangle and fill the untouched
  cells with their current values (post-images), which for wide columns such as
  embeddings means re-storing data that did not change. A sparse overlay stores
  exactly the changed cells.

## Protobuf

<details>
<summary>DataOverlayFile protobuf message</summary>

```protobuf
%%% proto.message.DataOverlayFile %%%
```

</details>

<details>
<summary>FieldCoverage protobuf message</summary>

```protobuf
%%% proto.message.FieldCoverage %%%
```

</details>

## Versioning and ordering

Overlays reuse the dataset version as their ordering clock rather than
introducing a separate generation counter.

`committed_version` is the dataset version at which an overlay **became
effective** — the version of the commit that introduced it, **not** the version
it was read from. It is stamped at commit time and re-stamped if the commit is
retried, in the same way as the created-at / last-updated-at version sequences.

This single value drives every ordering decision:

- **Overlay vs. overlay** (read precedence): higher `committed_version` wins.
- **Overlay vs. index** (query correctness): an index records the
  `dataset_version` it was built from. An index whose `dataset_version >=
  committed_version` already incorporates the overlay. An overlay whose
  `committed_version > index.dataset_version` is newer than the index and its
  cells must be excluded from index results and re-evaluated.
- **Scheduler signal**: the gap between an overlay's `committed_version` and an
  index's `dataset_version`, or between an overlay and the base, is a staleness
  measure the compaction scheduler can use.

!!! note "Why effective version, not read version"

    Suppose an overlay reads version 5 and commits at version 6, while an index
    is built reading version 5 (before the overlay) and commits at version 7 with
    `dataset_version = 5`. If the overlay stored its *read* version (5), the test
    `5 > 5` is false, the row would not be excluded, and the index — which never
    saw the overlay — would return a stale result. Storing the *effective*
    version (6) makes `6 > 5` true, the cell is excluded and re-evaluated, and the
    result is correct.

## Index integration

Building an index over a fragment that has overlays does **not** require dropping
the fragment from the index's coverage. The fragment stays indexed, and the query
path reconciles overlays at query time using an **exclusion set**.

The exclusion set for an index on field `F` is the union of the coverage bitmaps,
restricted to field `F`, of every overlay whose `committed_version >
index.dataset_version`. The exclusion is **field-aware**: an overlay that touches
only unrelated columns does not exclude anything from the index on `F`.

`F` here ranges over every field in the index's `fields`, not only the ones it is
keyed on. An index that carries columns it is not keyed on (see
[`covering_fields`](../index/index.md#serving-carried-columns)) depends on those
columns too: an overlay updating a merely-carried column leaves the keyed value
correct while making the carried value stale, so it must exclude those rows just
the same.

The query then proceeds as:

1. Run the index search as usual, producing candidate rows.
2. Remove any candidate in the exclusion set. (Its indexed value may be stale.)
3. **Re-evaluate** the excluded rows against their current values — the same flat
   path already used for the unindexed tail of fragments. For a scalar predicate
   this re-applies the filter; for a vector query it re-scores the row's current
   vector. Rows that still match are added back to the result.

Step 3 is what makes exclusion correct rather than merely safe: removing a row
from index candidates without re-evaluating it would silently drop a row that
should match under its new value.

Exclusion is always *sufficient* because a write changes a cell only by adding an
overlay, and that overlay's `committed_version` — the version of the commit that
adds it — necessarily exceeds the `dataset_version` of any pre-existing index. So
every cell a write changes is guaranteed to fall in that index's exclusion set.
Compaction may remove an overlay only if no index still relies on it for exclusion
(see [Compaction](#compaction)).

## Compaction

Overlays accumulate read cost — every overlay is a bitmap to test, a possible
file to open, and additional work to interleave values. Compaction bounds that cost in two modes:

- **Overlay → overlay.** Merge several overlays into fewer, computing the
  post-image per `(offset, field)` by walking the merged overlays newest-first.
  The merged overlay takes the **maximum** `committed_version` of its inputs, so
  the exclusion semantics are preserved. The merged overlays must be **contiguous
  in `committed_version`** — with overlays at v10, v30, and v50 you cannot merge
  just v10 and v50, because stamping the result v50 would incorrectly promote
  v10's values above the intervening v30 for any cell v30 also covers. Indexes can
  still be re-used, but they may now need to exclude more rows. This is cheap to
  write and does not touch the base.
- **Overlay → base.** Fold overlays into a fresh base data file, computing the
  post-image for every covered cell, then clear the overlays. The base is
  complete, so every post-image is well defined. Overlay offsets are physical, so
  they cannot survive a rewrite that reorders rows; folding therefore materializes
  values rather than carrying overlays forward.

!!! warning "Folding an indexed field must update its index"

    An overlay→base fold removes the overlay, which removes the exclusion signal
    that kept an index correct. Folding an overlay that covers an indexed field
    `F` is therefore equivalent to a column rewrite of `F` and must, in the same
    commit, either rebuild the index to a `dataset_version` at least the folded
    overlay's `committed_version`, or remove the fragment from the index's
    coverage so the rows fall to the flat path. Otherwise the index would serve
    stale values with no overlay to exclude them. This is the same rule that
    already governs rewriting a column that an index is built on.

When a fragment with overlays is compacted by a row-rewriting operation
(`RewriteRows`, which produces new fragments with new row addresses), the
overlays are folded into the new base as part of the rewrite, and existing
[fragment-reuse remapping](row_id_lineage.md) handles the row-address changes as
it does today.

## Row lineage

An overlay write updates the `last_updated_at_version` of every covered row, so
change-data-feed and time-travel queries observe the update. Because overlays are
addressed by physical offset, they do **not** require stable row IDs to be
enabled; lineage updates apply only when those features are on.

## Worked example

The following example illustrates how overlays function across their lifecycle, to make the rules above concrete.

A table `users` with stable row IDs enabled and these fields:

| field id | name      | type                    |
|----------|-----------|-------------------------|
| 1        | id        | `int32` (primary key)   |
| 2        | name      | `utf8`                  |
| 3        | age       | `int32`                 |
| 4        | embedding | `fixed_size_list<f32,4>`|

Created at version 1 as a single fragment `0` with one base data file
`data/file0.lance` holding all four columns. `physical_rows = 4`:

| offset | id | name  | age | embedding        |
|--------|----|-------|-----|------------------|
| 0      | 1  | Alice | 30  | …                |
| 1      | 2  | Bob   | 25  | …                |
| 2      | 3  | Carol | 40  | …                |
| 3      | 4  | Dave  | 22  | …                |

A BTree scalar index on `age` is built at version 1, covering fragment `0`
(`dataset_version = 1`).

### Step 1 — write an overlay

```sql
UPDATE users SET age = age + 1 WHERE id IN (2, 4);   -- Bob (offset 1), Dave (offset 3)
```

This touches one field (`age`) for two rows, so the writer emits a dense overlay —
one shared bitmap covering both offsets — and commits it as version 2. Fragment
`0` gains:

```text
DataOverlayFile {
  data_file: { path: "data/overlay-<uuid>.lance", fields: [3], column_indices: [0] }
  coverage:  shared_offset_bitmap = {1, 3}
  committed_version: 2
}
```

The overlay file stores a single `age` column with two values, `[26, 23]`, at
ranks `{1,3}.rank(1) = 0` and `{1,3}.rank(3) = 1`. `last_updated_at_version` is
set to 2 for offsets 1 and 3.

### Step 2 — read

`SELECT id, age FROM users` reads base ages `[30, 25, 40, 22]`. For `age`
(field 3), the overlay covers offsets 1 and 3, so `age[1]` is replaced with the
overlay value at rank `{1,3}.rank(1) = 0` → `26`, and `age[3]` with the value at
rank `{1,3}.rank(3) = 1` → `23`. Result ages: `[30, 26, 40, 23]`.

### Step 3 — index query

```sql
SELECT * FROM users WHERE age = 26;
```

The `age` index was built at `dataset_version = 1`; the overlay's
`committed_version` is 2. Since `2 > 1`, the overlay's coverage for `age`, `{1, 3}`,
is the exclusion set for this query.

- The index (built at v1) holds Bob's *old* `age = 25`, so a lookup for `26`
  returns nothing from the index.
- The whole exclusion set is re-evaluated on the flat path, not just the rows the
  index returned. Offset 1's current `age` (26, via the overlay) matches, so Bob
  is returned; offset 3's current `age` (23) does not match and is dropped.

The mirror case `WHERE age = 25` shows exclusion preventing a stale hit: the index
returns offset 1 (stale `25`), but offset 1 is excluded, re-evaluated to `26`, and
correctly dropped.

### Step 4 — a second, non-rectangular write

```sql
MERGE INTO users USING staged ON users.id = staged.id
WHEN MATCHED AND staged.kind = 'rename'  THEN UPDATE SET name = staged.name        -- Carol(2), Dave(3)
WHEN MATCHED AND staged.kind = 'embed'   THEN UPDATE SET embedding = staged.embedding -- Bob(1)
```

`name` is updated for offsets `{2, 3}` and `embedding` for offset `{1}` — different
fields over different rows. This is a sparse overlay, committed as version 3:

```text
DataOverlayFile {
  data_file: { path: "data/overlay-<uuid2>.lance", fields: [2, 4], column_indices: [0, 1] }
  coverage:  field_coverage { offset_bitmaps: [ {2,3},   {1} ] }
  //                                     name (field 2)  ^    ^ embedding (field 4)
  committed_version: 3
}
```

The file's `name` column has **two** values (`["Caroline", "David"]`, at
ranks 0 and 1 of `{2,3}`) and its `embedding` column has **one** value (at rank 0
of `{1}`) — columns of different lengths in one file.

### Step 5 — read after the second write

`SELECT name, age, embedding FROM users` resolves each field independently,
newest overlay first:

- `name`: the v3 overlay covers `{2,3}` → `["Alice", "Bob", "Caroline", "David"]`.
- `age`: the v3 overlay does not cover `age`; the v2 overlay still applies at
  offsets 1 and 3 → `[30, 26, 40, 23]`.
- `embedding`: the v3 overlay covers `{1}` → Bob's vector is the new one, others
  from base.

Overlays from different versions coexist and apply per field.

### Step 6 — compaction (overlay → base)

The scheduler folds both overlays into fragment `0` at version 4, computing
post-images for `age`, `name`, and `embedding`, and writing a new base data file
`data/file1.lance` with those columns. In the old file, fields 2, 3, and 4 are
marked with a tombstone (`-2`); field 1 (`id`) remains. The fragment's `overlays` list is
cleared. Row addresses are preserved (a column rewrite, not a row rewrite), so
stable row IDs and the deletion vector are untouched.

Because the fold removed the overlay that was excluding offsets 1 and 3 from the
`age` index, the commit must drop fragment `0` from its coverage so `age` queries
fall to the flat path.

## Guidance

!!! note "This section is a stub."

    The following are implementation considerations, not part of the on-disk
    specification.

### When to overlay vs. rewrite a column vs. move rows

<!-- TODO: Replace the rough heuristic below with concrete thresholds once we
     have benchmarked the crossover points between overlays, column rewrites,
     and row moves. -->

*(To be expanded.)* The choice between appending an overlay, rewriting a full
column (data evolution), and moving updated rows to a new fragment depends on the
fraction of rows changed, the fraction of columns changed, column width, the
presence of indexes on the changed columns, and the accumulated overlay read
cost. Roughly: few rows changed favors overlays; most rows in a few columns
favors a column rewrite; most columns changed favors moving rows to a new
fragment.

### Writer support

<!-- TODO: Fill in as writer implementation progresses, including the status of
     single-file sparse overlays (independent-length columns). -->

*(To be expanded.)* Dense (rectangular) overlays write with the existing
equal-length file writer today. Sparse overlays stored as a **single** file
require the writer to emit columns of independent lengths, which the current v2
writer does not yet do (it advances all columns from one global row counter).
Until that support lands, a writer can express a sparse update as multiple dense
overlays in one transaction.

### Scheduling compaction

<!-- TODO: Fill in with a concrete cost/benefit policy once compaction is
     implemented and benchmarked. -->

*(To be expanded.)* The overlay→overlay and overlay→base modes have very
different costs; a cost/benefit scheduler decides when each is worthwhile, using
the version gap as a staleness signal.

## Related specifications

- [Table format overview](index.md)
- [Transactions: DataOverlay operation](transaction.md#dataoverlay) — write path
  and conflict semantics
- [Row ID & Lineage](row_id_lineage.md)
- [Index Formats: handling overlay rows](../index/index.md#handling-deleted-and-invalidated-rows)
- [Format Versioning](versioning.md)

index.md 9.5 KB

# Lance Table Format

## Overview

The Lance table format organizes datasets as versioned collections of fragments, data files, deletion files, and indices. Each version is described by an immutable manifest that references the physical data for that snapshot.

The format is designed for machine learning and highly selective workloads where column additions, index maintenance, and partial rewrites must be cheap. It supports ACID transactions, schema evolution, time travel, and efficient incremental updates through Multi-Version Concurrency Control (MVCC).

## Design Goals

### Two-Dimensional Storage

Rows are partitioned into fragments, and each fragment can contain multiple data files that each provide one or more columns. This lets writers add or backfill columns by attaching new data files to existing fragments instead of rewriting the full table.

### First-Class Indices

Indices are part of the table format lifecycle. The table metadata describes index discovery and transactional coordination, while the detailed search structures remain separate index formats. This gives engines a uniform way to create, drop, update, and query indices without coupling the table format to any single indexing algorithm.

### External Manifest Store

Lance can commit directly to object storage, but deployments may also coordinate commits through an external manifest store. In that model, the external system helps serialize commits and apply governance checks, while the canonical table state is still persisted in the Lance table format.

## Manifest

![Overview](../../images/table_overview.png)

A manifest describes a single version of the dataset.
It contains the complete schema definition including nested fields, the list of data fragments comprising this version,
a monotonically increasing version number, and an optional reference to the index section that describes a list of index metadata.

<details>
<summary>Manifest protobuf message</summary>

```protobuf
%%% proto.message.Manifest %%%
```

</details>

## Schema & Fields

The schema of the table is written as a series of fields, plus a schema metadata map.
The data types generally have a 1-1 correspondence with the Apache Arrow data types.
Each field, including nested fields, have a unique integer id. At initial table creation time, fields are assigned ids in depth-first order.
Afterwards, field IDs are assigned incrementally for newly added fields.

Column encoding configurations are specified through field metadata using the `lance-encoding:` prefix.
See [File Format Encoding Specification](../file/encoding.md) for details on available encodings, compression schemes, and configuration options.

For complete schema specification details including supported data types, field ID assignment, and metadata handling,
see the [Schema Format Specification](schema.md).

<details>
<summary>Field protobuf message</summary>

```protobuf
%%% proto.message.lance.file.Field %%%
```

</details>

### Unenforced Primary Key

Lance supports defining an unenforced primary key through field metadata.
This is useful for deduplication during merge-insert operations and other use cases that benefit from logical row identity.
The primary key is "unenforced" meaning Lance does not always validate uniqueness constraints.
Users can use specific workloads like merge-insert to enforce it if necessary.
The primary key is fixed after initial setting and must not be updated or removed.

A primary key field must satisfy:

- The field, and all its ancestors, must not be nullable.
- The field must be a leaf field (primitive data type without children).
- The field must not be within a list or map type.

When using an Arrow schema to create a Lance table, add the following metadata to the Arrow field to mark it as part of the primary key:

- `lance-schema:unenforced-primary-key`: Set to `true`, `1`, or `yes` (case-insensitive) to indicate the field is part of the primary key.
- `lance-schema:unenforced-primary-key:position` (optional): A 1-based integer specifying the position within a composite primary key.

For composite primary keys with multiple columns, the position determines the primary key field ordering:

- When positions are specified, fields are ordered by their position values (1, 2, 3, ...).
- When positions are not specified, fields are ordered by their schema field id.
- Fields with explicit positions are ordered before fields without.

## Fragments

![Fragment Structure](../../images/fragment_structure.png)

A fragment represents a horizontal partition of the dataset containing a subset of rows.
Each fragment has a unique `uint32` identifier assigned incrementally based on the dataset's maximum fragment ID.
Each fragment consists of one or more data files storing columns, plus an optional deletion file.
If present, the deletion file stores the positions (0-based) of the rows that have been deleted from the fragment.
The fragment tracks the total row count including deleted rows in its physical rows field.
Column subsets can be read without accessing all data files, and each data file is independently compressed and encoded.

<details>
<summary>DataFragment protobuf message</summary>

```protobuf
%%% proto.message.DataFragment %%%
```

</details>

### Data Evolution

This fragment design enables a new concept called data evolution, which means efficient schema evolution (add column, update column, drop column) with backfill.
For example, when adding a new column, new column data are added by appending new data files to each fragment, with values computed for all existing rows in the fragment.
There is no need to rewrite the entire table to just add data for a single column.
This enables efficient feature engineering and embedding updates for ML/AI workloads.

Each data file should contain a distinct set of field ids.
It is not required that all field ids in the dataset schema are found in one of the data files.
If there is no corresponding data file, that column should be read as entirely `NULL`.

Field ids might be replaced with `-2`, a tombstone value.
In this case that column should be ignored. This used, for example, when rewriting a column:
The old data file replaces the field id with `-2` to ignore the old data, and a new data file is appended to the fragment.

## Data Files

Data files store column data for a fragment using the Lance file format.
Each data file stores a subset of the columns in the fragment.
Field IDs are assigned either sequentially based on schema position (for Lance file format v1)
or independently of column indices due to variable encoding widths (for Lance file format v2).

<details>
<summary>DataFile protobuf message</summary>

```protobuf
%%% proto.message.DataFile %%%
```

</details>

!!! note "Field-to-column mapping differs between data storage versions"

    In **2.0**, all fields (including non-leaf fields like struct and list containers) are assigned
    sequential column indices in `column_indices`.

    In **2.1+**, non-leaf fields (unpacked structs, list containers) are assigned `-1` in
    `column_indices` because their validity information is folded into repetition/definition
    levels. Only leaf fields and packed structs have column indices.

    See the [5.0.0 migration guide](../../guide/migration.md#500) for a detailed example.

## Deletion Files

Deletion files (a.k.a. deletion vectors) track deleted rows without rewriting data files.
Each fragment can have at most one deletion file per version.

Deletion files support two storage formats.
The Arrow IPC format (`.arrow` extension) stores a flat Int32Array of deleted row offsets and is efficient for sparse deletions.
The Roaring Bitmap format (`.bin` extension) stores a compressed roaring bitmap and is efficient for dense deletions.
Readers must filter rows whose offsets appear in the deletion file for the fragment.

Deletions can be materialized by rewriting data files with deleted rows removed.
However, this invalidates row addresses and requires rebuilding indices, which can be expensive.

<details>
<summary>DeletionFile protobuf message</summary>

```protobuf
%%% proto.message.DeletionFile %%%
```

</details>

## Data Overlay Files

!!! warning "Experimental"

    This feature is currently experimental and not yet supported in any library.

<!-- TODO: When overlay file support is implemented, update this note to state
     the released version that first supports the feature. -->

!!! note "Overlay files require feature flag 64 (data overlay files)"

Overlay files supply new values for a subset of cells within
a fragment without rewriting the base data files. They make updates cheap when only
a small percentage of rows and/or columns change: a writer appends a small file
carrying just the changed cells instead of rewriting whole columns or moving rows
to a new fragment.

For the full specification — coverage and resolution rules, dense vs. sparse layout,
versioning, index integration, compaction, and a worked example — see the
[Data Overlay Files Specification](data_overlay_file.md).



## Related Specifications

### Storage Layout

File organization, base path system, and multi-location storage.

See [Storage Layout Specification](layout.md)

### Transactions

MVCC, commit protocol, transaction types, and conflict resolution.

See [Transaction Specification](transaction.md)

### Row Lineage

Row address, Stable row ID, row version tracking, and change data feed.

See [Row ID & Lineage Specification](row_id_lineage.md)

### Indices

Vector indices, scalar indices, full-text search, and index management.

See [Index Formats](../index/index.md)

### Versioning

Feature flags and format version compatibility.

See [Format Versioning Specification](versioning.md)

layout.md 9.3 KB

# Storage Layout Specification

## Overview

This specification defines how Lance datasets are organized on object storage.
The layout design emphasizes portability, allowing datasets to be relocated or referenced across multiple storage systems with minimal metadata changes.

## Dataset Root

The dataset root is the location where the dataset was initially created.
Every Lance dataset has exactly one dataset root, which serves as the primary storage location for the dataset's files.
The dataset root contains the standard subdirectory structure (`data/`, `_versions/`, `_deletions/`, `_indices/`, `_refs/`, `tree/`) that organizes the dataset's files.

## Basic Layout

A Lance dataset in its basic form stores all files within the dataset root directory structure:

```
{dataset_root}/
    data/
        *.lance           -- Data files containing column data
    _versions/
        *.manifest                -- Manifest files (one per version)
        latest_version_hint.json  -- Optional hint of the latest version (see below)
    _transactions/
        *.txn             -- Transaction files for commit coordination
    _deletions/
        *.arrow           -- Deletion vector files (arrow format)
        *.bin             -- Deletion vector files (bitmap format)
    _indices/
        {UUID}/
            ...           -- Index content (different for each index type)
    _refs/
        tags/
            *.json        -- Tag metadata
        branches/
            *.json        -- Branch metadata
    tree/
        {branch_name}/
            ...           -- Branch dataset

```

## Base Path System

### BasePath Message

The manifest's `base_paths` field contains an array of `BasePath` entries that define alternative storage locations for dataset files.
Each base path entry has a unique numeric identifier that file metadata can reference to indicate where files are located.
The `path` field specifies an absolute path interpretable by the object store.
The `is_dataset_root` field determines how the path is interpreted: when true, the path points to a dataset root with standard subdirectories (`data/`, `_deletions/`, `_indices/`); when false, the path points directly to a file directory without subdirectories.
An optional `name` field provides a human-readable alias, which is particularly useful for referencing tags in shallow clones.

<details>
<summary>BasePath protobuf message</summary>

```protobuf
message BasePath {
  uint32 id = 1;
  optional string name = 2;
  bool is_dataset_root = 3;
  string path = 4;
}
```

</details>

### File Metadata Base References

Three types of files can specify alternative base paths: data files, deletion files, and index metadata.
Each of these file types includes an optional `base_id` field in their metadata that references a base path entry by its numeric identifier.
When a file's `base_id` is absent, the file is located relative to the dataset root.
When a file's `base_id` is present, readers must look up the corresponding base path entry in the manifest's `base_paths` array to determine where the file is stored.

At read time, path resolution follows a two-step process.
First, the reader determines the base path: if `base_id` is absent, the base path is the dataset root; otherwise, the reader looks up the base path entry using the `base_id` to obtain the path and its `is_dataset_root` flag.
Second, the reader constructs the full file path based on whether the base path represents a dataset root.
For dataset roots (when `is_dataset_root` is true), the full path includes standard subdirectories: data files are located under `data/`, deletion files under `_deletions/`, and indices under `_indices/`.
For non-root base paths (when `is_dataset_root` is false), the base path points directly to the file directory, and the file path is appended directly without subdirectory prefixes.

### Example Complex Layout Scenarios

#### Hot/Cold Tiering

```
Manifest base_paths:
[
  { id: 0, is_dataset_root: true, path: "s3://hot-bucket/dataset" },
  { id: 1, is_dataset_root: true, path: "s3://cold-bucket/dataset-archive" }
]

Fragment 0 (recent data):
  DataFile { path: "fragment-0.lance", base_id: 0 }
  → resolves to: s3://hot-bucket/dataset/data/fragment-0.lance

Fragment 100 (historical data):
  DataFile { path: "fragment-100.lance", base_id: 1 }
  → resolves to: s3://cold-bucket/dataset-archive/data/fragment-100.lance
```

This allows seamless querying across storage tiers without data movement.

#### Multi-Region Distribution

```
Manifest base_paths:
[
  { id: 0, is_dataset_root: true, path: "s3://us-east-bucket/dataset" },
  { id: 1, is_dataset_root: true, path: "s3://eu-west-bucket/dataset" },
  { id: 2, is_dataset_root: true, path: "s3://ap-south-bucket/dataset" }
]

Fragments distributed by data locality:
  Fragment 0 (US users): base_id: 0
  Fragment 1 (EU users): base_id: 1
  Fragment 2 (Asia users): base_id: 2
```

Compute jobs can read data from the nearest region without data transfer.

#### Shallow Clone

Shallow clones create a new dataset that references data files from a source dataset without copying:

**Example: Shallow Clone**

```
Source dataset: s3://production/main-dataset
Clone dataset:  s3://experiments/test-variant

Clone manifest base_paths:
[
  { id: 0, is_dataset_root: true, path: "s3://experiments/test-variant" },
  { id: 1, is_dataset_root: true, path: "s3://production/main-dataset",
    name: "v1.0" }
]

Original fragments (inherited):
  DataFile { path: "fragment-0.lance", base_id: 1 }
  → resolves to: s3://production/main-dataset/data/fragment-0.lance

New fragments (clone-specific):
  DataFile { path: "fragment-new.lance", base_id: 0 }
  → resolves to: s3://experiments/test-variant/data/fragment-new.lance
```

The clone can append new data, modify schemas, or delete rows without affecting the source dataset.
Only the manifest and new data files are stored in the clone location.

**Workflow:**

1. [Clone transaction](transaction.md#clone) creates new manifest in target location
2. Manifest includes base path pointing to source dataset
3. Original fragments reference source via `base_id: 1`
4. Subsequent writes reference clone location via `base_id: 0`
5. Source dataset remains immutable and can be garbage collected independently

## Dataset Portability

The base path system combined with relative file references provides strong portability guarantees for Lance datasets.
All file paths within Lance files are stored relative to their containing directory, enabling datasets to be relocated without file modifications.

To port a dataset to a new location, simply copy all contents from the dataset root directory.
The copied dataset will function immediately at the new location without any manifest updates, as all file references within the dataset root resolve through relative paths.

When a dataset uses multiple base paths (such as in shallow clones or multi-bucket configurations), users have flexibility in how to port the dataset.
The simplest approach is to copy only the dataset root, which preserves references to the original base path locations.
Alternatively, users can copy additional base paths to the new location and update the manifest's `base_paths` array to reflect the new base paths.
Since only the `base_paths` field in the manifest requires modification, this remains a lightweight metadata operation that does not require rewriting additional metadata or data files.

## File Naming Conventions

### Data Files

Pattern: `data/{uuid-based-filename}.lance`

Data files use UUID-based filenames optimized for S3 throughput.
The filename is generated from a UUID (16 bytes) by converting the first 3 bytes to a 24-character binary string and the remaining 13 bytes to a 26-character hex string, resulting in a 50-character filename.
The binary prefix (rather than hex) provides maximum entropy per character, allowing S3's internal partitioning to quickly recognize access patterns and scale appropriately, minimizing throttling.

Example: `data/101100101101010011010110a1b2c3d4e5f6g7h8i9j0.lance`

### Deletion Files

Pattern: `_deletions/{fragment_id}-{read_version}-{id}.{extension}`

Deletion files use two extensions: `.arrow` for Arrow IPC format (sparse deletions) and `.bin` for Roaring bitmap format (dense deletions).

Example: `_deletions/42-10-a1b2c3d4.arrow`

### Transaction Files

Pattern: `_transactions/{read_version}-{uuid}.txn`

Where `read_version` is the table version the transaction was built from.

Example: `_transactions/5-550e8400-e29b-41d4-a716-446655440000.txn`

### Manifest Files

Manifest files are stored in the `_versions/` directory with naming schemes that support atomic commits.

See [Manifest Naming Schemes](transaction.md#manifest-naming-schemes) for details on the V1 and V2 patterns and their implications for version discovery.

### Version Hint

The optional file `_versions/latest_version_hint.json` records the latest committed version as JSON:

```json
{"version": 42}
```

It exists to accelerate latest-version discovery on stores where listing `_versions/` is expensive: a reader can read the hint and probe higher versions with HEAD requests instead of listing the whole directory, falling back to a full listing if the hint is missing or stale.

The hint is purely an optimization. It is always safe to delete, never affects correctness, and can be ignored by readers that don't understand it. Writers may choose not to write it.

mem_wal.md 34.6 KB

# MemTable & WAL Specification (Experimental)

Lance MemTable & WAL (MemWAL) specification describes a Log-Structured-Merge (LSM) tree architecture for Lance tables, enabling high-performance streaming write workloads while maintaining indexed read performance for key workloads including
scan, point lookup, vector search and full-text search.

## Overall Architecture

![MemWAL Overview](../../images/mem_wal_overview.png)

A Lance table is called the **base table** in this document.
The base table may have an [unenforced primary key](index.md#unenforced-primary-key) in its schema.
Primary keys are required for primary-key lookups and last-write-wins upsert semantics.
Append-only MemWAL tables may omit a primary key.

MemWAL adds a set of shards on top of the base table.
Writers append to shards.
Each shard keeps recent data in an in-memory MemTable, persists writes to a per-shard WAL, flushes MemTables as small Lance datasets, and later compacts those SSTables into the base table.

The base table manifest contains one MemWAL system index entry named `__lance_mem_wal`.
This index stores MemWAL configuration and global progress metadata inline in `IndexMetadata.index_details`.
Each shard's own manifest remains authoritative for shard-local mutable state.

### MemWAL Shard

A **MemWAL shard** is the unit of horizontal write scaling.
Each shard has exactly one active writer epoch at a time.
Writers claim a shard, append WAL entries, update the in-memory MemTable, and publish SSTable generations by updating the shard manifest.

For primary-key tables, all rows for the same primary key must map to the same shard.
If one primary key can appear in multiple shards, asynchronous compaction order between shards can make an older row overwrite a newer row.
Append-only tables without a primary key do not rely on last-write-wins conflict resolution and may use any deterministic shard assignment suitable for the workload.

### MemWAL Index

The MemWAL index is a system index entry on the base table.
It has `name = "__lance_mem_wal"`, no indexed fields, and no index files.
`IndexMetadata.files` is `None`.
All MemWAL index data is stored in the `MemWalIndexDetails` protobuf message in `IndexMetadata.index_details`.

The index stores:

- **Configuration**: `sharding_specs`, `maintained_indexes`, and `writer_config_defaults`.
- **Compaction progress**: `compacted_sstables`, the last SSTable compacted into the base table for each shard.
- **Index catchup progress**: `index_catchup`, the compacted SSTable generation covered by each base-table index.
- **Shard snapshots**: optional point-in-time snapshot fields for read optimization.

Shard snapshots are not authoritative.
Readers that need the latest shard set list `_mem_wal/` and read each shard's latest manifest.

## Shard Architecture

![Shard Architecture](../../images/mem_wal_shard.png)

Within a shard, writes first enter an in-memory **MemTable** and are durably appended to the shard **write-ahead log (WAL)**.
The MemTable is periodically **flushed** to storage as a Lance dataset.
SSTables are asynchronously **compacted** into the base table.

### MemTable

A MemTable holds rows inserted into a shard before those rows are flushed to storage.
It serves two purposes:

1. It buffers data and per-MemTable indexes before an SSTable is written.
2. It lets readers access data that has not been flushed yet when strong consistency is required.

The storage format does not prescribe the in-memory MemTable layout.
Conceptually, a MemTable is an append log of Arrow record batches.
Later appends have larger in-memory row positions.
For primary-key tables, in-memory reads use the largest visible row position as the newest row for a key.

### SSTable Generation

Within each shard, SSTables have monotonically increasing generation numbers starting from 1.
When a MemTable is flushed, the resulting SSTable is assigned the shard manifest's `current_generation`, and `current_generation` advances to the next SSTable generation.
A MemTable does not have a generation.

SSTable generation numbers order persisted data freshness within one shard:

- Base table data is modeled as generation 0.
- Higher SSTable generations are newer.
- The active MemTable is newer than every published SSTable.
- Within the active MemTable, higher row positions are newer.
- Within an SSTable, flush-time deletion vectors hide older duplicate primary-key rows, so readers see at most the newest row for each primary key.

## WAL

The WAL is the durable append log for a shard.
Every durable WAL append creates one **WAL entry**.

### WAL Entry Positions

WAL entry positions are 1-based.
The first data entry is position 1.
Position 0 is reserved as the sentinel value meaning no WAL entry has been covered.

Writers append WAL entries in increasing position order.
If entry `N` is not fully written, entry `N + 1` must not exist.
Recovery replays from `replay_after_wal_entry_position + 1`.

### WAL Entry Format

Each WAL entry is an Apache Arrow IPC stream file.
The Arrow schema metadata includes:

- `writer_epoch`: decimal string containing the writer epoch that created the entry.
- `fence_sentinel`: optional marker for a data-less fence sentinel entry.

A normal WAL entry contains one or more record batches.
A fence sentinel entry contains no batches and is skipped during replay.
Sentinels are used so an older writer collides on the next WAL position and discovers that it has been fenced.

### WAL Storage Layout

WAL entries live under `_mem_wal/{shard_id}/wal/`.
Filenames use bit-reversed 64-bit binary names with the `.arrow` suffix:

```text
_mem_wal/{shard_id}/wal/{bit_reversed_position}.arrow
```

The bit-reversal spreads sequential positions across object-store keyspace.
For example, position 5 is encoded as:

```text
1010000000000000000000000000000000000000000000000000000000000000.arrow
```

## SSTable

An SSTable is the immutable result of flushing a MemTable.
It is stored as a Lance dataset under its shard directory.

!!! note
    Unlike a classic LSM sorted string table, a MemWAL SSTable is not sorted by key; random access is instead served by its BTree primary-key sidecar. It is called an SSTable because it is an immutable, persisted, indexed run.

### SSTable Storage Layout

An SSTable with generation `i` is written to:

```text
_mem_wal/{shard_id}/{random8}_gen_{i}/
```

`{random8}` is an 8-character random hex value generated for each flush attempt.
If a flush attempt fails, a retry writes a different directory instead of reusing a partially written one.
The shard manifest records the successful directory name in the SSTable's `path`.

### SSTable Accounting

Alongside `generation` and `path`, a shard manifest entry may record what the
SSTable holds, as the writer's MemTable accounted for it at flush:

- `in_memory_bytes`: payload size of the rows, summed over the MemTable's
  buffers as the window each one holds.
- `physical_rows`: rows held, counting the older duplicates of a primary key
  that the generation's deletion vector masks. A scan applying the deletion
  vector yields fewer.
- `primary_key_bytes`: total payload size of the primary-key columns over every
  row in `physical_rows`. Not a per-row size, which varies for a
  variable-length key.

All three are estimates of payload, not bounds on what reading the SSTable
costs: they exclude the per-array structure a reader materializes, and
`primary_key_bytes` is not the size of any encoded form of the key. A consumer
budgeting memory from them adds its own headroom.

All three are optional. An entry written before they existed records none of
them, and a reader must not treat an absent value as zero; `primary_key_bytes`
is also absent on a table with no primary key.

The SSTable directory is a standard Lance dataset written with the base table's data storage version.
Each SSTable is written as one fragment.
Additional MemWAL sidecars may be present:

```text
{random8}_gen_{i}/
├── _versions/
│   └── {version}.manifest
├── _deletions/                         # Present when within-SSTable dedup deletes rows
├── _indices/                           # Present when maintained user indexes are built
│   └── {index_uuid}/
├── _pk_index/                          # Primary-key sidecar BTree, not a manifest index
└── bloom_filter.bin                    # Primary-key bloom filter
```

The exact Lance dataset internals follow the [Lance table storage layout](layout.md).

### SSTable Row Order

SSTable rows are written in forward insert order.
Physical row offsets increase with write time.
For a duplicate primary key within one SSTable, the newest row has the largest physical offset.

Primary-key SSTables use a deletion vector to expose last-write-wins semantics.
During flush, the writer scans rows in forward order, keeps the last occurrence of each primary key, and marks all earlier duplicate offsets deleted.
The deletion vector is attached to fragment 0 in the SSTable's Lance manifest.

Append-only SSTables without a primary key do not perform primary-key deduplication and retain every row.

### Tombstone Rows

Delete operations are represented as rows with the internal `_tombstone` column.
Tombstone rows follow the same forward row ordering and deletion-vector rules as ordinary rows.
If the newest row for a primary key is a tombstone, the deletion vector keeps that tombstone row and hides older rows for the key.
Read planning then filters `_tombstone = false`, so the key is absent from query results.

### SSTable Primary-Key Sidecars

Primary-key MemTables maintain an implicit BTree for primary-key deduplication, independent of `maintained_indexes`.
When a primary-key MemTable is flushed, the SSTable writes two primary-key sidecars:

- `bloom_filter.bin` stores the SSTable's primary-key bloom filter and lets point lookups skip SSTables that cannot contain the queried key.
- `_pk_index/` stores a standalone BTree over primary-key values to forward row ids.

The `_pk_index/` sidecar is not a maintained user index, is not registered in the SSTable's Lance manifest, and has no manifest UUID.
Its identity is its immutable SSTable path.
Readers open it directly from `{sstable_path}/_pk_index`.

The `_pk_index/` directory is a Lance scalar BTree index store:

```text
_pk_index/
├── page_data.lance
└── page_lookup.lance
```

Readers load this directory as a BTree index using `BTreeIndexDetails` with default parameters.
The primary-key index type is the Arrow type of the primary-key column for a single-column primary key, or `Binary` for a composite primary key.

The `page_lookup.lance` file has the following schema:

| Column       | Type                    | Nullable | Description                                    |
|--------------|-------------------------|----------|------------------------------------------------|
| `min`        | {PrimaryKeyIndexType}   | true     | Minimum primary-key index value in the page    |
| `max`        | {PrimaryKeyIndexType}   | true     | Maximum primary-key index value in the page    |
| `null_count` | UInt32                  | false    | Number of null values in the page              |
| `page_idx`   | UInt32                  | false    | Page number pointing into `page_data.lance`    |

The `page_data.lance` file has the following schema:

| Column   | Type                  | Nullable | Description                                                       |
|----------|-----------------------|----------|-------------------------------------------------------------------|
| `values` | {PrimaryKeyIndexType} | true     | Sorted primary-key index values                                   |
| `ids`    | UInt64                | false    | Forward row ids corresponding to each primary-key index value     |

For a single-column primary key, the indexed value stores the primary-key scalar directly.
For a composite primary key, the indexed value stores an order-preserving binary tuple encoding of all primary-key columns in primary-key column order.
Each tuple column is encoded as:

- `0x00` for null.
- `0x01` followed by the non-null value encoding otherwise.

Supported non-null value encodings are:

- Signed integers and date values: sign-flipped 8-byte big-endian integer bytes.
- Unsigned integers: 8-byte big-endian unsigned integer bytes.
- Boolean: one byte, `0x00` for false and `0x01` for true.
- UTF-8 and binary values: raw bytes, with each `0x00` byte escaped as `0x00 0xff`, followed by a `0x00 0x00` terminator.

This encoding is injective and preserves primary-key tuple ordering under lexicographic byte comparison.
Composite primary-key columns must use one of the supported encodings above.

The sidecar row ids are in the same forward row-position space as the data files, deletion vector, and maintained user indexes.
The sidecar is used for cross-generation membership and block-list checks.
It is not used to choose the newest row inside the same SSTable; the deletion vector has already hidden older same-generation duplicates.

### Maintained User Indexes

When the MemWAL index lists `maintained_indexes`, flush may build matching indexes inside the SSTable.
These index files live in the SSTable's `_indices/{index_uuid}/` directory and are recorded in the SSTable's Lance manifest.
The implicit primary-key BTree sidecar is not included in `maintained_indexes` and does not live under `_indices/`.

These indexes use the same row-position space as the forward-written data files.
If the SSTable has a primary key, its deletion vector masks stale duplicate rows for indexed reads as well.

### SSTable Compaction

SSTables are compacted into the base table in ascending generation order within each shard.
Lower generation numbers are older and must be compacted before higher generation numbers.
SSTable compaction uses merge-insert semantics so newer rows overwrite older rows for the same primary key.

## Shard Manifest

Each shard has a versioned manifest.
The latest shard manifest is the source of truth for shard-local state.

### Shard Manifest Contents

The manifest contains:

- **Identity**: `shard_id`, `shard_spec_id`, and `shard_field_entries`.
- **Fencing state**: `writer_epoch`.
- **WAL pointers**: `replay_after_wal_entry_position` and `wal_entry_position_last_seen`.
- **SSTable generation state**: `current_generation` and `sstables`.
- **Lifecycle state**: `status`, either `ACTIVE` or `SEALED`.

`shard_field_entries` stores computed shard field values as raw Arrow scalar bytes keyed by `ShardingField.field_id`.
The matching `ShardingField.result_type` determines how to decode each value.
For example, `int32` values are four little-endian bytes and `utf8` values are raw UTF-8 bytes.

`replay_after_wal_entry_position` is the most recent 1-based WAL position covered by an SSTable.
The default value 0 means no WAL entry has been covered and recovery starts at position 1.

`wal_entry_position_last_seen` is a best-effort hint for the most recent WAL position observed at manifest update time.
It is not authoritative because it is not updated on every WAL write.
Recovery must still probe or list WAL files to find the actual tail.

`current_generation` is the generation number to assign to the next SSTable created by flushing the MemTable.
Each entry in `sstables` records a published SSTable's `generation` and `path`.

`status = SEALED` marks a reversible in-flight drop-table operation.
Sealed shards refuse new writer claims.

The manifest is serialized as the `ShardManifest` protobuf message.

<details>
<summary>ShardManifest protobuf message</summary>

```protobuf
%%% mem_wal.message.ShardManifest %%%
```

</details>

### Shard Manifest Versioning

Manifest versions start at 1.
Each update writes a new immutable protobuf file:

```text
_mem_wal/{shard_id}/manifest/{bit_reversed_version}.binpb
```

Writers use put-if-not-exists or atomic rename, depending on storage support.
If two processes race to write the same next version, one wins and the other reloads and retries.

After a successful version write, the writer best-effort updates:

```json
{"version": <new_version>}
```

in:

```text
_mem_wal/{shard_id}/manifest/version_hint.json
```

Readers use `version_hint.json` as a starting point and then probe subsequent versions until a version is missing.
The latest manifest is the last existing version.

## MemWAL Index Details

The MemWAL index is stored inline in the base table's `IndexMetadata`.
It is a system index with no file directory.
The `index_details` field contains a `MemWalIndexDetails` protobuf message.

Important fields:

- `sharding_specs`: sharding configuration used by writers and shard pruning.
- `maintained_indexes`: names of base-table indexes to maintain in MemTables and SSTables.
- `writer_config_defaults`: string map of default writer configuration values persisted for all writers.
- `compacted_sstables`: per-shard compaction progress, updated atomically with base-table compaction commits.
- `index_catchup`: per-index coverage progress after data has been compacted into the base table.
- `snapshot_ts_millis`, `num_shards`, and `inline_snapshots`: optional shard snapshot fields for read optimization.

A shard absent from `index_catchup` for an index means that index is *not* known
to have caught up, so the shard's SSTables must be retained until some commit
records that it has.

Catch-up is derived at commit time, not reported by the writer. An index whose segments together span every fragment live at the transaction's read version holds every row compaction had copied into the base table by then, so the commit records it as caught up to that version's `compacted_sstables`. That is the only proof available — nothing maps a compaction generation to the fragments its rows landed in — so covering the table as the transaction read it is how an index shows it covered those rows. Fragments appended since that read are a later catch-up gap and are not required.

Two rules bound what a commit may record. It never credits more than its own `compacted_sstables`, and it clamps to that value, so a position can only describe generations the base table has actually taken in. Otherwise it never lowers a position an index already held, provided that index is unchanged by this commit. "Unchanged" compares each segment's UUID together with its fragment bitmap, not the UUID alone, because an operation can prune the bitmap in place while keeping the UUID; the remaining metadata does not affect which rows the index answers for. An index this commit changes keeps no position it cannot re-earn.

Because the position is derived rather than transmitted, it cannot go stale between inspection and commit, and it survives rebase — `read_version` is fixed for a transaction's life, so what a commit can prove does not move, though a rebased attempt may record a different result because the head it commits against has changed. Any commit can earn a position, so an ordinary index build that happens to cover the table records catch-up as a side effect. A dedicated repair is still needed where no such commit occurs, or where an index does not yet span the table.

A read version with no fragments proves nothing, even though an index trivially covers an empty table. An empty fragment list is also what a manifest written before the `UpdateMemWalState` fragment fix looks like, where the SSTables are the last copy of those rows; crediting coverage there would retire them. The cost is that a table whose rows have all been deleted keeps its SSTables.

Shard snapshots, when present, use the following Lance file schema:

| Column                     | Type                         | Nullable | Description                                            |
|----------------------------|------------------------------|----------|--------------------------------------------------------|
| `shard_id`                 | Utf8                         | false    | Shard UUID string                                      |
| `shard_spec_id`            | UInt32                       | false    | Sharding spec that produced the shard                  |
| `shard_field_{field_id}`   | `ShardingField.result_type`  | false    | Computed shard field value for the given sharding field |

The MemWAL index data is stored inline.
Readers discover the latest shard set by listing `_mem_wal/` shard directories and reading shard manifests.

<details>
<summary>MemWalIndexDetails protobuf message</summary>

```protobuf
%%% mem_wal.message.MemWalIndexDetails %%%
```

</details>

## Sharding

A **ShardingSpec** defines how rows map to shards.
Each spec has a positive `spec_id` and one or more `ShardingField` entries.
Each shard manifest records the `shard_spec_id` and the computed shard field values for that shard.
`spec_id = 0` means the shard was manually created and is not governed by a sharding spec.

Each `ShardingField` contains:

- `field_id`: stable identifier for the computed shard field.
- `source_ids`: field IDs of source columns in the Lance schema.
- `transform`: well-known transform name, when using built-in transform evaluation.
- `expression`: reserved custom expression text, mutually exclusive with `transform`.
- `result_type`: Arrow type name for the computed value.
- `parameters`: transform-specific string parameters.

The supported built-in transforms are:

- `unsharded`: takes no source columns, always returns `int32` value 0, and creates one shard.
- `bucket`: takes one source column and `num_buckets`, hashes the value, and returns an `int32` bucket id in `[0, num_buckets)`.
- `identity`: takes one source column and returns the raw scalar value as the shard value.

`bucket` computes a deterministic 32-bit hash with seed 0 and then computes:

```text
(hash & i32::MAX) % num_buckets
```

`num_buckets` must be in `[1, 1024]`.
Null bucket values hash to 0 and therefore map to bucket 0.
See [Appendix 3: Bucket Hashing](#appendix-3-bucket-hashing) for the exact hash algorithm and test vectors.

The `bucket` transform supports scalar boolean, integer, floating-point, date32, time, timestamp, utf8, and large_utf8 source types.
The `identity` transform supports scalar boolean, integer, utf8, and large_utf8 source types.

The `year`, `month`, `day`, `hour`, `multi_bucket`, and `truncate` transform names are not supported MemWAL sharding transforms and must not be used in `ShardingSpec.transform`.

## Storage Layout

The MemWAL storage layout is:

```text
{table_path}/
├── _versions/
│   └── ...                              # Base table manifests, including __lance_mem_wal index metadata
├── _indices/
│   └── ...                              # Ordinary base table index files; MemWAL index has no files
└── _mem_wal/
    └── {shard_id}/
        ├── manifest/
        │   ├── {bit_reversed_version}.binpb
        │   └── version_hint.json
        ├── wal/
        │   ├── {bit_reversed_position}.arrow
        │   └── ...
        └── {random8}_gen_{generation}/
            ├── _versions/
            │   └── {version}.manifest
            ├── _deletions/
            ├── _indices/
            │   └── {index_uuid}/
            ├── _pk_index/
            └── bloom_filter.bin
```

Some SSTable subdirectories are conditional.
For example, `_deletions/` is present only when the SSTable's Lance manifest references a deletion vector, `_indices/` is present only when maintained user indexes are built, and `_pk_index/` plus `bloom_filter.bin` are meaningful for primary-key tables.

## Implementation Expectation

This document specifies the storage layout and observable reader and writer invariants.
Implementations may choose different in-memory structures, buffering policies, background scheduling, and query execution plans.

An implementation is compatible when it:

1. Writes WAL entries, shard manifests, SSTables, and MemWAL index metadata using the documented layout.
2. Preserves WAL position, writer fencing, and manifest versioning invariants.
3. Exposes last-write-wins semantics for primary-key tables.
4. Preserves append-only semantics for tables without primary keys.
5. Maintains generation ordering when compacting SSTables into the base table.

## Writer Expectations

A writer operates on one shard and is responsible for:

1. Claiming the shard with epoch-based fencing.
2. Appending WAL entries in sequential 1-based positions.
3. Maintaining in-memory MemTable state.
4. Flushing MemTables to SSTable Lance datasets.
5. Updating the shard manifest after an SSTable is durably written.

### Writer Fencing

Writers use `writer_epoch` to enforce single-writer semantics per shard.

To claim a shard:

1. Load the latest shard manifest.
2. Verify the shard is `ACTIVE`.
3. Increment `writer_epoch`.
4. Atomically write a new manifest version.
5. If the manifest write loses a race, reload and retry.

Before a manifest update, a writer verifies its local epoch is still current:

- If `local_writer_epoch == stored_writer_epoch`, the writer may proceed.
- If `local_writer_epoch < stored_writer_epoch`, the writer has been fenced and must abort.

WAL append conflicts also detect fencing.
If an older writer collides with a newer writer's WAL entry at the same position, it reloads the manifest and observes the higher epoch.
Fence sentinel entries make this collision path explicit without storing data batches.

## Background Job Expectations

Background jobs compact SSTables into the base table and remove obsolete shard data.

### SSTable Compactor

SSTables must be compacted into the base table in ascending generation order within each shard.
The compaction uses Lance merge-insert semantics and updates `compacted_sstables[shard_id]` atomically with the base-table commit.

On commit conflict, a compactor reloads the conflicting base-table version:

- If the committed `compacted_sstables[shard_id]` is already greater than or equal to the generation being compacted, the compactor skips that generation.
- Otherwise, the compactor retries from the latest base-table version.

### Garbage Collector

The garbage collector may remove obsolete SSTables after:

1. The SSTable has been compacted into the base table.
2. Every index a query may rely on has caught up to cover the SSTable's generation, or the SSTable is no longer needed for indexed reads. An index absent from `index_catchup` has *not* caught up, so this condition is not met for it.
3. No retained base-table version needs the SSTable for time travel or consistency.

!!! warning
    Deleting WAL files can weaken writer fencing.

    Fencing detects a stalled writer when its put-if-not-exists for the next WAL entry collides with a newer writer's entry at the same position.
    If garbage collection has removed that WAL file, the stalled writer may write into empty space with an old `writer_epoch`.
    Implementations that garbage collect WAL files must compensate by re-checking fence state after WAL writes, partitioning WAL positions by epoch, or otherwise preventing stale writers from landing at positions that have been garbage collected.

## Reader Expectations

### LSM Tree Merging Read

For primary-key tables, readers merge rows from the base table, SSTables, and optionally in-memory MemTables by primary key.
The newest row wins.

Freshness ordering within one shard is:

1. The active MemTable wins over every SSTable and the base table.
2. Among SSTables, higher generation wins.
3. Any uncompacted SSTable wins over the base table.
4. Within the active MemTable, higher row position wins.
5. Within an SSTable, its deletion vector has already hidden older duplicate primary-key rows.

For freshness comparisons, the base table uses the sentinel generation 0.
SSTable generations are positive.
This ordering applies only to sources selected for the same read plan.
Readers must not include an SSTable that is already covered by the base table according to `compacted_sstables[shard_id]`, because otherwise the positive SSTable generation would incorrectly outrank base-table rows during deduplication.
Rows from different shards do not need primary-key deduplication if the sharding spec guarantees that each primary key maps to exactly one shard.

Append-only tables without a primary key do not perform primary-key deduplication.
Rows from all selected sources are distinct appended rows.

### Tombstones

Readers must treat `_tombstone = true` rows as delete markers.
In SSTables, deletion vectors first resolve same-generation duplicate primary keys.
Then query planning filters tombstone rows from user-visible results.
In active in-memory MemTables, the newest visible row position for a primary key wins; if that row is a tombstone, the key is absent.

### Reader Consistency

Reader consistency depends on:

1. Whether the reader can access active in-memory MemTables.
2. Whether shard metadata comes from latest shard manifests or from an older MemWAL index snapshot.

Strong consistency requires active in-memory MemTable access for relevant shards and direct reads of latest shard manifests.
Otherwise, reads are eventually consistent because unflushed data or newly-created shards may be absent from the read plan.

Reading a stale MemWAL index snapshot does not corrupt last-write-wins ordering, but it can reduce freshness:

- If a compacted SSTable is still listed, readers must skip it when `generation <= compacted_sstables[shard_id]`.
    For primary-key tables, including it would let an older SSTable row outrank newer base-table contents because SSTable generations are positive and the base table is modeled as generation 0.
    For append-only tables, including it would return the same append twice.
- If a garbage-collected SSTable is still listed, readers may skip it after failing to open it because its data must already be in the base table or be filtered out by `compacted_sstables`.
- If a new SSTable is not listed, the read is consistent with the older snapshot but may miss fresher data.

Readers that require latest shard membership should list `_mem_wal/` and read shard manifests instead of relying only on snapshots.

### Query Planning

A query planner collects sources from:

1. The base table.
2. SSTables that are not yet safely replaceable by base-table indexed reads.
3. Active in-memory MemTables, when available and required by the requested consistency level.

Each source is tagged with its shard and freshness tier.
SSTable sources are also tagged with their generation.
For primary-key reads, the planner applies LSM deduplication across selected sources.
For append-only reads, the planner concatenates selected sources without primary-key deduplication.

Bloom filters and `_pk_index/` sidecars help prune SSTables during point lookups and cross-generation deduplication.

### Shard Pruning

When sharding specs are available, the planner evaluates query predicates against shard fields and skips shards whose computed shard values cannot match.

For example, with `bucket(user_id, 10)` and predicate `user_id = 123`:

1. Compute the bucket id for `123`.
2. Scan only shards whose manifest has the same computed bucket value.
3. Skip all other bucket shards.

### Indexed Read Plan

When data is compacted from an SSTable into the base table, base-table indexes may lag behind the data commit.
`index_catchup` records which compacted generation each base-table index covers.

If an indexed query needs index `I` and `I` has only caught up to generation `G` while `compacted_sstables[shard_id]` is higher, the planner should read the gap from SSTable indexes instead of scanning unindexed base-table rows.
Once index `I` catches up, the planner can use the base-table index for those compacted rows.

## Appendices

### Appendix 1: Writer Fencing Example

Initial shard manifest:

```text
version: 1
writer_epoch: 5
replay_after_wal_entry_position: 10
wal_entry_position_last_seen: 12
status: ACTIVE
```

Writer A loads version 1, claims epoch 6, and writes manifest version 2.
It appends WAL entries 13, 14, and 15 with `writer_epoch = 6`.

Writer B then loads version 2, claims epoch 7, and writes manifest version 3.
It appends WAL entries 16 and 17 with `writer_epoch = 7`.

When Writer A later tries to flush or update the shard manifest, it reloads the manifest and sees stored epoch 7 while its local epoch is 6.
Writer A is fenced and must abort.

Recovery starts from `replay_after_wal_entry_position + 1`, which is entry 11.
Entries 13, 14, 15, 16, and 17 are valid replay inputs because they were written by epochs that were valid at write time and are not greater than the current shard epoch.

### Appendix 2: Concurrent Compactor Example

Initial state:

```text
MemWAL index:
  compacted_sstables: {shard: 5}

Shard manifest:
  current_generation: 8
  sstables:
    - generation: 6, path: "abc12345_gen_6"
    - generation: 7, path: "def67890_gen_7"
```

Two compactors both try to compact the SSTable at generation 6.
Compactor A commits first and updates `compacted_sstables[shard]` to 6 in the same base-table commit as the data.
Compactor B then hits a commit conflict, reloads the latest MemWAL index, sees `compacted_sstables[shard] >= 6`, skips generation 6, and continues with generation 7.

The MemWAL index is the authoritative compaction-progress record because it is committed atomically with the base-table data changes.

### Appendix 3: Bucket Hashing

The bucket transform hash uses 32-bit wrapping arithmetic with these mixing functions.
Right shifts in `fmix` are logical shifts of the `u32` bit pattern.

```text
mix_k1(k) = rotl32(k * 0xcc9e2d51, 15) * 0x1b873593
mix_h1(h, k) = rotl32(h ^ k, 13) * 5 + 0xe6546b64
fmix(h, len) =
    h = h ^ len
    h = (h ^ (h >> 16)) * 0x85ebca6b
    h = (h ^ (h >> 13)) * 0xc2b2ae35
    h ^ (h >> 16)
```

Signed and unsigned casts use two's-complement wrapping.
Values are normalized and hashed as follows:

- `bool`: `false` as `0`, `true` as `1`, then `hash_i32`.
- `int8`, `int16`, `int32`, `uint8`, `uint16`, `uint32`, `date32`, `time32`: cast to `i32`, then `hash_i32`.
- `int64`, `uint64`, `timestamp`, `time64`: cast to `i64`, then `hash_i64`.
- `float32`: `-0.0` and `+0.0` normalize to bits `0`; all NaNs normalize to `0x7fc00000`; other values use IEEE 754 bits cast to `i32`, then `hash_i32`.
- `float64`: `-0.0` and `+0.0` normalize to bits `0`; all NaNs normalize to `0x7ff8000000000000`; other values use IEEE 754 bits cast to `i64`, then `hash_i64`.
- `utf8` and `large_utf8`: hash the UTF-8 bytes with `hash_bytes`.

The helper hashes are:

```text
hash_i32(v) = fmix(mix_h1(0, mix_k1(v)), 4)

hash_i64(v) =
    low = low 32 bits of v as i32
    high = high 32 bits of v as i32
    fmix(mix_h1(mix_h1(0, mix_k1(low)), mix_k1(high)), 8)

hash_bytes(bytes) =
    h = 0
    for each complete 4-byte little-endian chunk:
        h = mix_h1(h, mix_k1(chunk_as_i32))
    for each remaining byte:
        h = mix_h1(h, mix_k1(sign_extend_i8(byte)))
    fmix(h, byte_length)
```

Test vectors for `num_buckets = 8`:

- `int32` or `date32`: `1 -> 2`, `2 -> 7`, `null -> 0`, `3 -> 1`.
- `utf8`: `"a" -> 1`, `"b" -> 5`, `null -> 0`.
- `bool`: `true -> 2`.
- `float32`: `1.25 -> 0`.
- `float64`: `1.25 -> 0`.

row_id_lineage.md 14.2 KB

# Row ID and Lineage Specification

## Overview

Lance provides row identification and lineage tracking capabilities.
Row addressing enables efficient random access to rows within the table through a physical location encoding.
Stable row IDs provide persistent identifiers that remain constant throughout a row's lifetime, even as its physical location changes.
Row version tracking records when rows were created and last modified, enabling incremental processing, change data capture, and time-travel queries.

## Row Identifier Forms

A row in Lance has two forms of row identifiers:

- **Row address** - the current physical location of the row in the dataset.
- **Row ID** - a logical identifier of the row. When stable row IDs are enabled, this remains stable for the lifetime of a logical row. When disabled (default mode), it is exactly equal to the row address.


### Row Address

Row address is the physical location of a row in the table, represented as a 64-bit identifier composed of two 32-bit values:

```
row_address = (fragment_id << 32) | local_row_offset
```

This addressing scheme enables efficient random access: given a row address, the fragment and offset are extracted with bit operations.
Row addresses change when data is reorganized through compaction or updates.

Row address is currently the primary form of identifier used for indexing purposes.
Secondary indices (vector indices, scalar indices, full-text search indices) reference rows by their row addresses.

!!! note
      Work to support stable row IDs in indices is in progress.

### Row ID

Row ID is a logical identifier for a row.

#### Stable Row ID

When a dataset is created with stable row IDs enabled, each row is assigned a unique auto-incrementing `u64` identifier that remains constant throughout the row's lifetime, even when the row's physical location (row address) changes.
The `_rowid` system column exposes this logical identifier to users.
See the next section for more details on assignment and update semantics.

#### Historical/unstable usage

Historically, the term "row id" was often used to refer to the physical row address (`_rowaddr`), which is not stable across compaction or updates.

!!! warning
      With the introduction of stable row IDs, there may still be places in code and documentation that mix the terms "row ID" and "row address" or "row ID" and "stable row ID".
      Please raise a PR if you find any place incorrect or confusing.

## Stable Row ID

### Row ID Assignment

Row IDs are assigned using a monotonically increasing `next_row_id` counter stored in the manifest.

**Assignment Protocol:**

1. Writer reads the current `next_row_id` from the manifest at the read version
2. Writer assigns row IDs sequentially starting from `next_row_id` for new rows
3. Writer updates `next_row_id` in the new manifest to `next_row_id + num_new_rows`
4. If commit fails due to conflict, writer rebases:
   - Re-reads the new `next_row_id` from the latest version
   - Reassigns row IDs to new rows using the updated counter
   - Retries commit

This protocol mirrors fragment ID assignment and ensures row IDs are unique across all table versions.

### Enabling Stable Row IDs

Stable row IDs are a dataset-level feature recorded in the table manifest.

- Stable row IDs may be enabled when a dataset is created or by migrating an existing dataset.
- An ordinary write with `enable_stable_row_ids = true` does not migrate an existing dataset. Use the stable row ID migration operation instead; the Rust API exposes it as `Dataset::migrate_to_stable_row_ids`.
- Before migrating a dataset whose current manifest has no writer version, use the current Lance writer to commit an ordinary no-op deletion with predicate `false`, then reopen the latest version. This metadata-upgrade commit recomputes the authoritative physical row count for every fragment. Do not invoke stable row ID migration directly on such a legacy manifest: affected releases may have recorded stale counts, which would produce incomplete row ID sequences.
- Before migration, stop all index builds and index commits, drop every secondary index so no index entry remains in the dataset metadata, and keep index creation quiesced until migration completes. An in-flight index commit from a pre-migration snapshot can otherwise attach stale physical row addresses after activation. Recreate indices after migration.
- Quiesce data-modifying writers during migration. The migration uses a single atomic merge commit and does not retry when a concurrent write causes a conflict; the caller must retry the migration.
- Migration assigns an ID to every physical row position, including deleted positions, and atomically enables the feature and advances `next_row_id`. Migrating a dataset that already uses stable row IDs is a no-op.
- When stable row IDs are disabled, the `_rowid` column (if requested) is not stable and should not be used as a persistent identifier.

Row-level version tracking (`_row_created_at_version`, `_row_last_updated_at_version`) and the row ID index described below are only available when stable row IDs are enabled.

### Row ID Behavior on Updates

When stable row IDs are enabled, updates preserve the logical row ID and remap it to a new physical address instead of assigning a new ID.

**Update Workflow:**

1. Original row with `_rowid = R` exists at address `(F1, O1)`.
2. An update operation writes a new physical row with the updated values at address `(F2, O2)`.
3. The new physical row is assigned the same `_rowid = R`, so the logical identifier is preserved.
4. The original physical row at `(F1, O1)` is marked deleted using the deletion vector for fragment `F1`.
5. The row ID index for the new dataset version maps `_rowid = R` to `(F2, O2)`, and uses deletion vectors and fragment bitmaps to avoid returning the tombstoned row at `(F1, O1)`.

This design keeps `_rowid` stable for the lifetime of a logical row while allowing physical storage and secondary indices to be maintained independently.

### Row ID Sequences

#### Storage Format

Row ID sequences are stored using the `RowIdSequence` protobuf message.
The sequence is partitioned into segments, each encoded optimally based on the data pattern.

<details>
<summary>RowIdSequence protobuf message</summary>

```protobuf
%%% proto.message.RowIdSequence %%%
```

</details>

#### Segment Encodings

Each segment uses one of five encodings optimized for different data patterns:

##### Range (Contiguous Values)

For sorted, contiguous values with no gaps.
Example: Row IDs `[100, 101, 102, 103, 104]` → `Range{start: 100, end: 105}`.
Used for new fragments where row IDs are assigned sequentially.

<details>
<summary>Range protobuf message</summary>

```protobuf
%%% proto.message.Range %%%
```

</details>

##### Range with Holes (Sparse Deletions)

For sorted values with few gaps.
Example: Row IDs `[100, 101, 103, 104]` (missing 102) → `RangeWithHoles{start: 100, end: 105, holes: [102]}`.
Used for fragments with sparse deletions where maintaining the range is efficient.

<details>
<summary>RangeWithHoles protobuf message</summary>

```protobuf
%%% proto.message.RangeWithHoles %%%
```

</details>

##### Range with Bitmap (Dense Deletions)

For sorted values with many gaps.
The bitmap encodes 8 values per byte, with the most significant bit representing the first value.
Used for fragments with dense deletion patterns.

<details>
<summary>RangeWithBitmap protobuf message</summary>

```protobuf
%%% proto.message.RangeWithBitmap %%%
```

</details>

##### Sorted Array (Sparse Values)

For sorted but non-contiguous values, stored as an `EncodedU64Array`.
Used for merged fragments or fragments after compaction.

##### Unsorted Array (General Case)

For unsorted values, stored as an `EncodedU64Array`.
Rare; most operations maintain sorted order.

#### Encoded U64 Arrays

The `EncodedU64Array` message supports bitpacked encoding to minimize storage.
The implementation selects the most compact encoding based on the value range, choosing between base + 16-bit offsets, base + 32-bit offsets, or full 64-bit values.

<details>
<summary>EncodedU64Array protobuf message</summary>

```protobuf
%%% proto.message.EncodedU64Array %%%
```

</details>

#### Inline and External Storage

`DataFragment` defines inline and external metadata fields as valid wire alternatives for row ID sequences and row version sequences.
These fields do not currently imply a size-based switching threshold.
Current Lance writers store all three sequence types inline in the fragment metadata regardless of their encoded size and do not emit the external alternatives.

Current Lance readers can load externally stored row ID sequences.
The format also permits external created-at and last-updated-at version sequences, but current Lance readers cannot load them; this is an implementation limitation, not an invalid encoding.

<details>
<summary>DataFragment row_id_sequence field</summary>

```protobuf
message DataFragment {
  oneof row_id_sequence {
    bytes inline_row_ids = 5;
    ExternalFile external_row_ids = 6;
  }
}
```

</details>

### Row ID Index

#### Construction

The row ID index is built at table load time by aggregating row ID sequences from all fragments:

```
For each fragment F with ID f:
  For each (position p, row_id r) in F.row_id_sequence:
    index[r] = (f, p)
```

This creates a mapping from row ID to current row address.

#### Index Invalidation with Updates

When rows are updated and stable row IDs are enabled, the row ID index for a given dataset version only contains mappings for live physical rows. Tombstoned rows are excluded using deletion vectors, and logical row IDs whose contents have changed simply map to new row addresses.

**Example Scenario:**

1. Initial state (version V): Fragment 1 contains rows with IDs `[1, 2, 3]` at offsets `[0, 1, 2]`.
2. An update operation modifies the row with `_rowid = 2`:
    - A new fragment 2 is created with a row for `_rowid = 2` at offset `0`.
    - In fragment 1, the original physical row at offset `1` is marked deleted in the deletion vector.
3. Row ID index in version V+1:
    - `1 → (1, 0)` ✓ Valid
    - `2 → (2, 0)` ✓ Valid (updated row in fragment 2)
    - `3 → (1, 2)` ✓ Valid

The address `(1, 1)` is no longer reachable via the row ID index because it is filtered out by the deletion vector when the index is constructed.

#### Fragment Bitmaps for Index Masking

Secondary indices use fragment bitmaps to track which row IDs remain valid:

**Without Row Updates:**

```
String Index on column "str":
  Fragment Bitmap: {1, 2}  (covers fragments 1 and 2)
  All indexed row addresses are valid
```

**With Row Updates:**

```
Vector Index on column "vec":
  Fragment Bitmap: {1}  (only fragment 1)
  The row with _rowid = 2 was updated, so the index entry that points to its old physical address is stale
  Index queries filter out the stale address using deletion vectors while returning the row at its new address
```

This bitmap-based approach allows indices to remain immutable while accounting for row modifications.

## Row Version Tracking

Row version tracking is available for datasets that use stable row IDs. Version sequences are aligned with the stable `_rowid` ordering within each fragment.

### Created At Version

Each row tracks the version at which it was created.
For rows that are later updated, this creation version remains the version in which the row first appeared; updates do not change it.
The sequence uses run-length encoding for efficient storage, where each run specifies a span of consecutive rows and the version they were created in.

Example: Fragment with 1000 rows created in version 5:
```
RowDatasetVersionSequence {
  runs: [
    RowDatasetVersionRun { span: Range{start: 0, end: 1000}, version: 5 }
  ]
}
```

<details>
<summary>DataFragment created_at_version_sequence field</summary>

```protobuf
message DataFragment {
  oneof created_at_version_sequence {
    bytes inline_created_at_versions = 9;
    ExternalFile external_created_at_versions = 10;
  }
}
```

</details>

<details>
<summary>RowDatasetVersionSequence protobuf messages</summary>

```protobuf
%%% proto.message.RowDatasetVersionSequence %%%
```

</details>

### Last Updated At Version

Each row tracks the version at which it was last modified.
When a row is created, `last_updated_at_version` equals `created_at_version`.

When stable row IDs are enabled and a row is updated, Lance writes a new physical row for the same logical `_rowid` while tombstoning the old physical row. The `created_at_version` for that logical row is preserved from the original row, and `last_updated_at_version` is set to the current dataset version at the time of the update.

Example: Row created in version 3, updated in version 7:
```
Old physical row (tombstoned):
  _rowid: R
  created_at_version: 3
  last_updated_at_version: 3

New physical row (current):
  _rowid: R
  created_at_version: 3
  last_updated_at_version: 7
```

<details>
<summary>DataFragment last_updated_at_version_sequence field</summary>

```protobuf
message DataFragment {
  oneof last_updated_at_version_sequence {
    bytes inline_last_updated_at_versions = 7;
    ExternalFile external_last_updated_at_versions = 8;
  }
}
```

</details>

## Change Data Feed

Lance supports querying rows that changed between versions through version tracking columns.
These queries can be expressed as standard SQL predicates on the `_row_created_at_version` and `_row_last_updated_at_version` columns.

### Inserted Rows

Rows created between two versions can be retrieved by filtering on `_row_created_at_version`:

```sql
SELECT * FROM dataset
WHERE _row_created_at_version > {begin_version}
  AND _row_created_at_version <= {end_version}
```

This query returns all rows inserted in the specified version range, including the version metadata columns `_row_created_at_version`, `_row_last_updated_at_version`, and `_rowid`.

### Updated Rows

Rows modified (but not newly created) between two versions can be retrieved by combining filters on both version columns:

```sql
SELECT * FROM dataset
WHERE _row_created_at_version <= {begin_version}
  AND _row_last_updated_at_version > {begin_version}
  AND _row_last_updated_at_version <= {end_version}
```

This query excludes newly inserted rows by requiring `_row_created_at_version <= {begin_version}`, ensuring only pre-existing rows that were subsequently updated are returned.

schema.md 15.2 KB

# Schema Format Specification

## Overview

The schema describes the structure of a Lance table, including all fields, their data types, and metadata.
Schemas use a logical type system where data types are represented as strings that map to Apache Arrow data types.
Each field in the schema has a unique identifier (field ID) that enables robust schema evolution and version tracking.

!!! note

    Logical types are currently being simplified through discussion [#5864](https://github.com/lance-format/lance/discussions/5864).
    Proposed changes include consolidating encoding-specific variants (e.g., `large_string` and `string`, `large_binary` and `binary`)
    into single logical types with runtime optimization. Additionally, [#5817](https://github.com/lance-format/lance/discussions/5817) proposes adding
    `string_view` and `binary_view` types. This document describes the current implementation.

## Data Types

Lance supports a comprehensive set of data types that map to Apache Arrow types.
Data types are represented as strings in the schema and can be grouped into several categories.

### Primitive Types

| Logical Type | Arrow Type | Description |
|---|---|---|
| `null` | `Null` | Null type (no values) |
| `bool` | `Boolean` | Boolean (true/false) |
| `int8` | `Int8` | Signed 8-bit integer |
| `uint8` | `UInt8` | Unsigned 8-bit integer |
| `int16` | `Int16` | Signed 16-bit integer |
| `uint16` | `UInt16` | Unsigned 16-bit integer |
| `int32` | `Int32` | Signed 32-bit integer |
| `uint32` | `UInt32` | Unsigned 32-bit integer |
| `int64` | `Int64` | Signed 64-bit integer |
| `uint64` | `UInt64` | Unsigned 64-bit integer |

### Floating Point Types

| Logical Type | Arrow Type | Description |
|---|---|---|
| `halffloat` | `Float16` | IEEE 754 half-precision floating point (16-bit) |
| `float` | `Float32` | IEEE 754 single-precision floating point (32-bit) |
| `double` | `Float64` | IEEE 754 double-precision floating point (64-bit) |

### String and Binary Types

| Logical Type | Arrow Type | Description |
|---|---|---|
| `string` | `Utf8` | Variable-length UTF-8 encoded string |
| `binary` | `Binary` | Variable-length binary data |
| `large_string` | `LargeUtf8` | Variable-length UTF-8 string (supports large offsets) |
| `large_binary` | `LargeBinary` | Variable-length binary data (supports large offsets) |

### Decimal Types

Decimal types support arbitrary-precision numeric values. The format is: `decimal:<bit_width>:<precision>:<scale>`

| Logical Type | Arrow Type | Precision | Example |
|---|---|---|---|
| `decimal:128:P:S` | `Decimal128` | Up to 38 digits | `decimal:128:10:2` (10 total digits, 2 after decimal) |
| `decimal:256:P:S` | `Decimal256` | Up to 76 digits | `decimal:256:20:5` |

- **Precision (P)**: Total number of digits (1-38 for Decimal128, up to 76 for Decimal256)
- **Scale (S)**: Number of digits after the decimal point (0 ≤ S ≤ P)

### Date and Time Types

| Logical Type | Arrow Type | Description |
|---|---|---|
| `date32:day` | `Date32` | Date (days since epoch) |
| `date64:ms` | `Date64` | Date (milliseconds since epoch) |
| `time32:s` | `Time32` | Time (seconds since midnight) |
| `time32:ms` | `Time32` | Time (milliseconds since midnight) |
| `time64:us` | `Time64` | Time (microseconds since midnight) |
| `time64:ns` | `Time64` | Time (nanoseconds since midnight) |
| `duration:s` | `Duration` | Duration (seconds) |
| `duration:ms` | `Duration` | Duration (milliseconds) |
| `duration:us` | `Duration` | Duration (microseconds) |
| `duration:ns` | `Duration` | Duration (nanoseconds) |

### Timestamp Types

Timestamp types represent a point in time and may include timezone information.
Format: `timestamp:<unit>:<timezone>`

- **Unit**: `s` (seconds), `ms` (milliseconds), `us` (microseconds), `ns` (nanoseconds)
- **Timezone**: IANA timezone string (e.g., `UTC`, `America/New_York`) or `-` for no timezone

Examples:
- `timestamp:us:UTC` - Microsecond precision timestamp in UTC
- `timestamp:ms:America/New_York` - Millisecond precision timestamp in America/New_York timezone
- `timestamp:ns:-` - Nanosecond precision timestamp with no timezone

### Complex Types

#### Struct Type

A struct is a container for named fields with heterogeneous types.

| Logical Type | Arrow Type | Description |
|---|---|---|
| `struct` | `Struct` | Composite type containing multiple named fields |

Struct fields are represented as child fields in the schema.

Example schema with a struct:
```protobuf
Field {
    name: "address"
    type: "struct"
    children: [
        Field { name: "street", type: "string" },
        Field { name: "city", type: "string" },
        Field { name: "zip", type: "int32" }
    ]
}
```

#### List Types

Lists represent variable-length arrays of a single type.

| Logical Type | Arrow Type | Description |
|---|---|---|
| `list` | `List` | Variable-length list of values |
| `list.struct` | `List(Struct)` | Variable-length list of struct values |
| `large_list` | `LargeList` | Variable-length list (supports large offsets) |
| `large_list.struct` | `LargeList(Struct)` | Variable-length list of struct values (large offsets) |

The element type is specified as a child field.

#### Fixed-Size List Types

Fixed-size lists have a predetermined size known at schema definition time.
Format: `fixed_size_list:<element_type>:<size>`

| Logical Type | Description | Example |
|---|---|---|
| `fixed_size_list:float:128` | Fixed-size list of 128 floats | Vector embeddings (128-dimensional) |
| `fixed_size_list:int32:10` | Fixed-size list of 10 integers | |

Special extension types:
- `fixed_size_list:lance.bfloat16:256` - Fixed-size list of bfloat16 values

#### Fixed-Size Binary Type

Fixed-size binary data with a predetermined size in bytes.
Format: `fixed_size_binary:<size>`

| Logical Type | Description | Example |
|---|---|---|
| `fixed_size_binary:16` | Fixed-size binary of 16 bytes | MD5 hash |
| `fixed_size_binary:32` | Fixed-size binary of 32 bytes | SHA-256 hash |

#### Dictionary Type

Dictionary-encoded data with separate keys and values.
Format: `dict:<value_type>:<key_type>:<ordered>`

- **Value type**: The type of dictionary values
- **Key type**: The type used for dictionary indices (typically int8, int16, or int32)
- **Ordered**: Boolean indicating if dictionary values are sorted (currently not fully supported)

Example: `dict:string:int16:false` - Dictionary-encoded strings with int16 keys

#### Map Type

Key-value pairs stored in a structured format.

| Logical Type | Arrow Type | Description |
|---|---|---|
| `map` | `Map` | Key-value pairs (currently supports unordered keys only) |

Maps have key and value types specified as child fields.

### Extension Types

Lance supports custom extension types that provide semantic meaning on top of Arrow types.

#### Blob Type

Represents large binary data stored externally.

| Logical Type | Description |
|---|---|
| `blob` | Large binary data with external storage reference |
| `json` | JSON-encoded data stored as binary |

Blob types are stored as large binary data with metadata describing storage location.

#### BFloat16 Type

Brain float (bfloat16) is a 16-bit floating point format optimized for ML.
Used within fixed-size lists: `fixed_size_list:lance.bfloat16:SIZE`

## Field IDs

Field IDs are unique integer identifiers assigned to each field in a schema.
They are essential for robust schema evolution, as they allow fields to be renamed or reordered without breaking references.

### Field ID Assignment

**Initial assignment (depth-first order):**
When a table is created, field IDs are assigned to all fields in depth-first order, starting from 0.

Nested fields are linked via the `parent_id` field in the protobuf message. For example, if field "c" (id: 2) is a struct containing fields "x", "y", "z", those child fields will have `parent_id: 2`. Top-level fields have `parent_id: -1`.

Example with nested structure:
```
Field order: a, b, c.x, c.y, c.z, d

Assigned IDs with parent relationships:
- a: 0 (parent_id: -1)
- b: 1 (parent_id: -1)
- c: 2 (parent_id: -1, struct type)
- c.x: 3 (parent_id: 2)
- c.y: 4 (parent_id: 2)
- c.z: 5 (parent_id: 2)
- d: 6 (parent_id: -1)
```

Note: A `parent_id` of -1 indicates a top-level field. For nested fields, `parent_id` references the ID of the parent field. Child fields reference their parent via `parent_id` rather than being stored as separate "children" arrays in the protobuf message (though the Rust in-memory representation maintains a children vector for convenience).

**New field assignment (incremental):**
When fields are added later (e.g., through schema evolution), they receive the next available ID
incrementally. This preserves the history of field additions.

### Field ID Properties

- **Immutable**: Once assigned, a field's ID never changes
- **Unique**: Each field within a table has a unique ID
- **Stable**: IDs are preserved across schema evolution operations
- **Sparse**: Field IDs may not form a contiguous sequence after schema evolution

### Using Field IDs

When referencing fields internally within the format, use the field ids rather than field names or positions.

## Field Metadata

Fields can carry additional metadata as key-value pairs to configure encoding, primary key behavior, and other properties.

### Primary Key Metadata

Primary key configuration is handled by two protobuf fields in the Field message:
- **unenforced_primary_key** (bool): Whether this field is part of the primary key
- **unenforced_primary_key_position** (uint32): Position in primary key ordering (1-based for ordered, 0 for unordered)

For detailed discussion on primary key configuration, see [Unenforced Primary Key](index.md#unenforced-primary-key) in the table format overview.

### Clustering Key Metadata

Clustering key configuration uses a single protobuf field in the Field message:
- **unenforced_clustering_key_position** (uint32): 1-based position in clustering key ordering. 0 means not a clustering key field.

Clustering keys hint at the physical ordering of data within a table. Unlike primary keys,
clustering key fields may be nullable. This metadata enables query engines to perform
optimizations such as storage-partitioned joins.

### Encoding Metadata

Column encoding configurations are specified with the `lance-encoding:` prefix.
See [File Format Encoding Specification](../file/encoding.md) for complete details on available encodings.

### Arrow Extension Type Metadata

Custom Arrow extension types may have metadata under the `ARROW:extension:` namespace
(e.g., `ARROW:extension:name`).

## Schema Protobuf Definition

The schema is serialized using protobuf messages. Key messages include:

### Field Message

```protobuf
%%% proto.message.lance.file.Field %%%
```

The Field message contains:
- **id**: Unique field identifier (int32)
- **name**: Field name (string)
- **type**: Field type enum (PARENT, REPEATED, or LEAF)
- **logical_type**: Logical type string representation (string) - e.g., "int64", "struct", "list"
- **nullable**: Whether the field can be null (bool)
- **parent_id**: Parent field ID for nested fields; -1 for top-level fields (int32)
- **metadata**: Key-value pairs for additional configuration (map<string, bytes>)
- **unenforced_primary_key**: Whether this field is part of the primary key (bool)
- **unenforced_primary_key_position**: Position in primary key ordering (uint32, 0 = unordered)

### Schema Message

The complete schema is represented as a collection of top-level fields plus metadata.

## Schema Evolution

Field IDs enable efficient schema evolution:

- **Add Column**: Assign a new field ID and add to schema
- **Drop Column**: Remove field from schema; its ID may be reused in some systems
- **Rename Column**: Change field name; ID remains the same
- **Reorder Columns**: Change field order in schema; IDs remain the same
- **Type Evolution**: Data type can be changed. This might require rewriting the column in the data, depending on how the type was changed.

The use of field IDs ensures that data files can be correctly interpreted even as the schema changes over time.

## Example Schemas

The examples below use a simplified representation of the field structure. In the actual protobuf format, `type` refers to the field type enum (PARENT/REPEATED/LEAF) and `logical_type` contains the data type string representation.

### Simple Table

```
Field {
    id: 0
    name: "id"
    logical_type: "int64"
    nullable: false
    parent_id: -1
}
Field {
    id: 1
    name: "name"
    logical_type: "string"
    nullable: true
    parent_id: -1
}
Field {
    id: 2
    name: "created_at"
    logical_type: "timestamp:us:UTC"
    nullable: true
    parent_id: -1
}
```

### Nested Structure

```
Field {
    id: 0
    name: "id"
    logical_type: "int64"
    nullable: false
    parent_id: -1  // Top-level field
}
Field {
    id: 1
    name: "user"
    logical_type: "struct"
    nullable: true
    parent_id: -1  // Top-level field
}
Field {
    id: 2
    name: "name"
    logical_type: "string"
    nullable: true
    parent_id: 1  // Nested under "user" struct (id: 1)
}
Field {
    id: 3
    name: "email"
    logical_type: "string"
    nullable: true
    parent_id: 1  // Nested under "user" struct (id: 1)
}
Field {
    id: 4
    name: "tags"
    logical_type: "list"
    nullable: true
    parent_id: -1  // Top-level field
}
Field {
    id: 5
    name: "item"
    logical_type: "string"
    nullable: true
    parent_id: 4  // Nested under "tags" list (id: 4)
}
```

### With Vector Embeddings

```
Field {
    id: 0
    name: "id"
    logical_type: "int64"
    nullable: false
    parent_id: -1  // Top-level field
    unenforced_primary_key: true
    unenforced_primary_key_position: 1  // Ordered position in primary key
}
Field {
    id: 1
    name: "text"
    logical_type: "string"
    nullable: true
    parent_id: -1  // Top-level field
}
Field {
    id: 2
    name: "embedding"
    logical_type: "fixed_size_list:lance.bfloat16:384"
    nullable: true
    parent_id: -1  // Top-level field
}
```

## Type Conversion Reference

When converting between logical types and Arrow types, Lance uses the following mappings:

| Arrow Type | Logical Type Format |
|---|---|
| `Arrow::Null` | `null` |
| `Arrow::Boolean` | `bool` |
| `Arrow::Int8` to `Int64` | `int8`, `int16`, `int32`, `int64` |
| `Arrow::UInt8` to `UInt64` | `uint8`, `uint16`, `uint32`, `uint64` |
| `Arrow::Float16` | `halffloat` |
| `Arrow::Float32` | `float` |
| `Arrow::Float64` | `double` |
| `Arrow::Utf8` | `string` |
| `Arrow::LargeUtf8` | `large_string` |
| `Arrow::Binary` | `binary` |
| `Arrow::LargeBinary` | `large_binary` |
| `Arrow::Decimal128(p, s)` | `decimal:128:p:s` |
| `Arrow::Decimal256(p, s)` | `decimal:256:p:s` |
| `Arrow::Date32` | `date32:day` |
| `Arrow::Date64` | `date64:ms` |
| `Arrow::Time32(TimeUnit)` | `time32:s`, `time32:ms` |
| `Arrow::Time64(TimeUnit)` | `time64:us`, `time64:ns` |
| `Arrow::Timestamp(unit, tz)` | `timestamp:unit:tz` |
| `Arrow::Duration(unit)` | `duration:s`, `duration:ms`, `duration:us`, `duration:ns` |
| `Arrow::Struct` | `struct` |
| `Arrow::List(Element)` | `list` or `list.struct` if element is Struct |
| `Arrow::LargeList(Element)` | `large_list` or `large_list.struct` |
| `Arrow::FixedSizeList(Element, Size)` | `fixed_size_list:type:size` |
| `Arrow::FixedSizeBinary(Size)` | `fixed_size_binary:size` |
| `Arrow::Dictionary(KeyType, ValueType)` | `dict:value_type:key_type:false` |
| `Arrow::Map` | `map` |

transaction.md 30.8 KB

# Transaction Specification

## Transaction Overview

Lance implements Multi-Version Concurrency Control (MVCC) to provide ACID transaction guarantees for concurrent readers and writers.
Each commit creates a new immutable table version through atomic storage operations.
All table versions form a serializable history, enabling features such as time travel and schema evolution.

Transactions are the fundamental unit of change in Lance.
A transaction describes a set of modifications to be applied atomically to create a new table version.
The transaction model supports concurrent writes through optimistic concurrency control with automatic conflict resolution.

## Commit Protocol

### Storage Primitives

Lance commits rely on atomic write operations provided by the underlying object store:

- **rename-if-not-exists**: Atomically rename a file only if the target does not exist
- **put-if-not-exists**: Atomically write a file only if it does not already exist (also known as PUT-IF-NONE-MATCH or conditional PUT)

These primitives guarantee that exactly one writer succeeds when multiple writers attempt to create the same manifest file concurrently.

### Manifest Naming Schemes

Lance supports two manifest naming schemes:

- **V1**: `{version}.manifest` - Monotonically increasing version numbers (e.g., `1.manifest`, `2.manifest`)
- **V2**: `{u64::MAX - version:020}.manifest` - Reverse-sorted lexicographic ordering (e.g., `18446744073709551614.manifest` for version 1)

The V2 scheme enables efficient discovery of the latest version through lexicographic object listing.

### Transaction Files

Transaction files store the serialized transaction protobuf message for each commit attempt.
These files serve two purposes:

1. Enable manifest reconstruction during commit retries when concurrent transactions have been committed
2. Support conflict detection by describing the operation performed

### Commit Algorithm

The commit process attempts to atomically write a new manifest file using the storage primitives described above.
When concurrent writers conflict, the system loads transaction files to detect conflicts and attempts to rebase the transaction if possible.
If the atomic commit fails, the process retries with updated transaction state.
For detailed conflict detection and resolution mechanisms, see the [Conflict Resolution](#conflict-resolution) section.

## Transaction Types

The authoritative specification for transaction types is defined in [`protos/transaction.proto`](https://github.com/lancedb/lance/blob/main/protos/transaction.proto).

Each transaction contains a `read_version` field indicating the table version from which the transaction was built,
a `uuid` field uniquely identifying the transaction, and an `operation` field specifying one of the following transaction types:

In the following section, we will describe each transaction type and its compatibility with other transaction types. This
compatibility is not always bi-directional. We are describing it from the perspective of the operation being committed. For example, we say that an Append is not compatible with an Overwrite which means that if we are trying to commit an Append, and an
Overwrite has already been committed (since we started the Append), then the Append will fail. On the other hand, when describing the
Overwrite operation, we say that it does not conflict with Append. This is because, if we are trying to commit an Overwrite, and an
Append operation has occurred in the meantime, we still allow the Overwrite to proceed.

### Append

Adds new fragments to the table without modifying existing data.
Fragment IDs are not assigned at transaction creation time; they are assigned during manifest construction.

<details>
<summary>Append protobuf message</summary>

```protobuf
%%% proto.message.Append %%%
```

</details>

#### Append Compatibility

The append operation is one of the most common operations and is designed to be compatible with most other operations, even
itself. This is to ensure that multiple writers can append without worry about conflicts. These are the operations
that conflict with append:

- Overwrite
- Restore
- UpdateMemWalState

### Delete

Marks rows as deleted using deletion vectors.
May update fragments (adding deletion vectors) or delete entire fragments.
The `predicate` field stores the deletion condition, enabling conflict detection with concurrent transactions.

<details>
<summary>Delete protobuf message</summary>

```protobuf
%%% proto.message.Delete %%%
```

</details>

#### Delete Compatibility

Delete modifies an existing fragment, so there may be conflicts with other operations on overlapping fragments.
Generally these conflicts are rebaseable or retryable.

These are the operations that conflict with delete:

- Overwrite
- Restore
- UpdateMemWalState

These operations conflict with delete but can be retried:

- Merge (only if there are overlapping fragments)
- Rewrite (only if there are overlapping fragments)
- DataReplacement (only if there are overlapping fragments)

These operations conflict with delete but can potentially be rebased. The deletion
masks from the two operations will be merged. However, if both operations modified
the same rows, then the conflict becomes a retryable conflict.

- Delete
- Update

### Overwrite

Creates or completely overwrites the table with new data, schema, and configuration.

<details>
<summary>Overwrite protobuf message</summary>

```protobuf
%%% proto.message.Overwrite %%%
```

</details>

#### Overwrite Compatibility

An overwrite operation completely overwrites the table. Generally, we do not care what has happened since
the read version.

However, the overwrite does not necessarily rewrite the table config. As a result, we consider the following
to be retryable conflicts:

- UpdateConfig (only if the two operations modify the same config key)
- Overwrite (always)
- UpdateMemWalState (always)

### CreateIndex

Adds, replaces, or removes secondary indices (vector indices, scalar indices, full-text search indices).

<details>
<summary>CreateIndex protobuf message</summary>

```protobuf
%%% proto.message.CreateIndex %%%
```

</details>

#### CreateIndex Compatibility

Indexes record which fragments are covered by the index and we don't require all fragments be covered. As a result, it
is typically ok for an index to be created concurrently with the addition of new fragments. These new fragments will simply
be unindexed.

Updates and deletes are also compatible with index creation. This is because it is ok for an index to refer to deleted rows.
Those results will be filtered out after the index search. If an update occurs then the old value will be filtered out and the
new value will be considered part of the unindexed set.

If two CreateIndex operations are committed concurrently then it is allowed. If the indexes have different names this is no
problem. If the indexes have the same name then the second operation will win and replace the first.

These operations conflict with index creation:

- Overwrite
- Restore
- UpdateMemWalState

Data replacement operations will conflict with index creation if the column being replaced is being indexed. Rewrite operations
will conflict with index creation if the rewritten fragments are covered by the index. This is because an index refers to row
addresses and the rewrite operation changes the row addresses. However, if a fragment reuse index is being used, or if the stable
row ids feature is enable, then the rewrite operation is compatible with index creation. As a result, these are the operations
that are retryable conflicts with index creation:

- Rewrite (only if overlapping fragments, no stable row ids, and no fragment reuse index)
- DataReplacement (only if overlapping fragments and the column being replaced is being indexed)

Some indices are special singleton indices. For example, the fragment reuse index and the mem wal index. If a conflict occurs
between two operations that are modifying the same singleton index, then we must rebase the operation and merge the indexes.
As a result, these are the operations that are rebaseable conflicts with index creation:

- CreateIndex (only if both operations are modifying the same singleton index)

### Rewrite

Reorganizes data without semantic modification.
This includes operations such as compaction, defragmentation, and re-ordering.
Rewrite operations change row addresses, requiring index updates.
New fragment IDs must be reserved via `ReserveFragments` before executing a `Rewrite` transaction.

A rewrite that defers index remapping publishes its address mapping in the
same commit by replacing the [Fragment Reuse Index](../index/system/frag_reuse.md)
entry in the manifest's index section.

<details>
<summary>Rewrite protobuf message</summary>

```protobuf
%%% proto.message.Rewrite %%%
```

</details>

#### Rewrite Compatibility

Rewrite operations do not change data but they can materialize deletions and they do replace fragments. As a result,
they can potentially conflict with other operations that modify the fragments being rewritten.

These are the operations that conflict with rewrite:

- Overwrite
- Restore

Rewrite is not compatible with CreateIndex by default because the operation will change the row addresses that the CreateIndex
refers to. However, a fragment reuse index or the stable row ids feature can allow these operations to be compatible.

Several operations modify existing fragments. As a result, they can potentially conflict with Rewrite if they modify
the same fragments. However, Merge is [overly general](#overly-general-operation) and so no conflict detection is possible.
As a result, here are the operations that are retryable conflicts with Rewrite:

- Merge (always)
- DataReplacement (only if overlapping fragments)
- Delete (only if overlapping fragments)
- Update (only if overlapping fragments)
- Rewrite (if overlapping fragments or both carry a fragment reuse index)
- CreateIndex (overlapping fragments and no fragment reuse index or stable row ids)

There is one case where a Rewrite will rebase. This is when the Rewrite operation has a fragment reuse index and there is
a CreateIndex operation that is writing the fragment reuse index. In this case the Rewrite will rebase and update its
fragment reuse index to include the conflicting fragment reuse index.

As a result, these are the operations that are rebaseable conflicts with Rewrite:

- CreateIndex (if the CreateIndex is writing the fragment reuse index and the Rewrite is carrying a fragment reuse index)

### Merge

Adds new columns to the table, modifying the schema.
All fragments must be updated to include the new columns.

<details>
<summary>Merge protobuf message</summary>

```protobuf
%%% proto.message.Merge %%%
```

</details>

#### Overly General Operation

The Merge operation is a very generic operation. The set of fragments provided in the operation will be the final set of
fragments in the resulting dataset. As a result, it has a high potential for conflicts with other operations. If possible,
more restrictive operations such as Rewrite, DataReplacement, or Append should be preferred over Merge.

#### Merge Compatibility

As mentioned above, Merge is a very generic operation, as a result it has a high potential for conflicts with other operations.
The following operations conflict with Merge:

- Overwrite
- Restore
- UpdateMemWalState
- Project

These operations are retryable conflicts with Merge:

- Update (always)
- Append (always)
- Delete (always)
- Merge (always)
- Rewrite (always)
- DataReplacement (always)

### Project

Removes columns from the table, modifying the schema.
This is a metadata-only operation; data files are not modified.

<details>
<summary>Project protobuf message</summary>

```protobuf
%%% proto.message.Project %%%
```

</details>

#### Project Compatibility

Since project only modifies the schema, it is compatible with most other operations. However, it is not compatible with Merge
because the Merge operation modifies the schema (can potentially add columns) and the logic to rebase those changes does not
currently exist (project is cheap and easy enough to retry).

These are the operations that conflict with Project:

- Overwrite
- Restore
- UpdateMemWalState

The following operations are retryable conflicts with Project:

- Project (always)
- Merge (always)

### Restore

Reverts the table to a previous version.

<details>
<summary>Restore protobuf message</summary>

```protobuf
%%% proto.message.Restore %%%
```

</details>

#### Restore Compatibility

The Restore operation reverts the table to a previous version. It's generally assumed this trumps any
other operation. Here are the operations that conflict with Restore:

- UpdateMemWalState

### ReserveFragments

Pre-allocates fragment IDs for use in future `Rewrite` operations.
This allows rewrite operations to reference fragment IDs before the rewrite transaction is committed.

<details>
<summary>ReserveFragments protobuf message</summary>

```protobuf
%%% proto.message.ReserveFragments %%%
```

</details>

#### ReserveFragments Compatibility

The ReserveFragments operation is fairly trivial. The only thing it changes is the max fragment id. So this
only conflicts with operations that modify the max fragment id. Here are the operations that conflict with ReserveFragments:

- Overwrite
- Restore

### Clone

Creates a shallow or deep copy of the table.
Shallow clones are metadata-only copies that reference original data files through `base_paths`.
Deep clones are full copies using object storage native copy operations (e.g., S3 CopyObject).

<details>
<summary>Clone protobuf message</summary>

```protobuf
%%% proto.message.Clone %%%
```

</details>

#### Clone Compatibility

The Clone operation can only be the first operation in a dataset. If there is an existing dataset, then the Clone operation will fail.
As a result, there is no such thing as a conflict with Clone.

### Update

Modifies row values without adding or removing rows.
Supports two execution modes: REWRITE_ROWS deletes rows in current fragments and rewrites them in new fragments, which is optimal when the majority of columns are modified or only a small number of rows are affected; REWRITE_COLUMNS fully rewrites affected columns within fragments by tombstoning old column versions, which is optimal when most rows are affected but only a subset of columns are modified.

<details>
<summary>Update protobuf message</summary>

```protobuf
%%% proto.message.Update %%%
```

</details>

#### Update Compatibility

Here are the operations that conflict with Update:

- Overwrite
- Restore

An update operation is both a delete and an append operation. Like a Delete operation, it will modify fragments to change
the deletion mask. As a result, there will be a retryable conflict with other operations that modify the same fragments.
Here are the operations that are retryable conflicts with Update:

- Rewrite (only if overlapping fragments)
- DataReplacement (only if overlapping fragments)
- Merge (always)

Similar to Delete, the Update operation can rebase other modifications to the deletion mask. Here are the operations that
are rebaseable conflicts with Update:

- Delete
- Update

### UpdateConfig

Modifies table configuration, table metadata, schema metadata, or field metadata without changing data.

<details>
<summary>UpdateConfig protobuf message</summary>

```protobuf
%%% proto.message.UpdateConfig %%%
```

</details>

#### UpdateConfig Compatibility

An UpdateConfig operation only modifies table config and tends to be compatible with other operations. Here
are the operations that conflict with UpdateConfig:

- Overwrite
- UpdateConfig (only if the two operations modify the same config)

### DataReplacement

Replaces data in specific column regions with new data files.

<details>
<summary>DataReplacement protobuf message</summary>

```protobuf
%%% proto.message.DataReplacement %%%
```

</details>

#### DataReplacement Compatibility

A DataReplacement operation only replaces a single column's worth of data. As a result, it can be safer and simpler than Merge
or Update operations. It rewrites a column file positionally against the fragments it targets, so a concurrent operation only
conflicts when it removes one of those fragments or invalidates the rows the column file covers. Here are the operations that
conflict with DataReplacement (non-retryable):

- Overwrite
- Restore
- UpdateMemWalState
- Delete (only if it removes a target fragment outright)
- Update (only if it removes a target fragment outright)

The following operations are retryable conflicts with DataReplacement:

- DataReplacement (only if same field and overlapping fragments)
- CreateIndex (only if the field being replaced is being indexed)
- Rewrite (only if overlapping fragments)
- Update (only if it rewrites rows out of a target fragment, or rewrites one of the replaced fields in place)
- Merge (always)

A concurrent Delete or Update that only adds a deletion vector to a target fragment (without removing it) is compatible: the
positional column file stays aligned and the rebase preserves the deletion vector.

### DataOverlay

Attaches [overlay files](data_overlay_file.md) to fragments, supplying new values
for a subset of `(row offset, field)` cells without rewriting the fragments' base
data files. The overlays are appended to each fragment's existing `overlays` list,
so overlays written by concurrent commits are preserved. Each overlay's
`committed_version` is stamped to the new dataset version at commit time (and
re-stamped on retry), like the created-at / last-updated-at version sequences.

<details>
<summary>DataOverlay protobuf message</summary>

```protobuf
%%% proto.message.DataOverlay %%%

%%% proto.message.DataOverlayGroup %%%
```

</details>

#### DataOverlay Compatibility

A DataOverlay operation only changes cells within existing fragments and preserves
physical row addresses, so — like DataReplacement — it is intentionally permissive.
Because overlays stack and the higher `committed_version` wins each covered cell,
independent backfills never conflict, and a concurrent Delete simply makes the
overlay value for a deleted offset inert. Here are the operations that conflict
with DataOverlay:

- Overwrite
- Restore
- UpdateMemWalState

The following operations are retryable conflicts with DataOverlay:

- Rewrite (only if overlapping fragments) — row-rewriting compaction or an
  overlay→base fold changes physical row addresses or consumes the overlays, so
  the overlay's offsets are no longer valid; the writer must re-read the new
  fragment, recompute, and retry.
- Merge (always).
- A row-moving Update that touches an overlaid fragment — a delete-and-reinsert
  update (any update that is not a `REWRITE_COLUMNS` column rewrite) relocates the
  updated rows into new fragments, so the overlay's physical offsets no longer
  address them; the writer must re-read and retry.

DataOverlay is compatible with another DataOverlay (any fields), Append, Delete, a
`REWRITE_COLUMNS` column rewrite, and DataReplacement, because all of these
preserve physical row addresses: overlay offsets stay valid, the overlay is newer
and wins its covered cells, and the version gate excludes those cells from any
rebuilt index.

When a DataReplacement or a `REWRITE_COLUMNS` update writes new base values for a
field, it supersedes any older overlay on that field: the writer tombstones the
overlay's entry for the rewritten field — replacing the field id with the obsolete
sentinel, as with obsolete base columns — so the fresh base values are not silently
shadowed. Overlay entries for other fields are preserved, and an overlay left with
no live fields is dropped.

### UpdateMemWalState

Updates the state of MemWal indices (write-ahead log based indices).

<details>
<summary>UpdateMemWalState protobuf message</summary>

```protobuf
%%% proto.message.UpdateMemWalState %%%
```

</details>

### UpdateBases

Adds new base paths to the table, enabling reference to data files in additional locations.

<details>
<summary>UpdateBases protobuf message</summary>

```protobuf
%%% proto.message.UpdateBases %%%
```

</details>

#### UpdateBases Compatibility

An UpdateBases operation only modifies the base paths. As a result, it only conflicts with other
UpdateBases operations and even then only conflicts if the two operations have base paths with the
same id, name, or path.

## Conflict Resolution

### Terminology

When concurrent transactions attempt to commit against the same read version, Lance employs conflict resolution to determine whether the transactions can coexist.
Three outcomes are possible:

- **Rebasable**: The transaction can be modified to incorporate concurrent changes while preserving its semantic intent.
  The transaction is transformed to account for the concurrent modification, then the commit is retried automatically within the commit layer.

- **Retryable**: The transaction cannot be rebased, but the operation can be re-executed at the application level with updated data.
  The implementation returns a retryable conflict error, signaling that the application should re-read the data and retry the operation.
  The retried operation is expected to produce semantically equivalent results.

- **Incompatible**: The transactions conflict in a fundamental way where retrying would violate the operation's assumptions or produce semantically different results than expected.
  The commit fails with a non-retryable error.
  Callers should proceed with extreme caution if they decide to retry, as the transaction may produce different output than originally intended.

### Rebase Mechanism

The `TransactionRebase` structure tracks the state necessary to rebase a transaction against concurrent commits:

1. **Fragment tracking**: Maintains a map of fragments as they existed at the transaction's read version, marking which require rewriting
2. **Modification detection**: Tracks the set of fragment IDs that have been modified or deleted
3. **Affected rows**: For Delete and Update operations, stores the specific rows affected by the operation for fine-grained conflict detection
4. **Fragment reuse indices**: Accumulates fragment reuse index metadata from concurrent Rewrite operations

When a concurrent transaction is detected, the rebase process:

1. Compares fragment modifications to determine if there is overlap
2. For Delete/Update operations, compares `affected_rows` to detect whether the same rows were modified
3. Merges deletion vectors when both transactions delete rows from the same fragment
4. Accumulates fragment reuse index updates when concurrent Rewrites change fragment IDs
5. Modifies the transaction if rebasable, or returns a retryable/incompatible conflict error

### Conflict Scenarios

#### Rebasable Conflict Example

The following diagram illustrates a rebasable conflict where two Delete operations modify different rows in the same fragment:

```mermaid
gitGraph
    commit id: "v1"
    commit id: "v2"
    branch writer-a
    branch writer-b
    checkout writer-a
    commit id: "Delete rows 100-199" tag: "read_version=2"
    checkout writer-b
    commit id: "Delete rows 500-599" tag: "read_version=2"
    checkout main
    merge writer-a tag: "v3"
    checkout writer-b
    commit id: "Rebase: merge deletion vectors" type: HIGHLIGHT
    checkout main
    merge writer-b tag: "v4"
```

In this scenario:

- Writer A deletes rows 100-199 and successfully commits version 3
- Writer B attempts to commit but detects version 3 exists
- Writer B's transaction is rebasable because it only modified deletion vectors (not data files) and `affected_rows` do not overlap
- Writer B rebases by merging Writer A's deletion vector with its own, write it to storage
- Writer B successfully commits version 4

#### Retryable Conflict Example

The following diagram illustrates a retryable conflict where an Update operation encounters a concurrent Rewrite (compaction) that prevents automatic rebasing:

```mermaid
gitGraph
    commit id: "v1"
    commit id: "v2"
    branch writer-a
    branch writer-b
    checkout writer-a
    commit id: "Compact fragments 1-5" tag: "read_version=2"
    checkout writer-b
    commit id: "Update rows in fragment 3" tag: "read_version=2"
    checkout main
    merge writer-a tag: "v3: fragments compacted"
    checkout writer-b
    commit id: "Detect conflict: cannot rebase" type: REVERSE
```

In this scenario:

- Writer A compacts fragments 1-5 into a single fragment and successfully commits version 3
- Writer B attempts to update rows in fragment 3 but detects version 3 exists
- Writer B's Update transaction is retryable but not rebasable: fragment 3 no longer exists after compaction
- The commit layer returns a retryable conflict error
- The application must re-execute the Update operation against version 3, locating the rows in the new compacted fragment

#### Incompatible Conflict Example

The following diagram illustrates an incompatible conflict where a Delete operation encounters a concurrent Restore that fundamentally invalidates the operation:

```mermaid
gitGraph
    commit id: "v1"
    commit id: "v2"
    commit id: "v3"
    branch writer-a
    branch writer-b
    checkout writer-a
    commit id: "Restore to v1" tag: "read_version=3"
    checkout writer-b
    commit id: "Delete rows added in v2-v3" tag: "read_version=3"
    checkout main
    merge writer-a tag: "v4: restored to v1"
    checkout writer-b
    commit id: "Detect conflict: incompatible" type: REVERSE
```

In this scenario:

- Writer A restores the table to version 1 and successfully commits version 4
- Writer B attempts to delete rows that were added between versions 2 and 3
- Writer B's Delete transaction is incompatible: the table has been restored to version 1, and the rows it intended to delete no longer exist
- The commit fails with a non-retryable error
- If the caller retries the deletion operation against version 4, it would either delete nothing (if those rows don't exist in v1) or delete different rows (if similar row IDs exist in v1), producing semantically different results than originally intended

## External Manifest Store

If the backing object store does not support atomic operations (rename-if-not-exists or put-if-not-exists), an external manifest store can be used to enable concurrent writers.

An external manifest store is a key-value store that supports put-if-not-exists operations.
It is the concurrency coordinator and fast version index: its conditional write selects one
immutable staging manifest for each version. The canonical manifest bytes in object storage
remain authoritative, so the external store supplements but does not replace them.
A reader unaware of the external manifest store can still read the table, but may observe a version up to one commit behind the true latest version.

### Commit Process with External Store

The commit process follows a four-step protocol:

![External Store Commit Process](../../images/external_store_commit.gif)

1. **Stage manifest**: `PUT_OBJECT_STORE {dataset}/_versions/{version}.manifest-{uuid}`
   - Write the new manifest to object storage under a unique path determined by a new UUID
   - This staged manifest is not yet visible to readers

2. **Reserve version in external store**: `PUT_EXTERNAL_STORE base_uri, version, {dataset}/_versions/{version}.manifest-{uuid}`
   - Atomically reserve the version for this staged manifest using put-if-not-exists
   - The reservation selects one immutable staging object; it is not yet the canonical commit
   - If this operation fails due to conflict, another writer reserved this version

3. **Finalize in object store**: `COPY_OBJECT_STORE {dataset}/_versions/{version}.manifest-{uuid} → {dataset}/_versions/{version}.manifest`
   - Copy the staged manifest to the final path
   - Successful materialization at this deterministic path is the commit point
   - This makes the manifest discoverable by readers unaware of the external store

4. **Update external store pointer**: `PUT_EXTERNAL_STORE base_uri, version, {dataset}/_versions/{version}.manifest`
   - Update the external store to point to the finalized manifest path
   - After copying, read the canonical object's current metadata. Return its ETag to the caller as
     an opaque physical-generation observation so runtime caches do not collapse a newly committed
     Dataset into an older cached Dataset at the same URI and version
   - Do not persist that ETag in the external store. Concurrent finalizers can copy the same selected
     immutable bytes into different physical generations, and COPY plus external-store publication
     is not atomic. Every helper therefore publishes the same stable path-and-size tuple
   - Completes the synchronization between external store and object storage

**Fault Tolerance:**

If the writer fails after step 2 but before step 3, the external store contains a pending
reservation. Readers that use the external store detect this state and retry materialization.
If step 3 succeeds but step 4 fails, the canonical object remains committed; readers use it and
may repair the external index. Staging deletion is garbage collection and does not affect the
commit outcome.

**Rolling Upgrade:**

Roll this behavior out normally across the fleet. New readers ignore legacy stored
ETags, and legacy readers already accept finalized rows without an ETag, so mixed-version rows
remain compatible. While both legacy finalizers and legacy readers remain, the pre-existing race
can still republish a stale ETag that a legacy reader rejects. Full protection takes effect when
the rolling upgrade converges; no row migration or quiesced cutover is required.

### Reader Process with External Store

The reader follows a validation and synchronization protocol:

![External Store Reader Process](../../images/external_store_reader.gif)

1. **Query external store**: `GET_EXTERNAL_STORE base_uri, version` → `path`
   - Retrieve the manifest path for the requested version
   - If the path does not end with a UUID, validate the canonical object's size. Ignore any legacy
     stored ETag because it is neither content identity nor dataset-incarnation identity; the
     validation HEAD still returns the current canonical ETag to the caller
   - If the path ends with a UUID, synchronization is required

2. **Synchronize to object store**: `COPY_OBJECT_STORE {dataset}/_versions/{version}.manifest-{uuid} → {dataset}/_versions/{version}.manifest`
   - Attempt to finalize the staged manifest
   - This operation is idempotent

3. **Update external store**: `PUT_EXTERNAL_STORE base_uri, version, {dataset}/_versions/{version}.manifest`
   - Best-effort record the finalized path and size without an ETag while returning the observed
     destination ETag to the current caller
   - If this index repair fails, retain staging so a future reader can retry it

4. **Return finalized path**: Return `{dataset}/_versions/{version}.manifest`
   - Return once canonical materialization succeeds, even if index repair or staging cleanup fails
   - If canonical materialization cannot be established, or an observed size differs, return an error

This protocol ensures that datasets using external manifest stores remain portable: copying the dataset directory preserves all data without requiring the external store.

versioning.md 3.8 KB

# Format Versioning

## Feature Flags

As the table format evolves, new feature flags are added to the format.
There are two separate fields for checking for feature flags,
depending on whether you are trying to read or write the table.
Readers should check the `reader_feature_flags` to see if there are any flag it is not aware of.
Writers should check `writer_feature_flags`. If either sees a flag they don't know,
they should return an "unsupported" error on any read or write operation.

## Current Feature Flags

<style>
.feature-flags-table th:nth-child(2),
.feature-flags-table td:nth-child(2) {
  white-space: nowrap;
  min-width: 250px;
}
</style>

<div class="feature-flags-table" markdown="1">

| Flag Bit | Flag Name                       | Reader Required | Writer Required | Description                                                                                                 |
|----------|---------------------------------|-----------------|-----------------|-------------------------------------------------------------------------------------------------------------|
| 1        | `FLAG_DELETION_FILES`           | Yes             | Yes             | Fragments may contain deletion files, which record the tombstones of soft-deleted rows.                     |
| 2        | `FLAG_STABLE_ROW_IDS`           | Yes             | Yes             | Row IDs are stable for both moves and updates. Fragments contain an index mapping row IDs to row addresses. |
| 4        | `FLAG_USE_V2_FORMAT_DEPRECATED` | No              | No              | Files are written with the new v2 format. This flag is deprecated and no longer used.                       |
| 8        | `FLAG_TABLE_CONFIG`             | No              | Yes             | Table config is present in the manifest.                                                                    |
| 16       | `FLAG_BASE_PATHS`               | Yes             | Yes             | Dataset uses multiple base paths (for shallow clones or multi-base datasets).                               |
| 32       | `FLAG_DISABLE_TRANSACTION_FILE` | No              | Yes             | Transactions are recorded in the manifest rather than in a separate transaction file.                       |
| 64       | `FLAG_UNSTABLE_DATA_OVERLAY_FILES` | Yes          | Yes             | Fragments may carry data overlay files. Unstable: release builds reject it unless explicitly opted in.      |
| 128      | `FLAG_COVERED_INDEX_METADATA`   | Yes             | Yes             | Some index declares covering columns (`IndexMetadata.covering_fields`), so `fields` means keyed columns followed by carried ones. An implementation without this flag selects an index by membership of `fields` and would answer a query on a merely-carried column with an index keyed on a different one. |
| 256      | `FLAG_MIXED_DATA_FILE_VERSIONS` | Yes             | Yes             | The snapshot may reference recognized V2 data files with different exact versions. Both bits must be set and remain set on later versions. |
| 1024     | `FLAG_FRAGMENT_REUSE_INDEX`     | Yes             | Yes             | The fragment reuse index records tagged transitions (`IndexMetadata.index_version >= 1`). Readers must translate row addresses through them; writers must preserve them. An implementation without this flag would decode the details as the legacy format and silently drop the transitions when it next rewrites the fragment reuse index. See [FRI index versions](../index/system/frag_reuse.md#fri-index-versions). |

</div>

Flag bit 512 is reserved. Flags with bit values 2048 and above are unknown; unknown flags cause implementations to reject the dataset with an "unsupported" error. The paired mixed-version reader and writer bits must either both be set or both be clear; a half-set manifest is invalid.

index.md 3.4 KB

# Lance Lakehouse Format Specifications

Lance is a lakehouse format defined as a stack of interoperating specifications, rather than as a single file format or metadata layout. The storage-facing layers cover files, tables, indices, and catalogs. A unified namespace interface sits above those layers and gives engines a consistent way to work with Lance tables across catalog implementations.

## Architecture Overview

Modern lakehouses are built from complementary layers. Lance keeps those layers intentionally decoupled so that the file format, table metadata, indices, and catalogs can evolve independently without forcing lock-in across the stack.

![Lakehouse Stack](../images/lakehouse_stack.png)

At a high level:

- The **file format** stores column data in large pages optimized for random access and avoids row groups.
- The **table format** manages fragments, manifests, deletions, schema evolution, and ACID commits.
- The **index formats** define redundant search structures such as scalar, vector, full-text, and system indices.
- The **catalog specs** define how tables are discovered, registered, and coordinated across engines and services.
- The **namespace client spec** provides a unified interface for engines to interact with any catalog implementations.

The layers are designed so that only table readers, table writers, and index readers or writers need to understand the on-disk Lance file layout.

## Design Themes

### File Format

The Lance file format is optimized for cloud object storage and highly selective reads. It avoids Parquet-style row groups, uses structural encodings for efficient random access, and keeps statistics and search structures out of the file format so those concerns can evolve independently as indices.

### Table Format

The Lance table format organizes data in two dimensions: rows are grouped into fragments, and each fragment can contain multiple data files, each contributing a subset of columns. This makes column additions and backfills primarily metadata operations instead of data rewrites, which is especially useful for feature engineering and embedding workflows.

### Index Formats

Indices are first-class table objects. Lance tables define how indices are discovered, versioned, and coordinated transactionally. The index formats themselves remain decoupled from both the file encoding and the table manifest structure.

### Catalog Specs

Lance provides both storage-native and service-oriented catalog options. The [Directory Catalog](catalog/dir/index.md) supports zero-infrastructure deployments directly on object stores, while the [REST Catalog](catalog/rest/index.md) standardizes enterprise-facing APIs and can act as an external manifest store.

### Namespace Client Spec

The [Namespace Client Spec](namespace/index.md) provides a language-agnostic interface for engines to interact with any catalog implementation, including Lance-native catalogs and third-party catalog systems. This abstraction allows applications to switch between directory-based, REST-based, and third-party catalogs without changing their code.

## Specifications

The main specification entry points are:

1. **File Format**: [Lance file format](file/index.md)
2. **Table Format**: [Lance table format](table/index.md)
3. **Index Formats**: [Scalar, vector, and system index formats](index/index.md)
4. **Catalog Specs**: [Directory and REST catalog specs](catalog/index.md)
5. **Namespace Client Spec**: [Lance namespace interface](namespace/index.md)

guide

arrays.md 6.5 KB

# Extension Arrays

Lance provides extensions for Arrow arrays and Pandas Series to represent data
types for machine learning applications.

## BFloat16

[BFloat16](https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus)
is a 16-bit floating point number that is designed for machine learning use cases.
Intuitively, it only has 2-3 digits of precision, but it has the same range as
a 32-bit float: ~1e-38 to ~1e38. By comparison, a 16-bit float has a range of
~5.96e-8 to 65504.

Lance provides an Arrow extension array (`lance.arrow.BFloat16Array`)
and a Pandas extension array (`lance._arrow.PandasBFloat16Type`) for BFloat16.
These are compatible with the [ml_dtypes](https://github.com/jax-ml/ml_dtypes)
bfloat16 NumPy extension array.

If you are using Pandas, you can use the `lance.bfloat16` dtype string to create
the array:

```python
import lance.arrow

pd.Series([1.1, 2.1, 3.4], dtype="lance.bfloat16")
# 0    1.1015625
# 1      2.09375
# 2      3.40625
# dtype: lance.bfloat16
```

To create an Arrow array, use the `lance.arrow.bfloat16_array` function:

```python
from lance.arrow import bfloat16_array

bfloat16_array([1.1, 2.1, 3.4])
# <lance.arrow.BFloat16Array object at 0x000000016feb94e0>
# [
#   1.1015625,
#   2.09375,
#   3.40625
# ]
```

Finally, if you have a pre-existing NumPy array, you can convert it into either:

```python
import numpy as np
from ml_dtypes import bfloat16
from lance.arrow import PandasBFloat16Array, BFloat16Array

np_array = np.array([1.1, 2.1, 3.4], dtype=bfloat16)
PandasBFloat16Array.from_numpy(np_array)
# <PandasBFloat16Array>
# [1.1015625, 2.09375, 3.40625]
# Length: 3, dtype: lance.bfloat16
BFloat16Array.from_numpy(np_array)
# <lance.arrow.BFloat16Array object at 0x...>
# [
#   1.1015625,
#   2.09375,
#   3.40625
# ]
```

When reading, these can be converted back to to the NumPy bfloat16 dtype using
each array class's `to_numpy` method.

## ImageURI

`lance.arrow.ImageURIArray` is an array that stores the URI location of images
in some other storage system. For example, `file:///path/to/image.png` for a local
filesystem or `s3://bucket/path/image.jpeg` for an image on AWS S3. Use this
array type when you want to lazily load images from an existing storage medium.

It can be created by calling `lance.arrow.ImageURIArray.from_uris`
with a list of URIs represented by either `pyarrow.StringArray` or an
iterable that yields strings. Note that the URIs are not strongly validated and images
are not read into memory automatically.

```python
from lance.arrow import ImageURIArray

ImageURIArray.from_uris([
   "/tmp/image1.jpg",
   "file:///tmp/image2.jpg",
   "s3://example/image3.jpg"
])
# <lance.arrow.ImageURIArray object at 0x...>
# ['/tmp/image1.jpg', 'file:///tmp/image2.jpg', 's3://example/image3.jpg']
```

`lance.arrow.ImageURIArray.read_uris` will read images into memory and return
them as a new `lance.arrow.EncodedImageArray` object.

```python
from lance.arrow import ImageURIArray

relative_path = "images/1.png"
uris = [os.path.join(os.path.dirname(__file__), relative_path)]
ImageURIArray.from_uris(uris).read_uris()
# <lance.arrow.EncodedImageArray object at 0x...>
# [b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00...']
```

## EncodedImage

`lance.arrow.EncodedImageArray` is an array that stores jpeg and png images in
their encoded and compressed representation as they would appear written on disk.
Use this array when you want to manipulate images in their compressed format such as
when you're reading them from disk or embedding them into HTML.

It can be created by calling `lance.arrow.ImageURIArray.read_uris` on an existing
`lance.arrow.ImageURIArray`. This will read the referenced images into memory.
It can also be created by calling `lance.arrow.ImageArray.from_array` and passing
it an array of encoded images already read into `pyarrow.BinaryArray` or by
calling `lance.arrow.ImageTensorArray.to_encoded`.

A `lance.arrow.EncodedImageArray.to_tensor` method is provided to decode
encoded images and return them as `lance.arrow.FixedShapeImageTensorArray`, from
which they can be converted to numpy arrays.
For decoding images, it will first attempt to use a decoder provided via the optional
function parameter. If decoder is not provided it will attempt to use
[Pillow](https://pillow.readthedocs.io/en/stable/). If neither Pillow nor a custom
decoder is available an exception will be raised.

```python
from lance.arrow import ImageURIArray

uris = [os.path.join(os.path.dirname(__file__), "images/1.png")]
encoded_images = ImageURIArray.from_uris(uris).read_uris()
print(encoded_images.to_tensor())

def pillow_decoder(images):
    import io
    import numpy as np
    from PIL import Image

    return np.stack(
        np.asarray(Image.open(io.BytesIO(img.as_py()))) for img in images.storage
    )

print(encoded_images.to_tensor(pillow_decoder))
# <lance.arrow.FixedShapeImageTensorArray object at 0x...>
# [[42, 42, 42, 255]]
# <lance.arrow.FixedShapeImageTensorArray object at 0x...>
# [[42, 42, 42, 255]]
```

## FixedShapeImageTensor

`lance.arrow.FixedShapeImageTensorArray` is an array that stores images as
tensors where each individual pixel is represented as a numeric value. Typically images
are stored as 3 dimensional tensors shaped (height, width, channels). In color images
each pixel is represented by three values (channels) as per
[RGB color model](https://en.wikipedia.org/wiki/RGB_color_model).
Images from this array can be read out as numpy arrays individually or stacked together
into a single 4 dimensional numpy array shaped (batch_size, height, width, channels).

It can be created by calling `lance.arrow.EncodedImageArray.to_tensor` on a
previously existing `lance.arrow.EncodedImageArray`. This will decode encoded
images and return them as a `lance.arrow.FixedShapeImageTensorArray`. It can also be
created by calling `lance.arrow.ImageArray.from_array` and passing in a
`pyarrow.FixedShapeTensorArray`.

It can be encoded into to `lance.arrow.EncodedImageArray` by calling
`lance.arrow.FixedShapeImageTensorArray.to_encoded` and passing custom encoder
If encoder is not provided it will attempt to use
[Pillow](https://pillow.readthedocs.io/en/stable/). The default encoder will encode
to PNG. If neither Pillow nor a custom encoder is available it will raise an exception.

```python
from lance.arrow import ImageURIArray

uris = [image_uri]
tensor_images = ImageURIArray.from_uris(uris).read_uris().to_tensor()
tensor_images.to_encoded()
# <lance.arrow.EncodedImageArray object at 0x...>
# [...
# b'\x89PNG\r\n\x1a...'
```

blob.md 15.3 KB

# Blob Columns

Lance can store large binary objects (images, videos, audio, model artifacts) in blob columns, where they are treated like any other column payload in the dataset.
Blob columns support both planned full-payload reads and lazy file-like access.

!!! tip "Choosing between `read_blobs` and `take_blobs`"
    - For data loaders and batch processing that need complete byte payloads, use `read_blobs`.
    - Use `take_blobs` when you need a `BlobFile` handle for streaming, seeking, or partial reads.


![Blob v2 overview](../images/blob-v2-overview-light.png#only-light)
![Blob v2 overview](../images/blob-v2-overview-dark.png#only-dark)

If you're unsure about whether you need a blob column in the first place (and why it's useful), read the "[when to use blob column vs. inline binary](#when-to-use-a-blob-column-vs-inline-binary)" section below.

## Quick Start: Blob v2

This page focuses on blob workflows in Python and uses Lance file format terminology.

- `data_storage_version` means the Lance **file format version** of a dataset.
- A dataset's `data_storage_version` is fixed once the dataset is created.
- If you need a different file format version, write a **new dataset**.

```python
import lance
import pyarrow as pa
from lance import blob_array, blob_field

schema = pa.schema([
    pa.field("id", pa.int64()),
    blob_field("blob"),
])

table = pa.table(
    {
        "id": [1],
        "blob": blob_array([b"hello blob v2"]),
    },
    schema=schema,
)

ds = lance.write_dataset(table, "./blobs_v22.lance", data_storage_version="2.2")

_row_address, payload = ds.read_blobs("blob", indices=[0])[0]
assert payload == b"hello blob v2"
```

## Version Compatibility

Blob support is tied to the dataset's file format version. Earlier file format versions
(`< 2.2`) stored blobs using the `lance-encoding:blob` metadata field, while Blob
v2 introduces a new storage layout that requires file format `>= 2.2`.

The two
schemes are mutually exclusive: for file format `>= 2.2`, legacy blob metadata
(`lance-encoding:blob`) is rejected on write. The table below is the single
source of truth for which scheme is supported at each `data_storage_version`.

| Dataset `data_storage_version` | Legacy blob metadata (`lance-encoding:blob`) | Blob v2 (`lance.blob.v2`) |
|---|---|---|
| `0.1`, `2.0`, `2.1` | Supported for write/read | Not supported |
| `2.2+` | Not supported for write | Supported for write/read (recommended) |

## Blob v2: Write Patterns

Use `blob_field` and `blob_array` to build blob v2 columns.

### Logical Arrow schema

A blob v2 field is tagged with `ARROW:extension:name = "lance.blob.v2"`. Writers
accept these logical struct shapes:

| Shape | Children | Use |
|---|---|---|
| Minimal | `data: LargeBinary?`, `uri: Utf8?` | Inline bytes or a complete external object |
| Complete | Minimal fields plus `position: UInt64?`, `size: UInt64?` | An optional byte range within an external object |

Every non-null row must set exactly one of `data` and `uri`. For the complete
shape, `position` and `size` must either both be set or both be null, a range
requires `uri`, and an explicit range must have `size > 0`. Use inline `b""` for
an empty blob; a URI without range fields still represents the complete external
object, including an empty object. Python's `blob_field` and `BlobType` use the
complete shape. Lance preserves an accepted logical shape, including child
fields, nullability, and metadata, across create, append, and merge-insert
writes; descriptor scans still return the compact stored descriptor shape.

```python
import lance
import pyarrow as pa
from lance import Blob, blob_array, blob_field

schema = pa.schema([
    pa.field("id", pa.int64()),
    blob_field("blob", nullable=True),
])

# A single column can mix:
# - inline bytes
# - external URI
# - external URI slice (position + size)
# - null
rows = pa.table(
    {
        "id": [1, 2, 3, 4],
        "blob": blob_array([
            b"inline-bytes",
            "s3://bucket/path/video.mp4",
            Blob.from_uri("s3://bucket/archive.tar", position=4096, size=8192),
            None,
        ]),
    },
    schema=schema,
)

ds = lance.write_dataset(
    rows,
    "./blobs_v22.lance",
    data_storage_version="2.2",
)
```

Note:

- By default, external blob URIs must map to a registered non-dataset-root base path.
- If you need to reference external objects outside those bases, set
  `allow_external_blob_outside_bases=True` when writing.
- Blob v2 storage layout thresholds can be configured per column with
  `blob_field(..., inline_size_threshold=..., dedicated_size_threshold=...)`.
  The inline threshold controls when values move from the data file to packed
  `.blob` sidecar storage. The dedicated threshold controls when values move
  from packed sidecar storage to a dedicated `.blob` file. The dedicated
  threshold is checked first. For existing columns, these thresholds are stored
  in the dataset schema; appends that explicitly provide different threshold
  metadata for the same column are rejected.
- `blob_pack_file_size_threshold` is a write option for rolling packed `.blob`
  sidecar files. It does not control inline-vs-packed placement.
- Blob v2 fields can be nested inside structs and variable-length lists. Blob-aware
  scans preserve the surrounding nested layout; use `blob_handling="all_binary"`
  to materialize nested blob payloads as bytes.

### Example: packed external blobs (single container file)

```python
import io
import tarfile
from pathlib import Path
import lance
import pyarrow as pa
from lance import Blob, blob_array, blob_field

# Build a tar file with three payloads
payloads = {
    "a.bin": b"alpha",
    "b.bin": b"bravo",
    "c.bin": b"charlie",
}

with tarfile.open("container.tar", "w") as tf:
    for name, data in payloads.items():
        info = tarfile.TarInfo(name)
        info.size = len(data)
        tf.addfile(info, io.BytesIO(data))

# Capture offset/size for each member
blob_values = []
with tarfile.open("container.tar", "r") as tf:
    container_uri = Path("container.tar").resolve().as_uri()
    for name in payloads:
        m = tf.getmember(name)
        blob_values.append(Blob.from_uri(container_uri, position=m.offset_data, size=m.size))

schema = pa.schema([
    pa.field("name", pa.utf8()),
    blob_field("blob"),
])

rows = pa.table(
    {
        "name": list(payloads.keys()),
        "blob": blob_array(blob_values),
    },
    schema=schema,
)

ds = lance.write_dataset(
    rows,
    "./packed_blobs_v22.lance",
    data_storage_version="2.2",
    allow_external_blob_outside_bases=True,
)
```

## Blob v2: Read Patterns

Choose the read API based on the payload shape you want:

| API | Returns | Use When |
|---|---|---|
| `read_blobs` | `List[Tuple[int, Optional[bytes]]]` | You need complete blob payloads in memory, such as training loaders or batch preprocessing. |
| `read_blob_ranges` | `List[Tuple[int, int, Optional[bytes]]]` | You need selected byte ranges from multiple rows without materializing complete blobs. |
| `take_blobs` | `List[Optional[BlobFile]]` | You need file-like objects for streaming, seeking, or partial reads. |
| `scanner(..., blob_handling="all_binary")` | Arrow binary columns | You want blob columns in a scan result or `pyarrow.Table`. |

Do not wrap `take_blobs` in your own thread pool just to call `read()` or
`readall()` on every blob. Use `read_blobs` instead; it plans and executes
batched blob reads through Lance's scheduler.

Exactly one selector must be provided to `read_blobs` or `take_blobs`: `ids`,
`indices`, or `addresses`. `read_blob_ranges` accepts the same selector kinds
through its required `selector` argument.

| Selector | Typical Use | Stability |
|---|---|---|
| `indices` | Positional reads within one dataset snapshot | Stable within that snapshot |
| `ids` | Logical row-id based reads | Stable logical identity (when row ids are available) |
| `addresses` | Low-level physical reads and debugging | Unstable physical location |

### Read complete payloads by row indices

```python
import lance

ds = lance.dataset("./blobs_v22.lance")
rows = ds.read_blobs("blob", indices=[0, 1])
payloads = [payload for _row_address, payload in rows]
```

### Read complete payloads by row ids

```python
import lance

ds = lance.dataset("./blobs_v22.lance")
row_ids = ds.to_table(columns=[], with_row_id=True).column("_rowid").to_pylist()

rows = ds.read_blobs("blob", ids=row_ids[:2])
```

### Read complete payloads by row addresses

```python
import lance

ds = lance.dataset("./blobs_v22.lance")
row_addrs = ds.to_table(columns=[], with_row_address=True).column("_rowaddr").to_pylist()

rows = ds.read_blobs("blob", addresses=row_addrs[:2])
```

Blob selection APIs preserve logical result cardinality. `read_blobs()` and
`take_blobs()` return one element per selected row, and `read_blob_ranges()`
returns one element per request. A null blob is returned as `None`; a valid
empty blob remains a non-null empty payload or zero-length `BlobFile`.

### Read row-specific byte ranges

Use `read_blob_ranges` to read multiple blob-local ranges with one planned API
call. Each request is a `(row, offset, length)` tuple, and `selector` determines
whether every `row` is interpreted as a row ID, row address, or dataset index.

```python
import lance

ds = lance.dataset("./blobs_v22.lance")
results = ds.read_blob_ranges(
    "blob",
    requests=[
        (7, 0, 1024),
        (7, 4096, 1024),
        (12, 0, 0),
    ],
    selector="indices",
)

for request_index, row_address, data in results:
    if data is None:
        # The selected blob is null.
        continue
    print(request_index, row_address, len(data))
```

Each result contains the zero-based `request_index`, the resolved physical row
address, and the requested bytes. `request_index` identifies the original
request when the same row appears more than once.

A request on a null blob returns `None`, including when its range is empty. An
empty range on a non-null blob returns `b""` without payload I/O. For every
request, `offset + length` must fit in an unsigned 64-bit integer. A range on a
non-null blob must not extend beyond its logical size; blob-local bounds are not
evaluated for null blobs because they have no logical payload length.

### Read blob columns as Arrow binary

```python
import lance

ds = lance.dataset("./blobs_v22.lance")
table = ds.scanner(columns=["blob"], blob_handling="all_binary").to_table()
payloads = table.column("blob").to_pylist()
```

### Open file-like blob handles lazily

```python
import lance

ds = lance.dataset("./blobs_v22.lance")
blobs = ds.take_blobs("blob", indices=[0, 1])

blob = blobs[0]
if blob is not None:
    with blob as f:
        header = f.read(1024)
```

### Example: decode video frames lazily

```python
import av
import lance

ds = lance.dataset("./videos_v22.lance")
blob = ds.take_blobs("video", indices=[0])[0]
if blob is None:
    raise ValueError("video blob is null")

start_ms, end_ms = 500, 1000

with av.open(blob) as container:
    stream = container.streams.video[0]
    stream.codec_context.skip_frame = "NONKEY"

    start = (start_ms / 1000) / stream.time_base
    end = (end_ms / 1000) / stream.time_base
    container.seek(int(start), stream=stream)

    for frame in container.decode(stream):
        if frame.time is not None and frame.time > end_ms / 1000:
            break
        # process frame
        pass
```

## Legacy Compatibility (`data_storage_version` <= `2.1`)

If you need to keep writing legacy blob columns, use file format `0.1`, `2.0`, or `2.1`
and mark `LargeBinary` fields with a metadata kwarg `"lance-encoding:blob": true`.

```python
import lance
import pyarrow as pa

schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field(
        "video",
        pa.large_binary(),
        metadata={"lance-encoding:blob": "true"},
    ),
])

table = pa.table(
    {
        "id": [1, 2],
        "video": [b"foo", b"bar"],
    },
    schema=schema,
)

ds = lance.write_dataset(
    table,
    "./legacy_blob_dataset",
    data_storage_version="2.1",
)
```

As mentioned above, this write pattern is invalid for `data_storage_version >= 2.2`.
For new datasets, it's recommended to use Lance file format 2.2, which uses blob v2 by default.

## Rewrite to a New Blob v2 Dataset

If your current dataset consists of legacy blobs (stored in file formats <2.2) and you want to opt in to blob v2, you must rewrite it as a new dataset with `data_storage_version="2.2"`.

```python
import lance
import pyarrow as pa
from lance import blob_array, blob_field

legacy = lance.dataset("./legacy_blob_dataset")
raw = legacy.scanner(columns=["id", "video"], blob_handling="all_binary").to_table()

new_schema = pa.schema([
    pa.field("id", pa.int64()),
    blob_field("video"),
])

rewritten = pa.table(
    {
        "id": raw.column("id"),
        "video": blob_array(raw.column("video").to_pylist()),
    },
    schema=new_schema,
)

lance.write_dataset(
    rewritten,
    "./blob_v22_dataset",
    data_storage_version="2.2",
)
```

!!! warning
    - The example above materializes binary payloads in memory (`blob_handling="all_binary"` and `to_pylist()`).
    - For large datasets, prefer chunked/batched rewrite pipelines.

## When to Use a Blob Column vs. Inline Binary

Not every binary column needs to be a blob column. Plain Arrow `binary`/`large_binary` stores bytes *inline*, interleaved with your other columns, which is simplest and fastest for really small blobs (e.g., thumbnail images). Using a blob column to store the binary payload makes sense when either of these holds:

- **You need partial or streaming reads.** Inline binary is always read in full; there is no way to fetch a byte range without materializing the entire value. Blob columns expose `read_blob_ranges` for planned row-specific range reads and `take_blobs` → `BlobFile` handles for caller-driven seeks, so you pay only for the bytes you touch.
- **Your values are large (roughly 1 MB or more on average).** Operations that rewrite entire rows, such as compaction or some updates, must copy the large inline payloads forward into the new version — even when those bytes never changed. The bigger the payload, the more bytes you rewrite per logical change (write amplification). A blob column keeps large payloads in separate `.blob` files that are referenced rather than re-copied, so these operations don't rewrite the heavy bytes.

!!! tip
    As a rule of thumb, if average payload size is below a few tens of KB and you only ever read whole values, plain inline binary is fine. Above ~1 MB, or any time you want file-like access, prefer a blob column. Blob v2 also tunes this automatically: by default it keeps payloads under 16 KiB inline, packs mid-sized payloads into shared `.blob` sidecars, and gives payloads over 2 MiB their own dedicated `.blob` file.

## Troubleshooting

This section contains commonly noticed issues or errors, and explains how to address them.

### Blob v2 requires file version >= 2.2
**Cause**: You are writing blob v2 values into a dataset/file format below `2.2`.
**Fix**: Write to a dataset created with `data_storage_version="2.2"` (or newer).

### Legacy blob columns ... are not supported for file version >= 2.2
**Cause**: You are using legacy blob metadata (`lance-encoding:blob`) while writing `2.2+` data.
**Fix**: Replace legacy metadata-based columns with blob v2 columns (`blob_field` / `blob_array`).


### Exactly one of ids, indices, or addresses must be specified
**Cause**: `read_blobs` or `take_blobs` received none or multiple selectors.
**Fix**: Provide exactly one of `ids`, `indices`, or `addresses`.

data_evolution.md 8.7 KB

# Data Evolution

Lance supports traditional schema evolution: adding, removing, and altering columns in a
dataset. Most of these operations can be performed *without* rewriting the
data files in the dataset, making them very efficient operations.
In addition, Lance supports **data evolution**, which allows you to also backfill existing rows
with the new column data without rewriting the data files in the dataset,
making it highly suitable for use cases like ML feature engineering.

In general, schema changes will conflict with most other concurrent write
operations. For example, if you change the schema of the dataset while someone
else is appending data to it, either your schema change or the append will fail,
depending on the order of the operations. Thus, it's recommended to perform
schema changes when no other writes are happening.

## Adding new columns

### Schema only

A common use case we've seen in production is to add a new column to a dataset without
populating it. This is useful to later run a large distributed job to populate the column
lazily. To do this, you can use the `lance.LanceDataset.add_columns` method to
add columns with `pyarrow.Field` or `pyarrow.Schema`.

```python
table = pa.table({"id": pa.array([1, 2, 3])})
dataset = lance.write_dataset(table, "null_columns")

# With pyarrow Field
dataset.add_columns(pa.field("embedding", pa.list_(pa.float32(), 128)))
assert dataset.schema == pa.schema([
    ("id", pa.int64()),
    ("embedding", pa.list_(pa.float32(), 128)),
])

# With pyarrow Schema
dataset.add_columns(pa.schema([
    ("label", pa.string()),
    ("score", pa.float32()),
]))
assert dataset.schema == pa.schema([
    ("id", pa.int64()),
    ("embedding", pa.list_(pa.float32(), 128)),
    ("label", pa.string()),
    ("score", pa.float32()),
])
```

This operation is very fast, as it only updates the metadata of the dataset.

For Lance file format `<= 2.1`, adding sub-columns under an existing `struct` is not supported.
Starting with Lance file format `2.2`, schema-only add can also extend nested `struct` fields
(including `struct` fields nested inside list types), for example by adding
`people.item.location` under `list<struct<...>>`.

### With data backfill

New columns can be added and populated within a single operation using the
`lance.LanceDataset.add_columns` method. There are two ways to specify
how to populate the new columns: first, by providing a SQL expression for each
new column, or second, by providing a function to generate the new column data.

SQL expressions can either be independent expressions or reference existing
columns. SQL literal values can be used to set a single value for all
existing rows.

```python
table = pa.table({"name": pa.array(["Alice", "Bob", "Carla"])})
dataset = lance.write_dataset(table, "names")
dataset.add_columns({
    "hash": "sha256(name)",
    "status": "'active'",
})
print(dataset.to_table().to_pandas())
#     name                                               hash  status
# 0  Alice  b';\xc5\x10b\x97<E\x8dZo-\x8dd\xa0#$cT\xad~\x0...  active
# 1    Bob  b'\xcd\x9f\xb1\xe1H\xcc\xd8D.Z\xa7I\x04\xccs\x...  active
# 2  Carla  b'\xad\x8d\x83\xff\xd8+Z\x8e\xd4)\xe8Y+\\\xb3\...  active
```

You can also provide a Python function to generate the new column data. This can
be used, for example, to compute a new embedding column. This function should
take a PyArrow RecordBatch and return either a PyArrow RecordBatch or a Pandas
DataFrame. The function will be called once for each batch in the dataset.

If the function is expensive to compute and can fail, it is recommended to set
a checkpoint file in the UDF. This checkpoint file saves the state of the UDF
after each invocation, so that if the UDF fails, it can be restarted from the
last checkpoint. Note that this file can get quite large, since it needs to store
unsaved results for up to an entire data file.

```python
import lance
import pyarrow as pa
import numpy as np

table = pa.table({"id": pa.array([1, 2, 3])})
dataset = lance.write_dataset(table, "ids")

@lance.batch_udf(checkpoint_file="embedding_checkpoint.sqlite")
def add_random_vector(batch):
    embeddings = np.random.rand(batch.num_rows, 128).astype("float32")
    return pa.RecordBatch.from_arrays(
        [pa.FixedSizeListArray.from_arrays(embeddings.flatten(), 128)],
        names=["embedding"]
    )
dataset.add_columns(add_random_vector)
```

### Using merge

If you have pre-computed one or more new columns, you can add them to an existing
dataset using the `lance.LanceDataset.merge` method. This allows filling in
additional columns without having to rewrite the whole dataset.

To use the `merge` method, provide a new dataset that includes the columns you
want to add, and a column name to use for joining the new data to the existing
dataset.

For example, imagine we have a dataset of embeddings and ids:

```python
table = pa.table({
   "id": pa.array([1, 2, 3]),
   "embedding": pa.array([np.array([1, 2, 3]), np.array([4, 5, 6]),
                          np.array([7, 8, 9])])
})
dataset = lance.write_dataset(table, "embeddings", mode="overwrite")
```

Now if we want to add a column of labels we have generated, we can do so by merging a new table:

```python
new_data = pa.table({
   "id": pa.array([1, 2, 3]),
   "label": pa.array(["horse", "rabbit", "cat"])
})
dataset.merge(new_data, "id")
print(dataset.to_table().to_pandas())
#    id  embedding   label
# 0   1  [1, 2, 3]   horse
# 1   2  [4, 5, 6]  rabbit
# 2   3  [7, 8, 9]     cat
```

## Dropping columns

Finally, you can drop columns from a dataset using the `lance.LanceDataset.drop_columns`
method. This is a metadata-only operation and does not delete the data on disk. This makes
it very quick.

```python
table = pa.table({"id": pa.array([1, 2, 3]),
                 "name": pa.array(["Alice", "Bob", "Carla"])})
dataset = lance.write_dataset(table, "names", mode="overwrite")
dataset.drop_columns(["name"])
print(dataset.schema)
# id: int64
```

Starting with Lance file format `2.2`, nested sub-column removal is supported for
nested types (for example `people.item.city` on `list<struct<...>>`), instead of
being limited to `struct` only.

To actually remove the data from disk, the files must be rewritten to remove the
columns and then the old files must be deleted. This can be done using
`lance.dataset.DatasetOptimizer.compact_files()` followed by
`lance.LanceDataset.cleanup_old_versions()`.

!!! warning

    `drop_columns` is metadata-only and remains reversible as long as old versions are retained.
    After `compact_files()` rewrites data files and `cleanup_old_versions()` removes old manifests/files,
    removed data may become permanently unrecoverable.

    For production workflows, use a rollback window:
    - create a tag (or snapshot/backup) before nested column drops
    - delay cleanup until the rollback window has passed
    - only run aggressive cleanup after rollback validation

## Renaming columns

Columns can be renamed using the `lance.LanceDataset.alter_columns` method.

```python
table = pa.table({"id": pa.array([1, 2, 3])})
dataset = lance.write_dataset(table, "ids")
dataset.alter_columns({"path": "id", "name": "new_id"})
print(dataset.to_table().to_pandas())
#    new_id
# 0       1
# 1       2
# 2       3
```

This works for nested columns as well. To address a nested column, use a dot
(`.`) to separate the levels of nesting. For example:

```python
data = [
  {"meta": {"id": 1, "name": "Alice"}},
  {"meta": {"id": 2, "name": "Bob"}},
]
schema = pa.schema([
    ("meta", pa.struct([
        ("id", pa.int32()),
        ("name", pa.string()),
    ]))
])
dataset = lance.write_dataset(data, "nested_rename")
dataset.alter_columns({"path": "meta.id", "name": "new_id"})
print(dataset.to_table().to_pandas())
#                                  meta
# 0  {'new_id': 1, 'name': 'Alice'}
# 1    {'new_id': 2, 'name': 'Bob'}
```

## Casting column data types

In addition to changing column names, you can also change the data type of a
column using the `lance.LanceDataset.alter_columns` method. This
requires rewriting that column to new data files, but does not require rewriting
the other columns.

!!! note

    If the column has an index, the index will be dropped if the column type is
    changed.

This method can be used to change the vector type of a column. For example, we
can change a float32 embedding column into a float16 column to save disk space
at the cost of lower precision:

```python
table = pa.table({
   "id": pa.array([1, 2, 3]),
   "embedding": pa.FixedShapeTensorArray.from_numpy_ndarray(
       np.random.rand(3, 128).astype("float32"))
})
dataset = lance.write_dataset(table, "embeddings")
dataset.alter_columns({"path": "embedding",
                       "data_type": pa.list_(pa.float16(), 128)})
print(dataset.schema)
# id: int64
# embedding: fixed_size_list<item: halffloat>[128]
#   child 0, item: halffloat
```

data_types.md 14 KB

# Data Types

Lance uses [Apache Arrow](https://arrow.apache.org/) as its in-memory data format. This guide covers the supported data types with a focus on array types, which are essential for vector embeddings and machine learning applications.

## Arrow Type System

Lance supports the full Apache Arrow type system. When writing data through Python (PyArrow) or Rust (arrow-rs), the Arrow types are automatically mapped to Lance's internal representation.

### Primitive Types

| Arrow Type | Description | Example Use Case |
|------------|-------------|------------------|
| `Boolean` | True/false values | Flags, filters |
| `Int8`, `Int16`, `Int32`, `Int64` | Signed integers | IDs, counts |
| `UInt8`, `UInt16`, `UInt32`, `UInt64` | Unsigned integers | IDs, indices |
| `Float16`, `Float32`, `Float64` | Floating point numbers | Measurements, scores |
| `Decimal128`, `Decimal256` | Fixed-precision decimals | Financial data |
| `Date32`, `Date64` | Date values | Birth dates, event dates |
| `Time32`, `Time64` | Time values | Time of day |
| `Timestamp` | Date and time with timezone | Event timestamps |
| `Duration` | Time duration | Elapsed time |

### String and Binary Types

| Arrow Type | Description | Example Use Case |
|------------|-------------|------------------|
| `Utf8` | Variable-length UTF-8 string | Text, names |
| `LargeUtf8` | Large UTF-8 string (64-bit offsets) | Large documents |
| `Binary` | Variable-length binary data | Raw bytes |
| `LargeBinary` | Large binary data (64-bit offsets) | Large blobs |
| `FixedSizeBinary(n)` | Fixed-length binary data | UUIDs, hashes |

### Blob Type for Large Binary Objects

Lance provides a specialized **Blob** type for efficiently storing and retrieving very large binary objects such as videos, images, audio files, or other multimedia content. Blob columns support planned full-payload reads as well as lazy file-like access for streaming, seeking, and partial reads.

For new datasets, use blob v2 (`lance.blob.v2`) via `blob_field` and `blob_array`.

Blob versioning follows dataset file format rules:

- `data_storage_version` is the Lance file format version of a dataset.
- A dataset's `data_storage_version` is fixed once created.
- For `data_storage_version >= 2.2`, legacy blob metadata (`lance-encoding:blob`) is rejected on write.
- Legacy metadata-based blob write remains available for `0.1`, `2.0`, and `2.1`.

```python
import lance
import pyarrow as pa
from lance import blob_array, blob_field

schema = pa.schema([
    pa.field("id", pa.int64()),
    blob_field("video"),
])

table = pa.table(
    {
        "id": [1],
        "video": blob_array([b"sample-video-bytes"]),
    },
    schema=schema,
)

ds = lance.write_dataset(table, "./videos_v22.lance", data_storage_version="2.2")
_row_address, payload = ds.read_blobs("video", indices=[0])[0]
```

For legacy compatibility (`data_storage_version <= 2.1`), you can still write blob columns using `LargeBinary` with `lance-encoding:blob=true`.

To create a blob column with the legacy path, add the `lance-encoding:blob` metadata to a `LargeBinary` field:

```python
import pyarrow as pa
import lance

# Define schema with a blob column for videos
schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field("filename", pa.utf8()),
    pa.field("video", pa.large_binary(), metadata={"lance-encoding:blob": "true"}),
])

# Read video file
with open("sample_video.mp4", "rb") as f:
    video_data = f.read()

# Create and write dataset
table = pa.table({
    "id": [1],
    "filename": ["sample_video.mp4"],
    "video": [video_data],
}, schema=schema)

ds = lance.write_dataset(
    table,
    "./videos_legacy.lance",
    schema=schema,
    data_storage_version="2.1",
)
```

To read complete blob payloads into memory, use `read_blobs()`:

```python
rows = ds.read_blobs("video", indices=[0])
_row_address, payload = rows[0]
```

Use `take_blobs()` only when you need file-like objects for lazy, partial, or seek-based reading:

```python
# Retrieve blob as a file-like object (lazy loading)
blobs = ds.take_blobs("video", indices=[0])

# Use with libraries that accept file-like objects
import av  # pip install av
with av.open(blobs[0]) as container:
    for frame in container.decode(video=0):
        # Process video frames without loading entire video into memory
        pass
```

For more details, see the [Blob API Guide](blob.md).

## Array Types for Vector Embeddings

Lance provides excellent support for array types, which are critical for storing vector embeddings in AI/ML applications.

### FixedSizeList - The Preferred Type for Vector Embeddings

`FixedSizeList` is the recommended type for storing fixed-dimensional vector embeddings. Each vector has the same number of dimensions, making it highly efficient for storage and computation.

=== "Python"

    ```python
    import lance
    import pyarrow as pa
    import numpy as np

    # Create a schema with a vector embedding column
    # This defines a 128-dimensional float32 vector
    schema = pa.schema([
        pa.field("id", pa.int64()),
        pa.field("text", pa.utf8()),
        pa.field("vector", pa.list_(pa.float32(), 128)),  # FixedSizeList of 128 floats
    ])

    # Create sample data with embeddings
    num_rows = 1000
    vectors = np.random.rand(num_rows, 128).astype(np.float32)

    table = pa.Table.from_pydict({
        "id": list(range(num_rows)),
        "text": [f"document_{i}" for i in range(num_rows)],
        "vector": [v.tolist() for v in vectors],
    }, schema=schema)

    # Write to Lance format
    ds = lance.write_dataset(table, "./embeddings.lance")
    print(f"Created dataset with {ds.count_rows()} rows")
    ```

=== "Rust"

    ```rust
    use arrow_array::{
        ArrayRef, FixedSizeListArray, Float32Array, Int64Array, RecordBatch, StringArray,
    };
    use arrow_schema::{DataType, Field, Schema};
    use lance::dataset::WriteParams;
    use lance::Dataset;
    use std::sync::Arc;

    #[tokio::main]
    async fn main() -> lance::Result<()> {
        // Define schema with a 128-dimensional vector column
        let schema = Arc::new(Schema::new(vec![
            Field::new("id", DataType::Int64, false),
            Field::new("text", DataType::Utf8, false),
            Field::new(
                "vector",
                DataType::FixedSizeList(
                    Arc::new(Field::new("item", DataType::Float32, true)),
                    128,
                ),
                false,
            ),
        ]));

        // Create sample data
        let ids = Int64Array::from(vec![0, 1, 2]);
        let texts = StringArray::from(vec!["doc_0", "doc_1", "doc_2"]);

        // Create vector embeddings (128-dimensional)
        let values: Vec<f32> = (0..384).map(|i| i as f32 / 100.0).collect();
        let values_array = Float32Array::from(values);
        let vectors = FixedSizeListArray::try_new_from_values(values_array, 128)?;

        let batch = RecordBatch::try_new(
            schema.clone(),
            vec![
                Arc::new(ids) as ArrayRef,
                Arc::new(texts) as ArrayRef,
                Arc::new(vectors) as ArrayRef,
            ],
        )?;

        // Write to Lance
        let dataset = Dataset::write(
            vec![batch].into_iter().map(Ok),
            "embeddings.lance",
            WriteParams::default(),
        )
        .await?;

        println!("Created dataset with {} rows", dataset.count_rows().await?);
        Ok(())
    }
    ```

### Vector Search with Embeddings

Once you have vector embeddings stored in Lance, you can perform efficient vector similarity search:

```python
import lance
import numpy as np

# Open the dataset
ds = lance.dataset("./embeddings.lance")

# Create a query vector (same dimension as stored vectors)
query_vector = np.random.rand(128).astype(np.float32).tolist()

# Perform vector search - find 10 nearest neighbors
results = ds.to_table(
    nearest={
        "column": "vector",
        "q": query_vector,
        "k": 10,
    }
)
print(results.to_pandas())
```

For production workloads with large datasets, create a vector index for much faster search:

```python
# Create an IVF-PQ index for fast approximate nearest neighbor search
ds.create_index(
    "vector",
    index_type="IVF_PQ",
    num_partitions=256,  # Number of IVF partitions
    num_sub_vectors=16,  # Number of PQ sub-vectors
)

# Search with the index (automatically used)
results = ds.to_table(
    nearest={
        "column": "vector",
        "q": query_vector,
        "k": 10,
        "nprobes": 20,  # Number of partitions to search
    }
)
```

### List and LargeList - Variable-Length Arrays

For variable-length arrays where each row may have a different number of elements, use `List` or `LargeList`:

```python
import lance
import pyarrow as pa

# Schema with variable-length arrays
schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field("tags", pa.list_(pa.utf8())),      # Variable number of string tags
    pa.field("scores", pa.list_(pa.float32())), # Variable number of float scores
])

table = pa.Table.from_pydict({
    "id": [1, 2, 3],
    "tags": [["python", "ml"], ["rust"], ["data", "analytics", "ai"]],
    "scores": [[0.9, 0.8], [0.95], [0.7, 0.85, 0.9]],
}, schema=schema)

ds = lance.write_dataset(table, "./variable_arrays.lance")
```

## Nested and Complex Types

### Struct Types

Store structured data with multiple named fields:

```python
import lance
import pyarrow as pa

# Schema with nested struct
schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field("metadata", pa.struct([
        pa.field("source", pa.utf8()),
        pa.field("timestamp", pa.timestamp("us")),
        pa.field("embedding_model", pa.utf8()),
    ])),
    pa.field("vector", pa.list_(pa.float32(), 384)),  # 384-dim embedding
])

table = pa.Table.from_pydict({
    "id": [1, 2],
    "metadata": [
        {"source": "web", "timestamp": "2024-01-15T10:30:00", "embedding_model": "text-embedding-3-small"},
        {"source": "api", "timestamp": "2024-01-15T11:45:00", "embedding_model": "text-embedding-3-small"},
    ],
    "vector": [
        [0.1] * 384,
        [0.2] * 384,
    ],
}, schema=schema)

ds = lance.write_dataset(table, "./with_metadata.lance")
```

### Map Types

Store key-value pairs with dynamic keys:
Map writes require Lance file format version 2.2 or later.

```python
import lance
import pyarrow as pa

schema = pa.schema([
    pa.field("id", pa.int64()),
    pa.field("attributes", pa.map_(pa.utf8(), pa.utf8())),
])

table = pa.Table.from_pydict({
    "id": [1, 2],
    "attributes": [
        [("color", "red"), ("size", "large")],
        [("color", "blue"), ("material", "cotton")],
    ],
}, schema=schema)

ds = lance.write_dataset(table, "./with_maps.lance", data_storage_version="2.2")
```

## Data Type Mapping for Integrations

When integrating Lance with other systems (like Apache Flink, Spark, or Presto), the following type mappings apply:

| External Type | Lance/Arrow Type | Notes |
|--------------|------------------|-------|
| `BOOLEAN` | `Boolean` | |
| `TINYINT` | `Int8` | |
| `SMALLINT` | `Int16` | |
| `INT` / `INTEGER` | `Int32` | |
| `BIGINT` | `Int64` | |
| `FLOAT` | `Float32` | |
| `DOUBLE` | `Float64` | |
| `DECIMAL(p,s)` | `Decimal128(p,s)` | |
| `STRING` / `VARCHAR` | `Utf8` | |
| `CHAR(n)` | `Utf8` | Fixed-width in source system; stored as variable-length Utf8 |
| `DATE` | `Date32` | |
| `TIME` | `Time64` | Microsecond precision |
| `TIMESTAMP` | `Timestamp` | |
| `TIMESTAMP WITH LOCAL TIMEZONE` | `Timestamp` | With timezone info |
| `BINARY` / `VARBINARY` | `Binary` | |
| `BYTES` | `Binary` | |
| `BLOB` | Blob v2 extension type (`lance.blob.v2`) | Use `blob_field` / `blob_array` for new datasets; legacy metadata path applies to `data_storage_version <= 2.1` |
| `ARRAY<T>` | `List(T)` | Variable-length array |
| `ARRAY<T>(n)` | `FixedSizeList(T, n)` | Fixed-length array (vectors) |
| `ROW` / `STRUCT` | `Struct` | Nested structure |
| `MAP<K,V>` | `Map(K, V)` | Key-value pairs |

### Vector Embeddings in Integrations

For vector embedding columns, use `ARRAY<FLOAT>(n)` or `ARRAY<DOUBLE>(n)` where `n` is the embedding dimension:

```sql
-- Example: Creating a table with vector embeddings in SQL-compatible systems
CREATE TABLE embeddings (
    id BIGINT,
    text STRING,
    vector ARRAY<FLOAT>(384)  -- 384-dimensional vector
);
```

This maps to Lance's `FixedSizeList(Float32, 384)` type, which is optimized for:

- Efficient columnar storage
- SIMD-accelerated distance computations
- Vector index creation and search

## Best Practices for Vector Data

1. **Use FixedSizeList for embeddings**: Always use `FixedSizeList` (not variable-length `List`) for vector embeddings to enable efficient storage and indexing.

2. **Choose appropriate precision**:
   - `Float32` is the standard choice, balancing precision and storage
   - `Float16` or `BFloat16` can reduce storage by 50% with minimal accuracy loss
   - `Int8` for quantized embeddings

3. **Align dimensions for SIMD**: Vector dimensions divisible by 8 enable optimal SIMD acceleration. Common dimensions: 128, 256, 384, 512, 768, 1024, 1536.

4. **Create indices for large datasets**: For datasets with more than ~10,000 vectors, create an ANN index for fast search:

    ```python
    # IVF_PQ is recommended for most use cases
    ds.create_index("vector", index_type="IVF_PQ", num_partitions=256, num_sub_vectors=16)

    # IVF_HNSW_SQ offers better recall at the cost of more memory
    ds.create_index("vector", index_type="IVF_HNSW_SQ", num_partitions=256)
    ```

5. **Store metadata alongside vectors**: Lance efficiently handles mixed workloads with both vector and scalar data:

    ```python
    # Combine vector search with metadata filtering
    results = ds.to_table(
        filter="category = 'electronics'",
        nearest={"column": "vector", "q": query, "k": 10}
    )
    ```

## See Also

- [Vector Search Tutorial](../quickstart/vector-search.md) - Complete guide to vector search with Lance
- [Blob API Guide](blob.md) - Storing and retrieving large binary objects (videos, images)
- [Extension Arrays](arrays.md) - Special array types for ML (BFloat16, images)
- [Performance Guide](performance.md) - Optimization tips for large-scale deployments

distributed_indexing.md 7 KB

# Distributed Indexing

!!! warning
    Lance exposes public APIs that can be integrated into an external
    distributed index build workflow, but Lance itself does not provide a full
    distributed scheduler or end-to-end orchestration layer.

    This page describes the current model, terminology, and execution flow so
    that callers can integrate these APIs correctly.

## Overview

Distributed index build in Lance follows the same high-level pattern as distributed
write:

1. multiple workers build index data in parallel
2. the caller invokes Lance segment build APIs for one distributed build
3. Lance plans and builds index artifacts from the worker outputs supplied by the caller
4. the built artifacts are committed into the dataset manifest

For vector indices and segment-native scalar indices, the worker outputs are
segments stored directly under `indices/<segment_uuid>/`. Lance can turn these
outputs into one or more physical segments and then commit them as one logical
index.

![Distributed Vector Segment Build](../images/distributed_vector_segment_build.svg)

## Terminology

This guide uses the following terms consistently:

- **Segment**: one worker output written by `execute_uncommitted()` under
  `indices/<segment_uuid>/`
- **Physical segment**: one index segment that is ready to be committed into
  the manifest
- **Logical index**: the user-visible index identified by name; a logical index
  may contain one or more physical segments

For example, a distributed vector build may create a layout like:

```text
indices/<segment_uuid_0>/
├── index.idx
└── auxiliary.idx

indices/<segment_uuid_1>/
├── index.idx
└── auxiliary.idx

indices/<segment_uuid_2>/
├── index.idx
└── auxiliary.idx
```

After segment build, Lance produces one or more segment directories:

```text
indices/<physical_segment_uuid_0>/
├── index.idx
└── auxiliary.idx

indices/<physical_segment_uuid_1>/
├── index.idx
└── auxiliary.idx
```

These physical segments are then committed together as one logical index. In the
common no-merge case, the input segments are already the physical
segments and can be committed directly.

## Roles

There are two parties involved in distributed indexing:

- **Workers** build segments
- **The caller** launches workers, chooses how those segments should be turned
  into final segments, optionally merges caller-defined groups, and commits the
  final result

Lance does not provide a distributed scheduler. The caller is responsible for
launching workers and driving the overall workflow.

## Current Model

The current model for distributed indexing has two layers of parallelism.

### Worker Build

First, multiple workers build segments in parallel:

1. on each worker, call a shard-build API such as
   `create_index_builder(...).fragments(...).execute_uncommitted()`
   or Python `create_index_uncommitted(..., fragment_ids=...)`
2. each worker writes one segment under `indices/<segment_uuid>/`

### Segment Merge

Then the caller decides whether those existing segments should be committed as-is
or merged into larger segments:

1. keep the worker outputs as-is and commit them directly with
   `commit_existing_index_segments(...)`, or
2. group one or more existing segments and call
   `merge_existing_index_segments(...)` for each caller-defined group
3. commit the final segment list with `commit_existing_index_segments(...)`

Within a single commit, built segments must have disjoint fragment coverage.

`merge_existing_index_segments(...)` currently supports vector, inverted,
bitmap, BTree, and zone map segments. Other scalar index families can still
commit multiple compatible segments directly when their build path supports
fragment-scoped segments, but cannot be merged into a larger physical segment
until they add a merge implementation.

### Vector Model Scope

Distributed vector builds support two model scopes.

**Shared model artifacts**: the caller trains or provides IVF centroids once and
passes the same artifacts to every worker. For IVF-PQ segments that should be
physically mergeable, workers should also use the same PQ codebook. This makes
partition ids and quantizer state have the same meaning across segments.

**Independent segment models**: each worker trains the IVF/PQ model for its own
`fragment_ids`. The resulting segments can be committed together as one logical
index without sharing centroids or codebooks.

At query time, Lance searches each physical segment independently:

1. Lance opens each segment by index UUID
2. each segment ranks IVF partitions using its own centroids
3. each segment searches the selected partitions using its own quantizer storage
4. Lance merges the candidate rows from all segments by `_distance`

Because partition ids are interpreted only within a segment during this fanout
query path, independently trained committed segments can return valid results.
For L2 and cosine IVF-PQ, each segment computes residuals against its own IVF
centroid during both build and query, so distances remain estimates of the
original query-to-vector metric.

Physical merge is a separate operation. It rewrites several segment artifacts
into one artifact with one model metadata scope. Use shared compatible model
artifacts for segments you plan to merge physically, or keep independently
trained segments as separate physical segments.

## Internal Finalize Model

Internally, Lance models distributed segment build as:

1. **build** one uncommitted segment per worker
2. **optionally merge** caller-defined groups of existing segments
3. **commit** the resulting segments as one logical index

The merge step is driven directly by the `IndexMetadata` returned from
`execute_uncommitted()`.

This is intentionally a storage-level model:

- segments are worker outputs that are not yet published
- physical segments are durable artifacts referenced by the manifest
- the logical index identity is attached only at commit time

## Segment Grouping

The caller chooses the final segment grouping:

- keep segment boundaries, so each worker output is committed directly
- merge multiple existing segments into a larger segment before commit

The grouping decision is separate from worker build. Workers only build
segments; Lance applies the segment build policy when it plans
physical segments.

## Responsibility Boundaries

The caller is expected to know:

- which distributed build is ready for segment build
- the segment metadata returned by worker builds
- how the resulting physical segments should be published

Lance is responsible for:

- writing segment artifacts
- planning physical segments from the supplied segment set
- merging segment storage into physical segment artifacts
- committing physical segments into the manifest

If a staging root or built segment directory is never committed, it remains an
unreferenced index directory under `_indices/`. These artifacts are cleaned up
by `cleanup_old_versions(...)` using the same age-based rules as other
unreferenced index files.

This split keeps distributed scheduling outside the storage engine while still
letting Lance own the on-disk index format.

distributed_write.md 11.4 KB

# Distributed Write

!!! warning
    Lance provides out-of-the-box [Ray](https://github.com/lance-format/lance-ray) and [Spark](https://github.com/lance-format/lance-spark) integrations.

    This page is intended for users who wish to perform distributed operations in a custom manner, i.e. using `slurm` or `Kubernetes` without the Lance integration.

## Overview

The [Lance format](../format/index.md) is designed to support parallel writing across multiple distributed workers. A distributed write operation can be performed by two phases:

1. **Parallel Writes**: Generate new `lance.LanceFragment` in parallel across multiple workers.
2. **Commit**: Collect all the `lance.FragmentMetadata` and commit into a single dataset in a single `lance.LanceOperation`.

![Distributed Append](../images/distributed_append.png)

## Write new data

Writing or appending new data is straightforward with `lance.fragment.write_fragments`.

```python
import json
from lance.fragment import write_fragments

# Run on each worker
data_uri = "./dist_write"
schema = pa.schema([
    ("a", pa.int32()),
    ("b", pa.string()),
])

# Run on worker 1
data1 = {
    "a": [1, 2, 3],
    "b": ["x", "y", "z"],
}
fragments_1 = write_fragments(data1, data_uri, schema=schema)
print("Worker 1: ", fragments_1)

# Run on worker 2
data2 = {
    "a": [4, 5, 6],
    "b": ["u", "v", "w"],
}
fragments_2 = write_fragments(data2, data_uri, schema=schema)
print("Worker 2: ", fragments_2)
```

Output:
```
Worker 1:  [FragmentMetadata(id=0, files=...)]
Worker 2:  [FragmentMetadata(id=0, files=...)]
```

Now, use `lance.fragment.FragmentMetadata.to_json` to serialize the fragment metadata, and collect all serialized metadata on a single worker to execute the final commit operation.

```python
import json
from lance import FragmentMetadata, LanceOperation

# Serialize Fragments into JSON data
fragments_json1 = [json.dumps(fragment.to_json()) for fragment in fragments_1]
fragments_json2 = [json.dumps(fragment.to_json()) for fragment in fragments_2]

# On one worker, collect all fragments
all_fragments = [FragmentMetadata.from_json(f) for f in \
    fragments_json1 + fragments_json2]

# Commit the fragments into a single dataset
# Use LanceOperation.Overwrite to overwrite the dataset or create new dataset.
op = lance.LanceOperation.Overwrite(schema, all_fragments)
read_version = 0 # Because it is empty at the time.
lance.LanceDataset.commit(
    data_uri,
    op,
    read_version=read_version,
)

# We can read the dataset using the Lance API:
dataset = lance.dataset(data_uri)
assert len(dataset.get_fragments()) == 2
assert dataset.version == 1
print(dataset.to_table().to_pandas())
```

Output:
```
     a  b
0  1  x
1  2  y
2  3  z
3  4  u
4  5  v
5  6  w
```

## Append data

Appending additional data follows a similar process. Use `lance.LanceOperation.Append` to commit the new fragments, ensuring that the `read_version` is set to the current dataset's version.

```python
import lance

ds = lance.dataset(data_uri)
read_version = ds.version # record the read version

op = lance.LanceOperation.Append(all_fragments)
lance.LanceDataset.commit(
    data_uri,
    op,
    read_version=read_version,
)
```

## Add New Columns

[Lance Format excels at operations such as adding columns](../format/index.md). Thanks to its two-dimensional layout ([see this blog post](https://blog.lancedb.com/designing-a-table-format-for-ml-workloads/)), adding new columns is highly efficient since it avoids copying the existing data files. Instead, the process simply creates new data files and links them to the existing dataset using metadata-only operations.

```python
import lance
from pyarrow import RecordBatch
import pyarrow.compute as pc

dataset = lance.dataset("./add_columns_example")
assert len(dataset.get_fragments()) == 2
assert dataset.to_table().combine_chunks() == pa.Table.from_pydict({
    "name": ["alice", "bob", "charlie", "craig", "dave", "eve"],
    "age": [25, 33, 44, 55, 66, 77],
}, schema=schema)


def name_len(names: RecordBatch) -> RecordBatch:
    return RecordBatch.from_arrays(
        [pc.utf8_length(names["name"])],
        ["name_len"],
    )

# On Worker 1
frag1 = dataset.get_fragments()[0]
new_fragment1, new_schema = frag1.merge_columns(name_len, ["name"])

# On Worker 2
frag2 = dataset.get_fragments()[1]
new_fragment2, _ = frag2.merge_columns(name_len, ["name"])

# On Worker 3 - Commit
all_fragments = [new_fragment1, new_fragment2]
op = lance.LanceOperation.Merge(all_fragments, schema=new_schema)
lance.LanceDataset.commit(
    "./add_columns_example",
    op,
    read_version=dataset.version,
)

# Verify dataset
dataset = lance.dataset("./add_columns_example")
print(dataset.to_table().to_pandas())
```

Output:
```
      name  age  name_len
0    alice   25         5
1      bob   33         3
2  charlie   44         7
3    craig   55         5
4     dave   66         4
5      eve   77         3
```

## Update Columns

Currently, Lance supports the fragment level update columns ability to update existing columns in a distributed manner.

This operation performs a left-outer-hash-join with the right table (new data)
on the column specified by `left_on` and `right_on`. For every row in the current
fragment, the updated column value is:
1. If no matched row on the right side, the column value of the left side row.
2. If there is exactly one corresponding row on the right side, the column value
   of the matching row.
3. If there are multiple corresponding rows, the column value of a random row.

```python
import lance
import pyarrow as pa

# Create initial dataset with two fragments
# First fragment
data1 = pa.table(
    {
        "id": [1, 2, 3, 4],
        "name": ["Alice", "Bob", "Charlie", "David"],
        "score": [85, 90, 75, 80],
    }
)
dataset_uri = "./my_dataset.lance"
dataset = lance.write_dataset(data1, dataset_uri)

# Second fragment
data2 = pa.table(
    {
        "id": [5, 6, 7, 8],
        "name": ["Eve", "Frank", "Grace", "Henry"],
        "score": [88, 92, 78, 82],
    }
)
dataset = lance.write_dataset(data2, dataset_uri, mode="append")

# Prepare update data for fragment 0 using 'id' as join key
update_data1 = pa.table(
    {
        "id": [1, 3],
        "name": ["Alan", "Chase"],
        "score": [95, 85],
    }
)

# Prepare update data for fragment 1
update_data2 = pa.table(
    {
        "id": [5, 7],
        "name": ["Eva", "Gracie"],
        "score": [98, 88],
    }
)

# Update fragment 0
fragment0 = dataset.get_fragment(0)
updated_fragment0, fields_modified0 = fragment0.update_columns(
    update_data1, left_on="id", right_on="id"
)

# Update fragment 1
fragment1 = dataset.get_fragment(1)
updated_fragment1, fields_modified1 = fragment1.update_columns(
    update_data2, left_on="id", right_on="id"
)

union_fields_modified = list(set(fields_modified0 + fields_modified1))
# Commit the changes for both fragments
op = lance.LanceOperation.Update(
    updated_fragments=[updated_fragment0, updated_fragment1],
    fields_modified=union_fields_modified,
)
updated_dataset = lance.LanceDataset.commit(
    str(dataset_uri), op, read_version=dataset.version
)

# Verify the update
dataset = lance.dataset(dataset_uri)
print(dataset.to_table().to_pandas())
```

Output:
```
   id    name  score
0   1    Alan     95
1   2     Bob     90
2   3   Chase     85
3   4   David     80
4   5     Eva     98
5   6   Frank     92
6   7  Gracie     88
7   8   Henry     82
```

### Handling stable row id

On a dataset created with `enable_stable_row_ids=True`, each row keeps the same
`_rowid` for its lifetime, even when an update rewrites it into a different
fragment. Lance cannot infer which new row replaces which old one, so when you
assemble the transaction yourself, carrying those ids across is your job: read
the rows you are rewriting with `with_row_id=True` and attach their ids to the
new fragment with `lance.fragment.RowIdSequence`.

Rows you leave without an id are treated as newly inserted. That is not an error,
so a fragment written without `row_id_meta` commits successfully while silently
giving every rewritten row a fresh identity, breaking `_rowid` for anything
downstream that relies on it.

You do **not** need to supply `created_at_version_meta` or
`last_updated_at_version_meta`. Leave them as `None`. Lance derives both while
building the manifest: `last_updated_at_version_meta` becomes the version being
committed, and `created_at_version_meta` is copied from whichever existing row
carries the same stable row id, so a rewritten row keeps the version it first
appeared in.

#### Mixing updated and new rows

A single fragment may hold both rewritten rows and brand new ones. Order it so
that **the rewritten rows come first and the new rows last**, then pass only the
row ids of the rewritten rows. The row ids bind to the leading rows in fragment
order, and the commit generates new ids for the remaining rows.

Do not generate ids for the new rows yourself. Row ids are handed out from a counter
in the manifest, and a commit that loses a race is retried against the version
that won, which may have consumed the very ids you picked. Only the commit knows
which values are free, so it assigns them after conflict resolution has settled.
Supplying more row ids than the fragment has rows is rejected.

```python
import lance
import pyarrow as pa
import pyarrow.compute as pc
from lance.fragment import RowIdSequence, write_fragments

schema = pa.schema([("id", pa.int64()), ("score", pa.int64())])
dataset_uri = "./stable_row_ids.lance"
dataset = lance.write_dataset(
    pa.table({"id": [1, 2, 3, 4], "score": [85, 90, 75, 80]}, schema=schema),
    dataset_uri,
    enable_stable_row_ids=True,
)

# On a worker: read the rows to rewrite, keeping their stable row ids.
rows = dataset.to_table(columns=["id", "score"], with_row_id=True)
rewritten = rows.filter(pc.field("id").isin([2, 3]))

# Rewritten rows first, then the row that did not exist before.
new_data = pa.table(
    {
        "id": rewritten["id"].to_pylist() + [5],
        "score": [95, 70, 60],
    },
    schema=schema,
)
fragments = write_fragments(new_data, dataset_uri, schema=schema)
assert len(fragments) == 1

# Only the rewritten rows have ids. The trailing row gets one at commit time.
fragments[0].row_id_meta = RowIdSequence(rewritten["_rowid"]).to_inline_metadata()

# On the committing worker: tombstone the old copies of the rewritten rows.
updated_fragment = dataset.get_fragments()[0].delete("id in (2, 3)")

op = lance.LanceOperation.Update(
    updated_fragments=[updated_fragment],
    new_fragments=fragments,
)
dataset = lance.LanceDataset.commit(dataset_uri, op, read_version=dataset.version)

print(dataset.to_table(with_row_id=True).to_pandas())
```

Output:
```
   id  score  _rowid
0   1     85       0
1   4     80       3
2   2     95       1
3   3     70       2
4   5     60       4
```

Row ids 1 and 2 followed their rows into the new fragment, and the inserted row
received the next unused id. Reading the lineage columns shows that the rewritten
rows kept their original creation version while the inserted row is stamped with
the version that added it:

```python
print(
    dataset.to_table(
        columns=["id", "_row_created_at_version", "_row_last_updated_at_version"]
    ).to_pandas()
)
```

Output:
```
   id  _row_created_at_version  _row_last_updated_at_version
0   1                        1                             1
1   4                        1                             1
2   2                        1                             2
3   3                        1                             2
4   5                        2                             2
```

json.md 12 KB

# JSON Support

Lance provides comprehensive support for storing and querying JSON data, enabling you to work with semi-structured data efficiently. This guide covers how to store JSON data in Lance datasets and use JSON functions to query and filter your data.

## Getting Started

```python
import lance
import pyarrow as pa
import json

# Create a table with JSON data
json_data = {"name": "Alice", "age": 30, "city": "New York"}
json_arr = pa.array([json.dumps(json_data)], type=pa.json_())
table = pa.table({"id": [1], "data": json_arr})

# Write the dataset
lance.write_dataset(table, "dataset.lance")
```

## Storage Format

Lance stores JSON data internally as JSONB (binary JSON) using the `lance.json` extension type. This provides:

- Efficient storage through binary encoding
- Fast query performance for nested field access
- Compatibility with Apache Arrow's JSON type

When you read JSON data back from Lance, it's automatically converted to Arrow's JSON type for seamless integration with your data processing pipelines.

## JSON Functions

Lance provides a comprehensive set of JSON functions for querying and filtering JSON data. These functions can be used in filter expressions with methods like `to_table()`, `scanner()`, and SQL queries through DataFusion integration.

### Data Access Functions

#### json_extract

Extracts a value from JSON using JSONPath syntax.

**Syntax:** `json_extract(json_column, json_path)`

**Returns:** JSON-formatted string representation of the extracted value

**Example:**
```python
# Sample data: {"user": {"name": "Alice", "age": 30}}
result = dataset.to_table(
    filter="json_extract(data, '$.user.name') = '\"Alice\"'"
)
# Returns: "\"Alice\"" for strings, "30" for numbers, "true" for booleans
```

!!! note
    `json_extract` returns values in JSON format. String values include quotes (e.g., `"Alice"`),
    numbers are returned as-is (e.g., `30`), and booleans as `true`/`false`.

#### json_get

Retrieves a field or array element from JSON, returning it as JSONB for further processing.

**Syntax:** `json_get(json_column, key_or_index)`

**Parameters:**
- `key_or_index`: Field name (string) or array index (numeric string like "0", "1")

**Returns:** JSONB binary value (can be used for nested access)

**Example:**
```python
# Access nested JSON by chaining json_get calls
# Sample data: {"user": {"profile": {"name": "Alice"}}}
result = dataset.to_table(
    filter="json_get_string(json_get(json_get(data, 'user'), 'profile'), 'name') = 'Alice'"
)

# Access array elements by index
# Sample data: ["first", "second", "third"]
result = dataset.to_table(
    filter="json_get_string(data, '0') = 'first'"  # Gets first array element
)
```

### Type-Safe Value Extraction

These functions extract values with strict type conversion. The conversion uses JSONB's built-in strict mode, which requires values to be of compatible types:

#### json_get_string

Extracts a string value from JSON.

**Syntax:** `json_get_string(json_column, key_or_index)`

**Parameters:**
- `key_or_index`: Field name or array index (as string)

**Returns:** String value (without JSON quotes), null if conversion fails

**Type Conversion:** Uses strict conversion - numbers and booleans are converted to their string representation

**Example:**
```python
result = dataset.to_table(
    filter="json_get_string(data, 'name') = 'Alice'"
)

# Array access example
# Sample data: ["first", "second"]
result = dataset.to_table(
    filter="json_get_string(data, '1') = 'second'"  # Gets second array element
)
```

#### json_get_int

Extracts an integer value with strict type conversion.

**Syntax:** `json_get_int(json_column, key_or_index)`

**Returns:** 64-bit integer, null if conversion fails

**Type Conversion:** Uses JSONB's strict `to_i64()` conversion:
- Numbers are truncated to integers
- Strings must be parseable as numbers
- Booleans: true → 1, false → 0

**Example:**
```python
# {"age": 30} works, {"age": "30"} may work if JSONB allows string parsing
result = dataset.to_table(
    filter="json_get_int(data, 'age') > 25"
)
```

#### json_get_float

Extracts a floating-point value with strict type conversion.

**Syntax:** `json_get_float(json_column, key_or_index)`

**Returns:** 64-bit float, null if conversion fails

**Type Conversion:** Uses JSONB's strict `to_f64()` conversion:
- Integers are converted to floats
- Strings must be parseable as numbers
- Booleans: true → 1.0, false → 0.0

**Example:**
```python
result = dataset.to_table(
    filter="json_get_float(data, 'score') >= 90.5"
)
```

#### json_get_bool

Extracts a boolean value with strict type conversion.

**Syntax:** `json_get_bool(json_column, key_or_index)`

**Returns:** Boolean, null if conversion fails

**Type Conversion:** Uses JSONB's strict `to_bool()` conversion:
- Numbers: 0 → false, non-zero → true
- Strings: "true" → true, "false" → false (exact match required)
- Other values may fail conversion

**Example:**
```python
result = dataset.to_table(
    filter="json_get_bool(data, 'active') = true"
)
```

### Existence and Array Functions

#### json_exists

Checks if a JSONPath exists in the JSON data.

**Syntax:** `json_exists(json_column, json_path)`

**Returns:** Boolean

**Example:**
```python
# Find records that have an age field
result = dataset.to_table(
    filter="json_exists(data, '$.user.age')"
)
```

#### json_array_contains

Checks if a JSON array contains a specific value.

**Syntax:** `json_array_contains(json_column, json_path, value)`

**Returns:** Boolean

**Comparison Logic:**
- Compares array elements as JSON strings
- For string matching, tries both with and without quotes
- Example: searching for 'python' matches both `"python"` and `python` in the array

**Example:**
```python
# Sample data: {"tags": ["python", "ml", "data"]}
result = dataset.to_table(
    filter="json_array_contains(data, '$.tags', 'python')"
)
```

#### json_array_length

Returns the length of a JSON array.

**Syntax:** `json_array_length(json_column, json_path)`

**Returns:**
- Integer: length of the array
- null: if path doesn't exist
- Error: if path points to a non-array value

**Example:**
```python
# Find records with more than 3 tags
result = dataset.to_table(
    filter="json_array_length(data, '$.tags') > 3"
)

# Empty arrays return 0
result = dataset.to_table(
    filter="json_array_length(data, '$.empty_array') = 0"
)
```

## JSON Indexing

Lance supports indexing JSON columns to accelerate filters on frequently queried paths.

### Scalar Index on a JSON Path

For `pa.json_()` columns, create a scalar index with `IndexConfig` and specify the JSON
path to index. The query should use the same path literal that was indexed.

```python
import json
import lance
import pyarrow as pa
from lance.indices import IndexConfig

table = pa.table({
    "id": [1, 2, 3, 4],
    "data": pa.array([
        json.dumps({"x": 7, "y": 10}),
        json.dumps({"x": 11, "y": 22}),
        json.dumps({"y": 0}),
        json.dumps({"x": 10}),
    ], type=pa.json_()),
})

lance.write_dataset(table, "json-index.lance")
dataset = lance.dataset("json-index.lance")

dataset.create_scalar_index(
    "data",
    IndexConfig(
        index_type="json",
        parameters={
            "target_index_type": "btree",
            "path": "x",
        },
    ),
)

result = dataset.to_table(filter="json_get_int(data, 'x') = 10")
```

!!! note
    The JSON index matches queries by path literal. For example, if the index is built
    with `path="x"`, then the filter should also use `"x"` with a function such as
    `json_get_int(data, 'x')`. If the index is built with `path="$.user.name"`, then
    the filter should use `json_extract(data, '$.user.name')`.

### Full-Text Search on JSON Documents

If you want text search over the contents of a JSON document instead of scalar filtering
on a single path, create an `INVERTED` index on the JSON column.

```python
dataset.create_scalar_index(
    "data",
    index_type="INVERTED",
    base_tokenizer="simple",
    lower_case=True,
    stem=True,
    remove_stop_words=True,
)
```

!!! note
    JSON columns and nested struct columns are indexed differently. For nested struct
    fields, use dot notation such as `meta.lang`. For `pa.json_()` columns, use the JSON
    index shown above and query with `json_get_*` or `json_extract`.

## Usage Examples

### Working with Nested JSON

```python
import lance
import pyarrow as pa
import json

# Create nested JSON data
data = [
    {
        "id": 1,
        "user": {
            "profile": {
                "name": "Alice",
                "settings": {
                    "theme": "dark",
                    "notifications": True
                }
            },
            "scores": [95, 87, 92]
        }
    },
    {
        "id": 2,
        "user": {
            "profile": {
                "name": "Bob",
                "settings": {
                    "theme": "light",
                    "notifications": False
                }
            },
            "scores": [88, 91, 85]
        }
    }
]

# Convert to Lance dataset
json_strings = [json.dumps(d) for d in data]
table = pa.table({
    "data": pa.array(json_strings, type=pa.json_())
})

lance.write_dataset(table, "nested.lance")
dataset = lance.dataset("nested.lance")

# Query nested fields using JSONPath
dark_theme_users = dataset.to_table(
    filter="json_extract(data, '$.user.profile.settings.theme') = '\"dark\"'"
)

# Or using chained json_get
high_scorers = dataset.to_table(
    filter="json_array_length(data, '$.user.scores') >= 3"
)
```

### Combining JSON with Other Data Types

```python
# Create mixed-type table with JSON metadata
products = pa.table({
    "id": [1, 2, 3],
    "name": ["Laptop", "Phone", "Tablet"],
    "price": [999.99, 599.99, 399.99],
    "specs": pa.array([
        json.dumps({"cpu": "i7", "ram": 16, "storage": 512}),
        json.dumps({"screen": 6.1, "battery": 4000, "5g": True}),
        json.dumps({"screen": 10.5, "battery": 7000, "stylus": True})
    ], type=pa.json_())
})

lance.write_dataset(products, "products.lance")
dataset = lance.dataset("products.lance")

# Find products with specific specs
result = dataset.to_table(
    filter="price < 600 AND json_get_bool(specs, '5g') = true"
)
```

### Handling Arrays in JSON

```python
# Create data with JSON arrays
records = pa.table({
    "id": [1, 2, 3],
    "data": pa.array([
        json.dumps({"name": "Project A", "tags": ["python", "ml", "production"]}),
        json.dumps({"name": "Project B", "tags": ["rust", "systems"]}),
        json.dumps({"name": "Project C", "tags": ["python", "web", "api", "production"]})
    ], type=pa.json_())
})

lance.write_dataset(records, "projects.lance")
dataset = lance.dataset("projects.lance")

# Find projects with Python
python_projects = dataset.to_table(
    filter="json_array_contains(data, '$.tags', 'python')"
)

# Find projects with more than 3 tags
complex_projects = dataset.to_table(
    filter="json_array_length(data, '$.tags') > 3"
)
```

## Performance Considerations

1. **Choose the right function**: Use `json_get_*` functions for direct field access and type conversion; use `json_extract` for complex JSONPath queries.
2. **Index frequently queried paths**: Use a JSON scalar index on frequently filtered paths before creating computed columns for the same fields.
3. **Minimize deep nesting**: While Lance supports arbitrary nesting, flatter structures generally perform better.
4. **Understand type conversion**: The `json_get_*` functions use strict type conversion, which may fail if types don't match. Plan your schema accordingly.
5. **Array access**: When working with JSON arrays, you can access elements by index using numeric strings (e.g., "0", "1") with `json_get` functions.

## Integration with DataFusion

All JSON functions are available when using Lance with Apache DataFusion for SQL queries. See the [DataFusion Integration](../integrations/datafusion.md#json-functions) guide for more details on using JSON functions in SQL contexts.

## Limitations

- JSONPath support follows standard JSONPath syntax but may not support all advanced features
- Large JSON documents may impact query performance
- JSON functions are currently only available for filtering, not for projection in query results

migration.md 5.3 KB

# Migration Guides

Lance aims to avoid breaking changes when possible.  Currently, we are refining the Rust public API so that we can move
it out of experimental status and make stronger commitments to backwards compatibility.  The python API is considered
stable and breaking changes should generally be communicated (via warnings) for 1-2 months prior to being finalized to
give users a chance to migrate.  This page documents the breaking changes between releases and gives advice on how to
migrate.

## 9.0.0

* Unless overridden, newly created FTS indexes use format v2. The code analyzer
  and `block_size=256` require format v3, so readers must support v3 before an
  index using either option is created. `document_granularity="list_element"`
  also requires v3 reader capability, independently of the posting format.

* To keep new indexes readable by nodes that support at most format v1 or v2,
  set `format_version` in the index creation parameters, or set
  `LANCE_FTS_FORMAT_VERSION` for a rollout-wide override. Formats v1 and v2
  require the text analyzer and `block_size=128`.

* Operations that maintain an existing FTS index, including append, incremental
  indexing, optimize, and mem-wal maintained-index flush, preserve its format
  version.

## 7.2.0

* The `IndexSegmentBuilder` API has been removed from Rust, Python, and Java.
  This API was deprecated by the distributed indexing flow based on
  `create_index_uncommitted`, `merge_existing_index_segments`, and
  `commit_existing_index_segments`, but remained in the codebase as a parallel
  way to plan and publish staged index segments.

* Callers should now publish staged segment outputs directly with
  `commit_existing_index_segments(...)`. If multiple staged outputs should be
  combined into a larger physical segment first, callers should explicitly group
  those outputs and call `merge_existing_index_segments(...)` for each group
  before committing the final segment list.

* The old builder's `target_segment_bytes` automatic size-based grouping has no
  direct replacement. Distributed index drivers that used it should choose
  segment groups themselves, then pass each group to
  `merge_existing_index_segments(...)`.

## 5.0.0

* The default data storage version changed from 2.0 to 2.1. This affects the `column_indices`
  field in the `DataFile` protobuf message. In 2.0, every field (including non-leaf fields like
  struct containers and list containers) was assigned a sequential column index. In 2.1, non-leaf
  fields (unpacked structs, list containers) are assigned `-1` instead since their validity
  information is now folded into repetition/definition levels. Only leaf fields and packed structs
  are assigned column indices.

    For example, given the schema:

    ```
    x: i32, y: [f32], z: { a: i32 }
    ```

    The fields (in depth-first order) are:

    | Field ID | Field         |
    |----------|---------------|
    | 0        | `x` (i32)     |
    | 1        | `y` (list)    |
    | 2        | `y.item` (f32)|
    | 3        | `z` (struct)  |
    | 4        | `z.a` (i32)   |

    In **2.0**, `column_indices` = `[0, 1, 2, 3, 4]` — every field gets a column.

    In **2.1**, `column_indices` = `[0, -1, 1, -1, 2]` — non-leaf fields (`y` and `z`) get `-1`.

* This change only affects advanced users who construct `DataFile` messages directly, for example
  when building operations by hand for `Dataset.commit`. Normal read and write paths are
  unaffected.

* To opt back to 2.0 format, set `data_storage_version="2.0"` when creating a dataset.

## 1.0.0

* The `SearchResult` returned by scalar indices must now output information about null values.
  Instead of containing a `RowIdTreeMap`, it now contains a `NullableRowIdSet`. Expressions that
  resolve to null values must be included in search results in the null set. This ensures that
  `NOT` can be applied to index search results correctly.

## 0.39

* The `lance` crate no longer re-exports utilities from `lance-arrow` such as `RecordBatchExt` or `SchemaExt`.  In the
short term, if you are relying on these utilities,  you can add a dependency on the `lance-arrow` crate.  However, we
do not expect `lance-arrow` to ever be stable, and you may want to consider forking these utilities.

* Previously, we exported `Error` and `Result` as both `lance::Error` and `lance::error::Error`.  We have now reduced
this to just `lance::Error`.  We have also removed some internal error utilities (such as `OptionExt`) from the public
API and do not plan on reintroducing these.

* The Python and Rust `dataset::diff_meta` API has been removed in favor of `dataset::delta`,
which returns a `DatasetDelta` that offers both metadata diff through `list_transactions` and data diff
through `get_inserted_rows` and `get_updated_rows`.

* Some other minor utilities which had previously been public are now private.  It is unlikely anyone was utilizing'
these.  Please open an issue if you were relying on any of these.

* The `lance-namespace` Rust crate now splits into `lance-namespace` that contains the main `LanceNamespace` trait
and data models, and `lance-namespace-impls` that has different implementations of the namespace.
The `DirectoryNamespace` and `RestNamespace` interfaces have been refactored to be more user friendly.
The `DirectoryNamespace` also now uses Lance ObjectStore for IO instead of directly depending on Apache OpenDAL.

object_store.md 30.5 KB

# Object Store Configuration

Lance supports object stores such as AWS S3 (and compatible stores), Azure Blob Store,
and Google Cloud Storage. Which object store to use is determined by the URI scheme of
the dataset path. For example, `s3://bucket/path` will use S3, `az://bucket/path`
will use Azure, and `gs://bucket/path` will use GCS.

These object stores take additional configuration objects. There are two ways to
specify these configurations: by setting environment variables or by passing them
to the `storage_options` parameter of `lance.dataset` and
`lance.write_dataset`. So for example, to globally set a higher timeout,
you would run in your shell:

```bash
export TIMEOUT=60s
```

If you only want to set the timeout for a single dataset, you can pass it as a
storage option:

```python
import lance
ds = lance.dataset("s3://path", storage_options={"timeout": "60s"})
```

## General Configuration

These options apply to all object stores.

| Key                          | Description                                                                                                                                                                                                                                                                                             |
|------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `allow_http`                 | Allow non-TLS, i.e. non-HTTPS connections. Default, `False`.                                                                                                                                                                                                                                            |
| `download_retry_count`       | Number of times to retry a download. Default, `3`. This limit is applied when the HTTP request succeeds but the response is not fully downloaded, typically due to a violation of `timeout`.                                                                                                            |
| `allow_invalid_certificates` | Skip certificate validation on https connections. Default, `False`. Warning: This is insecure and should only be used for testing.                                                                                                                                                                      |
| `connect_timeout`            | Timeout for only the connect phase of a Client. Default, `5s`.                                                                                                                                                                                                                                          |
| `timeout`                    | Timeout for the entire request, from connection until the response body has finished. Default, `30s`. This applies to each individual request, so on a large write it must cover one complete multipart part upload; raise it alongside `LANCE_INITIAL_UPLOAD_SIZE`.                                    |
| `user_agent`                 | User agent string to use in requests.                                                                                                                                                                                                                                                                   |
| `proxy_url`                  | URL of a proxy server to use for requests. Default, `None`.                                                                                                                                                                                                                                             |
| `proxy_ca_certificate`       | PEM-formatted CA certificate for proxy connections                                                                                                                                                                                                                                                      |
| `proxy_excludes`             | List of hosts that bypass proxy. This is a comma separated list of domains and IP masks. Any subdomain of the provided domain will be bypassed. For example, `example.com, 192.168.1.0/24` would bypass `https://api.example.com`, `https://www.example.com`, and any IP in the range `192.168.1.0/24`. |
| `client_max_retries`         | Number of times for the object store client to retry the request. Default, `3`.                                                                                                                                                                                                                         |
| `client_retry_timeout`       | Timeout for the object store client to retry the request in seconds. Default, `180`.                                                                                                                                                                                                                    |

### Bulk copy strategy

Lance streams bulk index-file movement and dataset deep-clone files through
read and write APIs by default. This avoids requiring a provider-native copy
operation and works across different object stores.

Set `LANCE_IO_SERVER_SIDE_COPY_ENABLED` to a truthy value (`1`, `true`, `on`,
`yes`, or `y`, case-insensitive) to opt cloud copies whose source and destination
share the same object-store client into the provider-native server-side copy
operation. Cross-client, cross-store, and local copies do not use this setting.
Native copy can reduce client bandwidth and transfer cost, but it requires copy
support from the object-store integration and is subject to the provider
request's timeout and retry behavior.

Deep clone bounds non-local file movement to four concurrent files by default.
Set `LANCE_DEEP_CLONE_STREAM_CONCURRENCY` to a positive integer to override this
operation-specific limit. The bound also applies when server-side copy is
enabled because S3 and GCS copies above the provider's single-copy size limit
fall back to streaming through Lance.

## Per-Base Configuration

A dataset can register additional base paths that store part of its data, and each
base may live in a different bucket, account, or storage provider. A storage option
key of the form `base_<id>.<key>` applies `<key>` only to the base path with that
manifest id. Every base inherits the unscoped options; base-scoped entries add to or
override them for that base only.

```python
import lance
ds = lance.dataset(
    "az://account-a/path",
    storage_options={
        # Shared defaults, used by the primary dataset and inherited by bases
        "account_name": "account-a",
        "account_key": "key-a",
        # Overrides for the base path with id 1
        "base_1.account_name": "account-b",
        "base_1.account_key": "key-b",
    },
)
```

Base ids are assigned when bases are registered (`initial_bases` ids are assigned
sequentially starting at 1, in order) and can be inspected with
`ds.base_paths()`. The returned dictionary maps each base id to its registered
`DatasetBasePath`; its iteration order is unspecified, and it does not include the
primary storage unless that path was explicitly registered as a base.

Keys that do not match `base_<id>.<key>` exactly (e.g. `base_url`) are treated as
regular storage options. Exact per-base parameter maps (`base_store_params`, keyed
by base path URI) take precedence over base-scoped keys for that base.

## S3 Configuration

S3 (and S3-compatible stores) have additional configuration options that configure
authorization and S3-specific features (such as server-side encryption).

AWS credentials can be set in the environment variables `AWS_ACCESS_KEY_ID`,
`AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN`. Alternatively, they can be
passed as parameters to the `storage_options` parameter:

```python
import lance
ds = lance.dataset(
    "s3://bucket/path",
    storage_options={
        "access_key_id": "my-access-key",
        "secret_access_key": "my-secret-key",
        "session_token": "my-session-token",
    }
)
```

If you are using AWS SSO, you can specify the `AWS_PROFILE` environment variable.
It cannot be specified in the `storage_options` parameter.

The following keys can be used as both environment variables or keys in the
`storage_options` parameter:

| Key                                                                 | Description                                                                                                                                      |
|---------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
| `aws_region` / `region`                                             | The AWS region the bucket is in. This can be automatically detected when using AWS S3, but must be specified for S3-compatible stores.           |
| `aws_access_key_id` / `access_key_id`                               | The AWS access key ID to use.                                                                                                                    |
| `aws_secret_access_key` / `secret_access_key`                       | The AWS secret access key to use.                                                                                                                |
| `aws_session_token` / `session_token`                               | The AWS session token to use.                                                                                                                    |
| `aws_endpoint` / `endpoint`                                         | The endpoint to use for S3-compatible stores.                                                                                                    |
| `aws_virtual_hosted_style_request` / `virtual_hosted_style_request` | Whether to use virtual hosted-style requests, where bucket name is part of the endpoint. Meant to be used with `aws_endpoint`. Default, `False`. |
| `aws_s3_express` / `s3_express`                                     | Whether to use S3 Express One Zone endpoints. Default, `False`. See more details below.                                                          |
| `aws_server_side_encryption`                                        | The server-side encryption algorithm to use. Must be one of `"AES256"`, `"aws:kms"`, or `"aws:kms:dsse"`. Default, `None`.                       |
| `aws_sse_kms_key_id`                                                | The KMS key ID to use for server-side encryption. If set, `aws_server_side_encryption` must be `"aws:kms"` or `"aws:kms:dsse"`.                  |
| `aws_sse_bucket_key_enabled`                                        | Whether to use bucket keys for server-side encryption.                                                                                           |

### Credential provider selection

By default, Lance uses the standard AWS credential provider chain (environment
variables, shared config file, web identity tokens, ECS, EC2 instance metadata).

The `aws_provider_scheme` storage option pins a dataset to a specific credential
provider, which is useful when two datasets in the same process need different
AWS auth (for example, one bucket using IRSA and another using ECS container
credentials).

| Value | Behavior |
|-------|----------|
| `token` | Use static access-key credentials. Returns an error if `aws_access_key_id` and `aws_secret_access_key` are not set. |
| `ecs` | Use the ECS/Pod Identity container credential endpoint. Reads `AWS_CONTAINER_CREDENTIALS_FULL_URI` or `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` from the environment. |
| `irsa` | Use IRSA (IAM Roles for Service Accounts) web identity token credentials. Reads `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` from the environment. |

```python
import lance

# Bucket A — use IRSA (web identity token from the environment)
ds_a = lance.dataset(
    "s3://bucket-a/path",
    storage_options={"aws_provider_scheme": "irsa"},
)

# Bucket B — use ECS container credentials
ds_b = lance.dataset(
    "s3://bucket-b/path",
    storage_options={"aws_provider_scheme": "ecs"},
)
```

### S3-compatible stores

Lance can also connect to S3-compatible stores, such as MinIO. To do so, you must
specify both region and endpoint:

```python
import lance
ds = lance.dataset(
    "s3://bucket/path",
    storage_options={
        "region": "us-east-1",
        "endpoint": "http://minio:9000",
    }
)
```

This can also be done with the `AWS_ENDPOINT` and `AWS_DEFAULT_REGION` environment variables.

### S3 Express (Directory Bucket)

Lance supports [S3 Express One Zone](https://aws.amazon.com/s3/storage-classes/express-one-zone/) buckets,
a.k.a. S3 directory buckets.
S3 Express buckets only support connecting from an EC2 instance within the same
region.
By default, Lance automatically recognize the `--x-s3` suffix of an express bucket,
there is no special configuration needed.

In case of an access point or private link that hides the bucket name,
you can configure express bucket access explicitly through storage option `s3_express`.

```python
import lance
ds = lance.dataset(
    "s3://my-bucket--use1-az4--x-s3/path/imagenet.lance",
    storage_options={
        "region": "us-east-1",
        "s3_express": "true",
    }
)
```

## Google Cloud Storage Configuration

GCS credentials are configured by setting the `GOOGLE_SERVICE_ACCOUNT` environment
variable to the path of a JSON file containing the service account credentials.
Alternatively, you can pass the path to the JSON file in the `storage_options`

```python
import lance
ds = lance.dataset(
    "gs://my-bucket/my-dataset",
    storage_options={
        "service_account": "path/to/service-account.json",
    }
)
```

!!! note

    By default, GCS uses HTTP/1 for communication, as opposed to HTTP/2. This improves
    maximum throughput significantly. However, if you wish to use HTTP/2 for some reason,
    you can set the environment variable `HTTP1_ONLY` to `false`.

The following keys can be used as both environment variables or keys in the
`storage_options` parameter:

| Key | Description |
|-----|-------------|
| `google_service_account` / `service_account` | Path to the service account JSON file. |
| `google_service_account_key` / `service_account_key` | The serialized service account key. |
| `google_application_credentials` / `application_credentials` | Path to the application credentials. |

## Azure Blob Storage Configuration

Azure Blob Storage credentials can be configured by setting the `AZURE_STORAGE_ACCOUNT_NAME`
and `AZURE_STORAGE_ACCOUNT_KEY` environment variables. Alternatively, you can pass
the account name and key in the `storage_options` parameter:

```python
import lance
ds = lance.dataset(
    "az://my-container/my-dataset",
    storage_options={
        "account_name": "some-account",
        "account_key": "some-key",
    }
)
```

These keys can be used as both environment variables or keys in the `storage_options` parameter:

| Key | Description |
|-----|-------------|
| `azure_storage_account_name` / `account_name` | The name of the azure storage account. |
| `azure_storage_account_key` / `account_key` | The serialized service account key. |
| `azure_client_id` / `client_id` | Service principal client id for authorizing requests. |
| `azure_client_secret` / `client_secret` | Service principal client secret for authorizing requests. |
| `azure_tenant_id` / `tenant_id` | Tenant id used in oauth flows. |
| `azure_storage_sas_key` / `azure_storage_sas_token` / `sas_key` / `sas_token` | Shared access signature. The signature is expected to be percent-encoded, much like they are provided in the azure storage explorer or azure portal. |
| `azure_storage_token` / `bearer_token` / `token` | Bearer token. |
| `azure_storage_use_emulator` / `object_store_use_emulator` / `use_emulator` | Use object store with azurite storage emulator. |
| `azure_endpoint` / `endpoint` | Override the endpoint used to communicate with blob storage. |
| `azure_use_fabric_endpoint` / `use_fabric_endpoint` | Use object store with url scheme account.dfs.fabric.microsoft.com. |
| `azure_msi_endpoint` / `azure_identity_endpoint` / `identity_endpoint` / `msi_endpoint` | Endpoint to request a imds managed identity token. |
| `azure_object_id` / `object_id` | Object id for use with managed identity authentication. |
| `azure_msi_resource_id` / `msi_resource_id` | Msi resource id for use with managed identity authentication. |
| `azure_federated_token_file` / `federated_token_file` | File containing token for Azure AD workload identity federation. |
| `azure_use_azure_cli` / `use_azure_cli` | Use azure cli for acquiring access token. |
| `azure_disable_tagging` / `disable_tagging` | Disables tagging objects. This can be desirable if not supported by the backing store. |

## AliCloud Object Storage Service Configuration

OSS credentials can be set in the environment variables `OSS_ACCESS_KEY_ID`,
`OSS_ACCESS_KEY_SECRET`, `OSS_REGION`, and `OSS_SECURITY_TOKEN`. Alternatively, they can be
passed as parameters to the `storage_options` parameter:

```python
import lance
ds = lance.dataset(
    "oss://bucket/path",
    storage_options={
        "oss_region": "oss-region",
        "oss_endpoint": "oss-endpoint",
        "oss_access_key_id": "my-access-key",
        "oss_secret_access_key": "my-secret-key",
        "oss_security_token": "my-session-token",
    }
)
```

| Key | Description |
|-----|-------------|
| `oss_endpoint` | OSS endpoint. Required (for example, `https://oss-cn-hangzhou.aliyuncs.com`). |
| `oss_access_key_id` | Access key ID used for OSS authentication. Optional if credentials are provided by environment. |
| `oss_secret_access_key` | Access key secret used for OSS authentication. Optional if credentials are provided by environment. |
| `oss_region` | OSS region (for example, `cn-hangzhou`). Optional. |
| `oss_security_token` | Security token for temporary credentials (STS). Optional. |

## Volcengine TOS Configuration

TOS credentials can be set in the environment variables `TOS_ACCESS_KEY_ID`,
`TOS_SECRET_ACCESS_KEY`, `TOS_ENDPOINT`, `TOS_REGION`, and `TOS_SECURITY_TOKEN`.
Lance also accepts the corresponding `VOLCENGINE_` environment variable prefix.
Alternatively, credentials can be passed as parameters to the `storage_options`
parameter; explicit `storage_options` override environment variables:

```python
import lance
ds = lance.dataset(
    "tos://bucket/path",
    storage_options={
        "tos_endpoint": "https://tos-cn-beijing.volces.com",
        "tos_region": "cn-beijing",
        "tos_access_key_id": "my-access-key",
        "tos_secret_access_key": "my-secret-key",
        "tos_security_token": "my-session-token",
    }
)
```

| Key | Description |
|-----|-------------|
| `tos_endpoint` | TOS endpoint. Required (for example, `https://tos-cn-beijing.volces.com`). |
| `tos_region` | TOS signing region (for example, `cn-beijing`). Optional. |
| `tos_access_key_id` | Access key ID used for TOS authentication. Optional if credentials are provided by environment. |
| `tos_secret_access_key` | Secret access key used for TOS authentication. Optional if credentials are provided by environment. |
| `tos_security_token` | Security token for temporary credentials. Optional. |

## Tencent Cloud COS Configuration

[COS (Cloud Object Storage)](https://cloud.tencent.com/product/cos) credentials can be set in environment variables prefixed
with `COS_` or `TENCENTCLOUD_` (for example, `COS_ENDPOINT`, `COS_SECRET_ID`,
`COS_SECRET_KEY`, `TENCENTCLOUD_REGION`, `TENCENTCLOUD_SECURITY_TOKEN`).
Alternatively, credentials can be passed as parameters to the `storage_options`
parameter; explicit `storage_options` override environment variables:

=== "Python"

    ```python
    import lance
    ds = lance.dataset(
        "cos://bucket/path",
        storage_options={
            "cos_endpoint": "https://cos.ap-guangzhou.myqcloud.com",
            "cos_secret_id": "my-secret-id",
            "cos_secret_key": "my-secret-key",
        }
    )
    ```

=== "Rust"

    In this Lance distribution, `tencent` is already part of the **default
    features** of the `lance` crate, so simply depending on `lance` is enough:

    ```toml
    [dependencies]
    lance = "*"
    ```

    You only need to enable the `tencent` feature explicitly in the following
    cases:

    - You opted out of default features, e.g.
      `lance = { version = "*", default-features = false, features = ["tencent", ...] }`.
    - You depend on `lance-io` directly (without `lance`); `tencent` is **not**
      a default feature of `lance-io`:
      `lance-io = { version = "*", features = ["tencent"] }`.

| Key | Description |
|-----|-------------|
| `cos_endpoint` | COS endpoint. Required (for example, `https://cos.ap-guangzhou.myqcloud.com`). Can also be set via the `COS_ENDPOINT` environment variable. |
| `cos_secret_id` | Secret ID used for COS authentication. Optional if credentials are provided by environment. |
| `cos_secret_key` | Secret key used for COS authentication. Optional if credentials are provided by environment. |
| `cos_enable_versioning` | Whether to enable object versioning on the bucket. Optional. |

!!! warning

    Tencent COS does not reliably enforce put-if-not-exists on buckets that have
    ever had versioning enabled, even if versioning is now suspended. To prevent
    silent manifest overwrites, Lance requires a custom distributed commit lock
    for COS writes. Pass the same `commit_lock` implementation to every Python
    writer, or provide a custom `CommitHandler` in Rust. Reads do not require a
    commit lock.

!!! note

    The OpenDAL `CosConfig` currently exposes a limited set of options. Additional
    settings such as the security token (`TENCENTCLOUD_SECURITY_TOKEN`) and region
    (`TENCENTCLOUD_REGION`) must be configured via environment variables.

## Hugging Face Configuration

Use `hf://datasets/<owner>/<repo>/<path>` to read a Lance dataset hosted on
Hugging Face. Pass these options through `storage_options`:

| Key | Description |
| --- | --- |
| `hf_token` | Hugging Face access token. Falls back to `HF_TOKEN` or `HUGGINGFACE_TOKEN` when omitted. |
| `hf_revision` | Repository revision, such as a commit ID, branch, or tag. Defaults to `main`. |
| `hf_download_mode` | `http` (default) or `xet`. |
| `hf_enable_resolve_cache` | `"true"` reuses resolved HTTP download URLs and XET file metadata across readers. Defaults to `"false"`. |

These options also accept names without the `hf_` prefix. The prefixed name
takes precedence when both are supplied. `hf_enable_resolve_cache` accepts only
the strings `"true"` and `"false"`.

Enable the resolve cache only when existing files will not change. Updates,
including changes behind a moving branch or tag, may remain invisible while
cached results are reused. HTTP download URLs refresh near expiry, and issued
URLs may remain usable until expiry after Hub permissions change. The cache
reduces Hub resolution requests; it does not cache file contents.

```python
import lance

ds = lance.dataset(
    "hf://datasets/owner/repo/data.lance",
    storage_options={"hf_enable_resolve_cache": "true"},
)
```

## GooseFS Configuration

[GooseFS](https://cloud.tencent.com/product/goosefs) is a distributed caching
filesystem. Lance accesses GooseFS through its Master gRPC service. The URL format
is `goosefs://host:port/path`, where `host:port` is the GooseFS Master address
(default port: `9200`, may be omitted, e.g. `goosefs://10.0.0.1/path`) and
`/path` is the filesystem path within GooseFS.

Manifest commits on `goosefs://` use `ConditionalPutCommitHandler`
(`PutMode::Create` / if-not-exists), backed by GooseFS master's atomic
no-replace rename so concurrent writers cannot clobber each other's
versioned manifests.

!!! warning "Mixed-version writers are NOT safe"

    The `if-not-exists` guarantee only holds when **every** writer for a
    dataset routes through this new handler. A writer running an older
    Lance release still selects `UnsafeCommitHandler` for `goosefs://`
    and writes the version path unconditionally, which can overwrite a
    manifest that an upgraded writer has already won. Safe concurrent
    commits therefore require:

    - all writers for the dataset run a Lance release that includes this
      routing change, **or**
    - writers share an external coordination boundary (e.g. a single-writer
      queue, table-level lock, or a gateway that serializes commits) that
      prevents the old code path from racing the new one.

    When upgrading in place, quiesce all writers (drain jobs, scale
    clients to zero, or route traffic through a writer coordinator) before
    rolling out the new Lance version, then bring writers back on the new
    version together.

!!! note "About the dataset path"

    `/path` is just an arbitrary directory inside GooseFS — Lance does **not**
    require the path to end with a `.lance` suffix. Any valid GooseFS directory
    works, for example:

    - `goosefs://10.0.0.1:9200/data/my-dataset`
    - `goosefs://10.0.0.1:9200/data/my-dataset.lance`
    - `goosefs://10.0.0.1:9200/lance-test/lance-io`

    The `.lance` suffix used in the examples below is only a naming convention
    that makes it easy to recognize a Lance dataset directory at a glance; it
    has no special meaning to Lance itself. The only requirement is that the
    same path is used consistently for reads and writes of a given dataset.

=== "Python"

    ```python
    import lance

    ds = lance.dataset(
        "goosefs://10.0.0.1:9200/data/my-dataset.lance",
        storage_options={
            "goosefs_auth_type": "simple",
            "goosefs_auth_username": "lance",
        },
    )
    ```

=== "Rust"

    In this Lance distribution, `goosefs` is already part of the **default
    features** of the `lance` crate, so simply depending on `lance` is enough:

    ```toml
    [dependencies]
    lance = "*"
    ```

    You only need to enable the `goosefs` feature explicitly in the following
    cases:

    - You opted out of default features, e.g.
      `lance = { version = "*", default-features = false, features = ["goosefs", ...] }`.
    - You depend on `lance-io` directly (without `lance`); `goosefs` is **not**
      a default feature of `lance-io`:
      `lance-io = { version = "*", features = ["goosefs"] }`.

    Open the underlying `lance_io::object_store::ObjectStore` directly (mirrors
    the integration test in `rust/lance-io/tests/goosefs_integration.rs`):

    ```rust
    use lance_io::object_store::ObjectStore;

    let uri = "goosefs://10.0.0.1:9200/lance-test/lance-io";
    let (store, path) = ObjectStore::from_uri(uri).await?;

    // Read / write through the underlying `object_store::ObjectStore` API
    store.inner.put(&path, (&b"hello"[..]).into()).await?;
    let result = store.inner.get(&path).await?;
    let bytes = result.bytes().await?;
    ```

    Open a Lance dataset with custom storage options:

    ```rust
    use std::collections::HashMap;
    use lance::dataset::DatasetBuilder;

    let mut storage_options = HashMap::new();
    storage_options.insert("goosefs_master_addr".to_string(), "10.0.0.1:9200".to_string());
    storage_options.insert("goosefs_auth_type".to_string(), "simple".to_string());
    storage_options.insert("goosefs_auth_username".to_string(), "lance".to_string());

    let dataset = DatasetBuilder::from_uri("goosefs://10.0.0.1:9200/data/my-dataset.lance")
        .with_storage_options(storage_options)
        .load()
        .await?;
    ```

=== "Java"

    Pass the GooseFS configuration through `ReadOptions.setStorageOptions`
    when opening the dataset:

    ```java
    import org.lance.Dataset;
    import org.lance.ReadOptions;

    import java.util.HashMap;
    import java.util.Map;

    Map<String, String> storageOptions = new HashMap<>();
    storageOptions.put("goosefs_master_addr", "10.0.0.1:9200");
    storageOptions.put("goosefs_auth_type", "simple");
    storageOptions.put("goosefs_auth_username", "lance");

    ReadOptions options = new ReadOptions.Builder()
        .setStorageOptions(storageOptions)
        .build();

    try (Dataset dataset = Dataset.open()
            .uri("goosefs://10.0.0.1:9200/data/my-dataset.lance")
            .readOptions(options)
            .build()) {
        // ... use the dataset
    }
    ```

    For writes, the same `storageOptions(...)` setter is available on
    `WriteDatasetBuilder` and `WriteFragmentBuilder`.

The Master address can be resolved from (in priority order):

1. The `goosefs_master_addr` storage option (supports HA: `"addr1:port,addr2:port"`).
2. The `GOOSEFS_MASTER_ADDR` environment variable.
3. The host and port from the URL authority.

`storage_options` keys **must be lowercase**. Uppercase or mixed-case spellings
such as `GOOSEFS_MASTER_ADDR` are rejected with an explicit error — they are
not ignored, and they are not treated as the matching environment variable.
Environment variables keep the `GOOSEFS_*` form.

| storage_options key | env var | Description |
|---------------------|---------|-------------|
| `goosefs_master_addr` | `GOOSEFS_MASTER_ADDR` | GooseFS Master address. Supports a single address (`host:port`) or comma-separated HA addresses (`addr1:port,addr2:port`). Optional if the address is provided in the URL. |
| `goosefs_write_type` | `GOOSEFS_WRITE_TYPE` | Write type, e.g. `MUST_CACHE`, `CACHE_THROUGH`, `THROUGH`, `ASYNC_THROUGH`. Optional. |
| `goosefs_block_size` | `GOOSEFS_BLOCK_SIZE` | GooseFS block size (this is the GooseFS-side block size, not Lance's I/O block size). Accepts a raw byte count or GooseFS suffixes such as `64MB` (binary units: `1KB = 1024`). Optional. |
| `goosefs_chunk_size` | `GOOSEFS_CHUNK_SIZE` | Chunk size used when reading or writing files. Accepts a raw byte count or GooseFS suffixes such as `4MB` (binary units: `1KB = 1024`). Optional. |
| `goosefs_auth_type` | `GOOSEFS_AUTH_TYPE` | Authentication type. Either `nosasl` or `simple` (case-insensitive; the value is passed through to OpenDAL). Optional. |
| `goosefs_auth_username` | `GOOSEFS_AUTH_USERNAME` | Username used in `simple` authentication mode. Optional. |

!!! note "Running the GooseFS integration tests"

    The Rust integration tests for GooseFS live at
    `rust/lance-io/tests/goosefs_integration.rs` and are gated behind feature
    flags. They require a reachable GooseFS cluster (configured via the
    `GOOSEFS_MASTER_ADDR` and `GOOSEFS_AUTH_TYPE` environment variables) and
    can be run with:

    ```bash
    cargo test -p lance-io --features "goosefs goosefs-test" \
        --test goosefs_integration -- --ignored --nocapture --test-threads=1
    ```

observability.md 3.6 KB

# Observability

Lance can publish operational metrics to your monitoring stack. The table below
is the authoritative catalogue of the metrics Lance emits, shared verbatim with
the Rust [`lance::metrics`](https://github.com/lance-format/lance/blob/main/rust/lance/src/metrics.md)
module documentation.

--8<-- "rust/lance/src/metrics.md"

## Collecting metrics

Lance emits through the [`metrics`](https://docs.rs/metrics) crate facade, so it
is not tied to a specific backend — you install a recorder/exporter and route
the metrics wherever you like. Metrics are available from the Rust, Python, and
Java APIs.

### Rust

Enable the `metrics` feature on the `lance` crate:

```toml
lance = { version = "...", features = ["metrics"] }
```

Then install any `metrics`-compatible recorder once at startup, before opening
datasets. For example, with
[`metrics-exporter-prometheus`](https://docs.rs/metrics-exporter-prometheus):

```rust
metrics_exporter_prometheus::PrometheusBuilder::new()
    .install()
    .expect("install Prometheus recorder");
```

Any recorder works — Prometheus, StatsD, an OpenTelemetry bridge, and so on.
When no recorder is installed, emission is a cheap no-op.

### Python

Unlike Rust, the Python bindings do not let you plug in an arbitrary recorder:
bridging one across the FFI boundary into the Rust `metrics` facade would be
complicated and inefficient. Instead `pylance` standardizes on OpenTelemetry,
which has good Python support, as its recorder.

The `pylance` wheels are built with the `metrics` feature enabled. Install the
OpenTelemetry extra and call `instrument_lance_metrics`, which registers Lance's
metrics as observable instruments on your OpenTelemetry `MeterProvider`:

```bash
pip install "pylance[otel]"
```

```python
from lance.otel import instrument_lance_metrics

# Uses the global MeterProvider; pass meter_provider=... to target a specific one.
instrument_lance_metrics()
```

### Java

The Java SDK includes an OpenTelemetry bridge in `org.lance.otel`. Register it
before opening datasets or performing Lance IO so the process-global Rust
recorder sees every emitted metric:

```java
import org.lance.otel.LanceMetrics;

LanceMetrics.instrument();
```

The OpenTelemetry API is a dependency of the Java SDK. The application must
still configure an OpenTelemetry SDK, metric reader, and exporter for collection
and delivery.

The no-argument method uses OpenTelemetry's global `MeterProvider`. To register
with an explicitly configured provider, pass it directly:

```java
SdkMeterProvider provider = SdkMeterProvider.builder()
    .registerMetricReader(metricReader)
    .build();
LanceMetrics.instrument(provider);
```

Repeated calls with the same provider are idempotent. Passing a different
provider unregisters the existing callback before registering the new one. Call
`LanceMetrics.close()` to stop exporting while retaining the process-global Rust
metric state.

From there the metrics flow through whatever OpenTelemetry pipeline you have
configured (OTLP, Prometheus, console, …). Because OpenTelemetry has no
asynchronous histogram instrument, histograms are exported Prometheus-style as
three observable counters: `<name>_bucket`, `<name>_count`, and `<name>_sum`.
Each `<name>_bucket` sample carries an `le` ("less than or equal") attribute
giving that bucket's inclusive upper bound in the metric's unit; the bucket
count is cumulative, covering every observation at or below `le`. For example, a
`lance_object_store_request_duration_seconds_bucket` sample with `le="0.5"`
counts all requests that completed in 0.5 seconds or less, while `le="+Inf"` is
the total count.

performance.md 31.3 KB

# Lance Performance Guide

This guide provides tips and tricks for optimizing the performance of your Lance applications.

## Logging

Lance uses the `log` crate to log messages. Displaying these log messages will depend on the client
library you are using. For rust, you will need to configure a logging subscriber. For more details
ses the [log](https://docs.rs/log/latest/log/) docs. The Python and Java clients configure a default
logging subscriber that logs to stderr.

The Python/Java logger can be configured with several environment variables:

- `LANCE_LOG`: Controls log filtering based on log level and target. See the [env_logger](https://docs.rs/env_logger/latest/env_logger/) docs for more details. The `LANCE_LOG` environment variable replaces the `RUST_LOG` environment variable.
- `LANCE_TRACING`: Controls tracing filtering based on log level. Key tracing events described below are emitted at
  the `info` level. However, additional spans and events are available at the `debug` level which may be useful for
  debugging performance issues. The default tracing level is `info`.
- `LANCE_LOG_STYLE`: Controls whether colors are used in the log messages. Valid values are `auto`, `always`, `never`.
- `LANCE_LOG_TS_PRECISION`: The precision of the timestamp in the log messages. Valid values are `ns`, `us`, `ms`, `s`.
- `LANCE_LOG_FILE`: Redirects Rust log messages to the specified file path instead of stderr. When set, Lance will create the file and any necessary parent directories. If the file cannot be created (e.g., due to permission issues), Lance will fall back to logging to stderr.

## Trace Events

Lance uses tracing to log events. If you are running `pylance` then these events will be emitted
as log messages. For Rust connections you can use the `tracing` crate to capture these events.

Rust tracing targets are listed below. In `pylance` logs, trace events are emitted under a
`lance::events::` prefix so they can be filtered separately from normal log records. For example,
`LANCE_LOG="warn,lance::events::object_store::throttle=info"` shows storage throttling events
without enabling other Lance event logs.

### File Audit

File audit events are emitted when significant files are created or deleted.

| Event               | Parameter | Description                                                                |
| ------------------- | --------- | -------------------------------------------------------------------------- |
| `lance::file_audit` | `mode`    | The mode of I/O operation (create, delete, delete_unverified)              |
| `lance::file_audit` | `type`    | The type of file affected (manifest, data file, index file, deletion file) |

### Dataset Events

Dataset events are emitted when datasets are loaded, written, committed, deleted, compacted, or cleaned.

| Event                   | Parameter   | Description                                                               |
| ----------------------- | ----------- | ------------------------------------------------------------------------- |
| `lance::dataset_events` | `event`     | The dataset event type (loading, writing, committed, deleting, and others) |
| `lance::dataset_events` | `uri`       | The dataset URI                                                           |
| `lance::dataset_events` | `mode`      | The write mode                                                            |
| `lance::dataset_events` | `operation` | The committed transaction operation                                       |
| `lance::dataset_events` | `predicate` | The delete predicate                                                      |
| `lance::dataset_events` | `columns`   | The removed columns                                                       |

### Object Store Throttle Events

Object store throttle events are emitted when Lance observes cloud storage throttle responses and
reduces or retries request rates.

| Event                            | Parameter       | Description                              |
| -------------------------------- | --------------- | ---------------------------------------- |
| `lance::object_store::throttle`  | `previous_rate` | The request rate before AIMD adjustment  |
| `lance::object_store::throttle`  | `new_rate`      | The request rate after AIMD adjustment   |
| `lance::object_store::throttle`  | `attempt`       | The retry attempt for retry debug events |
| `lance::object_store::throttle`  | `error`         | The underlying object store throttle error      |

### I/O Events

I/O events are emitted when significant I/O operations are performed, particularly
those related to indices. These events are NOT emitted when the index is loaded from
the in-memory cache. Correct cache utilization is important for performance and these
events are intended to help you debug cache usage.

| Event              | Parameter | Description                                                                                          |
| ------------------ | --------- | ---------------------------------------------------------------------------------------------------- |
| `lance::io_events` | `type`    | The type of I/O operation (open_scalar_index, open_vector_index, load_vector_part, load_scalar_part) |

### Execution Events

Execution events are emitted when an execution plan is run. These events are useful for
debugging query performance.

| Event              | Parameter           | Description                                                    |
| ------------------ | ------------------- | -------------------------------------------------------------- |
| `lance::execution` | `type`              | The type of execution event (plan_run is the only type today)  |
| `lance::execution` | `output_rows`       | The number of rows in the output of the plan                   |
| `lance::execution` | `iops`              | The number of I/O operations performed by the plan             |
| `lance::execution` | `bytes_read`        | The number of bytes read by the plan                           |
| `lance::execution` | `indices_loaded`    | The number of indices loaded by the plan                       |
| `lance::execution` | `parts_loaded`      | The number of index partitions loaded by the plan              |
| `lance::execution` | `index_comparisons` | The number of comparisons performed inside the various indices |

## Threading Model

Lance is designed to be thread-safe and performant. Lance APIs can be called concurrently unless
explicitly stated otherwise. Users may create multiple tables and share tables between threads.
Operations may run in parallel on the same table, but some operations may lead to conflicts. For
details see [conflict resolution](../format/table/transaction.md/#conflict-resolution).

Most Lance operations will use multiple threads to perform work in parallel. There are two thread
pools in lance: the IO thread pool and the compute thread pool. The IO thread pool is used for
reading and writing data from disk. The compute thread pool is used for performing computations
on data. The number of threads in each pool can be configured by the user.

The IO thread pool is used for reading and writing data from disk. The number of threads in the IO
thread pool is determined by the object store that the operation is working with. Local object stores
will use 8 threads by default. Cloud object stores will use 64 threads by default. This is a fairly
conservative default and you may need 128 or 256 threads to saturate network bandwidth on some cloud
providers. The `LANCE_IO_THREADS` environment variable can be used to override the number of IO
threads. If you increase this variable you may also want to increase the `io_buffer_size`.

The compute thread pool is used for performing computations on data. The number of threads in the
compute thread pool is determined by the number of cores on the machine. The number of threads in
the compute thread pool can be overridden by setting the `LANCE_CPU_THREADS` environment variable.
This is commonly done when running multiple Lance processes on the same machine (e.g when working with
tools like Ray). Keep in mind that decoding data is a compute intensive operation, even if a workload
seems I/O bound (like scanning a table) it may still need quite a few compute threads to achieve peak
performance.

## Memory Requirements

Lance is designed to be memory efficient. Operations should stream data from disk and not require
loading the entire dataset into memory. However, there are a few components of Lance that can use
a lot of memory.

### Metadata Cache

Lance uses a metadata cache to speed up operations. This cache holds various pieces of metadata such
as file metadata, dataset manifests, etc. This cache is an LRU cache that is sized by bytes. The default
size is 1 GiB.

The metadata cache is not shared between tables by default. For best performance you should create a
single table and share it across your application. Alternatively, you can create a single session
and specify it when you open tables.

Keys are often a composite of multiple fields and all keys are scoped to the dataset URI. The following items are stored in the metadata cache:

| Item              | Key                                              | What is stored                      |
| ----------------- | ------------------------------------------------ | ----------------------------------- |
| Dataset Manifests | Dataset URI, version, and etag                   | The manifest for the dataset        |
| Transactions      | Dataset URI, version                             | The transaction for the dataset     |
| Deletion Files    | Dataset URI, fragment_id, version, id, file_type | The deletion vector for a frag      |
| Row Id Mask       | Dataset URI, version                             | The row id sequence for the dataset |
| Row Id Index      | Dataset URI, version                             | The row id index for the dataset    |
| Row Id Sequence   | Dataset URI, fragment_id, row_id_meta            | The row id sequence for a fragment  |
| Index Metadata    | Dataset URI, version                             | The index metadata for the dataset  |
| Index Details¹    | Dataset URI, index uuid                          | The index details for an index      |
| File Global Meta  | Dataset URI, file path                           | The global metadata for a file      |
| File Column Meta  | Dataset URI, file path, column index             | The search cache for a column       |

Notes:

1. This is only stored for very old indexes which don't store their details in the manifest.

### Index Cache

Lance uses an index cache to speed up queries. This caches vector and scalar indices in memory. The
max size of this cache can be configured when creating a `LanceDataset` using the `index_cache_size_bytes`
parameter. This cache is an LRU cached that is sized by bytes. The default size is 6 GiB.
You can view the size of this cache by inspecting the result of `dataset.session().size_bytes()`.

The index cache is not shared between tables. For best performance you should create a single table and
share it across your application.

**Note**: `index_cache_size` (specified in entries) was deprecated since version 0.30.0. Use
`index_cache_size_bytes` (specified in bytes) for new code.

### Scanning Data

Searches (e.g. vector search, full text search) do not use a lot of memory to hold data because they don't
typically return a lot of data. However, scanning data can use a lot of memory. Scanning is a streaming
operation but we need enough memory to hold the data that we are scanning. The amount of memory needed is
largely determined by the `io_buffer_size` and the `batch_size` variables.

Each I/O thread should have enough memory to buffer an entire page of data. Pages today are typically between
8 and 32 MB. This means, as a rule of thumb, you should generally have about 32MB of memory per I/O thread.
The default `io_buffer_size` is 2GB which is enough to buffer 64 pages of data. If you increase the number
of I/O threads you should also increase the `io_buffer_size`.

Scans will also decode data (and run any filtering or compute) in parallel on CPU threads. The amount of data
decoded at any one time is determined by the `batch_size` and the size of your rows. Each CPU thread will
need enough memory to hold one batch. Once batches are delivered to your application, they are no longer tracked
by Lance and so if memory is a concern then you should also be careful not to accumulate memory in your own
application (e.g. by running `to_table` or otherwise collecting all batches in memory.)

The default `batch_size` is 8192 rows. When you are working with mostly scalar data you want to keep batches
around 1MB and so the amount of memory needed by the compute threads is fairly small. However, when working with
large data you may need to turn down the `batch_size` to keep memory usage under control. For example, when
working with 1024-dimensional vector embeddings (e.g. 32-bit floats) then 8192 rows would be 32MB of data. If you
spread that across 16 CPU threads then you would need 512MB of compute memory per scan. You might find working
with 1024 rows per batch is more appropriate.

#### Tuning remote scans

An ordered dataset scan still overlaps I/O from multiple fragments. `scan_in_order=True` controls the order in
which batches are returned; it does not make fragment reads sequential. This is why a dataset scan can issue
more concurrent requests than scanning one fragment directly. The following controls tune different parts of
the scan:

* `fragment_readahead` limits how many fragments may have reads scheduled concurrently. Set it to `1` to match
  the fragment-level I/O pattern, then increase it if the storage connection has spare bandwidth.
* `LANCE_IO_THREADS` limits concurrent storage requests for the process. Cloud stores default to 64, which is
  intended for high-bandwidth, in-region access and can be too aggressive across regions or over the public
  internet.
* `io_buffer_size` limits buffered I/O bytes and applies backpressure when decoding falls behind.
* `batch_readahead` limits concurrent batch decoding. It does not control the size of storage range requests.

For a bandwidth-constrained remote connection, start with conservative settings and tune upward:

```shell
LANCE_IO_THREADS=8 python scan.py
```

```python
scanner = dataset.scanner(
    fragment_readahead=1,
    batch_readahead=2,
    io_buffer_size=64 * 1024 * 1024,
)
for batch in scanner.to_batches():
    process(batch)
```

Lance reads encoded pages from storage, so reducing `batch_size` changes the returned and decoded batch sizes
but may not reduce the initial range request. The first batch can require loading one encoded page for each
selected column.

In summary, scans could use up to `(2 * io_buffer_size) + (batch_size * num_compute_threads)` bytes of memory.
Keep in mind that `io_buffer_size` is a soft limit (e.g. we cannot read less than one page at a time right now)
and so it is not necessarily a bug if you see memory usage exceed this limit by a small margin.

### Cloud Store Throttling

Cloud object stores (S3, GCS, Azure) are automatically wrapped with an AIMD (Additive Increase / Multiplicative
Decrease) rate limiter. When the store returns throttle errors (HTTP 429/503), the request rate decreases
multiplicatively. During sustained success, the rate increases additively. This applies to all operations
(reads, writes, deletes, lists) and replaces the old `LANCE_PROCESS_IO_THREADS_LIMIT` process-wide cap.

Local and in-memory stores are **not** throttled.

The AIMD throttle can be tuned via storage options or environment variables. Storage options take precedence
over environment variables:

| Setting            | Storage Option Key              | Env Var                         | Default |
| ------------------ | ------------------------------- | ------------------------------- | ------- |
| Initial rate       | `lance_aimd_initial_rate`       | `LANCE_AIMD_INITIAL_RATE`       | 2000    |
| Min rate           | `lance_aimd_min_rate`           | `LANCE_AIMD_MIN_RATE`           | 1       |
| Max rate           | `lance_aimd_max_rate`           | `LANCE_AIMD_MAX_RATE`           | 5000    |
| Decrease factor    | `lance_aimd_decrease_factor`    | `LANCE_AIMD_DECREASE_FACTOR`    | 0.5     |
| Additive increment | `lance_aimd_additive_increment` | `LANCE_AIMD_ADDITIVE_INCREMENT` | 300     |
| Burst capacity     | `lance_aimd_burst_capacity`     | `LANCE_AIMD_BURST_CAPACITY`     | 100     |

These initial settings are balanced and should work for most
use cases. For example, S3 can typically get up to 5000
req/s and with these settings we should get there in about
10 seconds.

## Fragment Sizing

A Lance table is a collection of fragments tracked by a manifest. How you size those fragments
trades off two classes of work:

- **Manifest-level operations** scale with the *number* of fragments. Every dataset mutation
  (appends, metadata updates, schema changes, compactions, etc.) rewrites the manifest, so a
  larger fragment list makes every write slower. Reads pay a similar cost up front: opening a
  dataset, listing fragments, planning a scan, and resolving transaction conflicts at the
  dataset level all walk the manifest.
- **Fragment-level operations** scale with the *size* of a fragment. These include scans
  against a matching fragment, compaction, updates, deletes, and `merge_insert`. Conflict
  detection for these operations is also done at the fragment level.

Fewer, larger fragments make manifest-level operations cheap but make each fragment-level
operation heavier and increase the chance of conflicts when many writers target the same
fragment. More, smaller fragments do the reverse.

Practical guidance:

- The default of 1M rows per fragment works well up to ~1B rows. Past that, bumping toward
  ~100M rows per fragment is reasonable, though fragment-count limits are rarely the bottleneck
  in practice.
- Tens of thousands of fragments per table is generally fine.
- Keep individual fragments well under object-store object-size limits (S3 caps at 5 TB, and
  stores tend to misbehave well before that). 10 GB–100 GB per fragment is a reasonable upper
  range; 1 TB is a hard ceiling.
- If you run many concurrent updates, deletes, or `merge_insert` operations, err toward more
  fragments — conflict detection is per-fragment, so too few fragments leads to excess
  retries.

## Conflict Handling

Lance supports concurrent operations on the same table using optimistic concurrency control. When two
operations conflict, one of them must be retried. Retries are handled automatically but they repeat
work that has already been done, which can hurt throughput. Understanding and minimizing conflicts is
important for maintaining good performance in write-heavy workloads.

Common sources of conflicts include:

- Concurrent compaction and index building, since both need to modify the same indices
- Update operations that affect the same fragments, since both need to rewrite the same data files

For more details on which operations conflict with each other, see
[conflict resolution](../format/table/transaction.md#conflict-resolution).

### Fragment Reuse Index

Compaction is one of the most expensive write operations because it rewrites data files and, by
default, remaps all indices to reflect the new row addresses. When compaction and index building
run concurrently, they often conflict because both need to modify the same indices. This typically
causes the compaction to fail and retry, and repeated failures can cause table layout to degrade
over time.

The Fragment Reuse Index (FRI) solves this by allowing compaction to skip the index remap step.
Instead of immediately updating indices, compaction records a mapping from old fragment row
addresses to new ones. When indices are loaded into the cache, the FRI is applied to translate
the old row addresses to the current ones. This adds a small cost to index load time but does
not affect query performance once the index is cached.

This decoupling means compaction and index building no longer conflict, which is especially
valuable for tables that are continuously ingesting data while also maintaining indices.

To enable the FRI, set `defer_index_remap=True` when compacting:

```python
dataset.optimize.compact_files(defer_index_remap=True)
```

Rust callers can open the FRI for the dataset version they have loaded with
`Dataset::frag_reuse_index()`, which returns `None` when that version has no FRI. The returned
index exposes the raw remap: a physical row address is either unmapped, deleted by a recorded
compaction, or mapped to the last address reached through the retained mappings. Neither outcome
is validated against the loaded manifest. Unmapped addresses may still have moved in a compaction
whose history was trimmed, and mapped destinations may since have been removed, so callers that
need a complete translation must verify coverage and destinations themselves.

For details on the index format and usage patterns, see the
[Fragment Reuse Index specification](../format/index/system/frag_reuse.md).

## Indexes

Training and searching indexes can have unique requirements for compute and memory. This section provides some
guidance on what can be expected for different index types.

### BTree Index

The BTree index is a two-level structure that provides efficient range queries and sorted access.
It strikes a balance between an expensive memory structure containing all values and an expensive disk
structure that can't be efficiently searched.

Training a BTree index is done by sorting the column. This is done using an [external sort](https://en.wikipedia.org/wiki/External_sorting) to constrain the total memory usage to a reasonable amount. Updating a BTree index does not
require re-sorting the entire column. The new values are sorted and the existing values are merged into the new sorted
values in linear time.

#### Storage Requirements

The BTree index is essentially a sorted copy of a column. The storage requirements are therefore the same as the column
but an additional 4 bytes per value is required to store the row ID and there is a small lookup structure which
should be roughly 0.001% of the size of the column.

#### Memory Requirements

Training a BTree index requires some RAM but the current implementation spills to disk rather aggressively and so the
total memory usage is fairly low.

When searching a BTree index, the index is loaded into the index cache in pages. Each page contains 4096 values.

#### Performance

The sort stage is the most expensive step in training a BTree index. The time complexity is O(n log n) where n is the number of rows in the column. At very large scales this can be a bottleneck and a distributed sort may be necessary. Lance currently does
not have anything builtin for this but work is underway to add this functionality. Training an index in parts as the data grows
may be slightly more efficient than training the entire index at once if you have the flexibility to do so.

When the BTree index is fully loaded into the index cache, the search time scales linearly with the number of rows that match the
query. When the BTree index is not fully loaded into the index cache, the search time will be controlled by the number of pages
that need to be loaded from disk and the speed of storage. The parts_loaded metric in the execution metrics can tell you how many
pages were loaded from disk to satisfy a query.

### Bitmap Index

The Bitmap index is an inverted lookup table that stores a bitmap for each possible value in the column. These bitmaps are compressed and serialized as a [Roaring Bitmap](https://roaringbitmap.org/).

A bitmap index is currently trained by accumulating the column into a hash map from value to a vector of row ids. Each value
is then serialized into a bitmap and stored in a file.

### Storage Requirements

The size of a bitmap index is difficult to calculate precisely but will generally scale with the number of unique values in the
column since a unique bitmap is required for each value and a single bitmap with all rows will compress more efficiently than
many bitmaps with a small number of rows.

#### Memory Requirements

Since training a bitmap index requires collecting the values into a hash map you will need at least 8 bytes of memory per row.
In addition, if you have many unique values, then you will need additional memory for the keys of the hash map. Training large
bitmaps with many unique values at scale can be memory intensive.

When a bitmap index is searched, bitmaps are loaded into the session cache individually. The size of the bitmap will depend on
the number of rows that match the token.

### Performance

When the bitmap index is fully loaded into the index cache, the search time scales linearly with the number of values that the
query requires. This makes the bitmap very fast for equality queries or very small ranges. Queries against large ranges are
currently extremely slow and the btree index is much faster for large range queries.

When a bitmap index is not fully loaded into the index cache, the search time will be controlled by the number of bitmaps that
need to be loaded from disk and the speed of storage. The parts_loaded metric in the execution metrics can tell you how many
bitmaps were loaded from disk to satisfy a query.

### Vector Index

Vector indexes (IVF_PQ, IVF_HNSW_SQ, etc.) are built in multiple phases, each with different memory requirements.

#### IVF Training

The IVF (Inverted File) phase clusters vectors into partitions using KMeans. To train the KMeans model, a sample of the
dataset is loaded into memory. The size of this sample is determined by:

```
training_data = num_partitions * sample_rate * dimension * sizeof(data_type)
```

The default `sample_rate` is 256. For example, with 1024 partitions, 768-dimensional float32 vectors, and the default
sample rate:

```
1024 * 256 * 768 * 4 bytes = 768 MiB
```

In addition to the training data, each KMeans iteration allocates membership and distance vectors proportional to the
number of training vectors (8 bytes per vector). The centroids themselves require `num_partitions * dimension *
sizeof(data_type)` bytes. In practice, the training data dominates and these additional allocations are small in
comparison.

If the dataset has fewer rows than `num_partitions * sample_rate`, the entire dataset is used for training instead.

#### Quantizer Training

After IVF training, a quantizer (e.g. PQ, SQ) is trained to compress vectors. This phase may sample some of the
dataset, but the sample size is tied to properties of the quantizer and the vector dimension rather than the size of the
dataset. As a result, quantizer training typically requires very little RAM compared to the IVF phase.

#### Shuffling

The final phase scans the entire vector column, transforms each vector (assigning it to an IVF partition and quantizing
it), and writes the results into per-partition files on disk. This is a streaming operation — data is not accumulated in
memory.

The input scan uses a 2 GiB I/O readahead buffer by default (configurable via `LANCE_DEFAULT_IO_BUFFER_SIZE`) and reads
batches of 8,192 rows. Incoming batches are transformed in parallel, with `num_cpus - 2` batches in flight at a time
(configurable via `LANCE_CPU_THREADS`). Each batch is sorted by partition ID and the slices are written directly to the
corresponding partition file. The in-flight memory during this phase is roughly:

```
io_readahead_buffer + num_cpu_threads * batch_size * (raw_vector_size + transformed_vector_size)
```

Each partition has an open file writer with roughly 8 MiB of accumulation buffer. In practice there shouldn't be that
much data accumulated in a single partition anyways. Instead, the max accumulation will be roughly the final size of
the partitions which comes out to `num_rows * (num_sub_vectors + 8) bytes`. For example, 100M rows with a 1536-dimensional
vector will have 96 sub-vectors and so the max accumulation will be ~10GB. The additional 8 bytes per row is for the row ID.

#### Storage Requirements

The on-disk size of a vector index consists of the IVF centroids and the quantized vectors.

The centroids require:

```
num_partitions * dimension * sizeof(data_type)
```

This is typically small. For example, 10K partitions with 768-dimensional float32 vectors is only 30 MiB.

The quantized vectors make up the bulk of the index. Each row stores a quantized code plus an 8-byte row ID. The
exact size depends on the quantizer:

**PQ (Product Quantization):** Each sub-vector is quantized to a single byte, so each row requires
`num_sub_vectors + 8` bytes. For example, 100M rows with 96 sub-vectors:

```
100M * (96 + 8) = ~9.7 GiB
```

**SQ (Scalar Quantization):** Each dimension is independently quantized to a single byte, so each row requires
`dimension + 8` bytes. SQ preserves more information than PQ but requires more storage. For example, 100M rows with
768-dimensional vectors:

```
100M * (768 + 8) = ~72.3 GiB
```

**RQ (RaBitQ):** New indexes default to 5 bits per dimension. Every bit width stores a 1-bit sign
code plus three 4-byte correction factors. Multi-bit indexes also store the remaining bits in
64-dimension-padded blocks and two additional 4-byte correction factors. Including the 8-byte row
ID, the approximate size per row is:

- 1-bit: `dimension / 8 + 20` bytes
- Multi-bit: `dimension / 8 + round_up(dimension, 64) * (num_bits - 1) / 8 + 28` bytes

For example, the default 5-bit index for 100M rows with 768 dimensions requires:

```
100M * (768 / 8 + 768 * 4 / 8 + 28) = ~47.3 GiB
```

The 5-bit default retains more information for the higher-fidelity distance estimates used by
`Normal` and `Accurate` search modes, at the cost of more quantization work and index I/O during
the build and a larger index. `Fast` search mode uses only the 1-bit sign code even when the index
stores additional bits. Set `num_bits=1` explicitly to minimize index size and build I/O; the same
100M-row example uses about 10.8 GiB, but searches cannot use the multi-bit distance estimate and
may have lower recall.

#### AMX Acceleration

On Linux x86_64 with an AMX-FP16 CPU (Intel Granite Rapids / Xeon 6 and newer), a `float16`
vector column indexed with `dot` distance uses the AMX tile instructions, provided the build
machine had clang >= 16 or gcc >= 13 to compile the kernel. There is nothing to enable —
Lance checks the CPU at run time and falls back to the previous implementation everywhere else.

The accelerated paths are also shape-gated, because below these sizes a tile pass costs more
than it saves and the kernel declines the work:

| Condition | Why |
|---|---|
| `float16` vectors, `dot` distance | The kernel is fp16-specific; other types and metrics keep their existing paths |
| `dimension >= 32` | One tile pass covers 32 dimensions; a shorter vector would be all scalar cleanup |
| `num_centroids >= 32` | The GEMM steps its centroid loop by 32 and has no partial-tile path |

Anything outside them behaves exactly as it does today, so a small dataset or a low-dimensional
column simply keeps the previous implementation rather than changing behaviour.

Index build also changes algorithm where all of the above hold: comparing every vector against
every centroid becomes affordable, so partition assignment is exact instead of approximated with
a graph search over the centroids. Recall improves, and partition assignments differ from what an
older build produced.

Set `LANCE_DISABLE_AMX=1` to take the AMX paths out of service without rebuilding — for
A/B measurement, or to get the previous behaviour back. Because it also moves partition
assignment back to the approximate path, an index built with it set is not equivalent to one
built without it; compare recall, not just build time.

read_and_write.md 22.3 KB

# Read and Write Data

## Writing Lance Dataset

If you're familiar with [Apache PyArrow](https://arrow.apache.org/docs/python/getstarted.html), you'll find that creating a Lance dataset is straightforward.
Begin by writing a `pyarrow.Table` using the `lance.write_dataset` function.

```python
import lance
import pyarrow as pa

table = pa.Table.from_pylist([{"name": "Alice", "age": 20},
                              {"name": "Bob", "age": 30}])
ds = lance.write_dataset(table, "./alice_and_bob.lance")
```

If the dataset is too large to fully load into memory, you can stream data using `lance.write_dataset`
also supports `Iterator` of `pyarrow.RecordBatch` es.
You will need to provide a `pyarrow.Schema` for the dataset in this case.

```python
from typing import Iterator

def producer() -> Iterator[pa.RecordBatch]:
    """An iterator of RecordBatches."""
    yield pa.RecordBatch.from_pylist([{"name": "Alice", "age": 20}])
    yield pa.RecordBatch.from_pylist([{"name": "Bob", "age": 30}])

schema = pa.schema([
    ("name", pa.string()),
    ("age", pa.int32()),
])

ds = lance.write_dataset(producer(),
                         "./alice_and_bob.lance",
                         schema=schema, mode="overwrite")
print(ds.count_rows())  # Output: 2
```

`lance.write_dataset` supports writing `pyarrow.Table`, `pandas.DataFrame`,
`pyarrow.dataset.Dataset`, and `Iterator[pyarrow.RecordBatch]`.

## Choosing a data file version

`data_storage_version` selects the format of newly written data files. For an
existing V2 dataset, an operation can select `"2.0"`, `"2.1"`, `"2.2"`, or `"2.3"`
without rewriting the other files. The dataset's `data_storage_version` property
is the default for writes that omit a target, not a summary of its existing
files. Create and overwrite establish this default; append, update, merge-insert,
and compaction do not change it. V1 and V2 cannot be mixed.

```python
import lance
import pyarrow as pa

data = pa.table({"id": [1, 2], "value": [10, 20]})
ds = lance.write_dataset(data, "./versions.lance", data_storage_version="2.1")
ds.update({"value": "value + 1"}, data_storage_version="2.2")
assert ds.data_storage_version == "2.1"

ds.merge_insert("id").when_not_matched_insert_all().data_storage_version("2.2").execute(
    pa.table({"id": [3], "value": [30]})
)
ds.optimize.compact_files(data_storage_version="2.2")
```

The `"stable"` and `"next"` selectors resolve according to the engine release.
Use exact versions when the output identity must be independent of that release.
V2.3 is currently unstable: files written by one unstable revision may not be
readable by a later revision. Use it only for experimentation.
Compaction plans fix the target before distributing tasks, including when the
target comes from the dataset default. The target survives Python pickle and
Java serialization; workers do not reinterpret it using their own release defaults.

Java update, merge-insert, and compaction options use
`withDataStorageVersion(DataStorageVersion.V2_2)`. The shared `DataStorageVersion`
enum also provides `STABLE` and `NEXT` selectors.

Compaction can convert selected fragments to another supported V2 version.
Binary copy requires every selected file to match the output version, as well
as the usual eligibility checks. In particular, overlays need reencoding to
preserve updated values. `try_binary_copy` falls back to reencoding when inputs
are ineligible; `force_binary_copy` rejects them. A version mismatch error includes
the target version, actual version, and file path. A persistent compaction target
can be set through `lance.compaction.data_storage_version` in the table config;
an explicit operation target takes precedence.

### Upgrading clients before mixed-version writes

Before writing files that differ from the dataset default, upgrade every reader
and writer to a mixed-version-aware release. Drain, restart, or fence writers
that opened the dataset using an older release. The commit automatically sets
the paired mixed-version reader/writer feature flags when needed; there is no
separate activation API. These flags remain set even if compaction later makes
the files homogeneous again.

Older clients that do not recognize the flags reject the resulting snapshots.
The flags cannot retroactively fence an older writer that already read a prior
manifest. Historical homogeneous datasets remain readable by clients that
support their file versions; see [table versioning](../format/table/versioning.md)
for the feature flag contract.

## Adding Rows

To insert data into your dataset, you can use either `LanceDataset.insert`
or `lance.write_dataset` with `mode=append`.

```python
import lance
import pyarrow as pa

table = pa.Table.from_pylist([{"name": "Alice", "age": 20},
                              {"name": "Bob", "age": 30}])
ds = lance.write_dataset(table, "./insert_example.lance")

new_table = pa.Table.from_pylist([{"name": "Carla", "age": 37}])
ds.insert(new_table)
print(ds.to_table().to_pandas())
#     name  age
# 0  Alice   20
# 1    Bob   30
# 2  Carla   37

new_table2 = pa.Table.from_pylist([{"name": "David", "age": 42}])
ds = lance.write_dataset(new_table2, ds, mode="append")
print(ds.to_table().to_pandas())
#     name  age
# 0  Alice   20
# 1    Bob   30
# 2  Carla   37
# 3  David   42
```

## Deleting rows

Lance supports deleting rows from a dataset using a SQL filter, as described in [Filter push-down](#filter-push-down).
For example, to delete Bob's row from the dataset above, one could use:

```python
import lance

dataset = lance.dataset("./alice_and_bob.lance")
dataset.delete("name = 'Bob'")
dataset2 = lance.dataset("./alice_and_bob.lance")
print(dataset2.to_table().to_pandas())
#     name  age
# 0  Alice   20
```

!!! note

    [Lance Format is immutable](../format/index.md). Each write operation creates a new version of the dataset,
    so users must reopen the dataset to see the changes. Likewise, rows are removed by marking
    them as deleted in a separate deletion index, rather than rewriting the files. This approach
    is faster and avoids invalidating any indices that reference the files, ensuring that subsequent
    queries do not return the deleted rows.

## Updating rows

Lance supports updating rows based on SQL expressions with the
`lance.LanceDataset.update` method. For example, if we notice
that Bob's name in our dataset has been sometimes written as `Blob`, we can fix
that with:

```python
import lance

dataset = lance.dataset("./alice_and_bob.lance")
dataset.update({"name": "'Bob'"}, where="name = 'Blob'")
```

The update values are SQL expressions, which is why `'Bob'` is wrapped in single
quotes. This means we can use complex expressions that reference existing columns if
we wish. For example, if two years have passed and we wish to update the ages
of Alice and Bob in the same example, we could write:

```python
import lance

dataset = lance.dataset("./alice_and_bob.lance")
dataset.update({"age": "age + 2"})
```

If you are trying to update a set of individual rows with new values then it is often
more efficient to use the merge insert operation described below.

```python
import lance

# Change the ages of both Alice and Bob
new_table = pa.Table.from_pylist([{"name": "Alice", "age": 30},
                                  {"name": "Bob", "age": 20}])

# This works, but is inefficient, see below for a better approach
dataset = lance.dataset("./alice_and_bob.lance")
for idx in range(new_table.num_rows):
  name = new_table[0][idx].as_py()
  new_age = new_table[1][idx].as_py()
  dataset.update({"age": new_age}, where=f"name='{name}'")
```

## Merge Insert

Lance supports a merge insert operation. This can be used to add new data in bulk
while also (potentially) matching against existing data. This operation can be used
for a number of different use cases.

### Bulk Update

The `lance.LanceDataset.update` method is useful for updating rows based on
a filter. However, if we want to replace existing rows with new rows then a `lance.LanceDataset.merge_insert`
operation would be more efficient:

```python
import lance

dataset = lance.dataset("./alice_and_bob.lance")
print(dataset.to_table().to_pandas())
#     name  age
# 0  Alice   20
# 1    Bob   30

# Change the ages of both Alice and Bob
new_table = pa.Table.from_pylist([{"name": "Alice", "age": 2},
                                  {"name": "Bob", "age": 3}])
# This will use `name` as the key for matching rows.  Merge insert
# uses a JOIN internally and so you typically want this column to
# be a unique key or id of some kind.
rst = dataset.merge_insert("name") \
       .when_matched_update_all() \
       .execute(new_table)
print(dataset.to_table().to_pandas())
#     name  age
# 0  Alice    2
# 1    Bob    3
```

Note that, similar to the update operation, rows that are modified will
be removed and inserted back into the table, changing their position to
the end. Also, the relative order of these rows could change because we
are using a hash-join operation internally.

### Insert if not Exists

Sometimes we only want to insert data if we haven't already inserted it
before. This can happen, for example, when we have a batch of data but
we don't know which rows we've added previously and we don't want to
create duplicate rows. We can use the merge insert operation to achieve
this:

```python
# Bob is already in the table, but Carla is new
new_table = pa.Table.from_pylist([{"name": "Bob", "age": 30},
                                  {"name": "Carla", "age": 37}])

dataset = lance.dataset("./alice_and_bob.lance")

# This will insert Carla but leave Bob unchanged
_ = dataset.merge_insert("name") \
       .when_not_matched_insert_all() \
       .execute(new_table)
# Verify that Carla was added but Bob remains unchanged
print(dataset.to_table().to_pandas())
#     name  age
# 0  Alice   20
# 1    Bob   30
# 2  Carla   37
```

### Update or Insert (Upsert)

Sometimes we want to combine both of the above behaviors. If a row
already exists we want to update it. If the row does not exist we want
to add it. This operation is sometimes called "upsert". We can use
the merge insert operation to do this as well:

```python
import lance
import pyarrow as pa

# Change Carla's age and insert David
new_table = pa.Table.from_pylist([{"name": "Carla", "age": 27},
                                  {"name": "David", "age": 42}])

dataset = lance.dataset("./alice_and_bob.lance")

# This will update Carla and insert David
_ = dataset.merge_insert("name") \
       .when_matched_update_all() \
       .when_not_matched_insert_all() \
       .execute(new_table)
# Verify the results
print(dataset.to_table().to_pandas())
#     name  age
# 0  Alice   20
# 1    Bob   30
# 2  Carla   27
# 3  David   42
```

### Replace a Portion of Data

A less common, but still useful, behavior can be to replace some region
of existing rows (defined by a filter) with new data. This is similar
to performing both a delete and an insert in a single transaction. For
example:

```python
import lance
import pyarrow as pa

new_table = pa.Table.from_pylist([{"name": "Edgar", "age": 46},
                                  {"name": "Francene", "age": 44}])

dataset = lance.dataset("./alice_and_bob.lance")
print(dataset.to_table().to_pandas())
#       name  age
# 0    Alice   20
# 1      Bob   30
# 2  Charlie   45
# 3    Donna   50

# This will remove anyone above 40 and insert our new data
_ = dataset.merge_insert("name") \
       .when_not_matched_insert_all() \
       .when_not_matched_by_source_delete("age >= 40") \
       .execute(new_table)
# Verify the results - people over 40 replaced with new data
print(dataset.to_table().to_pandas())
#        name  age
# 0     Alice   20
# 1       Bob   30
# 2     Edgar   46
# 3  Francene   44
```

## Reading Lance Dataset

To open a Lance dataset, use the `lance.dataset` function:

```python
import lance
ds = lance.dataset("s3://bucket/path/imagenet.lance")
# Or local path
ds = lance.dataset("./imagenet.lance")
```

!!! note

    Lance supports local file system, AWS `s3` and Google Cloud Storage(`gs`) as storage backends
    at the moment. Read more in [Object Store Configuration](object_store.md).

The most straightforward approach for reading a Lance dataset is to utilize the `lance.LanceDataset.to_table`
method in order to load the entire dataset into memory.

```python
table = ds.to_table()
```

Due to Lance being a high-performance columnar format, it enables efficient reading of subsets of the dataset by utilizing
**Column (projection)** push-down and **filter (predicates)** push-downs.

```python
table = ds.to_table(
    columns=["image", "label"],
    filter="label = 2 AND text IS NOT NULL",
    limit=1000,
    offset=3000)
```

Lance understands the cost of reading heavy columns such as `image`.
Consequently, it employs an optimized query plan to execute the operation efficiently.

### Iterative Read

If the dataset is too large to fit in memory, you can read it in batches
using the `lance.LanceDataset.to_batches` method:

```python
for batch in ds.to_batches(columns=["image"], filter="label = 10"):
    # do something with batch
    compute_on_batch(batch)
```

Unsurprisingly, `lance.LanceDataset.to_batches` takes the same parameters
as `lance.LanceDataset.to_table` function.

### Filter push-down

Lance embraces the utilization of standard SQL expressions as predicates for dataset filtering.
By pushing down the SQL predicates directly to the storage system,
the overall I/O load during a scan is significantly reduced.

Currently, Lance supports a growing list of expressions.

* `>`, `>=`, `<`, `<=`, `=`
* `AND`, `OR`, `NOT`
* `IS NULL`, `IS NOT NULL`
* `IS TRUE`, `IS NOT TRUE`, `IS FALSE`, `IS NOT FALSE`
* `IN`
* `LIKE`, `NOT LIKE`
* `regexp_match(column, pattern)`
* `CAST`

For example, the following filter string is acceptable:

```sql
((label IN [10, 20]) AND (note['email'] IS NOT NULL))
    OR NOT note['created']
```

Nested fields can be accessed using the subscripts. Struct fields can be
subscripted using field names, while list fields can be subscripted using
indices.

If your column name contains special characters or is a [SQL Keyword](https://docs.rs/sqlparser/latest/sqlparser/keywords/index.html),
you can use backtick (`` ` ``) to escape it. For nested fields, each segment of the
path must be wrapped in backticks.

```sql
`CUBE` = 10 AND `column name with space` IS NOT NULL
  AND `nested with space`.`inner with space` < 2
```

!!! warning

    Field names containing periods (`.`) are not supported.

Literals for dates, timestamps, and decimals can be written by writing the string
value after the type name. For example

```sql
date_col = date '2021-01-01'
and timestamp_col = timestamp '2021-01-01 00:00:00'
and decimal_col = decimal(8,3) '1.000'
```

For timestamp columns, the precision can be specified as a number in the type
parameter. Microsecond precision (6) is the default.

| SQL | Time unit |
|-----|-----------|
| `timestamp(0)` | Seconds |
| `timestamp(3)` | Milliseconds |
| `timestamp(6)` | Microseconds |
| `timestamp(9)` | Nanoseconds |

Lance internally stores data in Arrow format. The mapping from SQL types to Arrow is:

| SQL type | Arrow type |
|----------|------------|
| `boolean` | `Boolean` |
| `tinyint` / `tinyint unsigned` | `Int8` / `UInt8` |
| `smallint` / `smallint unsigned` | `Int16` / `UInt16` |
| `int` or `integer` / `int unsigned` or `integer unsigned` | `Int32` / `UInt32` |
| `bigint` / `bigint unsigned` | `Int64` / `UInt64` |
| `float` | `Float32` |
| `double` | `Float64` |
| `decimal(precision, scale)` | `Decimal128` |
| `date` | `Date32` |
| `timestamp` | `Timestamp` (1) |
| `string` | `Utf8` |
| `binary` | `Binary` |

(1) See precision mapping in previous table.

### Random read

One distinct feature of Lance, as columnar format, is that it allows you to read random samples quickly.

```python
# Access the 2nd, 101th and 501th rows
data = ds.take([1, 100, 500], columns=["image", "label"])
```

The ability to achieve fast random access to individual rows plays a crucial role in facilitating various workflows
such as random sampling and shuffling in ML training.
Additionally, it empowers users to construct secondary indices,
enabling swift execution of queries for enhanced performance.

## Table Maintenance

Some operations over time will cause a Lance dataset to have a poor layout. For
example, many small appends will lead to a large number of small fragments. Or
deleting many rows will lead to slower queries due to the need to filter out
deleted rows.

To address this, Lance provides methods for optimizing dataset layout.

### Compact data files

Data files can be rewritten so there are fewer files. When passing a
`target_rows_per_fragment` to `lance.dataset.DatasetOptimizer.compact_files`,
Lance will skip any fragments that are already above that row count, and rewrite
others. Fragments will be merged according to their fragment ids, so the inherent
ordering of the data will be preserved.

!!! note

    Compaction creates a new version of the table. It does not delete the old
    version of the table and the files referenced by it.

```python
import lance

dataset = lance.dataset("./alice_and_bob.lance")
dataset.optimize.compact_files(target_rows_per_fragment=1024 * 1024)
```

During compaction, Lance can also remove deleted rows. Rewritten fragments will
not have deletion files. This can improve scan performance since the soft deleted
rows don't have to be skipped during the scan.

When files are rewritten, the original row addresses are invalidated. This means the
affected files are no longer part of any ANN index if they were before. Because
of this, it's recommended to rewrite files before re-building indices.

<!-- TODO: remove this last comment once stable row ids are default. -->

### Cleanup old versions

Lance is an immutable format — every write creates a new version. The new version
only writes the data that changed, so an insert writes the new rows and an update
rewrites the affected columns for the affected rows. Even a delete creates a
small deletion file. However, old versions still reference the previous data
files, so those files are kept on disk until explicitly removed. Over time this
means storage grows with each operation — inserts, updates, and deletes alike.

Keeping old versions has important benefits: readers that opened an older version
can continue reading it without interference from concurrent writers, providing
snapshot isolation. Old versions also enable time travel queries, letting you
read the dataset as it existed at any prior point in time.

`cleanup_old_versions` deletes old version metadata and any data files that are
no longer referenced by any version, reclaiming the accumulated storage.

!!! warning

    Once old versions are cleaned up, time travel queries to those versions are
    no longer possible. Choose your retention window (`older_than`) accordingly —
    any version removed by cleanup cannot be recovered.

```python
import lance

dataset = lance.dataset("./my_dataset.lance")
dataset.cleanup_old_versions()
```

By default, versions older than 7 days are removed. You can override this with
the `older_than` parameter (a `timedelta`):

```python
from datetime import timedelta

dataset.cleanup_old_versions(older_than=timedelta(days=1))
```

!!! note

    Tagged versions are exempt from cleanup. See [Tags and Branches](tags_and_branches.md)
    for details.

By default, Lance only removes files that it can **verify** are no longer needed.
A file is verified when Lance can see that it was referenced by an older version
and is no longer referenced by any newer version. However, some orphaned files
cannot be verified this way — for example, files left behind by aborted or failed
commits that were never recorded in any version. These files are
indistinguishable from files being written by an in-progress operation.

Cleanup will never delete the current (active) version. This means passing
`older_than=timedelta(0)` is safe and will delete all versions except the current
one.

The `delete_unverified` flag enables a more aggressive strategy that will also
delete these unverified files:

```python
dataset.cleanup_old_versions(
    older_than=timedelta(hours=2),
    delete_unverified=True,
)
```

!!! danger

    Only use `delete_unverified=True` when you are confident that no other
    concurrent operation has been in-progress for longer than the `older_than`
    duration. Lance uses the file's age to decide whether an unverified file is
    safe to remove, so any operation that is still running past the `older_than`
    window risks having its files deleted out from under it.

    In particular, combining `delete_unverified=True` with `older_than=timedelta(0)`
    is **extremely dangerous** — if any other operation is in-progress at all,
    its data files may be deleted, leading to dataset corruption.

### Automatic cleanup

Instead of calling `cleanup_old_versions` manually, you can configure Lance to
clean up old versions automatically during writes. When auto cleanup is enabled,
Lance will run cleanup every *N* commits (the **interval**), removing versions
older than a specified duration.

Auto cleanup can be enabled when creating a new dataset:

```python
import lance
import pyarrow as pa
from lance.dataset import AutoCleanupConfig

table = pa.table({"id": range(100)})
ds = lance.write_dataset(
    table,
    "./my_dataset.lance",
    auto_cleanup_options=AutoCleanupConfig(
        interval=20,             # run cleanup every 20 commits
        older_than_seconds=3600, # remove versions older than 1 hour
    ),
)
```

Or enabled on an existing dataset:

```python
ds = lance.dataset("./my_dataset.lance")
ds.optimize.enable_auto_cleanup(
    AutoCleanupConfig(
        interval=20,
        older_than_seconds=3600,
    )
)
```

And disabled again:

```python
ds.optimize.disable_auto_cleanup()
```

Auto cleanup parameters can also be set directly via dataset config keys:

```python
ds.update_config({
    "lance.auto_cleanup.interval": "20",
    "lance.auto_cleanup.older_than": "3600s",
})
```

!!! warning

    Auto cleanup runs as part of the commit path. If your writer does not have
    delete permissions, or you are doing high-frequency writes where the extra
    latency matters, pass `skip_auto_cleanup=True` to `write_dataset` to skip it
    on a per-write basis.

### Other cleanup strategies

It is common to run cleanup as a periodic background task on a dedicated server
(for example, via a cron job or scheduled workflow). This keeps cleanup off the
write path entirely, avoiding any impact to write latency, but requires setting
up and maintaining additional infrastructure.

tags_and_branches.md 4.4 KB

# Manage Tags and Branches

Lance provides Git-like tag and branch capabilities through the `LanceDataset.tags` and `LanceDataset.branches` properties.

## Tags
Tags label specific versions within a branch's history.

`Tags` are particularly useful for tracking the evolution of datasets,
especially in machine learning workflows where datasets are frequently updated.
For example, you can `create`, `update`,
and `delete` or `list` tags.

The `reference` parameter (used in `create`, `update`, and `checkout_version`) accepts:

- An **integer**: version number in the **current branch** (e.g., `1`)
- A **string**: tag name (e.g., `"stable"`)
- A **tuple** `(branch_name, version)`: a specific version in a named branch
  - `(None, 2)` means version 2 on the main branch
  - `("main", 2)` means version 2 on the main branch (explicit)
  - `("experiment", 3)` means version 3 on the experiment branch
  - `("branch-name", None)` means the latest version on that branch

!!! note

    Creating or deleting tags does not generate new dataset versions.
    Tags exist as auxiliary metadata stored in a separate directory.

```python
import lance
import pyarrow as pa

ds = lance.dataset("./tags.lance")
print(len(ds.versions()))
# 2
print(ds.tags.list())
# {}
ds.tags.create("v1-prod", (None, 1))
print(ds.tags.list())
# {'v1-prod': {'version': 1, 'created_at': ..., 'updated_at': ..., 'manifest_size': ...}}
ds.tags.update("v1-prod", (None, 2))
print(ds.tags.list())
# {'v1-prod': {'version': 2, 'created_at': ..., 'updated_at': ..., 'manifest_size': ...}}
ds.tags.delete("v1-prod")
print(ds.tags.list())
# {}
print(ds.tags.list_ordered())
# []
ds.tags.create("v1-prod", (None, 1))
print(ds.tags.list_ordered())
# [('v1-prod', {'version': 1, 'created_at': ..., 'updated_at': ..., 'manifest_size': ...})]
ds.tags.update("v1-prod", (None, 2))
print(ds.tags.list_ordered())
# [('v1-prod', {'version': 2, 'created_at': ..., 'updated_at': ..., 'manifest_size': ...})]
ds.tags.delete("v1-prod")
print(ds.tags.list_ordered())
# []
```

!!! note

    Tagged versions are exempted from the `LanceDataset.cleanup_old_versions()`
    process.

    To remove a version that has been tagged, you must first `LanceDataset.tags.delete()`
    the associated tag.

## Branches

Branches manage parallel lines of dataset evolution. You can create a branch from an existing version or tag, read and write to it independently, and checkout different branches. You can `create`, `delete`, `list`, and `checkout` branches.

The `reference` parameter works the same as for Tags (see above).

!!! note

    Creating or deleting branches does not generate new dataset versions.
    New versions are created by writes (append/overwrite/index operations).

    Each branch maintains its own linear version history, so version numbers may overlap across branches. Use `(branch_name, version_number)` tuples as global identifiers for operations like `checkout_version` and `tags.create`.

    "main" is a reserved branch name. Lance uses "main" to identify the default branch.

### Create and checkout branches
```python
import lance
import pyarrow as pa

# Open dataset
ds = lance.dataset("/tmp/test.lance")

# Create branch from the currently checked-out version
experiment_branch = ds.create_branch("experiment")
experimental_data = pa.Table.from_pydict({"a": [11], "b": [12]})
lance.write_dataset(experimental_data, experiment_branch, mode="append")

# Create tag on the latest version of the experimental branch
ds.tags.create("experiment-rc", ("experiment", None))

# Checkout by tag name
experiment_rc = ds.checkout_version("experiment-rc")
# Checkout the latest version of the experimental branch by tuple
experiment_latest = ds.checkout_version(("experiment", None))

# Create a new branch from a tag
new_experiment = ds.create_branch("new-experiment", "experiment-rc")
```

### List branches
```python
print(ds.branches.list())
# {'experiment': {...}, 'new-experiment': {...}}
```

### Delete a branch
```python
# Ensure the branch is no longer needed before deletion
ds.branches.delete("experiment")
print(ds.branches.list_ordered(order="desc"))
# {'new-experiment': {'parent_branch': 'experiment', 'parent_version': 2, 'create_at': ..., 'manifest_size': ...}, ...}
```

!!! note

    Branches hold references to data files. Lance ensures that cleanup does not delete files still referenced by any branch.

    Delete unused branches to allow their referenced files to be cleaned up by `cleanup_old_versions()`.

tokenizer.md 3.9 KB

# Tokenizers

Currently, Lance has built-in support for ICU, Jieba, and Lindera. ICU uses built-in segmenter data. Jieba and Lindera require external language models.
If tokenization is needed, you can download language models by yourself.
You can specify the location where the language models are stored by setting the environment variable LANCE_LANGUAGE_MODEL_HOME.
If it's not set, the default value is

```bash
${system data directory}/lance/language_models
```

It also supports configuring user dictionaries,
which makes it convenient for users to expand their own dictionaries without retraining the language models.

## Inspect Query Tokenization

Use `lance.tokenize` to inspect the tokens that a full-text query will produce
without creating a dataset or index:

```python
import lance

tokens = lance.tokenize("the Cats and Dogs")
[(token.text, token.position) for token in tokens]
# [("cat", 0), ("dog", 2)]
```

Positions start at the first retained query token and preserve gaps left by stop
word removal and other filters. This is the same representation used for phrase
matching. The function accepts the tokenizer-related options supported by
`LanceDataset.create_scalar_index`, including custom stop words, n-grams, and the
code analyzer:

```python
tokens = lance.tokenize(
    "getUserName::value42",
    analyzer="code",
    split_identifiers=True,
    index_operators=True,
)
```

Options set to `None` use the selected analyzer profile's default. For example,
the code analyzer disables stemming and stop-word removal unless explicitly
overridden. The exception is `max_token_length`: omitting it keeps the default
length limit of 40, while `max_token_length=None` disables the limit.

## ICU Tokenizer

ICU uses Unicode word boundary rules and bundled dictionary data for complex scripts. It is useful for mixed-language text and does not require downloading a language model.

```python
ds.create_scalar_index("text", "INVERTED", base_tokenizer="icu")
```

Use `icu/split` when mixed-language text also contains punctuation-delimited identifiers that should be searchable by part.

```python
ds.create_scalar_index("text", "INVERTED", base_tokenizer="icu/split")
```

## Language Models of Jieba

### Downloading the Model

```bash
python -m lance.download jieba
```

The language model is stored by default in `${LANCE_LANGUAGE_MODEL_HOME}/jieba/default`.

### Using the Model

```python
ds.create_scalar_index("text", "INVERTED", base_tokenizer="jieba/default")
```

### User Dictionaries

Create a file named config.json in the root directory of the current model.

```json
{
    "main": "dict.txt",
    "users": ["path/to/user/dict.txt"]
}
```

- The "main" field is optional. If not filled, the default is "dict.txt".
- "users" is the path of the user dictionary. For the format of the user dictionary, please refer to https://github.com/messense/jieba-rs/blob/main/jieba/src/data/dict.txt.

## Language Models of Lindera

### Downloading the Model

```bash
python -m lance.download lindera -l [ipadic|ko-dic|unidic]
```

Note that the language models of Lindera need to be compiled. Please install lindera-cli first. For detailed steps, please refer to https://github.com/lindera/lindera/tree/main/lindera-cli.

The language model is stored by default in ${LANCE_LANGUAGE_MODEL_HOME}/lindera/[ipadic|ko-dic|unidic]

### Using the Model

```python
ds.create_scalar_index("text", "INVERTED", base_tokenizer="lindera/ipadic")
```

### User Dictionaries

Create a file named config.yml in the root directory of your model, or specify a custom YAML file using the `LINDERA_CONFIG_PATH` environment variable.
If both are provided, the config.yml in the root directory will be used.
For more detailed configuration methods, see the lindera documentation at https://github.com/lindera/lindera/.

```yaml
segmenter:
    mode: "normal"
    dictionary: /path/to/lindera/ipadic/main
```

## Create your own language model

Put your language model into `LANCE_LANGUAGE_MODEL_HOME`.

integrations

datafusion.md 4 KB

# Apache DataFusion Integration

Lance datasets can be queried with [Apache Datafusion](https://datafusion.apache.org/),
an extensible query engine written in Rust that uses Apache Arrow as its in-memory format.
This means you can write complex SQL queries to analyze your data in Lance.

The integration allows users to pass down column selections and basic filters to Lance,
reducing the amount of scanned data when executing your query.
Additionally, the integration allows streaming data from Lance datasets,
which allows users to do aggregation larger-than-memory.

## Rust

Lance includes a DataFusion table provider `lance::datafusion::LanceTableProvider`.
Users can register a Lance dataset as a table in DataFusion and run SQL with it:

### Simple SQL

```rust
use datafusion::prelude::SessionContext;
use lance::datafusion::LanceTableProvider;

let ctx = SessionContext::new();

ctx.register_table("dataset",
    Arc::new(LanceTableProvider::new(
    Arc::new(dataset.clone()),
    /* with_row_id */ false,
    /* with_row_addr */ false,
    )))?;

let df = ctx.sql("SELECT * FROM dataset LIMIT 10").await?;
let result = df.collect().await?;
```

### Join 2 Tables

```rust
use datafusion::prelude::SessionContext;
use lance::datafusion::LanceTableProvider;

let ctx = SessionContext::new();

ctx.register_table("orders",
    Arc::new(LanceTableProvider::new(
    Arc::new(orders_dataset.clone()),
    /* with_row_id */ false,
    /* with_row_addr */ false,
    )))?;

ctx.register_table("customers",
    Arc::new(LanceTableProvider::new(
    Arc::new(customers_dataset.clone()),
    /* with_row_id */ false,
    /* with_row_addr */ false,
    )))?;

let df = ctx.sql("
    SELECT o.order_id, o.amount, c.customer_name
    FROM orders o
    JOIN customers c ON o.customer_id = c.customer_id
    LIMIT 10
").await?;

let result = df.collect().await?;
```

### Register UDF
Lance provides some built-in UDFs, which users can manually register and use in queries.
The following example demonstrates how to register and use ```contains_tokens```.

```rust
use datafusion::prelude::SessionContext;
use lance::datafusion::LanceTableProvider;
use lance_datafusion::udf::register_functions;

let ctx = SessionContext::new();

// Register built-in UDFs
register_functions(&ctx);

ctx.register_table("dataset",
    Arc::new(LanceTableProvider::new(
    Arc::new(dataset.clone()),
    /* with_row_id */ false,
    /* with_row_addr */ false,
    )))?;

let df = ctx.sql("SELECT * FROM dataset WHERE contains_tokens(text, 'cat')").await?;
let result = df.collect().await?;
```

### JSON Functions

Lance provides comprehensive JSON support through a set of built-in UDFs that are automatically registered when you use `register_functions()`. These functions enable you to query and filter JSON data efficiently.

For a complete guide to JSON functions including:
- `json_extract` - Extract values using JSONPath
- `json_get`, `json_get_string`, `json_get_int`, `json_get_float`, `json_get_bool` - Type-safe value extraction
- `json_exists` - Check if a path exists
- `json_array_contains`, `json_array_length` - Array operations

See the [JSON Support Guide](../guide/json.md) for detailed documentation and examples.

**Example: Querying JSON in SQL**
```rust
// After registering functions as shown above
let df = ctx.sql("
    SELECT * FROM dataset
    WHERE json_get_string(metadata, 'category') = 'electronics'
    AND json_array_contains(metadata, '$.tags', 'featured')
").await?;
```

## Python

In Python, this integration is done via [Datafusion FFI](https://docs.rs/datafusion-ffi/latest/datafusion_ffi/).
An FFI table provider `FFILanceTableProvider` is included in `pylance`.
For example, if I want to query `my_lance_dataset`:

```python
from datafusion import SessionContext # pip install datafusion
from lance import FFILanceTableProvider

ctx = SessionContext()

table1 = FFILanceTableProvider(
    my_lance_dataset, with_row_id=True, with_row_addr=True
)
ctx.register_table("table1", table1)
ctx.table("table1")
ctx.sql("SELECT * FROM table1 LIMIT 10")
```

quickstart

full-text-search.md 15.2 KB

---
title: Full-Text Search
description: Full-text search (FTS) with inverted BM25 indexes and N-gram search in Lance
---

# Full-Text Search in Lance

Lance provides powerful full-text search (FTS) capabilities using an inverted index. This tutorial guides you through building and using FTS indexes to dramatically speed up text search operations while maintaining high accuracy.

By the end of this tutorial, you'll be able to build and use an FTS index, understand performance differences between indexed and non-indexed searches, and learn how to tune search parameters for optimal performance.

## Install the Python SDK

First, install the required dependencies:

```bash
pip install pylance pyarrow
```

## Set Up Your Environment

Import the necessary libraries for working with Lance datasets:

```python
import lance
import pyarrow as pa
```

## Prepare Your Text Data

In this quickstart, we'll create a simple dataset with text documents:

```python
table = pa.table(
    {
        "id": [1, 2, 3],
        "text": [
            "I left my umbrella on the evening train to Boston",
            "This ramen recipe simmers the broth for three hours with dried mushrooms.",
            "This train is scheduled to leave for Edinburgh at 9:30 in the morning",
        ],
    }
)

# Write to a new Lance dataset
lance.write_dataset(table, "/tmp/fts.lance", mode="overwrite")
```

This creates a Lance dataset with three text documents containing overlapping keywords that we'll use to demonstrate different search scenarios.

## Explore Your Dataset Schema

Let's examine the structure of our dataset:

```python
ds = lance.dataset("/tmp/fts.lance")
print(ds.schema)
```

This prints the PyArrow schema of the dataset:

```
id: int64
text: large_string
```

## Build the Full-Text Search Index

Full-text search is created with an inverted scalar index on your text column. Choose the `INVERTED` index type when calling `create_scalar_index` on your Lance dataset. Lance uses the BM25 ranking algorithm for relevance scoring. Results are automatically ranked by relevance, with higher scores indicating better matches.

```python
ds.create_scalar_index(
    column="text",
    index_type="INVERTED"
)
```

The index creation process builds an efficient lookup structure that maps words to the documents containing them. This enables high-performance keyword-based search, even on large datasets.

!!! warning "Index Creation Time"
Index creation time depends on the size of your text data. For large datasets, this process may take several minutes, but the performance benefits at query time are substantial.

## Advanced Index Configuration

You can customize the index creation with various parameters to optimize for your specific use case:

```python
ds.create_scalar_index(
    column="text",
    index_type="INVERTED",
    name="text_idx",              # Optional index name (if omitted, default is "text_idx")
    with_position=False,          # Set True to enable phrase queries (stores token positions)
    base_tokenizer="simple",      # Tokenizer: "simple", "icu", "icu/split", "whitespace", "raw", or "ngram"
    language="English",           # Language used for stemming + stop words (only used if `stem` or `remove_stop_words` is True)
    max_token_length=40,          # Drop tokens longer than this length
    lower_case=True,              # Lowercase text before tokenization
    stem=True,                    # Stem tokens (language-dependent)
    remove_stop_words=True,       # Remove stop words (language-dependent)
    custom_stop_words=None,       # Optional additional stop words (only used if remove_stop_words=True)
    ascii_folding=True,           # Fold accents to ASCII when possible (e.g., "é" -> "e")
    block_size=128,               # Posting block size: 128 or 256; 256 is experimental
)
```

### Tokenizer Options

- **simple**: Splits tokens on whitespace and punctuation
- **whitespace**: Splits tokens only on whitespace
- **raw**: No tokenization (useful for exact matching)

Lance also supports multilingual tokenization:

- **icu**: Unicode word segmentation with built-in ICU dictionaries
- **jieba/default**: Chinese text tokenization using Jieba
- **lindera/ipadic**: Japanese text tokenization using Lindera with IPAdic dictionary
- **lindera/ko-dic**: Korean text tokenization using Lindera with Ko-dic dictionary
- **lindera/unidic**: Japanese text tokenization using Lindera with UniDic dictionary

### Language Processing Features

- **stemming**: Reduces words to their root form (e.g., "running" → "run")
- **stop words**: Removes common words like "the", "and", "is"
- **ascii folding**: Converts accented characters to ASCII (e.g., "é" → "e")

## Search With FTS Queries

Now you can run FTS queries using your inverted index:

```python
import lance

# Open dataset
ds = lance.dataset("/tmp/fts.lance")

# Specify keyword phrases when calling the `to_table` method
query_result = ds.to_table(
    full_text_query="umbrella train"
)
print(query_result)
```

This query returns documents that contain either "umbrella" or "train" (or both). The search is case-insensitive and uses the inverted index for fast retrieval.

```
id: [[1, 3]]
text: [["I left my umbrella on the evening train to Boston", "This train is scheduled to leave for Edinburgh at 9:30 in the morning"]]
_score: [[..., ...]]
```

## Combining Full-Text Search with Metadata

It can be useful to combine FTS with metadata filtering in a single query to find more relevant results.
You can do this by passing a filter expression to the `filter` parameter.

```python
import lance
import pyarrow as pa

table = pa.table(
    {
        "id": [1, 2, 3],
        "text": [
            "I left my umbrella on the morning train to Boston",
            "This ramen recipe simmers the broth for three hours with dried mushrooms.",
            "This train is scheduled to leave for Edinburgh at 9:30 AM",
        ],
        "category": ["travel", "food", "travel"],
    }
)

# Temp write dataset
lance.write_dataset(table, "./fts_test_with_metadata.lance", mode="overwrite")

ds = lance.dataset("./fts_test_with_metadata.lance")

# Create FTS index
ds.create_scalar_index(
    column="text",
    index_type="INVERTED",
)

# Run FTS query with metadata filter
query_result = ds.to_table(
    full_text_query="three",
    filter='category = "food"',
)

# Returns
# id: [[2]]
# text: [["This ramen recipe simmers the broth for three hours with dried mushrooms."]]
# category: [["food"]]
```

## Advanced Search Features

### Boolean Search Operators

You can use boolean search operators by constructing a structured query object.

#### All terms: `AND`

```python
from lance.query import FullTextOperator, MatchQuery

# Require the terms 'umbrella AND train AND boston' to be present
and_query = MatchQuery("umbrella train boston", "text", operator=FullTextOperator.AND)
query_result = ds.to_table(full_text_query=and_query)

# Returns
# text: [["I left my umbrella on the evening train to Boston"]]
```

#### Any terms: `OR`

```python
from lance.query import FullTextOperator, MatchQuery

# Require the terms 'morning OR evening' to be present
or_query = MatchQuery("morning evening", "text", operator=FullTextOperator.OR)
query_result = ds.to_table(full_text_query=or_query)

# Returns the Boston document that mentions 'evening', and the Edinburgh document that mentions 'morning'
# text: [["This train is scheduled to leave for Edinburgh at 9:30 in the morning", "I left my umbrella on the evening train to Boston"]]
```

#### Mix `AND`/`OR` queries via operators

You can mix `AND`/`OR` queries using operators in Python:

```python
from lance.query import FullTextOperator, MatchQuery

# Combine AND and OR semantics
# Require 'train' AND ('morning' OR 'evening')
q1 = MatchQuery("morning evening", "text", operator=FullTextOperator.OR)
q2 = MatchQuery("train", "text")
query_result = ds.to_table(full_text_query=(q1 & q2))

# Returns both the Boston and Edinburgh documents that mention 'train'
# text: [["I left my umbrella on the evening train to Boston", "This train is scheduled to leave for Edinburgh at 9:30 in the morning"]]
```

To combine `OR` queries via operators, use the pattern `q1 | q2`.

Every query combined with `AND` becomes a scoring `MUST` clause: all clauses must match,
and every matching clause contributes to the final `_score`.

#### Exclude terms: `NOT`

Queries that exclude specific keywords are explicitly written using `BooleanQuery`/`Occur`
as shown below.

```python
from lance.query import MatchQuery, BooleanQuery, Occur

# Require that 'umbrella' be present, but 'train' NOT be present
q = BooleanQuery(
    [
        (Occur.MUST, MatchQuery("umbrella", "text")),
        (Occur.MUST_NOT, MatchQuery("train", "text")),
    ]
)
query_result = ds.to_table(full_text_query=q)

# Returns empty result, as no document matches this condition
# text: []
```

### Phrase Search

For exact phrase matching, ensure you enable `with_position=True` during index creation, which is disabled by default.

```python
# Rebuild the index with positions enabled (required for phrase queries)
ds.create_scalar_index(
    "text",
    "INVERTED",
    with_position=True,
    remove_stop_words=False,
)
# Search for the exact phrase "train to boston"
table = ds.to_table(full_text_query="'train to boston'")

# If stopwords are removed, this phrase query would return an empty result
# text: [["I left my umbrella on the evening train to Boston"]]
```

!!! warning "Stop Words Are Removed by Default"
Common words like "to", "the", etc. are categorized as stop words and are removed by default when creating the index. If you want to search exact phrases that include stop words, set `remove_stop_words=False` when creating the index.

### Substring matches with N-gram indexing

`NGRAM` is a type of scalar index for **substring / pattern-style** searches over text. It is a good alternative to wildcard-style queries like `term*` / `*term` (which are not parsed by `full_text_query` in Lance).

The N-gram index creates a bitmap for each N-gram in the string. By default, Lance uses trigrams. This index can be used to speed up queries using the `contains` function in filters.

```python
import lance

ds = lance.dataset("/tmp/fts.lance")

# Build an NGRAM index for substring search (speeds up `contains(...)` filters)
# Give the index a distinct name so it won't replace your FTS index
ds.create_scalar_index(column="text", index_type="NGRAM", name="text_ngram")

# Substring search
q1 = ds.to_table(filter="contains(text, 'ramen')")

# Returns the document about ramen
# text: [["This ramen recipe simmers the broth for three hours with dried mushrooms."]]
```

You can explain the query plan to confirm the N-gram index's usage as shown below:

```python
# Inspect the query plan to confirm index usage
print(ds.scanner(filter="contains(text, 'train')").explain_plan())
```

### Fuzzy Search

Fuzzy search is supported for FTS `MatchQuery` on `INVERTED` indexes. It uses Levenshtein edit distance to match terms with typos or slight variations.

```python
from lance.query import MatchQuery

# Explicit edit distance (1)
query_result = ds.to_table(
    full_text_query=MatchQuery(
        "rammen",  # Misspelled 'ramen'
        "text",
        fuzziness=1,
        max_expansions=50,  # default: 50
    )
)
```

You can also set `fuzziness=None` to use automatic fuzziness:

- `0` for term length `<= 2`
- `1` for term length `<= 5`
- `2` for term length `> 5`

```python
query_result = ds.to_table(
    full_text_query=MatchQuery(
        "rammen",
        "text",
        fuzziness=None,
    )
)
```

To enforce exact prefixes during fuzzy matching, set `prefix_length`.
This means the first `N` characters must match exactly before fuzzy edits are allowed on the rest of the term.
For example, with `prefix_length=2`, `"rammen"` can match terms starting with `"ra"` (like `"ramen"`), but not terms starting with other prefixes.

```python
query_result = ds.to_table(
    full_text_query=MatchQuery(
        "rammen",
        "text",
        fuzziness=1,
        prefix_length=2,  # "ra" must match exactly
    )
)
```

## Performance Tips

### Index Maintenance

When you append new rows after creating an `INVERTED` index, Lance still returns those rows in `full_text_query` results. It searches indexed fragments using the FTS index, scans unindexed fragments with flat search, and then merges the results.

To keep FTS latency low as new data arrives, periodically add unindexed fragments into the existing FTS index by calling `ds.optimize.optimize_indices()`:

```python
# Append new data
new_rows = pa.table(
    {
        "id": [4],
        "text": ["The next train leaves at noon"],
    }
)
ds.insert(new_rows)

# Incrementally update existing indices (including "text_idx")
ds.optimize.optimize_indices(index_names=["text_idx"])

# Optional: monitor index coverage
stats = ds.stats.index_stats("text_idx")
print(stats["num_unindexed_rows"], stats["num_indexed_rows"])
```

!!! info
If you used a custom index name, replace `"text_idx"` with your index name.
If you did not set `name=...` when creating the FTS index on column `"text"`, the default index name is `"text_idx"`.

If you changed tokenizer settings (such as `with_position`, `base_tokenizer`, stop words, or stemming), rebuild the index with `create_scalar_index(..., replace=True)` so the full dataset is indexed with the new configuration.

### Index Configuration Best Practices

- Enable `with_position` when you need phrase queries, because it stores word positions within documents. For simple term searches, disabling this option can save considerable storage space without impacting performance.

- Keep `lower_case=True` enabled for most applications to ensure case-insensitive search behavior. This provides a better user experience and matches common search expectations, though you can disable it if case sensitivity is important for your use case.

- Enable stemming (`stem=True`) when you want better recall by matching word variations (e.g., "running" matches "run"). Disable stemming if you need exact term matching or if your domain requires precise terminology.

- Consider enabling `remove_stop_words=True` for cleaner search results, especially in content-heavy applications. This removes common words like "the", "and", and "is" from the index, reducing noise and improving relevance. Keep stop words if they carry important meaning in your domain.

### Query Optimization

Using specific, targeted search terms often yields better performance than broad, generic queries. More specific terms reduce the number of potential matches and allow the index to work more efficiently. Consider analyzing your most common search patterns and optimizing your index configuration accordingly.

Combining full-text search with metadata filters can significantly reduce the search space and improve performance. Use structured data filters to narrow down results before applying text search, or vice versa. This approach is particularly effective for large datasets where you can eliminate many irrelevant documents early in the query process.

### Further Reading

For advanced usage instructions with different tokenizers and more technical details on the index training process, including information about the expected memory and disk usage, visit the [full-text index](../format/index/scalar/fts.md) specification.

## Next Steps

Check out the **[User Guide](../guide/read_and_write.md)** and explore the Lance API in more detail.

index.md 3.3 KB

---
title: Quickstart
description: Get started with Lance - create datasets, convert from Parquet, and learn the basics
---

# Getting Started with Lance Tables

This quickstart guide will walk you through the core features of Lance including creating datasets, versioning, and vector search.

By the end of this tutorial, you'll be able to create Lance datasets from pandas DataFrames and convert existing Parquet files to Lance format. You'll also understand the basic workflow for working with Lance datasets and be prepared to explore advanced features like versioning and vector search.

## Install the Python SDK

The easiest way to get started with Lance is via our Python SDK `pylance`:

```bash
pip install pylance
```

For the latest features and bug fixes, you can install the preview version:

=== "pip"

    ```bash
    pip install --pre --extra-index-url https://pypi.fury.io/lance-format pylance
    ```

=== "uv"

    ```bash
    uv venv
    uv pip install --prerelease allow --index https://pypi.fury.io/lance-format pylance

    # To add to pyproject.toml, just do:
    uv add --prerelease allow --index https://pypi.fury.io/lance-format pylance
    ```

!!! note
    Preview releases receive the same level of testing as regular releases.

## Set Up Your Environment

First, let's import the necessary libraries:

```python
import shutil
import lance
import numpy as np
import pandas as pd
import pyarrow as pa
```

## Create Your First Dataset

Lance is built on top of Apache Arrow, making it incredibly easy to work with pandas DataFrames and Arrow tables. You can create Lance datasets from various data sources including pandas DataFrames, Arrow tables, and existing Parquet files. Lance automatically handles the conversion and optimization for you.

### Create a Simple Dataset

You'll create a simple dataframe and then write it to Lance format. This demonstrates the basic workflow for creating Lance datasets.

Create a simple dataframe:

```python
df = pd.DataFrame({"a": [5]})
df
```

Now you'll write this dataframe to Lance format and verify the data was saved correctly:

```python
shutil.rmtree("/tmp/test.lance", ignore_errors=True)

dataset = lance.write_dataset(df, "/tmp/test.lance")
dataset.to_table().to_pandas()
```

### Convert Your Existing Parquet Files

You'll convert an existing Parquet file to Lance format. This shows how to migrate your existing data to Lance.

First, you'll create a Parquet file and then convert it to Lance:

```python
shutil.rmtree("/tmp/test.parquet", ignore_errors=True)
shutil.rmtree("/tmp/test.lance", ignore_errors=True)

tbl = pa.Table.from_pandas(df)
pa.dataset.write_dataset(tbl, "/tmp/test.parquet", format='parquet')

parquet = pa.dataset.dataset("/tmp/test.parquet")
parquet.to_table().to_pandas()
```

Now you'll convert the Parquet dataset to Lance format in a single line:

```python
dataset = lance.write_dataset(parquet, "/tmp/test.lance")

# Make sure it's the same
dataset.to_table().to_pandas()
```

## Next Steps

Now that you've mastered the basics of creating Lance datasets, here's what you can explore next:

- **[Versioning Your Datasets with Lance](versioning.md)** - Learn how to track changes over time with native versioning
- **[Vector Indexing and Vector Search With Lance](vector-search.md)** - Build high-performance vector search capabilities with ANN indices

vector-search.md 9.8 KB

---
title: Vector Search
description: High-performance vector search with ANN indices, including IVF_PQ, IVF_HNSW_PQ, and IVF_HNSW_SQ
---

# Vector Indexing and Vector Search With Lance

Lance provides high-performance vector search capabilities with ANN (Approximate Nearest Neighbor) indices.

By the end of this tutorial, you'll be able to build and use ANN indices to dramatically speed up vector search operations while maintaining high accuracy. You'll also learn how to tune search parameters for optimal performance and combine vector search with metadata queries in a single operation.

## Install the Python SDK

```bash
pip install pylance
```

## Set Up Your Environment

First, import the necessary libraries:

```python
import shutil
import lance
import numpy as np
import pandas as pd
import pyarrow as pa
import duckdb
```

## Prepare Your Vector Embeddings

For this tutorial, download and prepare the SIFT 1M dataset for vector search experiments.

- Download `ANN_SIFT1M` from: http://corpus-texmex.irisa.fr/
- Direct link: `ftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz`

You can just use `wget`:

```bash
rm -rf sift* vec_data.lance
wget ftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz
tar -xzf sift.tar.gz
```

## Convert Your Data to Lance Format

Then, convert the raw vector data into Lance format for efficient storage and querying.

```python
from lance.vector import vec_to_table
import struct

uri = "vec_data.lance"

with open("sift/sift_base.fvecs", mode="rb") as fobj:
    buf = fobj.read()
    data = np.array(struct.unpack("<128000000f", buf[4 : 4 + 4 * 1000000 * 128])).reshape((1000000, 128))
    dd = dict(zip(range(1000000), data))

table = vec_to_table(dd)
lance.write_dataset(table, uri, max_rows_per_group=8192, max_rows_per_file=1024*1024)
```

Now you can load the dataset:

```python
uri = "vec_data.lance"
sift1m = lance.dataset(uri)
```

## Search Without an Index

You'll perform vector search without an index to see the baseline performance, then compare it with indexed search.

First, let's sample some query vectors:

```python
import duckdb
# Make sure DuckDB v0.7+ is installed
samples = duckdb.query("SELECT vector FROM sift1m USING SAMPLE 100").to_df().vector
```

```
0     [29.0, 10.0, 1.0, 50.0, 7.0, 89.0, 95.0, 51.0,...
1     [7.0, 5.0, 39.0, 49.0, 17.0, 12.0, 83.0, 117.0...
2     [0.0, 0.0, 0.0, 10.0, 12.0, 31.0, 6.0, 0.0, 0....
3     [0.0, 2.0, 9.0, 1.793662034335766e-43, 30.0, 1...
4     [54.0, 112.0, 16.0, 0.0, 0.0, 7.0, 112.0, 44.0...
                            ...
95    [1.793662034335766e-43, 33.0, 47.0, 28.0, 0.0,...
96    [1.0, 4.0, 2.0, 32.0, 3.0, 7.0, 119.0, 116.0, ...
97    [17.0, 46.0, 12.0, 0.0, 0.0, 3.0, 23.0, 58.0, ...
98    [0.0, 11.0, 30.0, 14.0, 34.0, 7.0, 0.0, 0.0, 1...
99    [20.0, 8.0, 121.0, 98.0, 37.0, 77.0, 9.0, 18.0...
Name: vector, Length: 100, dtype: object
```

Now, perform nearest neighbor search without an index:

```python
import time

start = time.time()
tbl = sift1m.to_table(columns=["id"], nearest={"column": "vector", "q": samples[0], "k": 10})
end = time.time()

print(f"Time(sec): {end-start}")
print(tbl.to_pandas())
```

Expected output:
```
Time(sec): 0.10735273361206055
       id                                             vector    score
0  144678  [29.0, 10.0, 1.0, 50.0, 7.0, 89.0, 95.0, 51.0,...      0.0
1  575538  [2.0, 0.0, 1.0, 42.0, 3.0, 38.0, 152.0, 27.0, ...  76908.0
2  241428  [11.0, 0.0, 2.0, 118.0, 11.0, 108.0, 116.0, 21...  92877.0
...
```

Without the index, the search will scan throughout the whole dataset to compute the distance between each data point. For practical real-time performance with, you will get much better performance with an ANN index.

## Build the Search Index

If you build an ANN index - you can dramatically speed up vector search operations while maintaining high accuracy. In this example, we will build the `IVF_PQ` index:

```python
sift1m.create_index(
    "vector",
    index_type="IVF_PQ", # specify the IVF_PQ index type
    num_partitions=256,  # IVF
    num_sub_vectors=16,  # PQ
)
```

The sample response should look like this:

```
Building vector index: IVF256,PQ16
CPU times: user 2min 23s, sys: 2.77 s, total: 2min 26s
Wall time: 22.7 s
Sample 65536 out of 1000000 to train kmeans of 128 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
Sample 65536 out of 1000000 to train kmeans of 8 dim, 256 clusters
```

!!! warning "Index Creation Performance"
    If you're trying this on your own data, make sure your vector (dimensions / num_sub_vectors) % 8 == 0, or else index creation will take much longer than expected due to SIMD misalignment.

## Vector Search with the ANN Index

You can now perform the same search operation using your newly created index and see the dramatic performance improvement.

```python
sift1m = lance.dataset(uri)

import time

tot = 0
for q in samples:
    start = time.time()
    tbl = sift1m.to_table(nearest={"column": "vector", "q": q, "k": 10})
    end = time.time()
    tot += (end - start)

print(f"Avg(sec): {tot / len(samples)}")
print(tbl.to_pandas())
```

Expected output:
```
Avg(sec): 0.0009334301948547364
       id                                             vector         score
0  378825  [20.0, 8.0, 121.0, 98.0, 37.0, 77.0, 9.0, 18.0...  16560.197266
1  143787  [11.0, 24.0, 122.0, 122.0, 53.0, 4.0, 0.0, 3.0...  61714.941406
2  356895  [0.0, 14.0, 67.0, 122.0, 83.0, 23.0, 1.0, 0.0,...  64147.218750
3  535431  [9.0, 22.0, 118.0, 118.0, 4.0, 5.0, 4.0, 4.0, ...  69092.593750
4  308778  [1.0, 7.0, 48.0, 123.0, 73.0, 36.0, 8.0, 4.0, ...  69131.812500
5  222477  [14.0, 73.0, 39.0, 4.0, 16.0, 94.0, 19.0, 8.0,...  69244.195312
6  672558  [2.0, 1.0, 0.0, 11.0, 36.0, 23.0, 7.0, 10.0, 0...  70264.828125
7  365538  [54.0, 43.0, 97.0, 59.0, 34.0, 17.0, 10.0, 15....  70273.710938
8  659787  [10.0, 9.0, 23.0, 121.0, 38.0, 26.0, 38.0, 9.0...  70374.703125
9  603930  [32.0, 32.0, 122.0, 122.0, 70.0, 4.0, 15.0, 12...  70583.375000
```

!!! note "Performance Note"
    Your actual numbers will vary by your storage. These numbers are from local disk on an M2 MacBook Air. If you're querying S3 directly, HDD, or network drives, performance will be slower.

## Tune the Search Parameters

You need to adjust search parameters to balance between speed and accuracy, finding the optimal settings for your use case.

The latency vs recall is tunable via:
- **nprobes**: how many IVF partitions to search
- **refine_factor**: determines how many vectors are retrieved during re-ranking

```python
sift1m.to_table(
    nearest={
        "column": "vector",
        "q": samples[0],
        "k": 10,
        "nprobes": 10,
        "refine_factor": 5,
    }
).to_pandas()
```

**Parameter Explanation:**
- `q` => sample vector
- `k` => how many neighbors to return
- `nprobes` => how many partitions (in the coarse quantizer) to probe
- `refine_factor` => controls "re-ranking". If k=10 and refine_factor=5 then retrieve 50 nearest neighbors by ANN and re-sort using actual distances then return top 10. This improves recall without sacrificing performance too much

!!! note "Memory Usage"
    The latencies above include file I/O as Lance currently doesn't hold anything in memory. Along with index building speed, creating a purely in-memory version of the dataset would make the biggest impact on performance.

## Combine Features and Vectors

You can add metadata columns to your vector dataset and query both vectors and features together in a single operation.

In real-life situations, users have other feature or metadata columns that need to be stored and fetched together. If you're managing data and the index separately, you have to do a bunch of annoying plumbing to put stuff together.

With Lance, you can add columns directly to the dataset using `add_columns()`. For basic use cases, you can use SQL:

```python
sift1m.add_columns(
    {
        "item_id": "id + 1000000",
        "revenue": "random() * 1000 + 5000",
    }
)
```
For more complex columns, you can provide a Python function to generate the new column data:
```python
@lance.batch_udf()
def add_columns_func(batch: pa.Table) -> pd.DataFrame:
    """Add item_id and revenue columns to a batch of data.

    Args:
        batch: PyArrow Table containing the original data

    Returns:
        Pandas DataFrame with added item_id and revenue columns
    """
    item_ids: np.ndarray = np.arange(batch.num_rows)
    revenue: np.ndarray = (np.random.randn(batch.num_rows) + 5) * 1000
    return pd.DataFrame({"item_id": item_ids, "revenue": revenue})


sift1m.add_columns(add_columns_func)
```
You can then query both vectors and metadata together:

```python
# Get vectors and metadata together
result = sift1m.to_table(
    columns=["item_id", "revenue"],
    nearest={"column": "vector", "q": samples[0], "k": 10}
)
print(result.to_pandas())
```

## Next Steps

Check out **[Full-text Search](../quickstart/full-text-search.md)**, where we show how to create and query a BM25 index for keyword-based search in Lance.

versioning.md 3.6 KB

---
title: Versioning
description: Learn how to version your Lance datasets with append, overwrite, tags, and branches
---

# Versioning Your Datasets with Lance

Lance supports versioning natively, allowing you to track changes over time.

In this tutorial, you'll learn how to append new data to existing datasets while preserving historical versions and access specific versions using version numbers or meaningful tags. You'll also understand how to implement proper data governance practices with Lance's native versioning capabilities.

## Install the Python SDK

```bash
pip install pylance
```

## Set Up Your Environment

First, you should import the necessary libraries:

```python
import shutil
import lance
import numpy as np
import pandas as pd
import pyarrow as pa
```

## Append New Data to Your Dataset

You can add new rows to your existing dataset, creating a new version while preserving the original data. Here is how to append rows:

```python
df = pd.DataFrame({"a": [10]})
tbl = pa.Table.from_pandas(df)
dataset = lance.write_dataset(tbl, "/tmp/test.lance", mode="append")

dataset.to_table().to_pandas()
```

## Overwrite Your Dataset

You can completely replace your dataset with new data, creating a new version while keeping the old version accessible.

Here is how to overwrite the data and create a new version:

```python
df = pd.DataFrame({"a": [50, 100]})
tbl = pa.Table.from_pandas(df)
dataset = lance.write_dataset(tbl, "/tmp/test.lance", mode="overwrite")

dataset.to_table().to_pandas()
```

## Access Previous Dataset Versions

You can also check what versions are available and then access specific versions of your dataset.

List all versions of a dataset with this request:

```python
dataset.versions()
```

If you only need version numbers, use the lightweight reference API. It lists manifest
locations without reading and deserializing every manifest:

```python
dataset.version_refs()
```

Use `dataset.latest_version` instead when only the latest version of the current branch
is needed.

You can also access any available version:

```python
# Version 1
lance.dataset('/tmp/test.lance', version=1).to_table().to_pandas()

# Version 2
lance.dataset('/tmp/test.lance', version=2).to_table().to_pandas()
```

## Tag Your Important Versions

Create named tags for important versions, making it easier to reference them by meaningful names.

```python
dataset.tags.create("stable", 2)
dataset.tags.create("nightly", 3)
dataset.tags.list()
```

Tags can be checked out like versions:

```python
lance.dataset('/tmp/test.lance', version="stable").to_table().to_pandas()
```

For advanced tag operations (e.g., tagging versions on specific branches), see [Tags and Branches](../guide/tags_and_branches.md).

## Work with Branches

Branches manage parallel lines of dataset evolution. You can create branches from existing versions or tags, read and write to them independently, and checkout different branches.

```python
# Create branch from current latest version
experiment_branch = ds.create_branch("experiment")

# Write to the branch (affects only that branch's history)
tbl = pa.Table.from_pandas(pd.DataFrame({"a": [42]}))
lance.write_dataset(tbl, experiment_branch, mode="append")
```

For more details, see [Tags and Branches](../guide/tags_and_branches.md).

## Next Steps

Now that you've mastered dataset versioning with Lance, check out **[Vector Indexing and Vector Search With Lance](vector-search.md)**. You can learn how to build high-performance vector search capabilities on top of your Lance tables.

This will teach you how to build fast, scalable search capabilities for your versioned datasets.

changelog-v7-v13.md 92.7 KB

# Lance changelog - v7 -> v12 (section 14)

Part of the Lance v13 reference (`lance-format/lance@v13.0.0-beta.4`). Citations are `path:line`
relative to the repo root; build a permalink as
`https://github.com/lance-format/lance/blob/v13.0.0-beta.4/<path>`. Line numbers drift between
tags - treat them as approximate. Cross-references written as "section N" use the original
16-section numbering; `lance-reference.md` maps every number to its file.

**Release-line shape.** The major is bumped by a bot, not a human: `ci/publish_beta.sh:65,87`
re-roots at `MAJOR+1` whenever any PR since the release root carries the GitHub
`breaking-change` label (`ci/check_breaking_changes.py:31`). The marker is the **label**, not a
conventional-commit `!` - of the 13 labeled PRs in the v11 window (#8024, #8025, #8026, #8027,
#8028, #8051, #8095, #8159, #8172, #8188, #8206, #8347, #8360) only two carry `!` in the
subject. It has now fired on two
consecutive lines: `9.1.0-beta.*` -> `10.0.0-beta.*` (2026-07-23), then `10.1.0-beta.*` ->
`11.0.0-beta.*` (2026-08-05, `649076df1 chore: bump to 11.0.0-beta.1 based on breaking change
detection`). So **neither `v9.1.0` nor `v10.1.0` was ever released**, and `v10.1.0-beta.2` is
the direct ancestor of `v11.0.0-beta.1`, one bump commit apart. The re-root renumbers in place:
`release-root/10.1.0-beta.N` and `release-root/11.0.0-beta.N` point at the same commit
(`ee0a60d0c`), both recording `Base: 10.0.0-rc.1`.

**`v10.0.0` final was tagged on 2026-08-08** - an annotated, PGP-signed tag ("Release version
10.0.0") on `release/v10.0`, one commit past `v10.0.0-rc.3` (2026-08-02). That branch forked at
`v10.0.0-beta.7` and took one substantive backport (`10d0c9f2e fix: backport encoding and FTS
fixes to release/v10.0`, #8146). It is **not** an ancestor of `main` - finals are cut on
`release/vX.Y` branches, so that is normal. `v10.0.0-beta.7` **is** an ancestor of
`v11.0.0-beta.16`, but `v10.0.0-rc.3` and `v10.0.0` are not.

**`v10.0.0` is the stable pin** (2026-08-08, superseding `v9.0.1`), and it is what GitHub
Releases marks `Latest`. `v9.0.1` (2026-08-06, superseding `v9.0.0`, 2026-07-24) shipped with
five sibling patch finals that day - `v8.0.1`, `v7.1.0`, `v6.1.0`, `v4.0.2`, `v3.0.2` - each on
its own `release/vX.Y` branch. `v5.0.0` still has no final despite `v5.0.0-rc.2`. crates.io
publishes **finals only** (`max_stable_version` = `10.0.0`; no 11.x, and the only pre-release
among ~186 versions is the ancient `0.0.1-alpha0`); PyPI `pylance` is likewise at `10.0.0`. So
any beta pin is a git dependency; beta artifacts publish to fury.io
(`.github/workflows/publish-beta.yml:114`) under the renamed org,
`https://pypi.fury.io/lance-format`.

## Contents

- [The v7.1.0-beta.1 delta](#the-v710-beta1-delta)
- [The v7.1.0-beta.2 delta](#the-v710-beta2-delta)
- [The v7.1.0-beta.2 -> v7.2.0-beta.5 delta](#the-v710-beta2---v720-beta5-delta)
- [The v7.2.0-beta.5 -> v8.0.0-beta.9 delta (major-version boundary)](#the-v720-beta5---v800-beta9-delta-major-version-boundary)
- [The v8.0.0-beta.9 -> v8.0.0-beta.14 delta](#the-v800-beta9---v800-beta14-delta)
- [The v8.0.0-beta.14 -> v9.0.0-beta.10 delta (v8 -> v9 major boundary)](#the-v800-beta14---v900-beta10-delta-v8---v9-major-boundary)
- [The v9.0.0-beta.10 -> v9.0.0-beta.16 delta](#the-v900-beta10---v900-beta16-delta)
- [The v9.0.0-beta.16 -> v9.0.0-beta.18 delta](#the-v900-beta16---v900-beta18-delta)
- [The v9.0.0-beta.18 -> v9.1.0-beta.8 delta](#the-v900-beta18---v910-beta8-delta)
- [The v9.1.0-beta.8 -> v10.0.0-beta.7 delta](#the-v910-beta8---v1000-beta7-delta)
- [The v10.0.0-beta.7 -> v11.0.0-beta.2 delta](#the-v1000-beta7---v1100-beta2-delta)
- [The v11.0.0-beta.2 -> v11.0.0-beta.6 delta](#the-v1100-beta2---v1100-beta6-delta)
- [The v11.0.0-beta.6 -> v11.0.0-beta.16 delta (the v11 beta frontier)](#the-v1100-beta6---v1100-beta16-delta-the-v11-beta-frontier)
- [v11 silent-corruption and wrong-results fixes](#v11-silent-corruption-and-wrong-results-fixes)
- [v11.0.0 final (the beta.16 -> final delta)](#v1100-final-the-beta16---final-delta)
- [v12 (release-root/12.0.0-beta.N -> v12.0.0-beta.6)](#v12-release-root1200-betan---v1200-beta6)
- [The v12.0.0-beta.6 -> v12.0.0-beta.15 delta](#the-v1200-beta6---v1200-beta15-delta)
- [The v12.0.0-beta.15 -> v12.0.0 final delta](#the-v1200-beta15---v1200-final-delta)
- [The v13 line (release-root/13.0.0-beta.N -> v13.0.0-beta.4)](#the-v13-line-release-root1300-betan---v1300-beta4)

Other files: `format-file.md` (1-4), `format-table.md` (5-10), `indexes.md` (11-12),
`ops.md` (13, 15, 16).

---

## 14. What changed (v7 -> v13)

The v7 tag line ran `v7.0.0-beta.1` through `v7.0.0-beta.17`, then `v7.0.0-rc.1` and
`v7.0.0`. The v7.1 line opened at `v7.1.0-beta.1`, continued through `v7.1.0-beta.4` and
`v7.1.0-rc.1`; the v7.2 line ran through `v7.2.0-beta.5`; the **v8 line** ran through
`v8.0.0-beta.19` to `v8.0.0` final (2026-07-01); the **v9 line opened** (auto-bumped from
a `breaking-change`-labeled PR) and ran through `v9.0.0-rc.2` to **`v9.0.0` final** (2026-07-24,
on the `release/v9.0` branch, later `v9.0.1` on 2026-08-06 - **the current stable pin**); the
**v9.1 line opened** at `9.1.0-beta.0` when `v9.0.0-rc.1` was cut and ran to `9.1.0-beta.8`; on
2026-07-23 that same dev line was **mechanically re-rooted as `10.0.0-beta.*`** by the
breaking-change detector, so `v9.1.0` was never tagged; the **v10 line** ran to
`v10.0.0-beta.7`, then stabilization forked to `release/v10.0`, reached `v10.0.0-rc.3` and
shipped **`v10.0.0` final on 2026-08-08 - the current stable pin** - while `main` opened
`10.1.0-beta.1/2`; and on 2026-08-05 **that** line was re-rooted in place as `11.0.0-beta.*`, so
`v10.1.0` was never tagged. This section keeps
the full v7 history below (still useful context), the **v7.2.0-beta.5 -> v8.0.0-beta.9 delta**
(the v7->v8 major boundary), the **v8.0.0-beta.9 -> v8.0.0-beta.14 delta**, the
**v8.0.0-beta.14 -> v9.0.0-beta.10 delta** (the v8->v9 major boundary), the
**v9.0.0-beta.10 -> v9.0.0-beta.16 delta**, the **v9.0.0-beta.16 -> v9.0.0-beta.18 delta**,
the **v9.0.0-beta.18 -> v9.1.0-beta.8 delta**, the **v9.1.0-beta.8 -> v10.0.0-beta.7 delta**,
the **v10.0.0-beta.7 -> v11.0.0-beta.2 delta**, the **v11.0.0-beta.2 -> v11.0.0-beta.6 delta**,
and finally the **v11.0.0-beta.6 -> v11.0.0-beta.16 delta** (most important
for a v11 reader) plus a consolidated list of v11's silent-corruption and wrong-results fixes
at the very end.

**The v6 -> v7 breaking change.** `feat!: make dataset object store access base-aware`
(PR #6647, commit `456198cd`), immediately followed by the automated bump to `7.0.0-beta.1`.
Object-store access is now scoped to a dataset *base* instead of a flat global path -
groundwork for multi-base storage (hot/cold tiering, multi-region, shallow clones). The
related `refactor!: vendor the tokenizer stack into lance` (PR #6512) is what created the
`lance-tokenizer` crate.

**MemWAL / LSM** is the dominant v7 theme: the WAL appender/tailer primitives (PR #6669), the
`shared-memory://` object-store scheme, `ShardWriter` manual-compaction APIs (#6766), a
builder-style MemWAL init API (#6815), append-only tables without primary keys (#6848), and
`ShardSpec` renamed to `ShardingSpec` (#6813). See section 10 - MemWAL is experimental.

**Lance-native in-memory HNSW** for the MemWAL shard writer (PR #6795).

**Indexes** - segmented btree indexes (#6605), zonemap index segments (#6593), incremental /
segmented FTS index merging (#6737, #6790), distributed bitmap index build (#6598), segmented
inverted index build and search (#6305), FTS exec internals exposed for distributed planning
(#6648). The geo / RTree index and the `lance-geo` crate; an RTree index-type parsing fix
(#6568).

**Branches and tags** - branch/tag metadata maps and tag timestamps (PR #6364); the `tree/`
and `_refs/branches/` layout (section 7).

**Commits** - manifest version hint for fast latest-version lookup (PR #6752); uncommitted
delete transactions exposed (#6781); the `Clone` transaction (shallow / deep).

**Spec restructuring** - the lakehouse spec was formally split into separate catalog /
namespace / table / index specifications (PR #6750x), reflected in `docs/src/format/`.

### The v7.1.0-beta.1 delta

The 19 commits in `v7.0.0-beta.16..v7.1.0-beta.1` are mostly bug fixes and internal
performance work (serializable BTree/Bitmap/LabelList index caches, deterministic HNSW
graph builds, roaring range-iterator speedups). The user-facing additions:

- **Materialized-view namespace API** (PR #6891) - `create_materialized_view` and
  `refresh_materialized_view` on the `LanceNamespace` trait. A materialized view is a
  query / UDTF / chunker backed by a stored spec, with an optional initial refresh. The
  `RestNamespace` implements both (`POST /v1/materialized_view/{id}/create` and
  `/refresh`); `DirectoryNamespace` and the default trait return `not_supported`.
- **Typed vector index details** (PR #6099) - the `VectorIndexDetails` and
  `HnswParameters` messages moved into `protos/index.proto` (section 11.1).
- **Multi-base `write_fragments`** (PR #6855) - multi-base storage config is now reachable
  from the Python and Java `write_fragments` API, not just Rust.
- **Granular tracing targets** (PR #6853) - `pylance` emits trace events under a
  `lance::events::` prefix so they filter separately from log records; new
  `lance::dataset_events` and `lance::object_store::throttle` targets. Example:
  `LANCE_LOG="warn,lance::events::object_store::throttle=info"`
  (`docs/src/guide/performance.md`).
- **MemWAL** - a sharding evaluator (PR #6854), L0 flushed-generation dataset caching
  (PR #6816), and exact primary-key dedup fixes for LSM point lookup and vector search
  (PR #6881).

### The v7.1.0-beta.2 delta

Seven commits in `v7.1.0-beta.1..v7.1.0-beta.2`, mostly MemWAL correctness work plus one
workspace refactor:

- **New `lance-select` crate** (PR #6879, commit `52c6ac34`) - mask code (`RowAddrMask`,
  `NullableRowAddrMask`, `RowIdMask`, set types, `bitmap_to_ranges`/`ranges_to_bitmap`) and
  scalar-index expression-result types (`IndexExprResult`, `NullableIndexExprResult` with
  their `Not`/`BitAnd`/`BitOr` boolean algebra) were extracted from `lance-core` and
  `lance-index` into `rust/lance-select/`. Downstream filtering code and the new
  `index_expr_result` / `row_addr_mask` benches can now depend on masks without pulling in
  either larger crate.
- **MemWAL: build secondary indexes when flushing the active memtable** (PR #6901, commit
  `cee7d32f`) - `MemTableFlushHandler` previously called `flush`, persisting the data file
  and bloom filter but **building no secondary indexes**, and never received the shard's
  `index_configs` in the first place. Over flushed generations this made flushed vector rows
  invisible to `fast_search()` (a correctness bug for KNN, not just perf), and point lookups
  fell back to a full scan instead of routing through a scalar index. The fix threads
  `index_configs` into the handler and calls `flush_with_indexes` when any index is
  configured, while keeping plain `flush` when none are so empty-index shards avoid an extra
  pass.
- **MemWAL: per-source PK-hash block-list post-filter** (PR #6899, commit `77db998a`)
  fixes a stale-read in LSM vector search. `LsmGlobalPkDedupExec` (introduced in #6881) is
  exact only over candidates each source surfaces; if a primary key's fresh row is pushed
  out of its source's top-k by closer rows, the dedup never sees it and a superseded copy
  from an older generation can win. The fix makes staleness a per-source PK-hash post-filter
  (`PkHashFilterExec`) applied to each source's KNN *before* the cross-source union, so a
  stale row never reaches the merge. Each generation's membership is an
  `Arc<HashSet<u64>>` of PK hashes (`compute_pk_hash`, the same hash the dedup nodes use).
- **Docs** - new integrations landing page at `docs/src/integrations/index.md` (PR #6915);
  Java doc URL updated from `com.lancedb` to `org.lance` (#6467).

Note: there is **no Tantivy-FTS-removal commit in the v7 range**. Lance FTS at this tag is
already its own native inverted-index implementation; the tokenizer vendoring (#6512)
predates `v7.0.0-beta.1`. Do not attribute a Tantivy removal to v7.

### The v7.1.0-beta.2 -> v7.2.0-beta.5 delta

66 commits, **no breaking change** (no `!:` commit, no `BREAKING CHANGE` footer), **no new
crate** (still 24), **no new transaction op** (still 15), no proto change. User-facing
additions:

- **ICU FTS tokenizer** (PR #6956) - `base_tokenizer="icu"`, ICU4X dictionary segmentation
  with bundled data, no external model. A PR making ICU the default (#6968) was reverted
  (#7006); the default base tokenizer stays `simple`. See section 11.3.
- **Scalar-index fast search** (PR #6784) - `fast_search` routes through scalar/BTREE-indexed
  fragments and skips unindexed ones (not on legacy file version). See section 11.2.
- **Batched vector queries** (PR #6828) - `Scanner::nearest` takes a batch of query vectors
  and exposes a synthetic 0-based `query_index` column. See section 11.1.
- **Streaming IVF k-means** (PR #6913) - `streaming_sample_rate` / `streaming_coreset_rate` /
  `streaming_refine_passes` for bounded-memory IVF training. See section 11.1.
- **Arrow view-type support** (PR #6985) - `Utf8View` / `BinaryView` now encode (fixes an
  encoder `todo!()` panic) and coerce correctly in filters.
- **HuggingFace `download_mode`** (PR #7022) - storage-option keys `hf_download_mode` /
  `download_mode` select the OpenDAL `http` (default) or `xet` backend on the existing
  `hf://` provider; not a new object-store scheme.
- **MemWAL LSM local-scoring FTS** (PR #6951) - `LsmScanner::full_text_search(column, query, k)`,
  contained entirely in the `mem_wal` module.
- Dependency bumps: pylance `lance-namespace>=0.8.0,<0.9` (PR #7031), `opendal 0.57`
  (PR #7018), `jieba-rs 0.10` (PR #6955).
- Doc clarification: RaBitQ (RQ) is documented 1-bit-only with multi-bit as future work, and
  the RQ metadata schema gained a `code_dim` field (`docs/src/format/index/vector/index.md`).

Unchanged and reverified at this tag: 15 transaction ops, the scalar/vector index-type set,
all `protos/*.proto`, file-format `version.rs`, `rust-version 1.91.0`, `resolver 3`, edition
2024, `CommitConfig num_retries=20`, MemWAL still experimental.

### The v7.2.0-beta.5 -> v8.0.0-beta.9 delta (major-version boundary)

86 commits. This is a **major version bump** whose unifying theme is moving *every* index
build onto one segment-based lifecycle. **Six breaking changes** (`!:` commits):

- **`feat!: migrate bitmap to index segment based`** (PR #6869) - the defining v8 change.
  Bitmap now flows through the segment workflow; the old public Python Bitmap shard path
  (`create_scalar_index(..., fragment_ids=)` + `merge_index_metadata(..., "BITMAP")`) "is no
  longer exposed; callers should use the segment workflow instead." `execute_uncommitted`
  writes canonical `bitmap_page_lookup.lance` segment roots
  (`rust/lance-index/src/scalar/bitmap.rs:59`).
- **`refactor!: remove index segment builder`** (PR #6997) - the `IndexSegmentBuilder` API
  was removed from Rust, Python, and Java; staged publishing routes through
  `create_index_uncommitted` / `execute_uncommitted` + `merge_existing_index_segments` +
  `commit_existing_index_segments`. `build_all()` and `target_segment_bytes` size-based
  grouping are gone with no direct replacement (`docs/src/guide/migration.md` "7.2.0").
- **`refactor(index)!: move distributed BTree build to segmented index framework`** (PR #7013)
  - distributed BTree now uses the same `create_index_uncommitted` / merge / commit path.
- **`feat!: return write summaries from file writers`** (PR #7096) - `finish()` changed from
  `Result<u64>` to `Result<FileWriteSummary>` (`{ num_rows: u64, size_bytes: u64 }`,
  `rust/lance-file/src/writer.rs:54-58,768`). Python `LanceFileWriter.finish` keeps its
  row-count return.
- **`fix(python)!: derive index type from details`** (PR #6903) - `describe_indices()` "now
  reports nested and special-character field names as full field paths (e.g. `meta.lang`)
  instead of just the leaf name"; `list_indices()` is a thin typed `IndexInformation` wrapper
  that no longer opens each index; the `load_indices()` Python binding was removed.
- **`perf!: avoid listing index files after writes`** (PR #7129) - `IndexFile` metadata is
  propagated from writer/builder APIs into manifest metadata instead of listing index
  directories after writes (a writer/builder trait-level break).

Net-new user-facing features:

- **`lance-derive` crate** (PR #6229) - `#[derive(DeepSizeOf)]` for Arrow-aware memory
  accounting, replacing the external `deepsize` crate. Crate workspace 24 -> 25. See section 2.
- **FM-Index scalar index** (`docs/src/format/index/scalar/fmindex.md`,
  `protos/index.proto` `FMIndexIndexDetails`) - BWT substring/prefix/regex search on raw
  bytes via the Segmented Index architecture (`num_segments`). See section 11.2.
- **Multi-bit IVF_RQ** (PR #7038) - RaBitQ `num_bits` 1..=9; ex-code bits in `__ex_codes`
  (+ `__add_factors_ex` / `__scale_factors_ex`). **Raw-query RQ search** (PR #7078) adds the
  `query_estimator` field and `__error_factors` lower-bound pruning. See section 11.1.
- **Independent per-worker vector index models** (PR #7148) for distributed builds; zone-map
  segments now mergeable via `merge_existing_index_segments` (PR #7128); HNSW segment merge
  (PR #7178); segmented BTree merge (PR #6889). See section 12.
- **Volcengine TOS** (`tos://`) and feature-gated **GooseFS** (`goosefs://`, PR #7034) object
  stores. See section 13.
- Smaller: `tracked_files` / `all_files` on `LanceDataset` (PR #6011); multi-segment FM-Index
  build config; `add_columns` UDFs no longer require pandas (PR #7131); FTS flat match now
  searches all unindexed fragments (PR #7188); AVX-512 distance tables compiled for the
  target CPU (PR #7121).
- Dependency facts: arrow 58, datafusion 53, opendal 0.57, jieba-rs 0.10,
  lance-namespace-reqwest-client 0.8.2; pylance `lance-namespace>=0.8.0,<0.9`.

Unchanged and reverified at `v8.0.0-beta.9`: **15 transaction ops** (`protos/transaction.proto`
diff empty); file-format `version.rs` (`Next => 2.3`, `#[default]` still `V2_1`, no 2.4 - so
section 3 holds unchanged); `CommitConfig num_retries = 20`
(`rust/lance-table/src/io/commit.rs:1530`); the feature-flag bits; the
`ConditionalPutCommitHandler` routing; `rust-version 1.91.0`, `resolver 3`, edition 2024;
MemWAL docs and system-index docs byte-identical; MemWAL still experimental.

### The v8.0.0-beta.9 -> v8.0.0-beta.14 delta

31 commits, **two breaking changes** - both vector/RaBitQ. No new crate (still 25), no new
transaction op (still 15), no file-format change.

- **`feat(vector)!: add approx mode for RaBitQ search`** (PR #7179) - a public
  `approx_mode` with values `fast` / `normal` / `accurate` for vector search "when the backing
  index supports it" (commit `e25620710`), threaded through the Rust scanner, Python query
  parsing, and ANN proto serialization. **Breaking proto change**: the ANN query proto now
  carries `VectorApproxMode approx_mode` (`protos/ann.proto:16,45`) - regenerate any consumer
  that matches the serialized ANN proto. See section 11.1.
- **`perf(vector)!: add dedicated SIMD kernels for RaBitQ ex-code reranking`** (PR #7205,
  `rust/lance-index/src/vector/bq/ex_dot.rs`).
- **IVF_RQ default `target_partition_size` is now 4096** (was the generic fallback, PR #7273).
- **Cleanup explain API** (PR #7147) - `Dataset::cleanup(policy)` splits into `explain()`
  (returns a `CleanupExplanation`, a dry run) and `execute()`. See section 7.
- **Object-store docs** (PR #7151) - the guide gained full Tencent COS and GooseFS config
  sections (`docs/src/guide/object_store.md:333,396`); GooseFS is no longer undocumented. See
  section 13.
- **Smaller adds**: Python zonemap segment builds exposed (PR #7177); per-query I/O metrics
  (`bytes_read` / `iops` / `requests`) on `ANNSubIndex` / `ANNIvfPartition` in EXPLAIN ANALYZE
  (PR #7204); branch-aware version ops in the Directory/REST namespaces (CreateTableBranch /
  ListTableBranches / DeleteTableBranch, PR #7166); enriched `IndexContent` fields in dir
  namespace `ListTableIndices` (PR #7109).
- **Fixes**: resolve Blob v2 external URIs and clean failed writes in `add_columns`
  (PR #7152); coerce filter literals for dictionary-encoded columns (PR #7003);
  composite-key `merge_insert` probes every indexed key column (PR #6878).
- **Removals**: `table_version_storage_enabled` and the `__manifest`-backed table-version path
  removed - version ops now use `_versions/` exclusively (PR #7222); brotli dropped from the
  dependency graph (PR #7270).
- **Dep pins**: `lance-namespace-reqwest-client` 0.8.2 -> 0.8.4; pylance `lance-namespace`
  `>=0.8.0,<0.9` -> `>=0.8.5,<0.9`. arrow 58 / datafusion 53 / opendal 0.57 / jieba-rs 0.10
  unchanged.

Unchanged and reverified at `v8.0.0-beta.14`: 25 crates; 15 transaction ops; file-format
`version.rs` (`Next => 2.3`, `#[default] V2_1`, no 2.4); `CommitConfig num_retries = 20`
(`rust/lance-table/src/io/commit.rs:1550`); `rust-version 1.91.0`, `resolver 3`, edition 2024;
feature-flag bits; `ConditionalPutCommitHandler` routing.

### The v8.0.0-beta.14 -> v9.0.0-beta.10 delta (v8 -> v9 major boundary)

129 commits. A **light major bump** - structurally v9 is nearly identical to v8. The major
version was auto-triggered by `ci/check_breaking_changes.py` (GitHub `breaking-change`-label
detection), fired by two PRs merged before the 2026-06-22 bump: the Python 3.9 drop (#7345)
and the `alter_columns` fail-fast cast (#7158). The FMIndex rename (#7397) carries the label
too but merged after the bump, so it rode the already-bumped 9.0.0 series rather than
triggering it. **Three breaking changes:**

- **`refactor!: rename FMIndexIndexDetails to FMIndexDetails`** (PR #7397) - proto message
  `protos/index.proto:251` `FMIndexDetails {}` (was `FMIndexIndexDetails`); Rust type
  `pb::FmIndexDetails`; the `get_plugin_name_from_details_name` `fmindex`->`fm` special-case
  was deleted. Author's note: "This change would be a breaking change to any existing FM
  indexes!" - existing FM indexes become unreadable. See section 11.2.
- **Drop Python 3.9** (PR #7345) - `python/pyproject.toml` `requires-python = ">=3.10"`; the
  `Python :: 3.9` classifier removed; PyO3 abi3 floor raised `abi3-py39` -> `abi3-py310`;
  release wheels no longer built for 3.9. See section 2.
- **`fix(dataset)!`: `alter_columns` cast fails fast with an attached index** (PR #7158) -
  previously a cast silently dropped/invalidated the index; now it errors and you must
  `drop_index()` first. See section 6.

**Removal (Rust API, not conventional-`!`):** `as_vector_index` removed from the public
`Index` trait (PR #7392) - callers downcast via `as_any()`. See section 11.1.

**Net-new features:**

- **Hamming clustering** (PR #7379) - SIMD near-duplicate detection over 64-bit binary
  hashes (`pairwise_hamming_distance`, `UnionFind`, `hamming_clustering_for_ivf_partition`).
  See section 11.1.
- **COUNT(*) pushdown on stable-row-id datasets** (PR #7360) - the fast path no longer falls
  back to a full scan when stable row IDs are enabled. See section 11.1.
- **Per-column blob size thresholds** (PR #7269) - `lance-encoding:blob-inline-size-threshold`
  / `...-dedicated-size-threshold`; appends with a different threshold are rejected. See
  section 3.5.
- **Tunable 32k miniblock chunks** via `LANCE_MINIBLOCK_MAX_VALUES` (PR #7356; default still
  4096). See section 3.3.
- **`icu/split` FTS tokenizer** (PR #7474) and **mixed-language stop words** (PR #7324). See
  section 11.3.
- **Distributed LabelList index builds** (PR #7223). See section 12.
- **ngram index accelerates regex + infix LIKE** (PR #7139). See section 11.2.
- `alter_columns` **Dict <-> value-type casts** (PR #7289, section 6); cleanup-explain
  exposed to **Python and Java** (PR #7248, section 7); Python **fragment-reuse remap +
  delete-by-offset** (PR #7438); v2 file writer/reader support **columns of unequal length**
  (PR #7406); a `SpillStore` trait with local-disk impl (PR #7311); a versioned cache-codec
  envelope (PR #7163).

**Notable fixes:** compaction rejects `defer_index_remap` with stable row IDs (#7468);
nested legacy blobs rejected in v2.2 and blob v2 supported in nested structs (#7278, #7281);
`merge_insert` no longer drops matches when a leading payload column is all-null (#7251); SQ
offset accounted for in dot distance (#7481); manifests >5 GB via size-aware copy (#7047);
double percent-encoding in object-store paths resolved (#6643/#6695).

**Dependency changes:** `lance-namespace-reqwest-client` 0.8.4 -> 0.8.6 (#7254); `itertools`
0.13 -> 0.14 (#7424). The pylance runtime pin `lance-namespace>=0.8.5,<0.9` is unchanged.

Unchanged and reverified at `v9.0.0-beta.10`: **25 crates**; **15 transaction ops**
(`protos/transaction.proto` unchanged); file-format `version.rs` (`Next => 2.3`,
`#[default] V2_1`, no 2.4); `CommitConfig num_retries = 20`
(`rust/lance-table/src/io/commit.rs:1550`); `rust-version 1.91.0`, `resolver 3`, edition 2024;
arrow 58 / datafusion 53 / opendal 0.57 / jieba-rs 0.10; feature-flag bits;
`ConditionalPutCommitHandler` routing; MemWAL still experimental.

### The v9.0.0-beta.10 -> v9.0.0-beta.16 delta

58 commits. **One breaking change**; no new crate (still 25), no new transaction op (still 15),
no file-format change, no dependency-pin change. **`v8.0.0` final also shipped** in this window
(tag `v8.0.0`, commit `15f2ff594`, 2026-07-01) - use it as the stable pin (section intro).

**Breaking change:**

- **`feat(fts)!: make v2 the default index format`** (PR #7512) - newly created FTS / inverted
  indexes default to on-disk **format v2**; `LANCE_FTS_FORMAT_VERSION` no longer controls new
  indexes; pass `format_version=1` for older-reader compatibility. Existing v1 indexes stay
  queryable and are maintained as v1. See section 11.3.

**Net-new features:**

- **Blob read-API rework** (PR #7530, #7558) - `read_blobs` is now the primary full-payload
  API, `take_blobs` is for streaming/seeking, `scanner(blob_handling="all_binary")` reads blobs
  as Arrow binary; documented auto-tiering defaults (16 KiB inline / 2 MiB dedicated) and a new
  `lance-encoding:blob-pack-file-size-threshold` field key (PR #7322). `dataset.update()` now
  works on blob-encoded columns (PR #7579). See section 3.5.
- **Per-base `storage_options`** via `base_<id>.<key>` keys (PR #7608); **multi-base
  merge-insert** with target-base routing (`MergeInsertBuilder::target_bases`, round-robin new
  fragments, `DataFile.base_id` stamped; PR #7610). See sections 13 and 6.
- **ZoneMap `value_range`** min/max without a scan (PR #7463); **BTREE + ZONEMAP accept
  `LargeUtf8`** (PR #7525). See section 11.2.
- **Prefiltered LSM vector + FTS search** across base/flushed/in-memory sources (PR #7138). See
  section 10.
- **Schema evolution allows all-null `Map` columns** (PR #7462). See section 6.
- **DirectoryNamespace** now implements `update_table` / `delete_from_table` (PR #6923) and
  `alter_transaction` (PR #6974) - previously `not_supported`.
- Compaction `RowAddrRemap` structure to avoid remap HashMap OOM (PR #7237); single-flight
  scalar-index opens (PR #7464); session-cached manifest reuse on open (PR #7576).

**Notable fixes:** `DataReplacement` commits preserve `DataFile.base_id` on multi-base datasets
(#7609); blob descriptor views kept opaque - the reader no longer recurses into `position`/
`size` child fields (#7618); stable row-id index tolerates sparse/overlapping chunks (#7480);
ngram posting-list writes chunked by byte size to avoid i32 offset overflow (#7607); scheduler
deadlock on same-priority chunks fixed (#7588).

Unchanged and reverified at `v9.0.0-beta.18`: **25 crates**; **15 transaction ops**
(`protos/transaction.proto` unchanged); file-format `version.rs` (`Next => 2.3`,
`#[default] V2_1`, no 2.4); `CommitConfig num_retries = 20`; `rust-version 1.91.0`,
`resolver 3`, edition 2024; arrow 58 / datafusion 53 / opendal 0.57 / jieba-rs 0.10 /
itertools 0.14 / lance-namespace-reqwest-client 0.8.6; pylance `lance-namespace>=0.8.5,<0.9`;
feature-flag bits; `ConditionalPutCommitHandler` routing; MemWAL still experimental.

### The v9.0.0-beta.16 -> v9.0.0-beta.18 delta

36 commits. **No breaking changes**; no new crate (still 25), no new transaction op
(`rust/lance/src/dataset/transaction.rs` untouched), no file-format change. Mostly fixes
plus three additive features:

- **pylance prewarm gains segment selection** (#7677) - warm only chosen index segments.
- **Object-store metrics published via the `metrics` crate** (#7533).
- **RLE v2 run-length widths** (#7376), with width selection by encoded size (#7636).

**Docs:** the performance guide gained a **Fragment Sizing** section (#6606); cleanup and
automatic-cleanup documentation added to the read-and-write guide (#6546); a new
`guide/observability.md` page; MemWAL format spec updated (#7655). All reproduced in this
skill's `references/docs/` mirror.

**Notable fixes:** FTS list columns indexed as row documents (#7656); fuzzy
`max_expansions` enforced globally across index partitions instead of per-partition; FTS
tail-partition merge split by the worker memory budget and a `num_tokens`-only DocSet
cached on `LazyDocSet` (#7600); MemWAL writer fenced on WAL persistence failure (#7547)
and slice-aware memtable flush-threshold size estimate; Arrow-JSON -> Lance-JSON
conversion fixed across the merge/update, single-fragment-create, and merge-insert
full-fragment-rewrite paths (`take` now returns Arrow JSON, #7470/#7471);
`object_store::Error::NotFound` mapped to `Error::NotFound` instead of a generic IO
error; PQ `num_bits` respected for numpy codebooks (#7586); hang fixed in
`train_streaming_coreset_ivf_model` (#7676).

### The v9.0.0-beta.18 -> v9.1.0-beta.8 delta

127 commits straddling the tail of the 9.0.0 beta line (through `rc.2`) and the new 9.1.0 dev
line. The 9.1.0 minor bump is **automatic release-train cadence**, not a breaking change: when
`v9.0.0-rc.1` was cut, `main` advanced to `9.1.0-beta.0`. **One breaking-labeled PR** in the
window: FTS `block_size` (#7466, below). Structural changes reverified at `v9.1.0-beta.8`:
**26 crates** (new `lance-index-core`, #7713); **16 transaction ops** (new `DataOverlay`);
**datafusion 53 -> 54** (#7793), geodatafusion 0.4 -> 0.5, build toolchain 1.91 -> 1.97 (#7712,
MSRV `rust-version` unchanged at 1.91.0). Unchanged: arrow 58, opendal 0.57, jieba 0.10,
`lance-namespace-reqwest-client` 0.8.6, itertools 0.14, edition 2024, resolver 3,
`lance-arrow-scalar =58.0.0`; `CommitConfig num_retries = 20`; file-format `version.rs`
(`Next => 2.3`, `#[default] V2_1`); Python min 3.10 (3.14 added, #7728).

**Breaking (labeled):**
- **FTS configurable posting `block_size`** (#7466) - `InvertedIndexParams` gains `block_size`
  (128/256, default 128, 512 rejected). `block_size=256` and the code analyzer require FTS
  on-disk **format v3** (#7866). Sections 11.3.

**Additive features:**
- **Data Overlay Files** (#7535 write path, #7536 read path, #7540 Python commit op) - the new
  16th transaction op `DataOverlay`, feature flag 64, spec `data_overlay_file.md`. Cell-level
  `(offset, field)` updates without rewriting base data files; **unstable**, env-gated by
  `LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES` (release builds refuse overlay datasets).
  Compaction folds fragments over an overlay-count limit (#7772). Section 5.5.
- **Sparse structural pages** (#7889) - first real 2.3 encoding; `structural-encoding=sparse`.
  Section 3.1.
- **Exact `IS NULL`** for ZONEMAP and BLOOM_FILTER via a new `null_bitmap`. Section 11.2.
- **Nested-field FTS** (#7686) index leaf fields like `data.text`; code-analyzer tokenizer
  (#7681); impact-skip / bulk MAXSCORE top-k / bulk conjunction FTS paths (#7602/#7603/#7624);
  read inverted-index params without opening the segment (#7816). Section 11.3.
- **OpenTelemetry metrics for Python** (`instrument_lance_metrics`, `pylance[otel]`, #7537);
  zone-map seeds written into data-file footers during append (#7427).
- **AWS creds via `AssumeRoleWithWebIdentity`** to avoid role chaining (#7757); batch/list
  blob reads (#7864, #7664) and a bulk packed-blob writer (#7743); MemWAL flush-interval
  ticker (#7894) and Python/Java shard delete (#7649, #7688); wider hamming hashes and
  multi-segment hamming clustering (#7767, #7758); RLE child-buffer zstd compression (#7663);
  cached file-metadata APIs on `FileFragment` (#7820); `TableProvider` write inputs for
  `merge_insert`/`insert` (#7368); runtime SIMD dispatch for pre-Haswell x86_64 from-source
  builds (#6630).

**Other:** writes now reject system column names (#7797). The TensorFlow integration moved
from built-in to an external `lance-tensorflow` package, and the image array decoder/encoder
is now Pillow-only (not vendored in this skill's docs mirror - integrations mirror is
`datafusion.md` only).

### The v9.1.0-beta.8 -> v10.0.0-beta.7 delta

78 commits. The major bump is **mechanical**, not a redesign: `ci/publish_beta.sh` re-roots at
`MAJOR+1` on any `breaking-change`-labeled PR, and `fb88621f8 chore: bump to 10.0.0-beta.1 based
on breaking change detection` landed immediately after `3a72f8a61 fix(blob)!: preserve null
selections across blob APIs (#7903)`. Only one bump happens per series, so the two later `!`
commits rode the already-bumped line. Structural invariants **all reverified unchanged**:
26 crates (no crate added or removed), 16 transaction ops, `CommitConfig num_retries = 20`,
file-format `version.rs` (`Next => 2.3`, `#[default] V2_1`), feature-flag bits (newest still 64,
data overlay), MSRV 1.91.0, toolchain 1.97.0, edition 2024, resolver 3, arrow 58 /
datafusion 54 / opendal 0.57 / jieba 0.10 / itertools 0.14 /
`lance-namespace-reqwest-client` 0.8.6, Python 3.10-3.14.

**Breaking (four):**

- **`fix(blob)!: preserve null selections across blob APIs`** (#7903) - the bump trigger. Every
  selection API returns one result per request, nulls as `None` instead of omitted. Rust,
  Python, and Java signatures all change. Section 3.5.
- **`perf(cache)!: use fixed-size cache keys`** (#7878) - opaque 16-byte BLAKE3 keys
  (`CACHE_KEY_FORMAT = "blake3-128-v1"`); all warm/persisted caches cold-miss, no legacy
  fallback; prefix-invalidation and key-inventory APIs removed. Section 9.4.
- **`perf(compaction)!: skip building row-address maps when index remapping is not needed`**
  (#7778) - `IndexRemapperOptions::create_remapper` becomes async and returns
  `Result<Option<Box<dyn IndexRemapper>>>`; compaction skips the `_rowid` scan and
  RoaringTreemap entirely for FRI-only or system-index-only datasets.
- **MemWAL rename** (#7943, #7957) - flushed generation -> SSTable, merge -> compaction, across
  spec, Rust, Python, Java, and protos. Wire-compatible, symbol-breaking, no shims. Section 10.

**Additive:**

- **`ConcreteFileVersion`** (#7879) exact file identity, unordered by design; manifests reject
  selector aliases; `try_from_major_minor` / `to_numbers` removed; byte-exact writer fixtures
  with SHA-256 locks (#8019). Section 3.6.
- **Sparse structural pages auto-select** in the 2.3 writer (#7756). Section 3.1.
- **Segment-native BLOOMFILTER / RTREE / NGRAM / LABEL_LIST** (#7925, #7932, #7244, #7884);
  `IndexSegment::new` 4 -> 6 params; merged segments inherit the minimum source
  `dataset_version`; concurrent LIST on segment commit, ~8x faster remote (#7657). Section 12.
- **ACORN-1 prefiltered HNSW** (#7927), opt-in via `approx_mode="fast"`, with a documented
  recall regression on uniform-random masks. Section 11.1.
- **FTS**: `total_tokens` metadata key and `bm25_search` removal (#7863),
  `LANCE_FTS_SEARCH_CHUNK` (#7950), top-k row-id resolution 26x (#7897), deterministic tie
  order (#8073), segment-uuid-scoped exec nodes (#7976). Section 11.3.
- **Data-overlay/index correctness** (#7549, #7926, #7918) - index results exclude
  overlay-superseded rows. Sections 5.5 and 9.1.
- **quick_cache** as the default index and metadata cache backend (#7953, #8013), with a
  per-shard admission ceiling that silently refuses oversized entries. Section 9.4.
- Cross-store `deep_clone` via `CommitBuilder::with_source_store` (#7545); commit-retry backoff
  overflow capped at `MAX_SLOTS = 128` (#7883); external-manifest finalization always HEADs
  (superseded at beta.8 by #8499 - the ETag from that HEAD must not be persisted; see below)
  (#7964); `memory://` `DatasetNotFound` fix (#8068); tokio-shutdown panic becomes an I/O error
  (#7478); `LANCE_CPU_THREADS` / `LANCE_IO_CORE_RESERVATION` validated (#7856); dir namespace
  surfaces throttles instead of `TableNotFound` (#7931) and honors `structured_query` FTS
  (#7592); Java `CacheStats` + `Session.metadataCacheStats()` (#7885); vector index append
  across heterogeneous segment models (#8047).

**Fixes worth knowing:** `Dataset::filter_deleted_ids` was wrong on stable-row-id datasets,
breaking `optimize_indices` (#7704); filtered scans and `add_columns(AllNulls)` returned a valid
struct with null children instead of a null struct on storage 2.1 (#8049); `LIKE ... ESCAPE ''`
was treated as no-escape and `ESCAPE 'ab'` silently truncated - both now error (#7810);
`list_indices` no longer backtick-quotes ordinary column names (#7503).

**Security:** `quinn-proto` 0.11.14 -> 0.11.16 via Dependabot security alert, applied to the
root workspace, `/python`, and `/java/lance-jni` (#7983, #7984, #7982) - "proto: yield error on
too many gaps in assembler". Plus bulk Dependabot cargo-group bumps (38 root, 28 python,
27 java-jni).

### The v10.0.0-beta.7 -> v11.0.0-beta.2 delta

128 commits. The bump is again **mechanical** - nine PRs carried the `breaking-change` label
(#8024, #8025, #8026, #8051, #8095, #8159, #8172, #8188, #8206), of which only two carry `!` in
the subject, and the bot re-rooted `10.1.0-beta.2` as `11.0.0-beta.1` in place. Structural
invariants **all reverified unchanged**: **26 crates** (the `rust/` `Cargo.toml` inventory is
byte-identical across the two tags), **16 transaction ops** (`protos/transaction.proto` is
byte-identical), `CommitConfig.num_retries` **20**, file-format enum `next => 2.3` / default 2.1
with no 2.4, manifest feature flags 1-128 unchanged, MSRV 1.91.0 / toolchain 1.97.0, Python
3.10-3.14, arrow 58 / datafusion 54 / `object_store` 0.13.2 / jieba 0.10 / blake3 1.8.5, and the
`=58.0.0` pins on `lance-arrow-scalar` / `lance-arrow-stats`. The only proto change in the whole
range is `protos/index_old.proto` (+17 lines).

**Breaking (labeled):**

- **#8206** - fragment ids became a dataset-lifetime high-water mark; overwrite no longer
  restarts at 0, overwrite fragments with deletion files are rejected, duplicate ids block all
  commits. A format invariant, not just an API change. Section 5.3.
- **#8024 / #8025 / #8026** - the exact-version reader/writer composition: `ReaderProjection`
  constructors became free functions, `FileReader::version()` and
  `Dataset::storage_version_or_default()` return `ConcreteFileVersion`,
  `FileReader::supports_projection` and `open_writer` were removed, and
  `lance-encoding::version` was deleted with no re-export. Section 2.1.
- **#8051** - `force_seal_active` returns `SealFence`. **#8095** -
  `MemIndexConfig::detect_index_type` replaced by `is_maintainable_index_type` + `MemIndexKind`.
  Section 10.
- **#8159** - `CacheBackend::deep_size_of_entries`; reported cache sizes shrink. Section 9.4.
- **#8172** - `DataBlockBuilder::append` is fallible; corrupt variable-width offsets now error
  instead of panicking. **#8188** - HNSW `try_with_capacity`, `m >= 4` enforced, persisted level
  layout corrected; different graphs and recall. Sections 2.1 and 11.1.

**Breaking (unlabeled but source-breaking - the #7877 series):** #8020 removed
`lance_io::encodings` and moved `lance-file::previous` to `versions::v1`; #8021 deleted the
`lance-encoding::previous` public encoder surface; #8023 turned `FileWriter` into an enum and
removed all its constructors plus two `FileWriterOptions` fields; #8038 added a context parameter
to `MiniBlockCompressor::compress`. Also `#8141` removed `GraphBuilderStats`, and `#7788` added a
third parameter to `load_segments`. Section 2.1.

**Net-new:**

- **FTS document granularity** (#7788) - `DocumentGranularity` ROW/LIST_ELEMENT,
  `posting_format_version`, `_doc_index` column, and a third FTS-v3 trigger. Section 11.3.
- **Compound FTS scoring core** (#8092, #8093, #8094, #8131, #8299) - Boolean/Phrase/Boost
  composition, public `CompoundQueryExec`, cost-ordered conjunctions, `AND` as scoring `MUST`.
  Section 11.3.
- **Zone maps**: `has_null_bitmap` making `IS NOT NULL` scan-free (#8088) and all-type support
  including nested (#8017). Section 11.2.
- **Manifest transaction spilling** above 20 MiB (#7881, ~50% manifest shrink measured).
  Section 9.1. **Pluggable cache backends** with a `moka://` URI form (#7683). Section 9.4.
- **`aws_provider_scheme`** token/ecs/irsa (#8103); **`goosefs://` via
  `ConditionalPutCommitHandler`** (#8134) with a mixed-version overwrite hazard; multipart
  part-identity retry fix (#8174) and removal of `LANCE_CONN_RESET_RETRIES`. Section 13.
- **Encoding performance**: exact decode-buffer preallocation via `decoded_size_bytes` (#8091,
  index-cache weight down up to 74% on IVF_SQ, no on-disk change); a zero-copy typed view for
  inline bitpacking (#7696, 13-22% faster unchunk); `O(n*m)` fragment compares removed from
  `build_manifest` for Update/Delete (#8210); parallel doc-length preload on the cold deferred
  FTS search path (#8119).
- **Python**: pydantic auto-conversion in `write_dataset` plus
  `LanceDataset.from_pydantic_model(model_class, data, uri=None, **kwargs)` (#7383);
  `LanceFileWriteSummary` giving `LanceFileWriter` a `size_bytes` (#7876); `max_source_fragments`
  on `compact_files` for incremental compaction, also settable via the manifest config key
  `lance.compaction.max_source_fragments` (#8116); `blob_handling` on the SQL/DataFrame builder
  (#8087).
- **Java** got the biggest build-out of any binding: an OpenTelemetry metrics bridge
  (`org.lance.otel.LanceMetrics`, #8064 - the docs now say metrics are available "from the Rust,
  Python, and Java APIs"); scanner tuning via `ScanOptions` (`batchSizeBytes`, `ioBufferSize`,
  `fragmentReadahead`, `scanInOrder`) and a typed `MaterializationStyle` (#8288);
  `FragmentStatistics` (#8072); a typed `LanceException` replacing bare `RuntimeException`
  (#8184); and `IndexBuildProgress` callbacks (#8090).
- Minor: `QuantizationType` accepts `"RQ"` (#8214); HNSW greedy descent stops at level 1
  (#8035, +3.7% recall@10); `BlobV2Layout` classification helper (#8266); a
  `ConditionalPutCommitHandler` test matrix covering every routed scheme.

**Security / supply chain:** `rust-stemmers 1.2.0` -> **`frostem`** (#8183) - the unmaintained
crate's "Greek implementation can retain stale UTF-8 byte offsets after shortening a word, then
panic while slicing the shortened string." `frostem` is generated from current upstream Snowball
and exposes the same 18 algorithms. `strum` and the direct `goosefs-sdk` dependency were dropped;
`crc32c` left the lockfile and `opendal-http-transport-reqwest` entered it.

**It is not a drop-in for existing FTS indexes.** The two stemmers disagree on a small but
non-trivial slice of ordinary English. Measured over `/usr/share/dict/words`, **484 of 235,976
words (0.2%) stem differently** - most are the `-ogist` family (`anthropologist` ->
`anthropolog`), but the rest are everyday vocabulary: `internal` (old `intern`, new `internal`),
`added` (`ad` vs `add`), `emergency`, `evening`, `interfering`, `erring`. An index built with
`.stem(true)` under v10 and queried by a v11 binary therefore **silently misses those forms** -
the query stems to `internal` while the index holds `intern`. There is no error and no version
check. Two consequences for a rollout: a mixed v10/v11 fleet appending FTS segments to the same
store produces segments stemmed two ways, so the upgrade wants to be fleet-coordinated, and any
stemmed FTS index built before the swap needs one rebuild to become self-consistent.

---

### The v11.0.0-beta.2 -> v11.0.0-beta.6 delta

94 commits, 90 PRs, **four `breaking-change`-labeled**: #8027, #8028, #8347, #8360. At that tag
the full v11 delta from `v10.0.0-beta.7` stood at **222 commits and 13 breaking PRs** (#8024,
#8025, #8026, #8027, #8028, #8051, #8095, #8159, #8172, #8188, #8206, #8347, #8360).

**Breaking:**

- **`LanceFileVersion` lost its ordering** (#8028). `PartialOrd`/`Ord` are gone, so
  `v >= LanceFileVersion::Next` no longer compiles, and both `From` conversions between selector
  and concrete version were deleted. Index readers, writers, shufflers and distributed mergers
  now take an exact `ConcreteFileVersion`. Per the PR: "Remaining version decisions are
  exhaustive matches at declared boundaries rather than `>=`, `max`, or selector round-trips."
- **`LanceFileVersion::resolve` changed signature** (#8027):
  `pub fn resolve(&self) -> Self` became `pub const fn resolve(self) -> ConcreteFileVersion`.
  Deleted: `iter_non_legacy()`, `support_add_sub_column()`, `support_remove_sub_column(&Field)`.
  Added: `stable_file_version() -> ConcreteFileVersion` (V2_1), `next_file_version()` (V2_3),
  `ConcreteFileVersion::to_selector()` and `::is_unstable()`. #8027 also centralized dataset
  version policies.
- **`Operation::Project` / `Merge` gained `preserves_nullability: bool`** (#8347). See section
  9.2 - a nullability tightening must not set it, and setting it makes the operation conflict
  with concurrent value-writes in either commit order.
- **`is_maintainable_index_type(&str)` removed** (#8360), replaced by
  `validate_maintained_indexes(dataset, index_names) -> Result<()>`. Type-URL filtering was
  unsound: an IVF-PQ over `FixedSizeList<Float64>` passed the check and then made the table
  unwritable. The replacement is all-or-nothing - it "reports the first index it cannot maintain
  rather than returning a usable subset". Error text: "index '{}' has type {}, which the MemWAL
  cannot maintain. Supported: BTree, Inverted, Vector".

**Format-level:**

- **New manifest feature flag at bit 128** (#8263); `FLAG_UNKNOWN` moved 128 -> 256. Reader and
  writer must both hold it. **The flag added here did not survive the major.** It was
  `FLAG_MEM_WAL_INDEX_CATCHUP` from `beta.4` to `beta.17`, then #8680 retired it and #8535 gave
  the reclaimed bit to `FLAG_COVERED_INDEX_METADATA`, which is what `v11.0.0` shipped. The proto
  field `Transaction.UpdateMemWalState.require_index_catchup` was deleted with it, and MemWAL
  catch-up lost its flag gate: an absent `index_catchup` shard now unconditionally means
  *unknown*. Builds pinned inside `beta.4`..`beta.17` still treat bit 128 as supported and will
  open a covering dataset instead of refusing it. Section 7.
- **Transaction proto field 9 deprecated** (#7432): `updated_fragment_offsets` gives way to
  field 10 `updated_fragment_offset_bitmaps`, "Per-fragment matched offsets as portable
  RoaringBitmap bytes". Writers emit field 10 only; readers prefer 10, falling back to 9 for
  manifests written before the change.
- `MemWalIndexDetails.index_catchup` added as `table.proto` field 10.
- **`IndexCatchupAdvance` never shipped.** #8263 added the message and
  `CreateIndex.mem_wal_index_catchup_advances`; #8481 deleted both within the same beta window,
  replacing them with catch-up derived from the version the transaction read. Present at
  `v11.0.0-beta.5`, absent at `v11.0.0-beta.6`.

**Net-new:**

- MemWAL backpressure is observable: `MemTableStats.frozen_count` / `frozen_bytes` and
  `ShardWriter::backpressure_stats()` (#8241) - "Heap bytes still owed to flush".
- MemWAL splits logical from storage schema, widening non-PK top-level fields to nullable, so
  `ShardWriter::delete` no longer requires nullable base columns (#8352).
- `write_fragments(session=...)` / Java `WriteFragmentBuilder.session(...)` (#8034); a foreign
  session against a dataset-backed target is rejected.
- `analyze_plan` appends `tokenized_query=` to FTS leaves (#8414); `explain_plan` deliberately
  unchanged. Python `lance.tokenize(...)` / `lance.FtsToken(text, position)` preview tokenization
  with no dataset or index (#8415).
- `LanceFragment.validate()` (#8428) validates one fragment rather than the whole dataset;
  `Dataset::validate()` gained stable-row-id invariant checks (#8258, no-op when unused).
- `BlobFile.read_ranges(ranges) -> list[bytes]` (#8319) - "The underlying physical reads may be
  reordered, coalesced, or split for efficiency."
- `lance.fragment.RowIdSequence` (#8356); duplicate ids now rejected.
- `LanceOperation.Update` carries `updated_fragment_offsets` in Python (#8447) and
  `updatedFragmentOffsets` in Java (#6748).
- Java: `Session.Builder` selects registered native cache backends by URI or
  `CacheBackendConfig`, e.g. `moka://?capacity=1048576` (#8446); `Index.getSizeBytes()` and
  `IndexDescription.getSegments()` (#8355).
- Blob v2 supported in `FileFragment::update_columns` (#8344).

**Performance / build:**

- FTS same-column `MUST + SHOULD` scores optional clauses lazily (#8448); conjunction
  confirmations ordered by `match_cost`, measured **200 -> 120** two-phase `matches()` calls per
  query (#8354).
- Release JNI cdylib stripped: `liblance_jni.so` linux-x86-64 **278.35 MB -> 221.0 MB**
  (-20.6%), `.dynsym` preserved (#8314).
- x86_64-linux build baseline dropped `target-cpu=haswell` -> **`x86-64-v2`** (#8377), so
  binaries no longer trap on import on pre-AVX2 hosts.
- The `time = "=0.3.47"` pin was removed from `lance-namespace-impls` (#8296).
- `retain_versions=0` now errors instead of panicking (#8467); deleting a branch referenced by a
  tag is rejected (#8365).

---

### The v11.0.0-beta.6 -> v11.0.0-beta.16 delta (the v11 beta frontier)

91 commits, **one newly `breaking-change`-labeled**: #8235. This brings the full v11 delta from
`v10.0.0-beta.7` to **313 commits and 14 breaking PRs** (#8024, #8025, #8026, #8027, #8028,
#8051, #8095, #8159, #8172, #8188, #8206, #8235, #8347, #8360). Structural invariants all held
at beta.16: 26 crates, 16 transaction ops, `num_retries` 20, `next => 2.3` / default 2.1 (no
2.4), arrow 58, datafusion 54, MSRV 1.91.0, Edition 2024, Python 3.10+, manifest feature flags
unchanged (bit 128 allocated, `FLAG_UNKNOWN` 256).

The **final** added two more breaking PRs (#8407, #8535) for **357 commits and 16 breaking PRs**
to `v11.0.0`, and reallocated bit 128 to `FLAG_COVERED_INDEX_METADATA` as described above. Every
structural invariant above still holds at `v13.0.0-beta.4`.

**Breaking:**

- **Compaction gained row and byte budgets** (#8235) - `max_source_rows: Option<usize>` and
  `max_source_bytes: Option<u64>` join `max_source_fragments` on `CompactionOptions`
  (`rust/lance/src/dataset/optimize.rs:278,287`), each with a matching `lance.compaction.*`
  config key. The label is on the options struct changing shape; the feature itself is additive.

**Net-new:**

- **Lightweight version references** (#8523) - `Dataset::version_refs()` -> `Vec<VersionRef>`
  lists manifest locations without deserializing every manifest, unlike `versions()`. Section 7.
- **`Dataset::migrate_to_stable_row_ids`** (#8521) - one `Merge` commit converts an existing
  dataset to stable row IDs and flips the flag atomically; `with_max_retries(0)`, so quiesce
  writers first. Supersedes "stable row IDs cannot be turned on later". Section 8.
- **Per-fragment column writes** (#8313, renamed `write_columns` by #8622) - survive compaction.
- **Compaction fragment exclusion** (#8532) - `excluded_fragment_ids: Vec<u32>` (`optimize.rs:295`).
- **AMX-FP16 IVF acceleration** (#8540) - ships `LANCE_DISABLE_AMX` and `LANCE_AMX_FP16_CC`, the
  only new `LANCE_*` env vars in the whole v11 line, and makes partition assignment exact where
  it engages. See `performance.md`.
- **External manifest stores: object storage became authoritative** (#8499). Section 9.3.
- FTS: MAXSCORE for pure SHOULD queries and its metrics (#8474, #8475), exact posting load
  policies (#8667), chunked posting reads during segment merge (#8668). Java: manifest writer
  version and location metadata (#8450, #8451), efficient dataset version count (#8453).
- `perf(dataset)`: `get_fragment` binary-searches the manifest (#8636), with a fall-back to
  linear scan for legacy manifests that are unsorted (Lance <= 0.10) or hold duplicate fragment
  ids (Lance <= 0.16) - the same legacy shapes #8206 made uncommittable.

**Correctness fixes in this window, split by whether upgrading is enough.**

*Requires rewriting or repairing data already on disk - upgrading alone does not heal it:*

- #8382 - malformed variable-width Arrow offsets. Reachable in practice via `slice_arrays`
  page-splitting, so not theoretical.
- #8669 - JSON columns updated from string expressions hold raw text. Rows updated with an
  explicit `jsonb '...'` were always fine.
- #8509 - reordered sources in indexed merge insert; re-run the affected merges (the old
  statistics were misleading too).
- #7703, #8539 - invalid manifests already committed; validation is commit-time only, so
  existing bad manifests stay bad.
- #8459 - non-atomic tag creation. A clobbered tag is **unrecoverable and undetectable**;
  re-create it manually.
- #8378 - Windows UNC share roots. Data "written to a UNC URL" actually landed on the local drive.
- #8482 - FTS metadata not written when a distributed build had no partitions; re-run the build.

*Read-path only, heals on upgrade:* #7371, #7966, #8443, #8525, #8534, #8536, #8542, #8577,
#8587, #8588, #8591, #8592, #8593, #8594, #8595, #8596, #8597, #8598, #8599, #8600, #8602,
#8603, #8609, #8613, #8618, #8620, #8636, #8650, #8666, #8668, #8682, #8687, #8388, #8381.

Three worth calling out individually:

- **#7966** is not purely a type-support widening. ZoneMap zones over Decimal128/256 columns
  written by **Lance 8.0.0** carry typed-null extrema despite holding real values, and the old
  `zone_has_finite_min` guard skipped those zones - silently dropping matching rows. Reading
  heals on upgrade, but **pruning selectivity stays degraded until the index is rebuilt**
  (`rust/lance-index/src/scalar/zonemap.rs:203-205`).
- **#8542** - `multivec_distance` with a query length that is not a positive multiple of `dim`
  silently scored every row `1.0 - 0.0` instead of erroring (`rust/lance-linalg/src/distance.rs:411`).
- **#8499** is the special case: legacy external-store rows still carry `e_tag`, but new readers
  set `e_tag: None` and ignore them, so mixed-version rows converge with **no migration**. The
  stale-ETag race can still fire while legacy *finalizers* remain in the fleet.

**Security:** #8613 bumps `h2` to 0.4.16 for RUSTSEC-2026-0258.

---

### v11 silent-corruption and wrong-results fixes

Eleven fixes in the v11 line address failures that produced **no error** - wrong data, missing
rows, or a hang. Each names the condition that triggers it, so you can tell whether a dataset
written on an earlier v11 beta is affected. The `beta.6 -> beta.16` window added more; they are
listed in that delta above, split by whether upgrading is enough.

**Data-loss class:**

- **Cleanup irreversibly deleted live overlay data** (#8267). Six manifest/fragment walkers read
  only `Fragment::files` and missed overlay data files; `process_manifest` builds the cleanup
  keep set, "so an overlay old enough to be a deletion candidate is irreversibly deleted from the
  live dataset". Affects datasets using data overlay files (`FLAG_UNSTABLE_DATA_OVERLAY_FILES`).
- **A fragment-less manifest could be published** (#8438). `Operation::UpdateMemWalState` rebuilt
  its manifest without `final_fragments`, "so the commit publishes a manifest with **no
  fragments**. Every row in the table disappears." MemWAL tables only.
- **Stale row-id sequences after overwrite** (#8078). `RowIdSequenceKey` was keyed on
  `fragment_id` alone, so after `WriteMode::Overwrite` with a shared `Session`, reads got the
  previous generation's sequence: "row ids are reported for rows that no longer exist, row counts
  disagree with the manifest, compaction rechunks more ids than the fragments physically hold,
  and the same id can look live in two fragments at once." The cache key now includes
  `row_id_meta`.
- **Tencent COS double-commit** (#8369) - see section 9.3.
- **MemWAL compaction generation mixup** (#8262): progress kept the larger generation, so a late
  job's rows were written under another job's marker - "mutations under a generation it did not
  produce, and anything reading only the marker could then stop serving SSTables whose rows were
  never inserted."
- **Stale `index_section` offset** (#8308): a manifest reused after its last index was dropped
  kept the prior offset, which "would point at unrelated bytes in the new manifest file".
- **Dictionary index width mismatch** (#8220): nullable dictionary normalization could widen
  indices to UInt32 while the page stayed declared Int8.

**Wrong-results class** - these directly contradict "this index returns correct results":

- **A KNN row returned twice, one ranked by a stale vector** (#8342).
  `optimize_indices(num_indices_to_merge >= 1)` "can leave two copies of the same row in a vector
  index, and a KNN query then returns that row twice, one ranked by its pre-update vector."
- **BloomFilter matches silently disappeared** (#8223): after deferred-remap compaction the index
  "returned the original zone ranges without applying that mapping", so matches on moved rows
  vanished.
- **Every approximate cosine distance was shifted** (#8393): IVF_RQ's query-factor match "still
  grouped Cosine with Dot and subtracted `1.0`".
- **FTS could prune a competitive document** (#8473) - WAND score upper bounds were not
  conservative against f32 accumulation order. Separately, fragment-restricted FTS scans used
  only the first segment's coverage bitmap, so "matching rows in later segments could be filtered
  out" (#8211).
- **A query could hang indefinitely** (#8350): an IVF delta taking the no-more-probes early
  return never decremented the late-search barrier. "Every delta must reach the barrier, even if
  it has no partitions left to search, so that siblings waiting for the initial search can
  proceed."

### v11.0.0 final (the beta.16 -> final delta)

Two more `breaking-change`-labeled PRs landed after `beta.16`, plus net-new surface the beta
never carried. 44 commits.

**Breaking:**

- **Bit 128 reallocated** - `FLAG_MEM_WAL_INDEX_CATCHUP` retired (#8680), replaced by
  `FLAG_COVERED_INDEX_METADATA` (#8535). Covered above and in section 7.
- **DataFusion filter planning uses the caller's session** (#8407, labeled `breaking-change` /
  `A-python`).

**Net-new:**

- **Covering indexes** (#8535). `IndexMetadata.covering_fields` (proto field 11) is "the
  trailing subset of `fields` whose values the index carries but is not keyed on, letting a
  query that only projects those columns be answered without a fragment take." `fields` is
  redefined - `fields[0]` is always keyed, trailing entries may be merely carried. Index
  invalidation widens to any column in `fields`, keyed or carried. Additive on the wire, and
  "no index builder writes carried values yet, so today every declaration is ahead of its
  storage." Section 11.
- **`merge_insert` gained `write_mode`** (#8423): `Auto` (default), `RewriteRows`, or
  `RewriteColumns`. Under `RewriteColumns` an updates-only partial-schema merge "patches the
  source columns into the fragments that already hold the matched rows instead of rewriting
  whole rows", through a new `InPlaceMergeInsertExec`.
- **`Scanner::with_row_addr_prefilter(RowAddrMask

format-file.md 37.4 KB

# Lance v12 reference - file format (sections 1-4)

Part of the Lance v13 reference (`lance-format/lance@v13.0.0-beta.4`). Citations are `path:line`
relative to the repo root; build a permalink as
`https://github.com/lance-format/lance/blob/v13.0.0-beta.4/<path>`. Line numbers drift between
tags - treat them as approximate. Cross-references written as "section N" use the original
16-section numbering; `lance-reference.md` maps every number to its file.

This is the Lance *format and engine*. LanceDB (`lancedb/lancedb`) is a separate database
product built on top of Lance and is out of scope - but Lance is what it stores into, so this
reference is still authoritative for the format underneath it.

## Contents

- [1. What Lance is](#1-what-lance-is)
- [2. The crate workspace](#2-the-crate-workspace)
  - [2.1 Module reorganization in v11 (PRs #8020-#8026)](#21-module-reorganization-in-v11-prs-8020-8026)
- [3. File format](#3-file-format)
  - [3.1 Versions](#31-versions)
  - [3.2 Container layout](#32-container-layout)
  - [3.3 Structural encoding (2.1)](#33-structural-encoding-21)
  - [3.4 Compression](#34-compression)
  - [3.5 Blob encoding](#35-blob-encoding)
  - [3.6 Exact file identity: `ConcreteFileVersion`](#36-exact-file-identity-concretefileversion-v10-relocated-in-v11)
- [4. Data types](#4-data-types)

Other files: `format-table.md` (5-10), `indexes.md` (11-12), `ops.md` (13, 15, 16),
`changelog-v7-v13.md` (14).

---

## 1. What Lance is

Lance is "a columnar data format that is 100x faster than Parquet for random access"
(`Cargo.toml:37`, workspace description). It is not a single format but a **stack of
interoperating specifications**, deliberately decoupled so each layer evolves independently
(`docs/src/format/index.md:3-19`):

- **File format** - stores column data in large random-access-friendly pages, no row groups.
  Only table readers/writers and index readers/writers need to know the on-disk layout.
- **Table format** - the dataset: manifests, fragments, deletion files, schema, transactions.
- **Index formats** - scalar, vector, full-text, geo, and system indexes. The file format
  deliberately keeps statistics and search structures *out* of the file so indexes evolve
  as independent specs (`docs/src/format/index.md:25`).
- **Catalog specs** - Directory Catalog and REST Catalog: how datasets are discovered.
- **Namespace client spec** - a unified client interface for engines to talk to any catalog,
  Lance-native or third-party, in any language.

Lance uses **Apache Arrow** as its in-memory type system and is consumed directly by DuckDB,
Polars, Ray, Spark, PyTorch, TensorFlow, and DataFusion, or by your own Rust/Python/Java
code. The format itself is the product - there is no server.

---

## 2. The crate workspace

26 crate directories under `rust/`. `[workspace.package]`: `version = "12.0.0-beta.15"`,
`edition = "2024"`, `rust-version = "1.91.0"` (MSRV; the pinned build toolchain in
`rust-toolchain.toml` is `1.97.0`, PR #7712), `license = "Apache-2.0"`,
`resolver = "3"` (`Cargo.toml:32-56`). `exclude = ["python", "java/lance-jni"]`. The crate set
is **unchanged from `v10.0.0-beta.7`** - the `Cargo.toml` inventory under `rust/` is
byte-identical between the two tags; the last addition was `lance-index-core` (PR #7713).
Module *layout inside* several crates changed substantially in v11 - see 2.1.

| Crate dir | Published name | Purpose |
|-----------|----------------|---------|
| `lance` | `lance` | **Public entry point.** `Dataset`, scanner, indexes, commits |
| `lance-table` | `lance-table` | Table format: `feature_flags`, manifest `format`, commit `io`, `rowids` |
| `lance-file` | `lance-file` | File format: file reader/writer, `LanceEncodingsIo`, MAGIC bytes |
| `lance-encoding` | `lance-encoding` | Structural encodings, compression. Internal - not for external use |
| `lance-index` | `lance-index` | Secondary indexes: scalar, vector, FTS, system |
| `lance-index-core` | `lance-index-core` | Shared index primitives extracted from `lance-index`. New in the 9.1/10.0 dev line (PR #7713) so lighter consumers can depend on core index types without the full index crate |
| `lance-io` | `lance-io` | Object store, I/O schedulers, local FS, FFI |
| `lance-core` | `lance-core` | Shared `Error`/`Result`, `cache`, `datatypes`, `traits`, `utils` |
| `lance-datafusion` | `lance-datafusion` | DataFusion glue: `exec`, `expr`, `planner`, `projection`, UDFs |
| `lance-linalg` | `lance-linalg` | SIMD L2 / dot / cosine / hamming kernels |
| `lance-arrow` | `lance-arrow` | Arrow extensions (`RecordBatchExt`, `SchemaExt`). Considered never-stable |
| `lance-select` | `lance-select` | Row-selection primitives: `RowAddrMask`/`NullableRowAddrMask`, `RowIdMask`, `IndexExprResult`. Extracted from `lance-core`/`lance-index` in v7.1.0-beta.2 (PR #6879) so benchmarks and filter consumers can depend on masks without pulling in either larger crate |
| `lance-tokenizer` | `lance-tokenizer` | FTS tokenizer stack: `TextAnalyzer`, jieba/lindera/ngram, filters |
| `lance-derive` | `lance-derive` | Proc-macro crate (`proc-macro = true`): `#[derive(DeepSizeOf)]` for Arrow-aware memory accounting. New in v8 (PR #6229), replacing the external `deepsize` crate, which double-counts Arrow buffers shared across `Arc` |
| `lance-geo` | `lance-geo` | Geospatial UDFs. Feature-gated `geo` |
| `lance-namespace` | `lance-namespace` | `LanceNamespace` trait + data models |
| `lance-namespace-impls` | `lance-namespace-impls` | `DirectoryNamespace`, `RestNamespace`, REST adapter, credential vendors |
| `lance-namespace-datafusion` | `lance-namespace-datafusion` | DataFusion catalog/schema provider bridge |
| `lance-tools` | `lance-tools` | `cli` / `meta` / `util`; ships a `lance-tools` binary |
| `lance-datagen` | `lance-datagen` | Random Arrow array/batch generation for tests/benchmarks |
| `lance-test-macros` | `lance-test-macros` | Test-only proc macros |
| `lance-testing` | `lance-testing` | Shared test helpers/fixtures |
| `compression/fsst` | `fsst` | FSST string compression |
| `compression/bitpacking` | `lance-bitpacking` | Vendored SIMD bit-packing (from spiraldb/fastlanes) |
| `arrow-scalar` | `lance-arrow-scalar` | Arrow scalar with `Ord`/`Hash`/`Eq`. Pinned `=58.0.0` (tracks Arrow) |
| `arrow-stats` | `lance-arrow-stats` | Statistics accumulator (min, max, null_count, nan_count). Also pinned `=58.0.0` (`Cargo.toml:86-87`) |

`rust/examples` (`lance-examples`) holds non-published example binaries. The workspace
`members` array lists 26 paths; `rust/lance-datafusion` is part of the workspace as a
path dependency rather than an explicit member.

**Bindings.** Python: package `pylance` (`python/pyproject.toml`), built with maturin, imported
as `lance`; the Rust extension crate is `pylance` (`[lib] name = "lance"`); supports Python
**3.10-3.14** (3.9 dropped in v9, PR #7345, breaking; PyO3 abi3 floor raised to `abi3-py310`);
runtime deps `pyarrow>=14`, `numpy>=1.22`, `lance-namespace>=0.11.1,<0.12`. Java: an
SDK under `java/` (Maven `org.lance`), bridged to Rust by the `lance-jni` crate
(`java/lance-jni/`, excluded from the Rust workspace). Notable workspace deps at this tag
(`Cargo.toml`): `arrow 58.0.0` (`:85`), `datafusion 54.0.0` (`:132`), `geodatafusion 0.5.0`,
**`opendal 0.58.1`** (`:180`, was `0.57` at v10 - PR #7823), `object_store 0.13.2` (`:179`),
`object_store_opendal 0.58` (`:181`), `jieba-rs 0.10` (`:166`), `itertools 0.14` (`:165`),
`lance-namespace-reqwest-client 0.12.0` (`:76`, bumped from 0.11.1 by #8915), and
`blake3 1.8.5` (`:113`, backing the cache-key digest, section 9.4). The
`lance-namespace`/`-impls` crates publish at the workspace version (`12.0.0-beta.15`).

**The `lance-namespace` version is no longer a single number.** #8915 moved the *Rust* client to
`0.12.0` while the Java pin (`java/pom.xml:113,118`, `0.11.1`) and the Python pin
(`python/pyproject.toml:4`, `lance-namespace>=0.11.1,<0.12`) deliberately stay on 0.11: "The Java
and Python `lance-namespace` pins stay on 0.11, so their generated models still send `on` as a
bare string and rely on the promotion described above." Quote a language-specific pin, not one
number for all three.

**Dependency deltas in the v11 range.** Only five non-version-bump lines changed in the root
`Cargo.toml`: `opendal`/`object_store_opendal` 0.57 -> 0.58.1/0.58, and **`strum 0.26` plus
`goosefs-sdk =0.1.5` were removed outright**. GooseFS is now reached purely through opendal
(`goosefs = ["dep:opendal", "opendal/services-goosefs", "dep:object_store_opendal"]`,
`rust/lance-io/Cargo.toml:73`), with `goosefs-sdk` pulled transitively at 0.1.8; the
`[patch.crates-io]` opendal git fork was dropped from the root, Python, and Java manifests.
Separately, `lance-tokenizer` swapped **`rust-stemmers 1.2.0` -> `frostem`**
(`rust/lance-tokenizer/Cargo.toml:18`, PR #8183): the unmaintained crate's Greek implementation
"can retain stale UTF-8 byte offsets after shortening a word, then panic while slicing the
shortened string". `frostem` is generated from current upstream Snowball, and only the same 18
Snowball algorithms already exposed by the `Language` API are enabled - the inverted-index
protobuf details and capability versions are unchanged. `Cargo.lock` also gained
`opendal-http-transport-reqwest` and dropped `crc32c`.

### 2.1 Module reorganization in v11 (PRs #8020-#8026)

If you depend on anything below the `lance` crate, this is the largest source-compatibility
event in the v10 -> v11 range. Six PRs moved the file-format reader/writer machinery out of
version-agnostic modules and into per-version ones, with **no re-exports left behind**:

| Was (v10) | Is (v11) |
|-----------|----------|
| `lance-encoding::version` (`LanceFileVersion`, `LEGACY_FORMAT_VERSION`, `V2_FORMAT_2_*`, `resolve`, `is_unstable`, ...) | `lance-file::version` - the module file is **deleted** from `lance-encoding` (#8026) |
| `lance-file::previous::*` | `lance-file::versions::v1::*` (#8020) |
| `lance_io::encodings` (`Encoder`, `Decoder`, `AsyncIndex`, `read_binary_array`, `read_fixed_stride_array`, `bytes_to_array`, ...) | **removed** (#8020) |
| `lance-encoding::previous` public encoder surface (`ArrayEncoder`, `ArrayEncodingStrategy`, `CoreFieldEncodingStrategy`, `EncodedArray`, `BitpackedArrayEncoder`, `FixedSizeBinaryEncoder`, ...) | **removed**; per-version `lance-file::versions::v2_{1,2,3}::compression` (#8021) |
| `struct FileWriter` with `try_new` / `new_lazy` / `create_file_with_batches` | `enum FileWriter { V2_0, V2_1, V2_2, V2_3 }` (`rust/lance-file/src/writer.rs:52,57`); construct via `versions::{create_writer, create_lazy_writer, encode_self_described_batch, encode_mini_batch}` (#8023) |
| `ReaderProjection::{from_field_ids, from_whole_schema, from_column_names}` | free fns `reader_projection_from_*` in `lance-file::versions` (`versions/mod.rs:119,154`) (#8024) |
| `FileReader::version() -> LanceFileVersion`; `FileReader::supports_projection` | `-> ConcreteFileVersion` (`reader.rs:2259`); `supports_projection` **removed** (#8024) |
| `Dataset::storage_version_or_default() -> LanceFileVersion`; `open_writer`; public `do_write_fragments` | `-> ConcreteFileVersion` (`write.rs:458`); `open_writer` **removed**; `do_write_fragments` crate-private (#8025) |

`FileWriterOptions` also lost `encoding_strategy` and `format_version`, and
`initialize_with_external_metadata` was renamed `initialize_with_external_columns`.
`CommitBuilder::with_storage_format(LanceFileVersion)` keeps its public signature (it now
`.into()`s), and `determine_file_version` stopped panicking on a failed `size()` call.
Physically, `lance-file/src/previous/` and `lance-encoding/src/previous/` (22 files) are gone,
replaced by `lance-file/src/versions/{v1,v2_0,v2_1,v2_2,v2_3}/` and
`lance-encoding/src/array_encoding/{logical,physical}/` plus a new `strategy.rs`.

Two more encoder-facing breaks rode along: `MiniBlockCompressor::compress` gained a
`MiniBlockCompressionContext` parameter (#8038 - "It intentionally changes no codec selection or
persisted bytes"), and `DataBlockBuilder::append` became fallible with `DataBlockBuilderImpl`
made private (#8172), so malformed variable-width offsets now return `Error::CorruptFile`
instead of panicking or yielding garbage - a file that previously "read" may now error.

**Published vs tagged.** crates.io carries only final releases - `lance 9.0.1` (2026-08-06) is
the newest, preceded that same day by the sibling patch finals 8.0.1, 7.1.0, 6.1.0, 4.0.2, and
3.0.2. **No 10.x or 11.x version, and no pre-release of any kind, is published.** Beta and rc
tags exist in git only (beta artifacts go to fury.io), so building against `v13.0.0-beta.4`
means a git dependency, not a registry one.

**Building.** Five workspace crates carry a protobuf build script - `lance-encoding`,
`lance-file`, `lance-index`, `lance-table`, `lance-datafusion` - so a `protoc` compiler must
be reachable to build them. The `lance` crate's `protoc` feature vendors one (`protobuf-src`)
and cascades it to the first four, but **not** to `lance-datafusion`, which still needs a
system `protoc` (`Cargo.toml:140-146`, `rust/lance-datafusion/Cargo.toml`).

---

## 3. File format

### 3.1 Versions

The file format has one major.minor version: the major changes when the container changes,
the minor when only the encoding strategy changes (`docs/src/format/file/versioning.md:3-5`).
The footer stores `u16` major and `u16` minor (`protos/file2.proto:90-91`).

| Version | Min Lance | Status | Description (`docs/src/format/file/versioning.md:18-26`) |
|---------|-----------|--------|-------------|
| `0.1` (`legacy`) | any | read-only, no longer writable | Initial Lance format |
| `2.0` | 0.16.0 | stable | Removed row groups; null support for lists, fixed-size lists, primitives |
| `2.1` | 0.38.1 | previous default | Adaptive structural encodings; better integer/string compression; nulls in struct fields; better nested random access |
| `2.2` | - | **current default** (`stable`) | Map type, Blob v2, `VariablePackedStruct`, larger mini-blocks |
| `2.3` | - | unstable (`next`) | The current `next` alias target. Ships **sparse structural pages** (PR #7889) - the first 2.3-specific encoding; **auto-selected** by the 2.3 writer under a budget heuristic (PR #7756), or forced via `lance-encoding:structural-encoding=sparse` |

`stable` resolves to **2.2** as of `v12.0.0-beta.15` (#8657); `next` resolves to the latest
unstable version. The enum declaration order is
`Legacy, V2_0, V2_1, Stable, V2_2 (#[default]), Next, V2_3`, with `Stable => 2.2` and
`Next => 2.3` (`rust/lance-file/src/version.rs:18-45` - the module **moved out of
`lance-encoding` in v11**, PR #8026, with no re-export). No 2.4 or new variant exists at this
tag.

`#[default]` moved from `V2_1` to `V2_2` in the same PR, and the default reaches new datasets
through `impl Default for DataStorageFormat { fn default() -> Self { Self::new(stable_file_version()) } }`
(`rust/lance-table/src/format/manifest.rs:677-680`). Upstream's framing was that the code lagged
the intent: "Lance 2.2 is the current stable file format, but the centralized release policy and
enum default still resolve new datasets to 2.1." The docs were **not** updated with the change -
`docs/src/format/file/versioning.md` is byte-identical across the range and still describes
`stable` only as an "Alias for the default version for new datasets in the Lance release you are
running", so read the code, not the table, for what `stable` means at a given tag.

Two consequences that surprise readers: (1) **`next` resolves to 2.3, not 2.2** - writing with
`next` produces a 2.3 file; (2) the code does **not** flag 2.2 as unstable, and now writes it by
default.

**`is_unstable()` is not an ordering comparison.** An earlier form of this note read
`is_unstable() = self >= Next`, which describes no recent tag and could not compile:
`LanceFileVersion` derives only `Debug, Default, Clone, Copy, PartialEq, Eq` - it lost
`PartialOrd`/`Ord` in #8027/#8028. The selector delegates
(`pub const fn is_unstable(self) -> bool { self.resolve().is_unstable() }`, `version.rs:71-74`)
and the concrete version matches exactly
(`pub const fn is_unstable(self) -> bool { matches!(self, Self::V2_3) }`, `version.rs:147-150`).
As of v9 the docs version table (`docs/src/format/file/versioning.md:18-27`) agrees: it lists
`2.3 (unstable)` and no longer labels 2.2 unstable (2.2 now reads "Adds support for newer
nested type/encoding capabilities (including map support) and 2.2-era storage features"). As
of v9.1 the 2.3 row reads **"Adds sparse structural pages and other experimental encodings"**
- 2.3 is no longer a placeholder: `V2_3` references in `lance-encoding` jumped 6 -> 59 with the
sparse-page encoding (`rust/lance-encoding/src/encodings/logical/primitive/sparse.rs`, PR
#7889). **Sparse pages** represent flat or nested Arrow structure directly as slot-domain
mappings instead of dense repetition/definition events (`docs/src/format/file/encoding.md:330`);
`structural-encoding` accepts `miniblock`, `fullzip`, or `sparse` (`sparse` requires 2.3).
2.2 still carries Map / Blob v2 / `VariablePackedStruct`.

**Sparse auto-selection (v10, PR #7756).** Sparse is no longer opt-in only. "Without an explicit
structural encoding, the Lance 2.3 writer selects sparse only when the dense mini-block
repetition/definition budget would split the page or one top-level row exceeds that budget, and
only when the value path is supported by the sparse writer"
(`docs/src/format/file/encoding.md:373-376`). Consequences worth knowing: "Unsupported sparse
value paths, including dictionary values and variable-width packed structs, retain their dense
behavior" (`encoding.md:378`), and "Lance 2.2 and earlier writers never select sparse"
(`encoding.md:375-376`). The field-metadata table reworded the key from *Select* to **"Force a
structural encoding; `sparse` requires Lance 2.3"** (`encoding.md:694`) - because leaving it
unset no longer means "never sparse". The policy decision is kept out of serialization: it
"adds no wire-format fields" (PR #7756).
`next` encodings can change and files written with them may become unreadable - "should only
be used for experimentation and benchmarking" (`docs/src/format/file/versioning.md:8-11`).
The default storage version became 2.1 in Lance 5.0.0 (`docs/src/guide/migration.md`), and 2.2
as of `v12.0.0-beta.15` (#8657); 2.2 is required for the Map type and Blob v2. Selected
per-dataset via `data_storage_version` - which, **as of the `v12.0.0` final, is no longer fixed
at creation**. It is the write-time default, not a description of the dataset: "The dataset's
`data_storage_version` property is the default for writes that omit a target, not a summary of
its existing files. Create and overwrite establish this default; append, update, merge-insert,
and compaction do not change it." An existing V2 dataset accepts `"2.0"`, `"2.1"`, `"2.2"` or
`"2.3"` per operation "without rewriting the other files", so one dataset can carry several exact
V2 versions at once and "each DataFile's version is authoritative for decoding". **V1 and V2
still cannot be mixed.** The capability is gated on the paired
`FLAG_MIXED_DATA_FILE_VERSIONS` bits (section 7), which the commit sets automatically - "there is
no separate activation API" - and which never clear afterwards.

Compaction can retarget versions by copying rather than reencoding: "`try_binary_copy` falls back
to reencoding when inputs are ineligible; `force_binary_copy` rejects them", and a persistent
target can be set with the `lance.compaction.data_storage_version` table-config key. Before any
of this, upgrade every reader and writer: "Drain, restart, or fence writers that opened the
dataset using an older release" - the flags cannot fence an older writer retroactively.

**A FixedSizeList whose inner values are all null is a two-way compatibility fence** (#9130,
`v12.0.0`). Before the fix, `FSL<2> = [[NULL, NULL], [NULL, NULL]]` - each list non-null, every
*element* null - was stored with `bits_per_values=0` and "the resulting file would be
unreadable"; such files must be **rewritten**, not just read by a newer build. After the fix the
fence points the other way: "Files containing this pattern written by Lance >= 11.1.0 are **not
readable by Lance < 11.1.0**. Old readers encounter the `Compression::Constant` inner encoding in
the FSL descriptor and panic rather than returning an error."

Treat that version number with suspicion: **`11.1.0` was never released** (the 11.1 line was
re-rooted into v12 without a tag), and the fix and this doc text landed in the same commit, which
shipped in `v12.0.0`. Read the fence as "pre-`v12.0.0` readers panic".

### 3.2 Container layout

A `.lance` file, top to bottom (`docs/src/format/file/index.md:123-161`, `protos/file2.proto`):

1. **Data pages** - sector-aligned data buffers.
2. **Column metadata** - one standalone protobuf `ColumnMetadata` per column. A subset of
   columns can be read without reading all metadata (column projection).
3. **Column metadata offset table** - position + size per column.
4. **Global buffers offset table** - position + size per global buffer (file schema, file
   indexes, column statistics).
5. **Footer** (fixed-size) - offsets to the above, column/buffer counts, `u16` major + `u16`
   minor version, magic `"LANC"`. All fields little-endian.

**No row groups.** "Unlike similar formats, there is no 'row group' concept, only pages. We
believe the concept of row groups to be fundamentally harmful to performance"
(`docs/src/format/file/index.md:41-42`). A disk page holds rows for a single column; each
column has its own page count. Default recommended page size is 8MB. A reader can split a
file at any row boundary via partial page reads with minimal read amplification - the unit of
parallelism is decoupled from physical layout.

Buffers are referenced by absolute offset, aligned to 64 bytes (SIMD) or 4096 (direct I/O).
The file container has **no type system** - columns are integer-indexed; the schema lives in
a global buffer and the file format is unaware of it. Encodings are extensions, designed to
be added/removed without recompiling the reader.

### 3.3 Structural encoding (2.1)

A structural encoding "breaks the data into smaller units which can be independently
decoded" and encodes structure (struct/list validity, list offsets) via **repetition and
definition levels** - one combined buffer instead of separate validity bitmaps and offset
arrays, to avoid multiple IOPS (`docs/src/format/file/encoding.md:48-69`). Note: Lance uses
**0 for the inner-most item** (Parquet uses 0 for the outer-most).

Data types and layouts are orthogonal. The top-level `PageLayout` has four page types
(`protos/encodings_v2_1.proto:197-210`):

- **Mini-block** - default for "smallish" types (integers, floats, booleans, small strings).
  Data split into mini-blocks of a power-of-two value count, each <32KiB compressed; reading
  any value reads the whole block, so blocks are kept small. Rep/def levels are sliced into
  the blocks. A random-access metadata buffer (2 bytes/block) is loaded into the search cache
  at init time. Default 4096 values/block (`LANCE_MINIBLOCK_MAX_VALUES`), **tunable up to
  32k** via that env var since v9 (PR #7356). 2.2 adds larger chunks (>=64KB) via
  `has_large_chunk`.
- **Full-zip** - for larger values (e.g. vector embeddings) above a 256-byte cutoff. Rep/def
  levels and compressed buffers are zipped into one buffer; a per-row repetition index gives
  random access. Requires *transparent* compression (individual values indexable after
  compression).
- **Constant** - all visible values in the page are the same scalar; also the all-null case.
  Generalizes the old `AllNullLayout` for file version >=2.2.
- **Blob** - large binary values stored out-of-line; the page stores `(position, size)`
  descriptions. See 3.5.

**Search cache.** Random access needs encoding + page-location info; this forms an LRU
"search cache" loaded during an initialization phase, amortized over the reader's lifetime.
Cold full scans can skip loading it.

Semi-structural transforms applied before structural encoding: **dictionary encoding**
(decided per leaf value page, so `List<u32>` can dictionary-encode its values), **struct
packing** (row-major struct storage - `PackedStruct` for fixed-width children in 2.1,
`VariablePackedStruct` for variable-width in 2.2), and **fixed-size-list flattening**.

### 3.4 Compression

Compression schemes and the contexts they apply in (`docs/src/format/file/encoding.md:441-450`):

| Scheme | Notes |
|--------|-------|
| Flat | Uncompressed fixed-width; bits-per-value need not be a multiple of 8 |
| Variable | Uncompressed variable-width (values + offsets) |
| Bitpacking | Drops unused high bits. `InlineBitpacking` (per-chunk width, opaque) and `OutOfLineBitpacking` (fixed width, transparent) |
| FSST | "The primary compression algorithm for variable-width data" - fast and transparent |
| RLE | Runs of identical values; applied when `run_count/num_values` < threshold (default 0.5) |
| ByteStreamSplit | Splits multi-byte values into per-byte streams; only helps if general compression also runs; f32/f64/timestamps only |
| General | Opaque back-referencing compressors: LZ4, ZStandard, Snappy. Auto-applied in full-zip for values >=32KiB; otherwise opt-in |

Configured via field metadata (`docs/src/format/file/encoding.md:536-552`): keys
`lance-encoding:compression` (`lz4`/`zstd`/`none`/`fsst`), `:compression-level`,
`:rle-threshold` (default 0.5), `:bss` (`off`/`on`/`auto`), `:general` (`off`/`on`),
`:packed`, plus dictionary tuning (`:dict-divisor`, `:dict-size-ratio`,
`:dict-values-compression`). Compression sub-crates: `fsst` (`rust/compression/fsst`) and
`lance-bitpacking` (`rust/compression/bitpacking`, a vendored copy of spiraldb/fastlanes).
`lance-encoding` default features: `lz4`, `zstd`, `bitpacking`.

The encoding strategy "tends to evolve more quickly than the file format itself"
(`encoding.md:3-4`); several layout details are explicitly marked as likely to change (the
FSST per-page symbol table, full-zip value-size encoding, constant-layout rep/def storage).
Only **1-dimensional random access** is currently supported.

### 3.5 Blob encoding

Blob page layout stores large binary values out-of-line (`docs/src/format/file/encoding.md:351-375`).
The disk page holds a struct array of `(position, size)` descriptions; actual bytes live in
external buffers. Validity is smuggled into the description: `size==0 && position==0` =
empty; `size==0 && position!=0` = null. Recommended only when one IOP per value is justified
(values >=1MiB).

**Blob v2** (`lance.blob.v2` extension type) is the path for file format >=2.2; for >=2.2 the
legacy `lance-encoding:blob` metadata is rejected on write (`docs/src/guide/blob.md:45-52`).

**The logical Arrow schema became documented contract in v12** (#8929, `v12.0.0-beta.11`).
"A blob v2 field is tagged with `ARROW:extension:name = "lance.blob.v2"`", and writers accept
two logical struct shapes - Minimal and Complete (`blob.md:68-71`). The row-level invariants are
now spelled out: "Every non-null row must set exactly one of `data` and `uri`. For the complete
shape, `position` and `size` must either both be set or both be null, a range requires `uri`, and
an explicit range must have `size > 0`" (`blob.md:78-81`).

The guarantee that matters for round-tripping is preservation: "Lance preserves an accepted
logical shape, including child fields, nullability, and metadata, across create, append, and
merge-insert writes; descriptor scans still return the compact stored descriptor shape"
(`blob.md:83-85`). So the shape you write is the shape you read back - except through a
descriptor scan, which always yields the compact stored form.

**Four read paths** (`docs/src/guide/blob.md:6-7,177-188`). `read_blobs` is the **primary** API -
"For data loaders and batch processing that need complete byte payloads, use `read_blobs`" - it
returns `List[Tuple[int, Optional[bytes]]]` (`(row_address, payload)`) and "plans and executes
batched blob reads through Lance's scheduler." `take_blobs` returns lazy `BlobFile` handles for
streaming/seeking/partial reads (`with blob as f: f.read()`) - "Do not wrap `take_blobs` in your
own thread pool just to call `read()` ... Use `read_blobs` instead." **`read_blob_ranges`**
(v10) returns `List[Tuple[int, int, Optional[bytes]]]` for "selected byte ranges from multiple
rows without materializing complete blobs" (`blob.md:197`) and "accepts the same selector kinds
through its required `selector` argument" (`blob.md:206-207`).
`scanner(..., blob_handling="all_binary")` reads blob columns as Arrow binary columns in a scan
/ `pyarrow.Table`; `LanceTableProvider::with_blob_handling` is the DataFusion-side equivalent
(v10). The selector-taking APIs take **exactly one** of `ids` (logical row-id), `indices`
(positional within a snapshot), or `addresses` (physical, debug). A blob v2 column can mix
inline bytes, an external URI, an external URI slice (`Blob.from_uri(uri, position=, size=)`),
and null - enabling many payloads packed into one container file referenced by
`(position, size)` slices.

**Blob v2 fields nest** (v11): "Blob v2 fields can be nested inside structs and variable-length
lists. Blob-aware scans preserve the surrounding nested layout" (`docs/src/guide/blob.md:137-138`).
v11 also taught `FileFragment::update_columns` to handle blob-v2 columns via their descriptor
representation (#8344), and added `BlobFile.read_ranges(ranges) -> list[bytes]` (#8319) for
vectored reads - "The underlying physical reads may be reordered, coalesced, or split for
efficiency."

**Null selections are preserved (v10, BREAKING, PR #7903).** This is the change that triggered
the major bump. "Blob selection APIs preserve logical result cardinality. `read_blobs()` and
`take_blobs()` return one element per selected row, and `read_blob_ranges()` returns one element
per request. A null blob is returned as `None`; a valid empty blob remains a non-null empty
payload or zero-length `BlobFile`" (`docs/src/guide/blob.md:247-250`). Previously null blobs were
**omitted**, so any caller zipping results positionally against its inputs was silently
misaligned whenever a null appeared. Signature changes:

| Surface | Before | After |
|---------|--------|-------|
| Rust `take_blobs` / `_by_addresses` / `_by_indices` | `Result<Vec<BlobFile>>` | `Result<Vec<Option<BlobFile>>>` (`rust/lance/src/dataset.rs:1757,1790,1817`) |
| Rust `ReadBlob::data`, `ReadBlobRange::data` | `Bytes` | `Option<Bytes>` (`rust/lance/src/dataset/blob.rs:1580,1642`) |
| Python `take_blobs` | `List[BlobFile]` | `List[Optional[BlobFile]]` |
| Python `read_blobs` | `List[Tuple[int, bytes]]` | `List[Tuple[int, Optional[bytes]]]` |
| Java `takeBlobs` | non-null elements | "null blob values are represented by null elements" |

Related v10 blob fixes: `merge_insert` no longer crashes with a `LargeBinary vs Struct schema
mismatch` when the source omits blob columns (PR #7615); storage-2.1 compaction no longer
surfaces a surviving null as a valid zero-length descriptor (PR #8070) and no longer classifies
an inline blob with `position=0/size=0` as null (PR #7965); blob selection by stable row ID no
longer drops deleted/unknown IDs or misattributes bytes to the wrong `request_index` (PR #8003).

**Auto-tiering.** Blob v2 tiers payloads by size (`docs/src/guide/blob.md:373`): "by default it
keeps payloads under 16 KiB inline, packs mid-sized payloads into shared `.blob` sidecars, and
gives payloads over 2 MiB their own dedicated `.blob` file." The blob column avoids the
row-rewrite write amplification that inline binary incurs on compaction/update. The cutoffs are
**per-column configurable** (PR #7269): field metadata `lance-encoding:blob-inline-size-threshold`
/ `lance-encoding:blob-dedicated-size-threshold` (Python `inline_size_threshold` /
`dedicated_size_threshold`), plus `lance-encoding:blob-pack-file-size-threshold`
(`rust/lance-arrow/src/lib.rs:69`; Python `blob_pack_file_size_threshold` on `write_dataset`,
PR #7322) which caps how large a shared packed `.blob` file grows before a new one starts.
Appends that specify a different threshold than the existing column are **rejected**, not
silently ignored.

### 3.6 Exact file identity: `ConcreteFileVersion` (v10; relocated in v11)

v10 split the file-version type in two (PR #7879). `LanceFileVersion` remains the user-facing
type carrying release *selectors* - `stable`, `next` - while `ConcreteFileVersion` is "the exact
persisted identity of a Lance file format ... this type cannot represent release selectors such
as `stable` or `next`. **Exact versions deliberately have no ordering because format
capabilities are not implied by release order.**" Variants: `V1, V2_0, V2_1, V2_2, V2_3`.

**In v11 both types live in `rust/lance-file/src/version.rs`** - `LanceFileVersion` at `:25`,
`ConcreteFileVersion` at `:117`. `lance-encoding::version` was deleted outright (PR #8026) with
no re-export, on the rationale that "`Stable` and `Next` are file-writing selectors, not
encoding mechanisms". A crate that reached `LanceFileVersion` through `lance-encoding` alone no
longer compiles and must add a `lance-file` dependency. Two methods were **removed** from
`LanceFileVersion` back in v10 (BREAKING): `try_from_major_minor` and `to_numbers`; their job
moved into `ConcreteFileVersion`'s persisted codecs.

`ConcreteFileVersion` also spread outward in v11: `FileReader::version()` and
`determine_file_version` now return it, `Dataset::storage_version_or_default()` returns it, and
the whole reader/writer construction path is typed on it (2.1).

Manifest version strings reject aliases: "Public selector aliases such as `legacy`, `0.3`,
`stable`, and `next` are intentionally rejected because manifests only store canonical exact
versions" (`version.rs:139-141`).

The `DataFile` wire mapping is now a locked contract (`version.rs:72-92`). Encode is exact;
decode accepts a wider set for historical files:

| Version | Encodes to | Decodes from |
|---------|-----------|--------------|
| `V1` | `(0,2)` | `(0, 0..=2)` |
| `V2_0` | `(2,0)` | `(0,3)` or `(2,0)` |
| `V2_1` | `(2,1)` | `(2,1)` |
| `V2_2` | `(2,2)` | `(2,2)` |
| `V2_3` | `(2,3)` | `(2,3)` |

Note the dual representation of 2.0: the **standard footer** encodes it as `(0,3)` while the
**embedded / self-described footer** uses `(2,0)`, and both are now pinned by checked-in
byte-exact fixtures with SHA-256 locks - "The compatibility tests require each stable writer to
reproduce its fixture byte-for-byte and each reader to open and read the baseline file"
(`rust/lance-file/test_data/exact_versions/README.md`, PR #8019). 2.3 is excluded from the
fixture set because it is unstable. Existing wire mappings, legacy empty-manifest recovery,
reader compatibility, and mixed-version rejection are otherwise unchanged.

A related policy note now lives in the repo's `AGENTS.md`: legacy is frozen - "Implement new
features in the current format and write paths. Do not extend legacy writers, retrofit new
capabilities into legacy readers, or reuse legacy implementations as the foundation for new
code" (PR #8039).

---

## 4. Data types

Lance supports the full Apache Arrow type system; Arrow types auto-map to Lance's internal
representation (`docs/src/guide/data_types.md`).

- **Primitive** - `Boolean`; `Int8/16/32/64`; `UInt8/16/32/64`; `Float16/32/64`;
  `Decimal128`, `Decimal256`; `Date32`, `Date64`; `Time32`, `Time64`; `Timestamp` (with
  timezone); `Duration`.
- **String/binary** - `Utf8`, `LargeUtf8` (64-bit offsets), `Binary`, `LargeBinary`,
  `FixedSizeBinary(n)`.
- **Nested** - `Struct` (arbitrarily nestable; 2.1 added null support in struct fields),
  `List` / `LargeList` (variable-length), `Map(K,V)` (**requires file format 2.2+**).
- **FixedSizeList** - the recommended type for fixed-dimension vector embeddings; optimized
  for columnar storage, SIMD distance computation, and vector indexing. Best practice:
  dimensions divisible by 8.
- **JSON** (`lance.json` extension type) - stored internally as **JSONB** (binary JSON),
  read back as Arrow's JSON type. Query functions: `json_extract` (JSONPath), `json_get`
  (returns JSONB for chaining), `json_get_string/int/float/bool`, `json_exists`,
  `json_array_contains`, `json_array_length`. Indexable: a scalar index on a JSON path, or an
  inverted (FTS) index over JSON contents (`docs/src/guide/json.md`). **As of v13 only the four
  typed accessors reach a scalar index** - `json_extract` and `json_get` were de-routed in #9101
  and now fall back to a full scan with no error (section 11).

  The Rust helpers live in **`lance_arrow::json`**, not `lance::arrow::json`: the `lance` facade
  has an unrelated same-named submodule for schema serialization, so the obvious import path
  compiles as a different thing or fails to resolve. `json_field`, `encode_json` (JSON text ->
  JSONB on the write side) and `decode_json` (read side, called by `to_logical_stream`) are at
  `rust/lance-arrow/src/json.rs:79,284,290`.
  **"Filter-only" is a hard limitation, not a stylistic note**: "JSON functions are currently
  only available for filtering, not for projection in query results" (`guide/json.md:433`). You
  cannot `SELECT json_get_string(col, 'k')` - extract the whole JSON column and unpack it
  client-side, or materialize the field into its own column at write time if you need it
  projected or grouped.
- **Blob** (`lance.blob.v2` extension type) - large binary objects, lazy file-like loading
  (section 3.5). Migrating an existing dataset is a rewrite, not an alter: the guide has a
  dedicated "Rewrite to a New Blob v2 Dataset" procedure (`docs/src/guide/blob.md:377`) plus a
  troubleshooting section (`:404`). Note Blob v2 needs file format **2.2**, while the default is
  2.1 - so a dataset created without an explicit `data_storage_version` cannot take it, and the
  version is fixed at creation.
- **ML extension arrays** (`docs/src/guide/arrays.md`) - `BFloat16` (16-bit ML float,
  `lance.arrow.BFloat16Array`), `ImageURI`, `EncodedImage` (jpeg/png on disk),
  `FixedShapeImageTensor`.

format-table.md 65 KB

# Lance v12 reference - table format (sections 5-10)

Part of the Lance v13 reference (`lance-format/lance@v13.0.0-beta.4`). Citations are `path:line`
relative to the repo root; build a permalink as
`https://github.com/lance-format/lance/blob/v13.0.0-beta.4/<path>`. Line numbers drift between
tags - treat them as approximate. Cross-references written as "section N" use the original
16-section numbering; `lance-reference.md` maps every number to its file.

## Contents

- [5. Table format](#5-table-format)
  - [5.1 Dataset directory layout](#51-dataset-directory-layout)
  - [5.2 Manifest](#52-manifest)
  - [5.3 Fragments](#53-fragments)
  - [5.4 Deletion files](#54-deletion-files)
  - [5.5 Data overlay files (unstable, v9.1)](#55-data-overlay-files-unstable-v91)
- [6. Schema evolution](#6-schema-evolution)
- [7. Versioning, tags, branches](#7-versioning-tags-branches)
- [8. Row IDs and lineage](#8-row-ids-and-lineage)
- [9. Transactions and concurrency](#9-transactions-and-concurrency)
  - [9.1 Commit protocol](#91-commit-protocol)
  - [9.2 OCC retry and conflict resolution](#92-occ-retry-and-conflict-resolution)
  - [9.3 Commit handlers](#93-commit-handlers)
  - [9.4 Cache keys and backend (v10, BREAKING)](#94-cache-keys-and-backend-v10-breaking)
- [10. MemWAL](#10-memwal)

Other files: `format-file.md` (1-4), `indexes.md` (11-12), `ops.md` (13, 15, 16),
`changelog-v7-v13.md` (14).

---

## 5. Table format

A Lance **dataset** (a "table") is a directory of immutable files plus a sequence of
versioned manifests.

### 5.1 Dataset directory layout

`docs/src/format/table/layout.md:18-42`:

```
{dataset_root}/
  data/          *.lance                    -- column data files
  _versions/     *.manifest                 -- one manifest per version
                 latest_version_hint.json   -- optional latest-version hint
  _transactions/ *.txn                      -- serialized Transaction protobuf per commit
  _deletions/    *.arrow / *.bin            -- deletion vectors (Arrow IPC / roaring bitmap)
  _indices/      {UUID}/...                 -- index content, one dir per index segment
  _refs/         tags/*.json branches/*.json -- tag and branch metadata
  tree/          {branch_name}/...          -- per-branch datasets (v7)
```

All file paths inside Lance files are stored **relative to their containing directory** -
copying the dataset root relocates it with no manifest edits.

**Base paths.** The manifest's `base_paths` array defines alternative storage locations
(`docs/src/format/table/layout.md:46-79`, `protos/table.proto:211-222`). A `BasePath` has an
`id` (uint32, from 0), optional `name`, `is_dataset_root` (true = standard subdirectory
layout; false = a flat file directory), and an absolute `path`. Data files, deletion files,
and index metadata each carry an optional `base_id` referencing a base; absent means relative
to the dataset root. Use cases: hot/cold tiering, multi-region distribution, shallow clones.
Gated by feature flag `FLAG_BASE_PATHS`.

### 5.2 Manifest

A manifest describes a single immutable version of the dataset (`protos/table.proto:36-208`).
Key fields: `fields` (the schema, all fields including nested), `fragments` (the
`DataFragment` list for this version), `version` (monotonically increasing u64), `timestamp`,
`writer_version`, `index_section` (file position of index metadata), `data_format`
(`file_format` + version string - every file in a version shares one format version),
`config` and `table_metadata` (string maps; `lance.`-prefixed config keys reserved),
`base_paths`, `branch` (optional; absent = main), `next_row_id` (only with stable row IDs),
`reader_feature_flags` / `writer_feature_flags`, `transaction_file`.

Two manifest-adjacent messages worth naming explicitly. **`IndexSection`** is what
`index_section` points at - "a list of index metadata for one dataset version"
(`protos/table.proto:311-314`), so index metadata is versioned with the table rather than
living beside it. **`VersionAuxData`** (`:236-241`) attaches arbitrary key/value metadata to a
version and is explicitly "Only load on-demand", i.e. it is *not* paid for on every dataset
open the way `config` and `table_metadata` are - the right place for per-version annotations
that most readers never need.

**Manifest naming** has two schemes (`transaction.md:24-32`): **V1** = `{version}.manifest`;
**V2** = `{u64::MAX - version:020}.manifest` (20-digit, reverse-sorted, so the latest version
sorts first lexicographically - enables O(1) latest-version discovery on ordered stores).

### 5.3 Fragments

A `DataFragment` is a horizontal partition holding a subset of rows
(`protos/table.proto:308-349`): `id` (unique, incrementally assigned), `files` (one or more
`DataFile`s, each storing a subset of columns), an optional `deletion_file` (at most one per
fragment per version), `physical_rows` (total including tombstoned rows - live count =
`physical_rows - deletion_file.num_deleted_rows`), and optional inline-or-external row-id /
version sequences.

A `DataFile` stores a subset of columns in the Lance file format, with `fields` (the field
IDs it contains; `-2` = tombstoned), `column_indices`, file major/minor version, and optional
`base_id`. **A field with no backing data file reads as entirely NULL** - this is the
mechanism behind zero-copy schema evolution.

**Fragment ids are a dataset-lifetime high-water mark (v11, BREAKING, PR #8206).** Previously
an overwrite restarted ids at 0; now the first fragment an overwrite writes takes the next id
after the dataset's highest ever used, because "an id must never name two different sets of
rows, or per-fragment state keyed by id (caches, deletion files, row addresses) can be
attributed to the wrong rows" (`rust/lance-table/src/transaction/manifest_build.rs:538`). Three
consequences:

- Code that assumes a known id after an overwrite (`dataset.get_fragment(0)`) breaks - read ids
  from the manifest instead.
- An overwrite fragment carrying a deletion file is now **rejected**: "Overwrite fragments must
  be newly written, but fragment {} carries deletion file {}. Use Delete to commit deletions
  against existing fragments, or Merge to change their schema"
  (`rust/lance-table/src/transaction/validate.rs:109`). The reason is structural - a deletion file
  cannot follow its fragment to a new id, because its path embeds the old one
  (`_deletions/{fragment_id}-{read_version}-{id}`).
- **Any** commit producing duplicate ids is rejected (`check_fragment_ids`,
  `rust/lance/src/io/commit.rs:687`). Datasets written by Lance 0.16 and earlier could contain
  duplicates: they still read, but "can no longer be committed to" - they must be rewritten or
  rolled back to a version without the duplicate.

`ManifestNamespace::manifest_from_overwrite_transaction` is carved out and still restarts at 0.

#### Legacy manifests and fragment resolution (read this before trusting a fragment subset)

Two manifest shapes predate the current invariants and are **still readable**: fragments not in
id order (Lance 0.10 and earlier) and duplicate fragment ids (Lance 0.16 and earlier). Neither
is rejected on read. `find_fragment` handles both - it binary-searches, then checks the result
and falls back to a linear scan, because "returning some other fragment's data would be silent
corruption" (`rust/lance/src/dataset.rs:2871-2888`).

**Its sibling `Dataset::get_frags_from_ordered_ids` does not have that guard**
(`rust/lance/src/dataset.rs:2850-2867`). It resolves each id as
`manifest.fragments[fragment_bitmap.rank(id) - 1]`. `fragment_bitmap` is built by collecting
fragment ids into a `RoaringBitmap` (`:885`), which is inherently sorted, so `rank(id) - 1` is
the id's index in the **sorted** id set - while the subscript indexes `manifest.fragments` in
**stored** order. The two agree only if the stored vector is sorted by id. On an unsorted legacy
manifest they diverge, and the only guard is a `debug_assert_eq!`, which compiles out in release.
Concretely, a manifest storing `[id=7, id=3, id=5]` resolves 7 -> fragment 5, 3 -> fragment 7,
and 5 -> fragment 3.

Which legacy shape can actually reach this differs:

- **Duplicate ids** - largely self-limiting, since `check_fragment_ids` makes the dataset
  uncommittable, so index-building paths fail loudly first. Note that check scans
  `manifest.fragments.windows(2)` for adjacent equal ids, so it detects duplicates **only when
  the manifest is sorted**; non-adjacent duplicates on an unsorted manifest pass it.
- **Unsorted fragments, no duplicates** - passes every commit-time check and is the shape that
  breaks the rank arithmetic.

Consequences differ by caller, and the difference matters:

- `dataset/take.rs:305` re-looks-up offsets by the returned fragment's own `id()`, so a
  misresolution drops slots rather than mis-attributing values - the failure mode is **silently
  missing rows**, not wrong values.
- `index/scalar.rs:233` zips the requested ids against the resolved fragments and takes
  `frag.metadata()` with **no id re-check**, so a scalar index built over a fragment subset would
  train on the wrong fragments while recording the requested ids as its coverage.
- `index/create.rs:937` discards the result (existence check only) and is unaffected.

**Status: a static finding, not a demonstrated defect.** The mechanism and the call sites above
were read at `v11.0.0-beta.16`; no legacy dataset was constructed to drive the path, and it is
not a reported upstream issue. The upstream test that looks like it covers this,
`test_get_frags_from_ordered_ids_accepts_unsorted_duplicates` (`rust/lance/src/index/create.rs:1348`),
writes a **fresh** dataset and varies only the *query array* order - which is the documented
flexibility ("The ids do not need to be sorted or deduplicated"); the manifest-ordering
assumption is untested. Practical guidance: if you operate datasets written by Lance 0.10 or
earlier, rewrite them before building indexes over fragment subsets, and do not assume a
fragment-filtered index covers the fragments you named.

**Do not key application state on fragment ids.** Independently of the above, any operation
that rewrites a fragment mints new ids, so caches, delta detectors, or coverage checks that
compare "are the base version's fragment ids still a subset of current?" will spuriously
invalidate. Note this is about fragment *identity*, not write volume: a matched `merge_insert`
updating one column writes a new per-column data file plus a deletion vector rather than
rewriting the whole fragment, so the physical cost is small even though the id churns.

### 5.4 Deletion files

Deletes are **soft**: a deletion file (deletion vector) marks deleted row offsets without
rewriting data files; at most one per fragment per version (`docs/src/format/table/index.md:149-160`).
Two formats: **Arrow IPC** (`.arrow`, a flat `Int32Array` of offsets - sparse deletions) and
**roaring bitmap** (`.bin` - dense deletions). Offsets are 0-based within the fragment. Path:
`_deletions/{fragment_id}-{read_version}-{id}.{ext}`. Gated by `FLAG_DELETION_FILES`. Deletes
avoid invalidating indexes; accumulating deletions slow scans until compacted.

### 5.5 Data overlay files (unstable, v9.1)

Overlay files supply **new values for a subset of `(row offset, field)` cells within a
fragment without rewriting the fragment's base data files** (`docs/src/format/table/data_overlay_file.md`)
- the cheap-cell-update counterpart to soft-delete's cheap-row-removal. Written and committed
via the new `DataOverlay` transaction op (section 9.1); a reader resolves each cell by taking
the highest-`committed_version` overlay that covers it, falling back to the base data file.
Overlays interact with indexes: for each overlay whose `committed_version` exceeds an index
segment's `dataset_version`, the covered rows are excluded from index results (they carry
updated values the index has not seen).

Gated by **feature flag 64** (`FLAG_UNSTABLE_DATA_OVERLAY_FILES`,
`rust/lance-table/src/feature_flags.rs:32`). This is **not a released feature**: writes require
`LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES` (`feature_flags.rs:38`), and in release builds the
flag is treated as unknown so a release reader/writer **refuses** an overlay dataset rather than
silently ignoring an overlay (`feature_flags.rs:109` - the gate is
`cfg!(debug_assertions) || env::var_os(...).is_some()`). Compaction can fold fragments over an
overlay-count limit (PR #7772).

**Two overlay shapes.** "A single overlay is one of two shapes"
(`docs/src/format/table/data_overlay_file.md:72-74`): a **dense** overlay replaces a contiguous
run of row offsets, while a **sparse** overlay addresses scattered offsets and therefore carries
its own offset mapping. Which shape a writer emits determines how cheaply a reader can skip the
overlay for a given row range. The spec also has a dedicated "Scheduling compaction" section
(`:372`) covering when accumulated overlays should be folded back into base files.

**Writer support is still incomplete upstream**, and the spec says so in-line: "TODO: Fill in as
writer implementation progresses, including the status of single-file sparse overlays
(independent-length columns)" (`data_overlay_file.md:361-362`). Treat the read path as the
better-specified half.

**Overlays vs indexes (v10).** A batch of correctness work made index-served queries
overlay-aware, since an overlay can change a cell the index was built against. Index results now
exclude overlay-superseded rows: "`WHERE age = 25` after an overlay sets a row's age to 26 must
not return that row from the index; `WHERE age = 26` must find it" (PR #7549). The machinery is
a new module `rust/lance/src/dataset/overlay.rs` (`overlay_exclusion_offsets`,
`overlaid_fragments`, `collect_overlay_stale_frags`, `collect_overlay_stale_rows_for_segment`)
plus `with_overlay_block(RowAddrMask)` builders on `DatasetPreFilter`, `FilteredReadOptions`,
`MaterializeIndexExec`, and `ANNIvfSubIndexExec`. Query plans change shape only when stale
overlays exist - BTree and ANN gain a targeted `TakeExec` re-evaluation, FTS drops whole stale
segments to `FlatMatchQueryExec`; with no overlays it is "O(num_fragments) boolean check, zero
allocations". Two follow-up fixes: a `RewriteRows` UPDATE touching only a non-indexed column no
longer drops overlaid rows from index-path results (PR #7926), and with stable row IDs a
fragment carrying both a deletion and an overlay no longer masks the wrong row and leaks the
stale indexed value - unmapped offsets are now a hard error (PR #7918).

---

## 6. Schema evolution

Every field, including nested fields, has a unique integer **field ID**, assigned in
depth-first order from 0 at table creation; new fields get the next available ID
(`docs/src/format/table/schema.md:195-236`). Field IDs are immutable, unique, stable across
evolution, and sparse. Internal references always use field IDs, never names or positions.
Nested fields link via `parent_id` (`-1` for top-level).

Schema changes are **metadata-only** wherever possible (`docs/src/guide/data_evolution.md`):

**Per-fragment column writes (v11, #8313; renamed to `write_columns` by #8622).**
`FileFragment::write_columns` writes new column data for a *single* fragment and the result
survives compaction, so a distributed backfill can have each worker materialize its own
fragment's columns independently instead of funneling through one whole-dataset pass. The
API is new enough that it had not appeared in a release tag when it was renamed - pin exactly
if you build on it.

- **Add column** - assign a new field ID, update the schema. Schema-only add is very fast.
  File format <=2.1 cannot add sub-columns under an existing struct; 2.2 can extend nested
  struct fields (including structs nested in lists). Since v9, adding an **all-null `Map`
  column** is allowed (PR #7462; previously rejected because Arrow's non-null `entries`/`key`
  child failed the nullable check).
- **Drop column** - remove the field from the schema; metadata-only, does not delete data on
  disk; reversible while old versions are retained. Physical removal happens only after
  compaction + version cleanup. 2.2 supports nested sub-column removal.
- **Rename / reorder** - change `name` or order; field IDs unchanged.
- **Type change / cast** - may require rewriting that column to new data files (other columns
  untouched). Since v9, `alter_columns` **fails fast** if the column has an index attached -
  it no longer silently drops/invalidates the index; you must `drop_index()` first (PR #7158,
  breaking: `Error::invalid_input("Cannot cast column(s) [...]: they have N index(es) attached
  ... Drop the index(es) with drop_index() before altering")`). v9 also **allows Dict <->
  value-type casts** via `alter_columns` (PR #7289).

**Zero-copy data evolution.** Because each data file holds a distinct set of field IDs and a
missing field reads as NULL, a writer can add and backfill a column by **appending new data
files to existing fragments** with computed values - no full table rewrite. This is the
mechanism for ML feature engineering and adding embeddings to an existing dataset. When a
column is rewritten, the old data file's field ID becomes the tombstone `-2` and a new data
file is appended.

Lance also supports an **unenforced primary key** and **clustering key**, declared via field
metadata (`lance-schema:unenforced-primary-key`). "Unenforced" - Lance does not always
validate uniqueness; it is used for merge-insert dedup and last-write-wins. PK fields must be
non-nullable leaf primitives; clustering-key fields may be nullable.

**Merge-insert (upsert / find-or-create).** `MergeInsertBuilder` defaults to find-or-create
semantics ("By default this will build a job that has the same semantics as find-or-create",
`rust/lance/src/dataset/write/merge_insert.rs:418`); enable `when_matched(WhenMatched::UpdateAll)`
for upsert - note `UpdateAll` rewrites whole fragments. The default behavior for **duplicate
source rows that match the same target** is to **fail the operation**
(`SourceDedupeBehavior::Fail`, `merge_insert.rs:322,472`); opt into `SourceDedupeBehavior::FirstSeen`
to keep the first and skip later duplicates. Empty `on` keys fall back to the schema's
unenforced primary key.

**That failure is deterministic, and it is not a conflict - do not let an OCC retry loop eat
it.** The error reads "Ambiguous merge inserts are prohibited: multiple source rows match the
same target row on ({})" (`merge_insert.rs:292`) and is raised as `Invalid user input`, i.e. it
describes the *source batch*, not the state of the table. Retrying it re-runs the same
comparison against the same input and fails identically every time, so a generic
"retry the Lance operation on error" wrapper turns one clear error into N attempts and a
misleading exhausted-retries message. Match on the error class at the Lance boundary and retry
only genuine commit conflicts; fix ambiguous input by deduping the source or selecting
`SourceDedupeBehavior::FirstSeen`.

**The third clause: `when_not_matched_by_source_*`.** Beyond "row in both" (`when_matched`) and
"row only in source" (`when_not_matched`), merge-insert can act on **target rows the source did
not mention**. `when_not_matched_by_source_delete()` removes them, and the predicate form
`when_not_matched_by_source_delete("age >= 40")` (`docs/src/guide/read_and_write.md:266`) deletes
only those matching a condition. This is what makes the **"replace a portion of data"** pattern
work (`read_and_write.md:241`): scope a merge to a slice of the table by combining an unmatched
source with a predicate, so one atomic commit replaces exactly that slice rather than requiring
a separate delete-then-append. Without it, "sync this partition to look like my source" is two
transactions and a window where the table is inconsistent.

---

## 7. Versioning, tags, branches

Every commit creates a new immutable version with a monotonically increasing `version`
number; all versions form a serializable history enabling time travel
(`docs/src/format/table/transaction.md:5-7`). Writes (append, overwrite, index ops,
compaction) create versions; **creating or deleting tags or branches does not**. Time travel
is `checkout_version` by version number, tag name, or `(branch, version)` tuple.

**Listing versions cheaply (v11, #8523).** `versions()` reads and deserializes *every* manifest,
which is why it is expensive on a long-lived dataset over object storage. `version_refs()`
returns `VersionRef`s by listing manifest locations only - use it whenever you need version
numbers rather than full metadata (`rust/lance/src/dataset.rs:259,2644`; Python
`dataset.version_refs()`). When only the current branch tip matters, `latest_version` is
cheaper still. `get_fragment` also became a binary search over the manifest at `beta.16`
(#8636), with a check-and-fall-back-to-linear-scan guard for legacy manifests whose fragment
lists are unsorted (Lance <= 0.10) or contain duplicate ids (Lance <= 0.16).

### Feature flags

The manifest carries `reader_feature_flags` and `writer_feature_flags` bitmaps; an
implementation seeing an unknown flag must return "unsupported" (`docs/src/format/table/versioning.md`):

| Bit | Flag | Meaning |
|-----|------|---------|
| 1 | `FLAG_DELETION_FILES` | Fragments may carry deletion files |
| 2 | `FLAG_STABLE_ROW_IDS` | Stable row IDs; fragments carry a row-id-to-address index |
| 4 | `FLAG_USE_V2_FORMAT_DEPRECATED` | Deprecated, unused |
| 8 | `FLAG_TABLE_CONFIG` | Table config present in the manifest |
| 16 | `FLAG_BASE_PATHS` | Dataset uses multiple base paths |
| 32 | `FLAG_DISABLE_TRANSACTION_FILE` | Transaction recorded in the manifest, not a separate `.txn` file (writer-only) |
| 64 | `FLAG_UNSTABLE_DATA_OVERLAY_FILES` | Fragments may carry data overlay files; **unstable** - release builds reject it unless explicitly opted in |
| 128 | `FLAG_COVERED_INDEX_METADATA` | Some index declares covering columns (`IndexMetadata.covering_fields`); `fields` means keyed columns plus carried ones (v11 final; `covering_fields` redefined at v13, see section 11) |
| 256 | `FLAG_MIXED_DATA_FILE_VERSIONS` | **Supported** as of the `v12.0.0` final - the snapshot may reference recognized V2 data files at different exact versions; both reader and writer bits must be set and stay set |
| 1024 | `FLAG_FRAGMENT_REUSE_INDEX` | **Documented, not implemented** - the spec page lists it reader/writer `Yes`, the code declares it above `FLAG_UNKNOWN` and never reads it, so this build refuses it (see below) |

**Flags at or above 512 are unknown** and must be rejected as "unsupported" - the boundary moved
from 128 in v11 when bit 128 was allocated, and again to `1 << 9` in the `v12.0.0` final when bit
256 was spent. Bits 32 and 64 existed in Rust before v11 but
were undocumented until the v11 docs catch-up. At v12 the constants are written as bit shifts
(`1 << 7`) rather than decimals; the serialized values and compatibility behavior are unchanged.

**Bit 128 changed hands mid-v11 - this is the trap.** v11 first allocated it as
`FLAG_MEM_WAL_INDEX_CATCHUP` (#8263), then **retired that flag** (#8680) and handed the
reclaimed bit to `FLAG_COVERED_INDEX_METADATA` (#8535) before `v11.0.0` shipped. The
index-catchup flag does not exist at the final or at v12, and the proto field
`Transaction.UpdateMemWalState.require_index_catchup` was deleted from the wire with it.

A reader without bit 128 "would answer a query on a merely-carried column with an index keyed on
a different column and return wrong neighbours with no error"; a writer without it "would
maintain the index as though every entry of `fields` were keyed". **Both must refuse the table.**

**The reclamation has a named exposure window.** Upstream chose the bit precisely because the
current released build already treats it as unknown - but "builds from the window where the bit
was allocated to index catch-up (v11.0.0-beta.4 through beta.17) still count it as supported and
will open a covering dataset rather than refuse it; that exposure comes with the reclamation and
is inherited by whichever flag takes the bit." Do not run a pin inside that window against data
a newer writer may touch.

**MemWAL catch-up is no longer flag-gated.** With the flag gone, the semantics are
unconditional: "A shard absent from `index_catchup` for an index means that index is *not* known
to have caught up, so the shard's SSTables must be retained until some commit records that it
has." There is no longer a legacy "absence means fully caught up" reading and no one-way flag to
set - an absent shard simply means *unknown*, and a repair is scheduled.

**Bit 256 was spent in the `v12.0.0` final.** `FLAG_MIXED_DATA_FILE_VERSIONS` is still declared
`1 << 8`, but the compile-time assert relaxed from `== FLAG_UNKNOWN` to `< FLAG_UNKNOWN`, and
`FLAG_UNKNOWN` moved `1 << 8` -> `1 << 9`. Tests now assert both
`can_read_dataset(FLAG_MIXED_DATA_FILE_VERSIONS)` and `can_write_dataset(...)`, so this layer
accepts mixed manifests. The follow-ups landed with it: #8581 (validation), #8582 (per-operation
V2 write targets), #8583 (propagation across dataset operations) and #8584 (compaction
targeting) are in `v12.0.0`; #8585 exposed it in the bindings in the v13 line.

`STICKY_PAIRED_FLAGS` survives and is still exactly this bit. A **half-set** manifest is now a
hard error rather than an ambiguity: "Manifest has only one of the mixed data-file-version reader
and writer feature bits set, so its semantics are undefined." The flags also do not come back off
- they "remain set even if compaction later makes the files homogeneous again."

**Bit 1024 is documented but not implemented - trust the code.** `FLAG_FRAGMENT_REUSE_INDEX` is
declared `1 << 10` at `rust/lance-table/src/feature_flags.rs:69`, and at `v13.0.0-beta.4` that
declaration is its **only occurrence in the tree**. Because `supported_flags()` is computed as
`FLAG_UNKNOWN - 1` and `FLAG_UNKNOWN` is `1 << 9`, bit 1024 is outside the supported set and a
manifest setting it is refused. The spec page says the opposite - reader `Yes`, writer `Yes`,
unknown starting at 2048 - and also states "Flag bit 512 is reserved", which the code does not
reflect either. The docs describe where tagged FRI is going; the constants describe what this
build does.

### Tags

A tag labels a specific version. Stored as JSON under `_refs/tags/`, always at the root
regardless of branch. Tag JSON: `branch` (optional; absent = main), `version`, `createdAt` /
`updatedAt` (RFC 3339), `manifestSize`, `metadata`. **Tagged versions are exempt from
`cleanup_old_versions()`** - to remove a tagged version you must delete the tag first
(`docs/src/guide/tags_and_branches.md:59-65`). Tag names: alphanumeric, `.`, `-`, `_`; no
`/`.

### Branches (v7)

Branches are Git-like parallel histories (`docs/src/format/table/branch_tag.md`). A branch
dataset is technically a **shallow clone** of its source, with version-specific files under
`tree/{branch_name}/` carrying their own `_versions/`, `_transactions/`, `_deletions/`,
`_indices/`. Branch metadata is JSON at `_refs/branches/{name}.json` (`/` URL-encoded as
`%2F`): `parentBranch`, `parentVersion`, `createAt`, `manifestSize`, `metadata`. Each branch
has its **own linear version history** - version numbers can overlap across branches, so use
`(branch_name, version)` tuples as global identifiers. `main` is the reserved default branch.
Branches hold references to data files - cleanup will not delete files still referenced by a
branch, so unused branches must be deleted to reclaim space.

`cleanup_old_versions(policy)` deletes old manifests, unreferenced data/deletion/index files.
A file referenced by no manifest is deleted only if >=7 days old unless `delete_unverified`
is set. `CleanupPolicy` knobs: `before_timestamp`, `before_version`, `delete_unverified`,
`error_if_tagged_old_versions` (default true), `clean_referenced_branches`,
`delete_rate_limit` (max delete requests/sec, to avoid S3 throttling). A newer
`Dataset::cleanup(policy)` API (new in v8, PR #7147) splits this into `explain()` (returns a
`CleanupExplanation` of what would be removed - a dry run) and `execute()`; v9 exposes both to
**Python and Java** (PR #7248).

---

## 8. Row IDs and lineage

A row has two identifier forms (`docs/src/format/table/row_id_lineage.md`):

- **Row address** - the current physical location. A 64-bit value:
  `row_address = (fragment_id << 32) | local_row_offset`. Exposed as `_rowaddr`. Changes when
  data is reorganized by compaction or updates. Secondary indexes currently reference rows by
  row address.
- **Row ID** - a logical identifier. With **stable row IDs disabled (the default), the row ID
  equals the row address.** With stable row IDs enabled, each row gets a unique
  auto-incrementing u64 (exposed as `_rowid`) that stays constant for the row's lifetime even
  as physical location changes.

**Stable row IDs** are normally enabled at dataset creation (manifest flag bit 2). Since
`v11.0.0-beta.15` (#8521) an existing dataset can also be migrated in place with
`Dataset::migrate_to_stable_row_ids`, which supersedes the older "cannot be turned on later"
rule. It is one `Merge` commit that assigns row-id sequences to every fragment and flips the
feature flag atomically; because `Merge` conflicts with all data-modifying operations, "a
successful commit guarantees no concurrent write occurred". Two operational catches:
**no retries are attempted** (`with_max_retries(0)`), so quiesce concurrent writers first and
retry yourself on conflict; and it is idempotent, returning `Ok(())` immediately if the table
already uses stable row IDs (`rust/lance/src/dataset.rs:3216`).

Assignment uses a monotonic `next_row_id` counter in the manifest; on a
commit conflict the writer rebases by re-reading the latest counter. On update, Lance writes a
new physical row, keeps the same `_rowid`, marks the old physical row deleted, and the row-id
index maps `_rowid -> (new fragment, new offset)`.

Row-id sequences are stored per fragment as a `RowIdSequence` protobuf (`protos/rowids.proto`)
with five compact segment encodings (Range, RangeWithHoles, RangeWithBitmap, SortedArray,
Array) - bitpacked. The wire format defines both inline and external alternatives, but upstream
now explicitly disclaims a size rule: "These fields do not currently imply a size-based switching
threshold. Current Lance writers store all three sequence types inline in the fragment metadata
regardless of their encoded size and do not emit the external alternatives." Readers *can* load
an externally stored row-id sequence; they **cannot** load external created-at or
last-updated-at version sequences, "an implementation limitation, not an invalid encoding". A
row-id index is built at table load by aggregating all fragments' sequences.

**Change data feed** (stable row IDs only): each row tracks `created_at_version` and
`last_updated_at_version`, queryable via SQL predicates on `_row_created_at_version` and
`_row_last_updated_at_version` to find rows inserted or updated between two versions.

### Stable row IDs in hand-assembled transactions

If you build fragments yourself and commit them (distributed write, Ray/Spark workers), stable
row IDs stop being automatic and become **your** responsibility. Three rules, all of them
silent-failure modes if broken (`docs/src/guide/distributed_write.md`):

- **Populate `row_id_meta` on every fragment you write.** A fragment written without it commits
  successfully while **silently giving every rewritten row a fresh identity**, breaking `_rowid`
  for anything downstream that depends on it. There is no error.
- **Never mint row ids yourself.** Ids come from a counter in the manifest, and a commit that
  loses a race is retried against the version that won - which may have consumed the very ids
  you picked. The commit assigns them after conflict resolution.
- **Leave `created_at_version_meta` and `last_updated_at_version_meta` as `None`.** Lance
  derives both at manifest-build time; supplying them is not needed.

Build the metadata with `lance.fragment.RowIdSequence` (v11, #8356). Duplicate ids are now
rejected outright - previously `[1, 1, 2]` silently encoded to `[1]`, because the segment
encodings represent a sorted run as a range plus its holes, so a repeated value became a shorter
sequence with a spurious hole.

---

## 9. Transactions and concurrency

Lance uses **MVCC**: each commit creates a new immutable version; concurrency is **optimistic
with automatic conflict resolution** (`docs/src/format/table/transaction.md`).

### 9.1 Commit protocol

A transaction commits by writing the next manifest file, which must be written exactly once
even under concurrent writers. This relies on atomic object-store primitives -
**rename-if-not-exists** or **put-if-not-exists** (conditional PUT). A `Transaction` protobuf
is written to `_transactions/{read_version}-{uuid}.txn` first, then the manifest. A
conflict-free commit is 1 read IOP + 2 write IOPs.

The `Transaction` message carries `read_version`, `uuid`, optional `tag`, a
`transaction_properties` string map, and a `oneof operation` - **16 operation types**
(`protos/transaction.proto`):

`Append`, `Delete`, `Overwrite`, `CreateIndex`, `Rewrite`, `DataReplacement`, `DataOverlay`,
`Merge`, `Restore`, `ReserveFragments`, `Update`, `Project`, `UpdateConfig`,
`UpdateMemWalState`, `Clone`, `UpdateBases`.

`DataOverlay` arrived in the 9.1 dev line (PR #7535/#7536): it attaches overlay files supplying
new values for a subset of `(row offset, field)` cells without rewriting a fragment's base data
files (section 5). It is **unstable** - env-gated by `LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES`,
and release builds refuse overlay datasets (feature flag 64 treated as unknown). No transaction
op was added in the v10 or v11 ranges; the count stands at 16, `protos/transaction.proto` is
byte-identical between `v10.0.0-beta.7` and `v11.0.0-beta.2`, and the `oneof operation` tag
range still ends at `DataOverlay data_overlay = 115` (`:371`). Do **not** count `RewriteRows` /
`RewriteColumns` toward the total - those are variants of `UpdateMode`, a different enum living
in the same Rust file.

**Large transactions spill out of the manifest (v11, PR #7881).** Transactions whose serialized
size exceeds `MAX_INLINE_TRANSACTION_BYTES` are no longer inlined into the manifest and live
only in their external `_transactions/` file; readers use the existing fallback path. The
shipped threshold is **20 MiB** - `rust/lance/src/io/commit.rs:338` under `#[cfg(not(test))]`;
the 64 KiB figure quoted in the PR description is the `#[cfg(test)]` value only. Applied in all
three commit paths (`:357`, `:1105`, `:1433`). There is no new configuration knob, the
transaction file is retained as long as the manifest referencing it, and the measured effect on
a large workload was a full-commit manifest shrinking from 1576 MiB to ~790 MiB.

**Proto renames (v10, BREAKING for proto consumers).** The MemWAL vocabulary change (section 10)
touched `protos/table.proto` and `protos/transaction.proto`: message `FlushedGeneration` ->
`SsTable`, `MergedGeneration` -> `CompactedSsTable`; `ShardManifest.flushed_generations` ->
`sstables` (tag 8), `MemWalIndexDetails.merged_generations` -> `compacted_sstables` (tag 9),
`Transaction.Merge` field 5 and `Transaction.UpdateMemWalState` field 1 likewise renamed, and
`IndexCatchupProgress.caught_up_generations` keeps its name but changes element type. **No
message or field was deleted and no tag number was reused or renumbered** - "Proto field numbers
are unchanged (wire-compatible), and `ShardManifest` persists as protobuf, so there is no
on-disk change" (PR #7943). The break is at the generated-symbol level: anything compiling these
protos must regenerate. Separately, `cache_key_prefix = 8` was removed and reserved in the
non-`protos/` file `rust/lance-index/protos-cache/cache.proto` (PR #7878).

Notable semantics: `Rewrite` reorganizes data without semantic change (compaction) and
changes row addresses; `Merge` adds columns and is "overly general" / high-conflict (prefer
`Rewrite`/`DataReplacement`/`Append`); `Update` has two modes - `REWRITE_ROWS` (optimal when
few rows change) and `REWRITE_COLUMNS` (optimal when few columns change across many rows);
`Clone` (shallow = metadata-only referencing the source via `base_paths`, or deep = native
object-store copy) can only be the first operation in a dataset so it never conflicts.

### 9.2 OCC retry and conflict resolution

`commit_transaction` computes `target_version = read_version + 1`, then runs a retry loop
(`rust/lance/src/io/commit.rs`). Each attempt loads concurrent transactions since
`read_version`, builds a `TransactionRebase`, and produces a rebased transaction. The retry
budget is `CommitConfig.num_retries`, **default 20** (settable via
`CommitBuilder::with_max_retries`). Backoff is slot-based, seeded from the first attempt's
observed commit latency. `num_retries == 0` triggers strict-overwrite mode (an `Overwrite`
not subject to any rebasing).

Three conflict outcomes:

- **Rebasable** - the transaction is transformed to incorporate the concurrent change while
  preserving intent, then retried automatically inside the commit layer.
- **Retryable** - cannot rebase but can be re-executed at the application level against the
  new version; returns a retryable conflict error.
- **Incompatible** - a fundamental conflict; the commit fails non-retryably.

Compatibility is per-operation and not bidirectional. Examples: `Append` is compatible with
almost everything including itself (conflicts only with `Overwrite`/`Restore`/
`UpdateMemWalState`); `Rewrite` is incompatible with `CreateIndex` by default because it
changes row addresses - **unless a fragment reuse index or stable row IDs are in use**, which
decouple logical identity from physical address and let those operations proceed without
conflict.

**`preserves_nullability` on `Project` / `Merge`** (v11, #8347, `protos/transaction.proto:152,
164`). The default `false` means "this operation makes no nullability assertion". A nullability
*tightening* must **not** set it: the producer proved the claim by scanning at its read version,
so a concurrent write can falsify it - which is why such a projection now **conflicts with any
value-write in either commit order**. The hole this closed: `alter_columns` proved NOT NULL by
scanning, then committed a `Project` that conflicted with nothing, so a write racing the scan
could land nulls that then fail to read under the tightened schema.

**Dataset *creation* is not covered by any of this.** OCC protects *commits*; the create path
has no retry loop at all - `do_commit_new_dataset` carries an in-repo
`// TODO: Allow Append or Overwrite mode to retry using` comment
(`rust/lance/src/io/commit.rs:483`) and returns `DatasetAlreadyExists` on collision (`:531`).
The retry classifier matches exactly one arm, `Error::RetryableCommitConflict`
(`rust/lance/src/dataset/write/retry.rs:73`), returning every other error immediately (`:98`);
and `execute_with_retry` takes an `Arc<Dataset>`, so it structurally cannot cover creation.
Upstream's own concurrency test creates the dataset first and *then* races appends. Racing
`create` from multiple processes is an application-level problem: create once, or treat
`DatasetAlreadyExists` as "someone else won" and re-open.

### 9.3 Commit handlers

The commit strategy is pluggable via the `CommitHandler` trait. Routing by URI scheme
(`rust/lance-table/src/io/commit.rs`):

| Scheme | Handler |
|--------|---------|
| `file` (non-Windows), `s3`, `gs`, `az`, `abfss`, `oss`, `tos`, `memory`, `shared-memory`, `goosefs` | `ConditionalPutCommitHandler` (`rust/lance-table/src/io/commit.rs:1115-1116`) |
| `cos` (Tencent) | `TencentCosCommitHandler` - **fails closed**, see below (v11, #8369) |
| `file` (Windows) | `RenameCommitHandler` |
| `s3+ddb` | `ExternalManifestCommitHandler` (DynamoDB; requires the `dynamodb` feature) |
| anything else | `UnsafeCommitHandler` (no concurrency check; logs a warning) |

`goosefs` joined that list in v11 (PR #8134); `abfss`, `tos`, and `shared-memory` were already
routed there before v10 despite earlier editions of this table omitting them.

**Tencent COS is the one store that cannot self-coordinate** (v11, #8369). COS silently ignores
put-if-not-exists on buckets that have *ever* had versioning enabled - even if versioning is now
suspended - so under the old `ConditionalPutCommitHandler` routing two concurrent writers could
both report success while one manifest overwrote the other. `cos://` now fails closed: a write
**requires** a custom distributed `commit_lock` or a custom `CommitHandler`, and errors without
one. If you have been writing to COS on an older Lance, treat past concurrent commits as
suspect.

`ConditionalPutCommitHandler` is the current default for nearly all stores. It uses the
object store's native conditional write (`PutMode::Create`, i.e. `If-None-Match: *`):

- **Plain `s3://`** works for safe concurrent writes - S3 supports conditional PUT natively,
  so no external lock is needed.
- **S3 Express** (directory buckets) - auto-detected; routed the same way.
- **GCS / Azure** - native atomic writes.
- **`s3+ddb://`** remains available for environments where conditional writes are
  unavailable: a DynamoDB table coordinates commits via conditional writes
  (`?ddbTableName=...`). The `ExternalManifestStore` remembers `(uri, version) -> manifest
  path` and stages then finalizes the manifest in the object store.

Note: the `commit.rs` module doc still says the S3 default is `UnsafeCommitHandler` - that
comment is stale; the actual routing sends `s3://` to `ConditionalPutCommitHandler`.

**v10 commit-path changes.** `CommitBuilder::with_source_store` (`rust/lance/src/dataset/write/
commit.rs:115`, PR #7545) enables cross-store and cross-account `deep_clone`: per-file copy
streams source -> target when the stores differ and keeps the server-side `ObjectStore::copy`
fast path when `store_prefix` matches. The retry backoff gained a cap - `slot_i * unit` could
overflow `u32` and panic in debug or wrap into a tiny sleep in release, so `MAX_SLOTS = 128`
now bounds it (`rust/lance-core/src/utils/backoff.rs:87`, PR #7883); attempts 0-4 are
unchanged, and because "the cap is proportional to `unit`, not absolute, a slow first attempt
can still produce a multi-minute single sleep". External-manifest finalization now always HEADs
the destination after copy: it previously reused staging object metadata for manifests under
5 MiB, which could reject valid tables as corrupt (PR #7964). **Superseded at v11.0.0-beta.8 by
PR #8499** - the HEAD still happens, but its ETag is now returned to the caller as an opaque
physical-generation observation and deliberately *not* persisted; see the protocol change
below. Two correctness fixes:
`Dataset::filter_deleted_ids` returned wrong results on stable-row-id datasets, breaking
`optimize_indices` with `batch.num_rows() != chunk.len()` (PR #7704), and `filter_addr_or_ids`
now errors on mismatched input lengths instead of silently truncating.

**v11 (beta.8): object storage became authoritative for external manifest stores (PR #8499).**
The external store is now described as "the concurrency coordinator and fast version index",
not the commit itself. The protocol re-labels its steps:

1. Stage the manifest under `{dataset}/_versions/{version}.manifest-{uuid}`.
2. **Reserve** the version in the external store with put-if-not-exists. This "selects one
   immutable staging object; it is not yet the canonical commit" - the previous wording called
   this step the commit.
3. Copy staging -> `{dataset}/_versions/{version}.manifest`. "Successful materialization at
   this deterministic path is the commit point."
4. Update the external-store pointer to the finalized path.

The load-bearing rule is about the ETag: **"Do not persist that ETag in the external store."**
Concurrent finalizers can copy the *same* selected immutable bytes into different physical
generations, and COPY plus external-store publication is not atomic, so a retained ETag makes
later readers reject a perfectly good canonical manifest with `Manifest e_tag mismatch` - a
dataset that reads as broken while its bytes are fine. Every helper therefore publishes the
same stable path-and-size tuple, and readers "ignore any legacy stored ETag because it is
neither content identity nor dataset-incarnation identity", validating **size** instead. The
HEAD's ETag is still handed to the current caller, purely so runtime caches do not collapse a
newly committed `Dataset` into an older cached one at the same URI and version.

Fault tolerance was re-cut along the same seam: a failure between steps 2 and 3 leaves a
pending reservation that readers retry; a failure between 3 and 4 leaves the canonical object
**committed** (readers use it and may repair the index); staging deletion is garbage collection
and never affects the commit outcome. Rollout needs no migration or quiesced cutover - new
readers ignore legacy stored ETags and legacy readers already accept finalized rows without
one - but while legacy *finalizers* remain in the fleet the pre-existing race can still
republish a stale ETag that a legacy reader rejects. Full protocol:
`references/docs/format/table/transaction.md`.

**v12 (beta.12): predecessor-conditioned publication (PR #8800), a `breaking-change`-labeled PR.**
`ExternalManifestStore::put_if_predecessor` "reserves a version only if the store's record for the
predecessor still carries the identity the writer observed", and `CommitHandler::commit_after`
publishes on that condition. Its contract is explicit that this is not the ordinary conflict path:
"Commit only if `predecessor` is still the manifest at its version, decided with the reservation;
otherwise [`Error::PrerequisiteFailed`], never a conflict"
(`rust/lance-table/src/io/commit.rs:1036-1041`).

The identity comes from a new `ManifestLocation` field - "A token unique to this manifest record
in the commit handler's store ... A dataset recreated at the same version has a different one"
(`pub identity: Option<String>`). **That field is the actual compile break**: the new trait
methods are all default-implemented, so only code constructing `ManifestLocation` with a struct
literal must change. No built-in store implements the conditioned contract, and unconditioned
commits behave exactly as before.

### v2 manifest paths are a compatibility fence

`CommitBuilder::enable_v2_manifest_paths` is **default true** for new datasets. The v2 naming
scheme is what "allow[s] constant-time lookups for the latest manifest on object storage" instead
of listing `_versions/`, but its own doc carries a warning worth reading before writing a dataset
other systems must read: "turning this on will make the dataset unreadable for older versions of
Lance (prior to 0.17.0)" (`rust/lance/src/dataset/write/commit.rs:171-180`). The parameter has no
effect on an existing dataset - migrate one with `Dataset::migrate_manifest_paths_v2`.

### 9.4 Cache keys and backend (v10, BREAKING)

Cache keys became an opaque 16-byte BLAKE3 digest (PR #7878). The format is stamped as
`pub const CACHE_KEY_FORMAT: &str = "blake3-128-v1"`
(`rust/lance-core/src/cache/key.rs:23`), and `InternalCacheKey` is now a newtype over
`[u8; 16]` (`key.rs:91`) rather than a three-field struct. **There is no runtime legacy-key
fallback**: every warm or persisted cache cold-misses after the upgrade. Upstream's guidance is
"Persistent backends should include `CACHE_KEY_FORMAT` in their physical namespace and allow
entries from older formats to age out."

Removed with it: `CacheBackend::invalidate_prefix`, `LanceCache::keys`, `CacheKeyIterator`,
`LanceCache::with_backend_and_prefix`, `Session::index_cache_keys`,
`Session::metadata_cache_keys`. Migration: "Replace `with_backend_and_prefix` with
`with_backend(...).with_key_prefix(...)`." New exports: `CacheKeySchema`, `CacheNamespace`,
`InternalCacheKey`, `KeyBuilder`, plus `QuickCacheBackend` and `recommended_cache_shards`.

**v11 added a component to the Row Id Sequence key** (#8078): it is now
`Dataset URI, fragment_id, row_id_meta`, up from `Dataset URI, fragment_id`. This was a
correctness fix, not a tuning change - keyed on `fragment_id` alone, a `WriteMode::Overwrite`
against a shared `Session` served the *previous* generation's sequence, corrupting stable row
ids. See the data-loss roundup in `changelog-v7-v13.md`. Java can now also select a registered
native cache backend by URI (e.g. `moka://?capacity=1048576`) or `CacheBackendConfig`, mutually
exclusive with the size options (#8446).

**quick_cache is now the default backend** for both the index cache (PR #7953) and the metadata
cache (PR #8013) - hard-wired in `Session::new`, with no env var or Cargo feature to opt out.
The operationally important consequence is an **admission ceiling**: quick_cache splits its
weight budget evenly across shards with no borrowing and *silently refuses* entries heavier than
a shard's share. Shards are `min(cpus / 2, capacity / 4 GiB)` with a floor of 1, so the
per-shard share bounds the largest cacheable entry - an oversized index partition simply never
caches, with no error. The io_uring handle cache (needs TTL) and the MemWAL SSTable cache (needs
predicate invalidation) stay on moka. Measured FTS effect at concurrency 128: 180.7 -> 1340.6
qps, 710 -> 96 ms, 47% -> 93% CPU.

**Cache backends became pluggable in v11 (PR #7683)**, which reopens the "no opt-out" statement
above. A `BackendConfig { kind, options: HashMap<String, String> }` plus a process-wide registry
(`register_backend`, `build_from_config`, `rust/lance-core/src/cache/registry.rs:29,118`) let
you supply your own implementation, and `parse_backend_uri` / `build_from_uri`
(`cache/backend_uri.rs:51`) accept a compact single-string form -
`moka://?capacity=1073741824`. Wired into sessions via `Session::with_cache_backends`
(`rust/lance/src/session.rs:209`). The only built-in `kind` is `moka`, whose sole option is
`capacity`. Duplicate `kind` registration returns an error rather than silently overriding an
existing constructor.

**Reported cache sizes changed in v11 (PR #8159, behaviourally breaking).** `CacheBackend` gained
a defaulted `deep_size_of_entries` method (`rust/lance-core/src/cache/backend.rs:130`), because
`LanceCache` previously "returned the backend weighted size, whose per-entry weights are each
computed with a fresh `DeepSizeOf` context. Shared `Arc` and Arrow allocations were therefore
charged once per cache entry, and shared `LanceCache` handles could charge the full cache
repeatedly." `approx_size_bytes` is now documented as a fallback "when exact entry traversal is
unavailable". Source-compatible, but **anything budgeting or alerting against
`LanceCache::deep_size_of()` will see smaller numbers after upgrade** - recalibrate thresholds
rather than treating the drop as a cache regression.

---

## 10. MemWAL

**MemWAL is experimental.** It is an LSM-tree architecture layered on a normal Lance table to
absorb high-throughput streaming writes while keeping indexed read performance
(`docs/src/format/table/mem_wal.md`). The Lance table is the **base table**; on top sit
**shards** that take writes and are asynchronously merged back. The spec is an on-disk-layout
contract; in-memory buffering and scheduling are implementation-defined.

### Architecture

> **Terminology changed in v10 (PR #7943, #7957).** What earlier versions called a *flushed
> MemTable* / *flushed generation* is now an **SSTable**, and *merge* into the base table is now
> **compaction**. The rename runs through the spec, Rust, Python, Java, and the protos. Field
> numbers are unchanged, so on-disk data and the wire format are compatible - but every symbol
> and binding name changed, with **no deprecation shims**. Mapping table at the end of this
> section.

- **Shard** - the unit of write scale-out; exactly one active writer per shard. For
  primary-key tables, all rows of a PK must map to one shard (otherwise inter-shard compaction
  order can resurrect stale rows). Append-only MemWAL tables may omit the primary key.
  Sharding is also a **read** optimization, not only a write one: "When sharding specs are
  available, the planner evaluates query predicates against shard fields and skips shards whose
  computed shard values cannot match" (`mem_wal.md:609-611`). A predicate over the shard field
  therefore prunes whole shards before any data is touched - so choosing shard fields that
  appear in common filters buys read selectivity, not just write parallelism.
- **MemTable** - holds rows before flush; a list of Arrow record batches. **A MemTable does not
  have a generation** - generation numbers belong to SSTables. `current_generation` in the shard
  manifest "is the generation number to assign to the next SSTable created by flushing the
  MemTable" (`docs/src/format/table/mem_wal.md:308`).
- **WAL** - durable storage of all MemTables in a shard, ordered by generation. Each WAL
  entry is an Arrow IPC stream file at `_mem_wal/{shard_id}/wal/`, named with bit-reversed
  64-bit binary (spreads sequential writes across S3 partitions). The writer epoch is in the
  Arrow schema metadata under `writer_epoch` for fencing.
- **SSTable** - "the immutable result of flushing a MemTable" (`mem_wal.md:130`), itself a Lance
  table at `_mem_wal/{shard_id}/{hex}_gen_{i}/`, with pre-built indexes and a PK bloom filter.
  The name is deliberate despite the layout: "Unlike a classic LSM sorted string table, a MemWAL
  SSTable is not sorted by key; random access is instead served by its BTree primary-key
  sidecar. It is called an SSTable because it is an immutable, persisted, indexed run"
  (`mem_wal.md:134`).

  **v12 added three optional accounting fields** to the `SsTable` proto message (#8981,
  `v12.0.0-beta.14` - the only `format-change`-labeled PR in that range):
  `in_memory_bytes` (field 3), `physical_rows` (field 4), `primary_key_bytes` (field 5)
  (`protos/table.proto:781,786,797`). They record "what the SSTable holds, as the writer's
  MemTable accounted for it at flush" (`mem_wal.md:148-151`).

  Three traps. (1) These are **payload estimates, not read-cost bounds**: `in_memory_bytes`
  "excludes the per-array structure a reader materializes, so a consumer budgeting memory from it
  must add its own headroom." (2) `physical_rows` counts "the older duplicates of a primary key
  that the generation's deletion vector masks. A scan applying the deletion vector yields fewer."
  (3) **Absence is not zero**: "An entry written before they existed records none of them, and a
  reader must not treat an absent value as zero; `primary_key_bytes` is also absent on a table
  with no primary key" (`mem_wal.md:167-170`). Downstream code constructing `SsTable` with a
  struct literal needs updating.
- **Shard manifest** - source of truth per shard: `writer_epoch`, shard assignment, WAL
  pointers, and "**SSTable generation state**: `current_generation` and `sstables`"
  (`mem_wal.md:294`). Versioned, immutable, committed via put-if-not-exists.
  **v12 renamed and narrowed `ShardManifestStore`** (#8640): `read_latest` -> `latest`,
  `read_latest_uncached` -> `refresh_latest`, and `write` became crate-private - callers reach
  it through `commit_update`, `claim_epoch`, or `initialize_shard`
  (`rust/lance/src/dataset/mem_wal/manifest.rs:134,151,251`). Existing `commit_update` closures
  need no change: setting `version: current.version + 1` is exactly what the check expects.
- **MemWAL Index** - one per table, centralizing config, **compaction progress**
  (`compacted_sstables`, "the last SSTable compacted into the base table for each shard",
  `mem_wal.md:43`), index catchup, and shard snapshots. Tied to the `UpdateMemWalState`
  transaction.

**Two MemWAL internals live only in the mirror.** `references/docs/format/table/mem_wal.md`
carries the reader-side **"Query Planning" / "Indexed Read Plan"** model - how sources are
collected and ranked, where "Each source is tagged with its shard and freshness tier. SSTable
sources are also tagged with their generation" - and, in Appendix 3, the exact **bucket transform
hash** (32-bit wrapping `fmix`/`rotl32` mixing) that decides shard bucketing. Read the mirror for
either; reimplementing bucketing from anything else will not interoperate.

### MemWAL is a parallel stack, not an integration into `Dataset`

Three properties decide whether MemWAL is adoptable for a given system, and none of them are
stated in the spec pages. All three were re-verified at `v13.0.0-beta.4`.

- **`Dataset::scanner()` has no MemWAL integration.** The whole of
  `rust/lance/src/dataset/scanner.rs` mentions MemWAL exactly once, in an unrelated comment about
  phrase matching (`:290`). Fresh rows - MemTable, WAL, and un-compacted SSTables - are reachable
  only through `LsmScanner` (`rust/lance/src/dataset/mem_wal/scanner/builder.rs:193`), which is a
  **narrower** API than the normal scanner: no SQL, no joins, no aggregation, no `take`. A system
  needing DataFusion-grade query over fresh data does not get it from MemWAL.
- **No manifest feature flag marks a MemWAL table.** The flag set is bits 0-6 plus bit 7
  (`rust/lance-table/src/feature_flags.rs:11-52`) and none of them means "this table has a
  MemWAL" - the one that briefly existed, `FLAG_MEM_WAL_INDEX_CATCHUP`, was retired before the
  `v11.0.0` final. So an ordinary reader opens the dataset happily and **silently sees only
  base-table data**, missing everything not yet compacted. That is a correctness-of-expectation
  hazard with no error surface, not a format hazard.
- **No SSTable compactor ships in-tree.** The spec fully describes an SSTable Compactor and a
  Garbage Collector, but the workspace contains no struct implementing either - adopting MemWAL
  means writing and operating both yourself.

**Read freshness.** The ordering rules grew from three to five, with the MemTable given its own
explicit tier: "The active MemTable is newer than every published SSTable" (`mem_wal.md:81`),
and any uncompacted SSTable wins over the base table. The background job formerly called the
*MemTable Merger* is now the **SSTable Compactor**: "The compaction uses Lance merge-insert
semantics and updates `compacted_sstables[shard_id]` atomically with the base-table commit"
(`mem_wal.md:508`).

### The appender/tailer/flusher model

Rust write path (`rust/lance/src/dataset/mem_wal/`):

- **`ShardWriter`** - the main per-shard writer interface. `ShardWriter::open` does
  epoch-based fencing once.
- **`WalAppender`** - the lowest-level primitive: single-entry synchronous atomic appends via
  put-if-not-exists, no buffering, owns the object store + epoch + position state.
- **`WalFlusher`** - buffers the WAL for durability.
- **`WalTailer`** - ordered reader of WAL entries from one shard.
- **`MemTableFlusher`** - flushes a frozen MemTable to a Lance file (producing an SSTable).

`ShardWriterConfig.enable_memtable` (default `true`) controls whether a MemTable layer is
maintained. With `enable_memtable == false` (**WAL-only mode**) no MemTable/index is
allocated and `index_configs` must be empty. Key defaults: `durable_write` true,
`max_wal_buffer_size` 10MB, `max_wal_flush_interval` 100ms, `max_memtable_size` 256MB,
`max_memtable_rows` 100,000, `max_unflushed_memtable_bytes` 1GB (backpressure budget - writes
block, never fail).

### In-memory HNSW

The in-memory MemTable can carry a **Lance-native HNSW vector index** (`MemIndexConfig::hnsw`,
new in v7 - PR #6795). HNSW is self-contained (no centroids/codebook needed); only the
distance metric is inherited from the base index. Also supported as MemTable indexes: BTree
scalar and FTS. Since v9, **prefiltered LSM vector and full-text search** is supported across
all

indexes.md 58.3 KB

# Lance v12 reference - indexes and distributed builds (sections 11-12)

Part of the Lance v13 reference (`lance-format/lance@v13.0.0-beta.4`). Citations are `path:line`
relative to the repo root; build a permalink as
`https://github.com/lance-format/lance/blob/v13.0.0-beta.4/<path>`. Line numbers drift between
tags - treat them as approximate. Cross-references written as "section N" use the original
16-section numbering; `lance-reference.md` maps every number to its file.

## Contents

- [11. Indexes](#11-indexes)
  - [11.1 Vector indexes](#111-vector-indexes)
  - [11.2 Scalar indexes](#112-scalar-indexes)
  - [11.3 Full-text search](#113-full-text-search)
  - [11.4 Geo / RTree](#114-geo--rtree)
  - [11.5 Index updates and reindexing](#115-index-updates-and-reindexing)
- [12. Distributed write and indexing](#12-distributed-write-and-indexing)

Other files: `format-file.md` (1-4), `format-table.md` (5-10), `ops.md` (13, 15, 16),
`changelog-v7-v13.md` (14).

---

## 11. Indexes

Lance treats indexes as **independent, redundant structures layered on top of row
identifiers** - the file format has no built-in search structures, so index formats evolve
independently (`docs/src/format/index/index.md`). Three categories: scalar, vector, system.

Index design: loaded on demand (a dataset opens without loading any index), loaded
progressively, immutable once written. An index is composed of **segments**, each with a
UUID, each covering a disjoint subset of fragments recorded in a `fragment_bitmap`. **Segments
need not cover all fragments** - an index can lag; engines split queries into indexed and
unindexed subplans and merge results. When a column has **no index at all**, both vector
search and full-text search transparently fall back to a flat scan rather than erroring
(`rust/lance/src/dataset/scanner.rs:3419,3697`). Index content lives at `_indices/{UUID}`.
`IndexMetadata` carries `uuid`, `name`, `fields`, `fragment_bitmap`, `index_details` (a typed
`Any`), `version`, and - since v11 - `covering_fields` (proto field 11).

**Covering indexes / carried columns** (v11, PR #8535; **redefined at v13**, PR #8856; manifest
flag `FLAG_COVERED_INDEX_METADATA = 128`, section 7). `covering_fields` names the columns an
index carries values for, letting a query that only projects those columns be answered without a
fragment take.

**It is no longer a trailing suffix of `fields`.** The v11 wording ("must be a suffix of
`fields`") was replaced: it "must be a subset of `fields`, in the order the index emits them. A
column is carried if and only if it is named here, so a column the index is both keyed on and
carries is listed once in `fields` and named here; no id repeats in `fields`, and `fields[0]`
remains a column the index is keyed on." So a keyed column may *also* be carried, and membership
of `covering_fields` - not position in `fields` - is what decides. "Every id in `covering_fields`
names a top-level field. Covering a struct column carries the whole struct, its children
included, as one column." A reader without the flag would select an index by plain membership of
`fields` and answer a query on a merely-carried column with an index keyed on a different one -
wrong neighbours, no error - which is why both reader and writer must refuse a covering dataset
without the bit.

**Carried values are now really written, per segment.** The v11-era "no index builder writes
carried values yet" is gone from upstream. V3 IVF auxiliary files can hold them, and "a reader
discovers carried columns by exclusion, not by position: any column in the auxiliary file's
schema that is not one of the quantizer's internal columns is a carried column", bound to source
fields by a `covering_field_ids` schema-metadata key. A merge "must not combine shards whose
carried columns disagree on these ids, even when those columns match by name and type". Coverage
is therefore a per-segment property: "one logical index may hold values for some of its segments
and not others", depending on "the index type, on the writer that produced the segment, and on
what later maintenance did to it".

Query-side, `VectorQueryProto.covering_projection` (field 15) reserves the narrowing tag as a
message rather than a bare list, because empty is meaningful: "absent: no narrowing computed;
materialize every covering column declared. present and empty: materialize nothing, though the
index does declare covering. present and non-empty: materialize exactly these."

**The fence is deliberately not permanent, which is why the flag costs nothing today.** Bit 128
is set only while some index actually carries values:

```rust
if final_indices.iter().any(|index| !index.covering_fields.is_empty()) {
    manifest.reader_feature_flags |= FLAG_COVERED_INDEX_METADATA;
    manifest.writer_feature_flags |= FLAG_COVERED_INDEX_METADATA;
}
```

(`rust/lance-table/src/transaction/manifest_build.rs:1323-1329`.) Upstream un-sets it rather than
inheriting it, precisely so the fence lifts again: "fence by simply not setting it again.
Inheriting it from the previous manifest instead would make the fence permanent." The practical
consequence is that because no builder writes `covering_fields`, the bit is never set in normal
operation - so v11/v12-written datasets stay openable by older builds despite the flag existing.

### 11.1 Vector indexes

Every vector index has **three orthogonal parts: clustering, sub-index, quantization**, named
`{clustering}_{sub_index}_{quantization}` (`docs/src/format/index/vector/index.md`).

- **Clustering** - only **IVF** (Inverted File): k-means partitioning; search examines only
  the most relevant clusters.
- **Sub-index** - `FLAT` (exact, scans all vectors) or `HNSW` (approximate graph search).
- **Quantization** - `FLAT` (none, exact), `PQ` (Product Quantization), `SQ` (Scalar
  Quantization), `RQ` (RaBitQ - random rotation + binary quantization).

The seven documented combinations: `IVF_FLAT`, `IVF_PQ`, `IVF_SQ`, `IVF_RQ`,
`IVF_HNSW_FLAT`, `IVF_HNSW_SQ`, `IVF_HNSW_PQ`.

`protos/index.proto:115` also carries a **`DiskAnn`** stage message. It is part of the persisted
wire format rather than one of the seven documented builder combinations - expect to encounter
it when reading index protos, not when choosing an index type.

**Distance metrics** (`VectorMetricType`): `L2` (0), `Cosine` (1), `Dot` (2), `Hamming` (3).
SIMD kernels in `lance-linalg`; the `fp16kernels` feature compiles C SIMD kernels for fp16.

**Compression** (bytes per vector vs float32):

| Quantization | Storage | Ratio |
|--------------|---------|-------|
| FLAT | `dimension * 4` | 1x (exact) |
| SQ (8-bit) | `dimension` | ~4x |
| PQ | `num_sub_vectors` (one uint8 code per sub-vector) | ~`(dimension*4)/m` |
| RQ (RaBitQ, `num_bits` bits/dim) | `ceil(dimension * num_bits / 8)` + correction factors | ~32x at 1 bit; ~6.5x at the 5-bit default |

**IVF_RQ requires the vector dimension to be divisible by 8** - enforced with the error
"vector dimension must be divisible by 8 for IVF_RQ" (`rust/lance-index/src/vector/bq/builder.rs`).
**RaBitQ is now multi-bit** (new in v8, PR #7038): `num_bits` is "in the range 1..=9"
(`docs/src/format/index/vector/index.md:255`). IVF_RQ always stores the 1-bit binary sign
code in `_rabit_codes`; for `num_bits > 1`, the remaining `num_bits - 1` ex-code bits are stored
in a separate column instead of widening the binary code path, alongside `__add_factors_ex` /
`__scale_factors_ex` correction columns.

**The ex-code column was renamed in v11** (#8407). Writers now emit **`__blocked_ex_codes`**,
sized `next_multiple_of(code_dim, 64) * (num_bits - 1) / 8`. The old column is still readable:
"Indexes written before the blocked ex-code layout store the same bits in `__ex_codes`, sized
`ceil(dimension * (num_bits - 1) / 8)`. Readers still accept that column and repack it at load
time; writers no longer emit it" (`docs/src/format/index/vector/index.md:207-209`). Two sibling
sizing corrections landed with it: `__pq_code` is
`list<uint8>[num_sub_vectors * num_bits / 8]` (not `[m]`), and `_rabit_codes` is
`ceil(code_dim / 8)` (not `dimension / 8`). Every vector-index storage column is also now
declared **nullable**, and the RQ rows gained a separate "Present when" column.
A new `query_estimator` metadata field selects the distance-estimator layout: "`residual_query`
or `raw_query`. Missing values are read as `residual_query` for compatibility with released
1-bit IVF_RQ indexes" (`index.md:258`); raw-query search (PR #7078) adds an `__error_factors`
column "for raw-query lower-bound pruning" (`index.md:201`). The metadata schema also carries
`code_dim` (u32, the rotated-vector dimension).

**`num_bits` now defaults to 5, not 1** (#8936, `v12.0.0-beta.12`) - the PR carries a
conventional-commit `!` but no `breaking-change` label, so the release bot did not count it.
`RABIT_DEFAULT_NUM_BITS: u8 = 5` (`rust/lance-index/src/vector/bq.rs`) drives both
`RQBuildParams::default` and `RabitBuildParams::default`, and the Python `build_rq_model` stub
default moved with it. Per-row storage (`docs/src/guide/performance.md:483-501`):

- 1-bit: `dimension / 8 + 20` bytes
- Multi-bit: `dimension / 8 + round_up(dimension, 64) * (num_bits - 1) / 8 + 28` bytes

"Every bit width stores a 1-bit sign code plus three 4-byte correction factors." Multi-bit adds
the `__blocked_ex_codes` and ex-factor columns on top of the sign code.

**Budget for a ~4.4x jump if you relied on the default.** Upstream's own worked example moved
from `100M * (768 / 8 + 16) = ~10.8 GiB` to
`100M * (768 / 8 + 768 * 4 / 8 + 28) = ~47.3 GiB`. The escape hatch is explicit and documented:
"`Fast` search mode uses only the 1-bit sign code even when the index stores additional bits. Set
`num_bits=1` explicitly to minimize index size and build I/O; the same 100M-row example uses about
10.8 GiB, but searches cannot use the multi-bit distance estimate and may have lower recall."
So `Fast` mode gains nothing from the wider default while paying its full storage cost.

**A separate multi-bit correctness fix landed at beta.15**: RaBitQ FastScan above 1024 rotated
dimensions overflowed its accumulator (#8842) - "the scalar path saturates and the AVX2/AVX-512/NEON
kernels wrap, so a distance becomes `true_sum % 65536` and the ranking collapses. At rotated dim
4096 a full-range sum reaches 261120, four times the ceiling." Query-time only, so it heals on
upgrade - but any recall figure measured on an affected index before the fix is invalid.

**bfloat16 is not usable for vector indexes**, despite the docs recommending it as an embedding
type directly above an IVF_PQ `create_index` example (`docs/src/guide/data_types.md:406`). The
`lance.bfloat16` extension stores as `FixedSizeBinary(2)`, and the accepted element types are
only `Float16 | Float32 | Float64 | UInt8 | Int8`
(`rust/lance/src/index/vector/utils.rs:244`). The rejection fires on **both** the index-build
and the query path (`scanner.rs:1647`, `knn.rs:397`), so it is not a build-time-only limit. The
in-tree error message is itself stale - it omits `Int8` from the list it actually accepts.

**Approx mode** (new in `v8.0.0-beta.10`, PR #7179). Vector search takes a public
`approx_mode` with three values - "`fast`, `normal`, and `accurate`" - to pick the
speed/accuracy tradeoff "when the backing index supports it" (RaBitQ today). "The public API
avoids exposing RaBitQ/HACC terminology" (commit `e25620710`). It threads through the Rust
scanner, the ANN proto, and Python query parsing; serialized as `VectorApproxMode approx_mode`
(`protos/ann.proto:16,45`) - a **breaking ANN-proto change**, so any consumer matching Lance's
serialized ANN query proto must regenerate. Multi-bit RaBitQ ex-code reranking also got
dedicated SIMD kernels (PR #7205, `rust/lance-index/src/vector/bq/ex_dot.rs`). **IVF_RQ now
defaults `target_partition_size` to 4096** (was the generic fallback, PR #7273).

**Query-time knobs: `nprobes` and `refine_factor`.** Distinct from the build-time parameters
above, and the two the docs put front-and-center for tuning a live query: "The latency vs recall
is tunable via: **nprobes**: how many IVF partitions to search / **refine_factor**: determines
how many vectors are retrieved during re-ranking"
(`docs/src/quickstart/vector-search.md:208-210`). `nprobes` trades read volume for recall by
widening the partition sweep; `refine_factor` over-fetches quantized candidates and re-ranks
them against full-precision vectors, so it recovers accuracy lost to PQ/SQ/RQ compression at the
cost of extra `take`s. Reach for these before rebuilding an index with different parameters -
they need no reindex.

On-disk layout (format V3): each vector index is two Lance files - an **index file**
(`index.idx`, the search structure: IVF metadata, HNSW graph) and an **auxiliary file**
(`auxiliary.idx`, quantized vector storage). HNSW construction defaults: `max_level` 7, `m`
20, `ef_construction` 150. The PQ codebook and the RaBitQ rotation matrix are stored as
tensors in the auxiliary file's global buffer.

**HNSW construction changed in v11 (PR #8188, BREAKING).** Three things at once:
`OnlineHnswBuilder::with_capacity` is `#[deprecated]` in favour of a fallible
`try_with_capacity` that runs shared parameter validation
(`rust/lance-index/src/vector/hnsw/online.rs:161,174`); **`m < 4` is now rejected** ("must be at
least {MIN_HNSW_M} to avoid severely fragmented graphs", `builder.rs:144`); and the persisted
level layout was corrected from `max_level` offsets to `max_level + 1`, because "truncating
persisted levels to the sampled height also broke version-1 readers that index the configured
level count directly". `HnswBuildParams::default()` is unchanged (`m: 20`). **Expect different
graphs and different recall from the same inputs** after upgrading. Separately, greedy descent
now stops before level 0 - "the greedy (ef=1) descent should cover only levels `max_level-1`
down to 1; level 0 must be searched solely by the ef-bounded beam search" (PR #8035,
`builder.rs:396,546`) - worth +3.7% recall@10 at ef=16.

Minor v11 fix: `QuantizationType`'s `FromStr` now accepts `"RQ"` as well as `"RABIT"`
(`rust/lance-index/src/vector/quantizer.rs:86`, PR #8214) - `Display` wrote `"RQ"` but `FromStr`
rejected it, so the two never round-tripped.

**"partition N is empty, skipping" is benign by design.** During an IVF build, an empty
partition emits `log::warn!("partition {} is empty, skipping", part_id)`
(`rust/lance/src/index/vector/builder.rs:1174`) and then registers a zero-sized partition in both
the storage and index IVF models and continues the merge loop (`:1176-1180`). There is no
`Result::Err` and no exception to catch, so on a skewed or sparse dataset this floods logs
without indicating a failure - suppress it with a scoped tracing filter rather than hunting a
bug. (The one genuine error mentioning empty partitions is a guarded inconsistency case at
`:1100`, reachable only when a partition reports a non-zero size yet reads back `None`.)

**Typed index details.** A vector index records a typed `VectorIndexDetails` message in the
manifest's `index_details` field (`protos/index.proto:188-241`; moved out of `table.proto`
in `v7.1.0-beta.1`): `metric_type`, `target_partition_size` (0 = unset), an optional
`HnswParameters` (`max_connections` = M, `construction_ef`, `max_level`), a `compression`
oneof (`ProductQuantization` / `ScalarQuantization` / `RabitQuantization` with a `FAST` or
`MATRIX` rotation / `FlatCompression`), and a free-form `runtime_hints` string map. Hint
keys use reverse-DNS namespacing (e.g. `lance.ivf.max_iters`) and unrecognized keys must be
silently ignored by all runtimes.

**Build prerequisites.** A vector index cannot be built on an empty table -
`build_empty_vector_index` returns `not_supported` ("Creating empty vector indices with
train=False is not yet implemented", `rust/lance/src/index/vector.rs:1437`). PQ training
needs at least `2^num_bits` rows for its codebook centroids, so a default 8-bit PQ index
hard-errors below **256 rows** ("Not enough rows to train PQ. Requires {n} rows but only {m}
available", `rust/lance-index/src/vector/pq/builder.rs:177`); IVF k-means separately needs at
least `num_partitions` rows. Build vector indexes lazily, once the table holds data.

**Batched vector queries** (PR #6828). `Scanner::nearest` accepts a batch of query vectors on
a fixed-size-list column - there is no separate `nearest_batch` API. Batched results carry a
synthetic 0-based `query_index` discriminator column (`QUERY_INDEX_COL`) so each result row is
attributable to its source query (`rust/lance/src/dataset/scanner.rs:104,1972`).

**Streaming IVF k-means training** (PR #6913). For bounded-memory IVF training on large
datasets, the IVF builder exposes `streaming_sample_rate`, `streaming_coreset_rate`, and
`streaming_refine_passes` (exposed through Python). When set, training loads at most
`num_partitions * streaming_sample_rate` vectors and keeps the total sampled set bounded
(`rust/lance-index/src/vector/ivf/builder.rs:44-51`).

**ACORN-1 prefiltered HNSW traversal (v10, PR #7927).** A prefilter-aware graph traversal for
HNSW, **opt-in only**: it is gated on `ApproxMode::Fast` (Python `approx_mode="fast"`), with no
env var. "`Normal` (the default) and `Accurate` keep the existing traversal, so default behavior
is unchanged." Do not reach for it reflexively - upstream documents a real regression on
low-selectivity uniform-random masks: "uniform random masks at 2% selectivity drop recall (0.775
vs 0.975 on GIST1M), and at 50% random the waypoint bookkeeping makes it slower than the current
traversal (15.3 vs 4.1 ms)." It pays off on *clustered* prefilters, not random ones. Its
constants are not tunable: 16 mask-sampled seeds (`ACORN_SEED_COUNT`,
`rust/lance-index/src/vector/graph.rs:550`) and a `4 * ef` waypoint budget. Narrow API break: the
all-public `HnswQueryParams` gained a required `use_acorn: bool` field
(`rust/lance-index/src/vector/hnsw/builder.rs:1104`), so struct-literal construction no longer
compiles.

Two runtime escapes make `approx_mode="fast"` weaker than "ACORN is on": even when requested,
ACORN is **skipped** when the mask passes all rows or when fewer than 10% of rows survive it,
and an under-delivering traversal falls back to `search_basic`
(`rust/lance-index/src/vector/hnsw/builder.rs:1367-1388`). So a benchmark that sees no change
from `fast` may simply never have entered the ACORN path. The opt-in gate itself is unchanged in
v11 despite the surrounding HNSW rework (PRs #8188, #8035 both leave `use_acorn: query.approx_mode
== ApproxMode::Fast` at `:1170` intact).

**Vector index append across heterogeneous models (v10, PR #8047).** Appending to a vector index
whose segments were trained with different IVF/quantizer models previously failed. Append now
writes **one** new segment over the unindexed fragments, using the manifest-suffix segment's
model. Explicit `OptimizeOptions::retrain` remains the only operation that rebuilds from source
and unifies models.

**v9 vector changes.** `as_vector_index` was **removed from the public `Index` trait**
(PR #7392) - downcast via `as_any()` instead. A new **hamming clustering** utility (PR #7379,
`rust/lance-linalg` + `rust/lance`) does SIMD-accelerated (AVX-512/AVX2) pairwise Hamming
distance over 64-bit binary hashes plus union-find grouping for near-duplicate detection:
`pairwise_hamming_distance[_parallel]`, `UnionFind`, and `hamming_clustering_for_ivf_partition`
/ `_for_sample` / `_for_range` / `_from_hashes` returning a `RecordBatchReader` of clusters.
This is a clustering *utility* over binary vectors, distinct from the `Hamming` distance
*metric* already listed above. Separately, `COUNT(*)` pushdown now works on **stable-row-id**
datasets (PR #7360) - the `CountFromMaskExec` fast path no longer falls back to a full scan
when stable row IDs are enabled. A vector-search correctness fix also accounts for the SQ
offset in dot-distance (PR #7481).

### 11.2 Scalar indexes

`docs/src/format/index/scalar/`. Results are **exact** (BTREE, BITMAP, LABEL_LIST) or
**inexact / AtMost** (BLOOM_FILTER, NGRAM, ZONEMAP, RTREE) - except that ZONEMAP and
BLOOM_FILTER now answer **`IS NULL` exactly** (see the `null_bitmap` note below).

| Index | For | Structure |
|-------|-----|-----------|
| BTREE | Range queries, sorted access, high-cardinality columns | Two-level: in-memory page lookup + on-disk sorted leaves, default 4096 rows/page |
| BITMAP | Low-cardinality columns, fast set membership | One bitmap (serialized `RowAddrTreeMap`) per distinct value |
| LABEL_LIST | Multi-value / tag columns | Built on a bitmap index; supports `array_has`/`_all`/`_any` |
| NGRAM | Substring matching | Overlapping trigrams (ASCII-folded, lowercased); query `contains`. Since v9 also accelerates regex and infix `LIKE` (PR #7139) |
| ZONEMAP | Scan pruning / predicate pushdown | Per-zone min/max/null stats (default 8192 rows/zone); a *primary skipping structure* |
| BLOOM_FILTER | Probabilistic membership | Zone-based Split Block Bloom Filters (xxHash64; default 8192 items/zone, FPP 0.00057) |
| FM_INDEX | Substring / prefix / regex search on raw bytes | Compressed BWT index over raw byte arrays; built on the Segmented Index architecture (see 11.5). New in v8 |
| RTREE | 2D spatial pruning | See 11.4 |

NGRAM, ZONEMAP, and BLOOM_FILTER are newer additions. A JSON scalar index wraps another
index's details with a JSON path.

**As of v13, only four UDFs reach a JSON index** (#9101). The routing allow-list was cut from
six names to `json_get_int`, `json_get_float`, `json_get_bool` and `json_get_string`;
**`json_extract` and `json_get` are no longer routed at all** and fall back to a full scan. This
was a correctness fix, not a regression - the three symptoms it removed were a predicate that
"searched for a quoted key and matched nothing, where the unindexed scan matched", a `Utf8`
literal pushed into an `Int64` btree that "panicked", and a range that "returned every row"
because "quoting is not order-preserving (`ab` < `ab!` but `"ab"` > `"ab!"`), so a decoded-key
btree cannot answer a text-ordered range, and its page min/max pruning is unsound for one". The
cost lands silently: the query still succeeds, just without the index. Rewrite `json_extract`
predicates onto the typed accessors if they were carrying an index.

One more routing prerequisite, easy to miss and now doubly load-bearing: a DataFusion JSON
predicate only reaches the index if the UDF the planner parsed is the **same `ScalarUDF` object**
Lance registered. Register Lance's functions on the same `SessionContext` that owns the table
provider; otherwise the name matches, the object does not, and the predicate is evaluated
post-scan with no error. Since v9, **BTREE and ZONEMAP accept `large_string`
(`LargeUtf8`) columns** (PR #7525), not just `Utf8`. A ZoneMap index also exposes a column's
global **min/max without a scan** via `zonemap_value_range(column)` (`DatasetIndexExt`;
`ZoneMapIndex::value_range` / `value_range_over(segments)`, PR #7463) - cheap stats and a
range-pruning planning input.

As of v9.1, both **ZONEMAP and BLOOM_FILTER carry a `null_bitmap`** (a serialized
`RowAddrTreeMap` of null rows) that upgrades **`IS NULL` from inexact to Exact** - "since
finding NULLs is a common query pattern, the index also maintains a bitmap of null rows which
allows it to return exact results for IS NULL queries" (`docs/src/format/index/scalar/bloom_filter.md`,
`.../zonemap.md`). Other predicates on these indexes stay inexact/`AtMost`.

**For SBBF internals, read the spec page directly.** `references/docs/format/index/scalar/bloom_filter.md`
carries the block structure ("The SBBF divides the bit array into blocks of 256 bits, where each
block consists of 8 contiguous 32-bit words"), the 8 salt constants, and the binary-search sizing
algorithm behind the FPP figure. None of that is restated here - go to the mirror when you need to
reimplement or validate a filter rather than just use one.

**In v11 that capability became wire-explicit and grew a second half** (PR #8088). The zone-map
details proto gained `optional bool has_null_bitmap = 3` (`protos/index_old.proto:44`),
documented as: "Absent or false means legacy format: null positions are not tracked and IS NULL
searches fall back to approximate zone-level statistics. Present true means IS NULL is exact
**and IS NOT NULL can be answered without a full scan**" (`:42-47`). The flag is set from
`builder.null_rows.is_some()` (`rust/lance-index/src/scalar/zonemap.rs:1189`), so an index built
before v11 keeps the legacy behaviour until rebuilt - check the flag before assuming exactness.

**Zone maps now cover every data type, including nested ones** (PR #8017): "We are using zonemap
as our 'statistics' and it is also important for recording nullability bitmaps. As a result, we
need it to support all types. For nested types we don't track the min/max but we still track the
nullability bitmap and the null count." For List, FixedSizeList, Struct, and Map columns, min and
max are stored as typed null values (`rust/lance-index/src/scalar/zonemap.rs`) - so a zone map on
a nested column buys null statistics and nothing range-related.

**Scalar-index pushdown is volume-independent.** The planner emits a `ScalarIndexQuery` whenever
an index exists for the column - `apply_scalar_indices` is driven purely by whether
`index_info.get_index(col)` answers (`rust/lance-index/src/scalar/expression.rs:2612`), with no
row-count, cardinality, or selectivity heuristic anywhere in the module. The plan for a 4-row
table and a 2000-row table is identical. Only two non-volume gates exist: `use_scalar_index` is
suppressed for a non-prefiltered vector search (`let use_scalar_index = self.use_scalar_index &&
(self.prefilter || self.nearest.is_none())`, `rust/lance/src/dataset/scanner.rs:2812`), and
scalar indexes are skipped entirely when any fragment lacks `physical_rows` metadata ("We need
row counts to use scalar indices", `scanner.rs:2668`) - that is metadata *presence*, not
magnitude. Practical consequence: "indexes only pay off at scale" is false here; a scalar index
on a small table is used, for better or worse.

**FM-Index** (new in v8, `docs/src/format/index/scalar/fmindex.md`,
`protos/index.proto:251` `FMIndexDetails` - **renamed from `FMIndexIndexDetails` in v9**,
PR #7397, a **breaking change that makes existing FM indexes unreadable**; the rename also
dropped the `get_plugin_name_from_details_name` `fmindex`->`fm` special-case). The
Ferragina-Manzini index is "a compressed
substring index based on the Burrows-Wheeler Transform (BWT)" that "enables efficient
**arbitrary substring search**, **prefix match**, and **suffix/regular-expression search**
directly on raw bytes" (`fmindex.md:3`) - unlike the NGRAM index (fixed trigrams) or FTS
(distinct words). It indexes columns of strings or binary as raw byte arrays, so it is
**normalization-independent by design**: any case-folding / Unicode / stemming normalization
is the caller's job and must be applied identically to the column at build time and to the
query (`fmindex.md`). Two bytes are reserved as BWT sentinels (`\x00` terminator, `\xFF`
row separator) and any incoming `\x00`/`\xFF` are sanitized to space (`\x20`) at build time.
Because a BWT suffix array cannot be merged by concatenation, FM-Index is partitioned via
the Segmented Index architecture: a `num_segments` parameter (set at index creation) splits
fragments into disjoint subsets, each a self-contained FM-Index; appends build a new segment
over the unindexed fragments, and `merge_segments` re-reads the covered fragments' raw text
to rebuild a unified segment (`fmindex.md:53`). Queries (`CONTAINS(column, "...")`) return an
inexact candidate set; the engine verifies.

**Sizing and residency, field-measured - `num_segments` does not bound query memory.** An
FM-Index is roughly **1:1 on disk with the raw column** it covers (measured: a 2,793.8 MiB text
column produced a 2,617 MiB index, 0.937x), so it roughly doubles the footprint of the data it
indexes rather than compressing it, and peak build RSS exceeded the column size (~3.1 GB). More
surprising at query time: the wavelet-tree blocks are **heap-resident, not mmap'd**, `prewarm()`
calls `wavelet.load_all()`, and `prewarm_partitions` warms **every** partition
(`rust/lance-index/src/scalar/fmindex.rs:1146,1546`, concurrently via `buffer_unordered`). So
`num_segments` lowers *build* RSS per segment but **not** query RSS - one `CONTAINS` can pull the
whole index resident, per process, unshared. On object storage that is also a full cold download
on first query. The practical mitigation is to index a narrow materialized column rather than a
wide blob.

**Choosing between NGRAM, FM-Index, and FTS for substring work.** These three are not
interchangeable, and two column-level constraints decide the choice before performance does:

- **NGRAM requires a real text column.** "A ngram index can only be created on a Utf8 or
  LargeUtf8 field" (`rust/lance-index/src/scalar/ngram.rs:1690-1692`) - a `LargeBinary` column
  (a JSONB/`lance.json` column, for instance) is rejected outright. FM-Index *will* build on
  binary, but DataFusion's `contains` refuses to coerce `LargeBinary` to `Utf8`, so the query
  cannot reach it; a `CAST(... AS VARCHAR)` makes the plan legal but defeats index pushdown and
  falls back to a full scan. If you need substring search on a column, store it as text.
- **NGRAM and FM-Index disagree on matching semantics, silently.** NGRAM is always case- and
  accent-insensitive: "Currently we ALWAYS use trigrams with ascii folding and lower casing. We
  may want to make this configurable in the future" (`ngram.rs:93`), applied through the shared
  `TEXT_PREPPER` (`LowerCaser` + `AsciiFoldingFilter`, `:87-91`) on **both** the build and query
  paths (`:1150`, `:507`). FM-Index has no case handling at all - queries go straight to
  `search_string_contains(pattern.as_bytes())` (`fmindex.rs:1638`). The same needle therefore
  returns different result sets from the two indexes, with no warning.
- **FTS answers a different question.** It matches whole tokens, so it is the wrong tool for
  identifier or mid-token lookup: a needle that appears inside larger tokens matches nothing
  under FTS while NGRAM/FM find it, and a needle that is also a common token can match orders of
  magnitude more rows than the exact substring occurrences.

**v10 scalar-index changes.** BTREE + ZONEMAP `large_string` support (v9) is joined by
**LABEL_LIST accepting `LargeList`** (PR #7884) - previously a `LargeList` LABEL_LIST filter
silently fell back to a full scan because `ScalarValue::LargeList` was not accepted. Two
planner fixes: same-column range predicates with *differing* index fragment coverage were
merged unconditionally, claiming coverage they did not have - `index_type` is now part of the
merge key (PR #6782); and an index built from a stale handle and committed after a concurrent
`Operation::Update` no longer retains coverage over fragments whose indexed column changed -
"The fix applies to all index types" (PR #8011). A new optimizer rule drops an exact
`IS NOT NULL` when another exact same-index query is already null-intolerant (disabled under
`NOT`). Two new per-query metrics, `index_cache_hits` / `index_cache_misses`, surface in
EXPLAIN ANALYZE, `ExecutionSummaryCounts`, Python `ScanStatistics`, and Java `ScanStats`
(PR #7862) - with a caveat: "IVF v2 streaming scans and legacy v1 IVF partitions run
`load_partition` with `write_cache=false`. Those loads always execute the loader and never write
the result back, so they are reported as a miss on every call."

**Scalar-index fast search** (PR #6784). `fast_search` now routes through scalar/BTREE-indexed
fragments and skips unindexed ones, so a filtered query can return from the index without a
flat scan of recently appended (still-unindexed) fragments. Not supported on the legacy file
version (`LanceFileVersion::Legacy`). This mirrors the long-standing vector `fast_search`
behavior - both return only what the index covers, trading completeness for latency until
`optimize_indices` folds the new fragments in.

### 11.3 Full-text search

The FTS (inverted) index maps terms to documents with **BM25** scoring
(`docs/src/format/index/scalar/fts.md`). It is **Lance-native** - there is no Tantivy
dependency; the tokenizer stack lives in the `lance-tokenizer` crate (one tokenizer, the
ngram tokenizer, is noted as adapted from Tantivy, but the FTS engine is Lance's own).

Storage: `tokens.lance` (dictionary), `docs.lance` (doc metadata), `invert.lance`
(compressed posting lists, optional positions), `metadata.lance`. An FTS index may be
**partitioned** - every partition is searched at query time and results combined.

**On-disk format version (v9, breaking).** Newly created FTS / inverted indexes now default to
**format v2** (PR #7512, `docs/src/guide/migration.md:9-30`): "Newly created FTS / inverted
indexes now default to format v2 instead of v1. The `LANCE_FTS_FORMAT_VERSION` environment
variable no longer controls the format used for newly created indexes ... pass the index
creation parameter `format_version` explicitly." Pass `format_version=1` (accepts `1`/`2`/`"v1"`
/`"v2"`, `create_scalar_index("text", "INVERTED", format_version=1)`) when older Lance readers
must read the index - "older Lance readers may not be able to read them" otherwise. Existing v1
indexes stay queryable and are **maintained as v1**: "append, incremental indexing, optimize,
and mem-wal maintained-index flush ... continue preserving the v1 format."

**Configurable posting `block_size` (v9.1, breaking, PR #7466).** FTS index creation takes a
`block_size` param - documents per compressed posting block, "must be 128 or 256" (default
128; missing values in older indexes read as 128; unsupported values like 512 are rejected by
`validate_format_version_block_size`, `rust/lance-index/src/scalar/inverted/builder.rs:229`).
"`256` is experimental and may introduce breaking changes." Both `block_size=256` and the new
**code analyzer** require FTS on-disk **format v3** (the capability gate moved to v3, PR #7866):
"The code analyzer and `block_size=256` require format v3, so readers must support v3 before an
index using either option is created" (`docs/src/guide/migration.md`). As of v11 there is a
**third** v3 trigger, independent of the other two: "`document_granularity=\"list_element\"` also
requires v3 reader capability, independently of the posting format"
(`docs/src/guide/migration.md:13-14`).

**Document granularity (v11, PR #7788).** FTS gained an explicit document-boundary axis. The
index details proto carries `DocumentGranularity document_granularity = 14` with values
`ROW = 0` / `LIST_ELEMENT = 1` - "The logical FTS document boundary. The protobuf default
preserves the legacy row-document behavior when this field is absent"
(`protos/index_old.proto:96-97`) - so existing indexes keep row-granularity semantics with no
migration. `LIST_ELEMENT` treats each element of a list column as its own document, which
changes what BM25 scores and what a phrase can span. A companion output column
`_doc_index` (`DOC_INDEX_COL`, storage prefix `_doc_index_`,
`rust/lance-index/src/scalar/inverted/index.rs:140-141`) identifies which element matched.
Exposed as `create_scalar_index(..., document_granularity=...)` plus `MatchQuery` / `PhraseQuery`
parameters in Python (`python/python/lance/query.py:25,108,169`), and
`InvertedIndexParams.Builder.documentGranularity(..)` with `FullTextQuery` overloads in Java.
Rust callers note: `load_segments` gained a third `document_granularity` parameter.

The same PR added **`posting_format_version = 15`**, deliberately separate from `index_version`:
"The posting-list payload format. This is separate from index_version, which identifies the
overall inverted-index layout" (`protos/index_old.proto:99-100`). Do not conflate the two when
reasoning about reader compatibility - the maximum inverted-list format is still V3.

**Nested-field FTS (PR #7686).** FTS can now index leaf fields inside nested columns
(e.g. `data.text`), not just top-level string columns. Query-side bulk paths were added too:
impact-skip data for posting lists (#7602), a bulk MAXSCORE top-k path for disjunctions
(#7603), and a bulk conjunction path for AND / phrase queries (#7624).

**v10 FTS changes.** Format versions at this tag are V1/V2/V3; the **written default is still
V2**, max supported is V3, and `block_size=256` still requires `format_version=3` ("FTS
format_version={} is incompatible with block_size=256; use format_version=3",
`rust/lance-index/src/scalar/inverted/index.rs:101,279`). Compatibility fixtures pin v1 and v2
only - "FTS v3 is intentionally deferred until it is written by a stable release" (PR #7890) -
and readers must keep tolerating the retired `skip_merge` parameter written by Lance 3.0.1.

- **Optional `total_tokens` metadata key** (PR #7863): "Partitioned `docs.lance` files may
  include the optional schema metadata key `total_tokens`. Its decimal `UInt64` value is the sum
  of `_num_tokens` in that file" (`docs/src/format/index/scalar/fts.md:37-38`); when absent,
  readers compute the sum. Back-compatible on-disk addition, no format bump.
- **`LANCE_FTS_SEARCH_CHUNK`** (PR #7950) - partitions searched per CPU-pool task, default
  **16**, minimum 1, non-numeric values ignored (`inverted/index.rs:115`). Chunking stops query
  concurrency from flooding the pool with one small task per partition; `=1` restores the old
  per-partition shape. Measured 227 -> 428 qps at concurrency 16.
- **Row-id resolution moved after the global top-k merge** (PR #7897) - at most `limit` lookups
  per query instead of per-partition resolution: 4.1 -> 107.7 qps (26x), 3.9 s -> 148 ms on a
  100M-doc benchmark. The cost is that each partition's ROW_ID column is now its own weighed
  index-cache entry (~8 bytes/doc per partition; ~800 MB for a 100M-doc index), and it now
  counts against `index_cache_size_bytes` - budget for it.
- **Deterministic tie ordering** - compound FTS ties are now ordered `_score DESC, _rowid ASC`
  (PR #8073), which also fixed nested `MultiMatchQuery` taking its fetch limit from the ambient
  scanner limit instead of the recursive FTS params (omitting the true top-k under
  MUST/SHOULD/BoostQuery).
- **BREAKING (Rust)**: public `InvertedPartition::bm25_search` was removed, split into private
  `bm25_search_legacy` / `bm25_search_modern` (PR #7863). New legacy indexes are no longer
  written; `metadata.lance` is unchanged. New public
  `MatchQueryExec::new_with_segment_uuids` / `PhraseQueryExec::new_with_segment_uuids` plus an
  `fts_segment_bind_duration` metric (PR #7976).

Tokenizer pipeline (`InvertedIndexParams`): a base tokenizer (`simple`, `whitespace`, `raw`,
`ngram`, `icu`, `icu/split`, `code` (the v9.1 code analyzer / `CodeLexTokenizer` +
`WordDelimiterFilter`, PR #7681 - splits identifiers like `getUserName`/`snake_case`),
`jieba/*` for Chinese, `lindera/*` for Japanese) followed by
token filters (`RemoveLong`, `LowerCase`, `Stemmer`, `StopWords`, `AsciiFolding`) - stop
words can now be **mixed-language** (PR #7324). The `icu` tokenizer
(PR #6956) does ICU4X dictionary-based Unicode word segmentation with **bundled segmenter
data** - unlike jieba/lindera it needs no external language model; the v9 **`icu/split`**
variant (PR #7474) applies ICU segmentation with simple-style delimiter splitting
(`rust/lance-index/src/scalar/inverted/tokenizer.rs:390`, `docs/src/guide/tokenizer.md:1`). The
default base tokenizer remains **`simple`** (`tokenizer.rs:187`); a PR making ICU the default
(#6968) was reverted (#7006) because "ICU showed behavior differences that are too large for
the default path." Config keys:
`base_tokenizer`, `language`, `with_position` (store positions for phrase queries),
`lower_case`, `stem`, `remove_stop_words`, `ascii_folding`, ngram `min_gram`/`max_gram`/
`prefix_only`.

**The 18 stemming / stop-word languages**, in full: Arabic, Danish, Dutch, English, Finnish,
French, German, Greek, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish,
Swedish, Tamil, Turkish (`docs/src/format/index/scalar/fts.md:159`). Anything outside that set
needs a base tokenizer that segments it (`icu`, `jieba/*`, `lindera/*`) rather than a stemmer.

**User dictionaries and custom language models.** The jieba and lindera tokenizers both accept
**user dictionaries** to override segmentation of domain terms, and the guide has a dedicated
"Create your own language model" procedure (`docs/src/guide/tokenizer.md:45,77,89`) for building
a model Lance can load from disk. This is the escape hatch when the bundled Chinese/Japanese
models split your vocabulary wrongly - reach for it before abandoning FTS for ngram.

**Two document types, two tokenization rules.** "Lance supports 2 kinds of documents: text and
json. Different document types have different tokenization rules, and parse tokens in different
format" (`docs/src/format/index/scalar/fts.md:162-163`). FTS over JSON documents breaks them
into `path,type,value` triplet tokens rather than plain terms, so a JSON-typed FTS index and a
text FTS index over the same bytes are not query-compatible. The spec also covers distributed
training for FTS at `fts.md:262`.

Query types (all inexact): `contains_tokens`, `match` (AND/OR), `phrase` (requires
`with_position`), `boolean` (must/should/must_not), `multi_match`, `boost`.

**Compound scoring core (v11, PRs #8092-#8094, #8131, #8299).** Compound FTS queries moved onto
a posting-backed scoring core: an internal `ComposableScorer` protocol, a cross-partition
`TopKCollector`, incremental `WandCursor` support, and `CompoundQueryExec` integration
(`rust/lance-index/src/scalar/inverted/compound.rs`, new at this tag). Semantics to know:

- **Boolean composition** - `SHOULD` clauses sum scores; `MUST` clauses intersect membership
  while preserving the existing first-`MUST` scoring contract; `MUST NOT` filters confirmed
  matches. Concretely, at the query level: "Every query combined with `AND` becomes a scoring
  `MUST` clause: all clauses must match, and every matching clause contributes to the final
  `_score`" (`docs/src/quickstart/full-text-search.md:245-246`) - so adding an `AND` term changes
  ranking, not just filtering.
- **Phrase leaves** use a two-phase `matches()` hook for positional confirmation, so
  **phrase-containing compound queries require an index built `with_position`**.
- **Boost** composes as a nested node inside same-column Boolean/Phrase scorer trees, with
  signed score bounds; match and boost multipliers "are rejected unless finite and non-negative".
- **`CompoundQueryExec` is now public** (`rust/lance/src/io/exec/fts.rs:378-478`) with an
  injectable base scorer and `new_with_segments` / `new_with_segment_uuids` constructors, for
  distributed engines driving FTS themselves.
- Conjunction approximations are ordered by iterator cost (#8299): "Keep scorer children in query
  order so score summation, score bounds, document keys, and ties remain bit-for-bit stable" -
  1.50x fewer comparisons, 1.12x speedup, with results unchanged.
- Conjunction confirmations are ordered by `match_cost` (#8354), measured **200 -> 120**
  two-phase `matches()` calls per query; same-column `MUST + SHOULD` scores the optional side
  lazily, "only touch[ing] the optional side when it can change competitiveness or an exact
  score is requested" (#8448).

**When the posting-backed scorer bails out.** `CompoundQueryExec` refuses the fast path and
reverts to the materializing hash-join plan whenever the table has **any unindexed fragment**,
the query spans **more than one column**, granularity is **`ListElement`**, or the overlay plan
is not `Unchanged` (`rust/lance/src/dataset/scanner.rs:3940-3948`). The first condition is the
one that bites in practice: a table with an unindexed tail silently gets the slower plan, so
benchmark compound FTS only after the backlog is folded in.

#### Inspecting tokenization (v11)

`lance.tokenize(...)` previews the tokens a full-text query will produce **without creating a
dataset or index** (#8415): "Use `lance.tokenize` to inspect the tokens that a full-text query
will produce without creating a dataset or index" (`docs/src/guide/tokenizer.md:17-18`). It
returns `lance.FtsToken(text, position)`. On the query side, `analyze_plan` appends
`tokenized_query=` to FTS leaves (#8414); `explain_plan` is deliberately unchanged.

**`max_token_length` is the one tri-state option.** Every other analyzer option treats `None` as
"use the analyzer default"; this one does not: "omitting it keeps the default length limit of
40, while `max_token_length=None` disables the limit" (`tokenizer.md:45-46`).

**`InvertedIndexParams` is a closed set.** It exposes base-tokenizer selection, ngram range,
`lower_case`, `ascii_folding`, and stemming - there is **no** hook for a custom token filter and
**no non-English stemmer**. If your corpus needs either (camelCase splitting, a non-English
language), Lance FTS cannot be configured for it; the options are to pre-tokenize into your own
column or to use ngrams.

**The `Fixed32` empty-segment merge failure.** A fold whose batch contributes zero tokens - an
all-NULL tail, but also empty or whitespace-only documents - still writes a delta segment. On
read-back, `posting_tail_codec()` aggregates over partitions and a partitionless segment falls
back to `PostingTailCodec::default()` (`VarintDelta`) rather than reading its file metadata
(`rust/lance-index/src/scalar/inverted/index/inverted_index.rs:72`). The root cause is two
disagreeing defaults: the `Default` impl is `VarintDelta` while `parse_posting_tail_codec`
defaults to `Fixed32` when the key is absent. **Scope: this only bites `Fixed32` indexes** - FTS
`format_version=1` or legacy indexes missing the `posting_tail_codec` key. The V2 default is
`VarintDelta`, which happens to match the `Default` impl, so a default-configured FTS index never
trips it. Where it applies, once delta segments reach the merge threshold with an empty one among
them, `merge_segments` fails deterministically every time. Guard by skipping the fold when the
unindexed tail has no non-null, non-empty values.

**The `fts()` table function does not declare `_score`.** `FtsTableProvider` declares its logical
schema as dataset columns plus optional `_rowid`/`_rowaddr` and never `_score`
(`rust/lance/src/dataset/udtf.rs:39`), while the scan's scoring autoprojection force-appends
`_score` physically. So `COUNT(*)` and `GROUP BY` over `fts()` are **rejected by DataFusion**
(the logical and physical column counts disagree), and `_score` cannot be named in a projection
or ordered by. Plain `SELECT` shapes work only because nothing downstream validates - the
`_score` you see is the undeclared column leaking through.

### 11.4 Geo / RTree

New in v7. The **RTREE** index is a static immutable 2D spatial index on bounding boxes
(`docs/src/format/index/scalar/rtree.md`): a multi-level packed hierarchy, items sorted by
**Hilbert curve** value, leaf pages of `(bbox, rowid)`. Files: `page_data.lance` (all pages)
and `nulls.lance`. Accelerated queries return a candidate set (AtMost), with exact geometry
verification done by the engine: `Intersects`, `Contains`, `Within`, `Touches`, `Crosses`,
`Overlaps`, `Covers`, `CoveredBy`.

The **`lance-geo`** crate provides geospatial UDFs registered into a DataFusion context -
measurement (`Area`, `Distance`, `Length`), relationships (`Contains`, `Intersects`,
`Within`, ...), and validation (`IsValid`) - via `geodatafusion` and `geoarrow`. Gated behind
the `geo` feature.

### 11.5 Index updates and reindexing

There is no monolithic delta-index format - new data is folded in at the **segment** level. A
new index segment covers new fragments; engines query indexed and unindexed subplans and
merge. In-place updates to an indexed column remove the affected fragment IDs from the
covering segment's `fragment_bitmap`, flagging them for reindexing. **Since v11 that rule keys
off any column in `fields`, not just the keyed one**: the engine "must remove the affected
fragment IDs from the `fragment_bitmap` field of any index segment whose `fields` include that
column - whether the index is keyed on it or merely carries it." The same widening applies to
overlay exclusion, where the field range "is every field in the index's `fields`, not only the
ones it is keyed on."

After compaction, three strategies handle changed row addresses: do nothing (segment stops
covering those fragments), rewrite segments with remapped addresses, or use a **fragment reuse
index** (remap in memory at read time). Stable row IDs avoid remapping entirely at the cost of a
lookup.

**Address-domain indexes under compaction on a stable-row-id dataset** (v11 fix). Stable row IDs
keep *row ids* stable across a rewrite, but not row *addresses* - so an index whose payload is
address-domain cannot simply follow its data. The Rewrite commit path now branches on
`index.results_are_row_addrs()`: a row-id-domain index gets `recalculate_fragment_bitmap` and
follows its data to the new fragments, while an address-domain one gets
`drop_rewritten_fragments`, because "its stored addresses point into the fragments the rewrite
dropped. Claiming coverage of the new fragments would make it answer queries with addresses that
no longer resolve, so drop the rewritten fragments from its coverage instead and let the scanner
fall back to a full scan for them"
(`rust/lance-table/src/transaction/manifest_build.rs:818-833`).

ZoneMap is the address-domain case: `results_are_row_addresses() -> true`, `can_remap() ->
false`, and `remap()` returns `"ZoneMapIndex does not support remap"`
(`rust/lance-index/src/scalar/zonemap.rs:732,736,743`). Two consequences worth planning around:
before v11 the bitmap was advanced unconditionally, so a compacted ZoneMap claimed fragments its
zones had no entries for - **that damage is not repaired by upgrading, and such an index must be
recreated**. After v11 the compacted fragments read as *unindexed* instead, so an incremental
`optimize_indices(append)` picks them up on the next maintenance pass rather than no-oping.

The caller-facing API for folding new data in is `optimize_indices(&OptimizeOptions)`
(`rust/lance/src/index/api.rs:297`). `OptimizeOptions` (`rust/lance-index/src/optimize.rs:65`)
has three constructors: `append()` adds a new delta segment over the new fragments;
`merge(N)` folds the delta updates plus the latest N segments into one; `retrain()` rebuilds
the whole index from current data (v3 vector indices only). This is incremental maintenance -
distinct from dropping and recreating an index from scratch.

**`append()` never collapses anything.** It sets `num_indices_to_merge: Some(0)`
(`rust/lance-index/src/optimize.rs:79-80`), and the merge path short-circuits on
`num_to_merge == 0` (`rust/lance/src/index/append.rs:408`) - "append mode (`num_to_merge == 0`)
defers cleanup to a real merge". The *default* `OptimizeOptions` is more useful than that:
`num_indices_to_merge` unset means `unwrap_or(1)`, so a plain `optimize_indices` collapses the
trailing segment on every call (`append.rs:400-403`). `retrain: true` ignores
`num_indices_to_merge` entirely and merges everything into one. Critically, **no automatic
count- or size-based collapse threshold exists anywhere in the codebase** - nothing stops delta
segments accumulating if a pipeline only ever calls `append()`. Bounding segment count is the
caller's responsibility; see `performance.md` Part B for the operational consequences.

**A fragment reuse index is not per-index coverage.** `unindexed_fragments` is computed purely
from the union of each index's own `fragment_bitmap` and does not consult the FRI at all
(`rust/lance/src/index.rs:2977-2985`). With `defer_index_remap = true` the remapper is `None`,
so `rewritten_indices` is empty and `handle_rewrite_indices` is a no-op - the per-index bitmaps
keep the *old* fragment ids while the FRI is appended separately as its own index entry
(`rust/lance-table/src/transaction/index_maintenance.rs:338`). Consequence: deferring remap does **not**
stop `unindexed_fragments(<idx>)` from reporting the new fragments, so a "is my index caught up?"
check built on that call will still see churn after every compaction. The FRI removes the cost of
*rewriting* index entries, not the bookkeeping that says an index does not cover a fragment.

**The FRI has an operational cost model the spec page states and this file does not restate.**
`references/docs/format/index/system/frag_reuse.md` covers index *load* cost - every reader pays
remap work proportional to the accumulated history - and the cleanup duty. An FRI left untrimmed
is a slow, silent tax on open, so read that page before running compaction continuously.

**The cleanup rule changed at v13 and is stricter than the old one.** The v11-era text ("Once all
scalar and vector indices have been rebuilt past a given reuse version, that version is no longer
needed and can be trimmed") is **gone**. Rebuilding past a version is no longer sufficient on its
own: "Cleanup must retain intermediate transitions still needed to translate old addresses", and
a second rule now governs the external payloads - "External mapping files can be deleted only
when no retained dataset version references them." A trimming job written against the old
sentence can delete a transition a retained version still needs.

**The FRI became a versioned format at v13** (#9136), with a second use case. `InlineContent`
field 1 is now `legacy_versions` and tagged `transitions` arrived at field 2 under
`index_version >= 1`, as a oneof of `OrderedCompaction` or `StablePartition`. The new one covers
**reclustering**: "A stable partition assigns source rows to destination fragments while
preserving their relative source order within each destination. This lets FRI reuse existing
indices after reclustering." Its payload is an immutable row-map Lance file with `uint16` labels
and an `LSPC`-magic counts matrix. Two constraints to plan around: "Tables using stable row IDs
do not support tagged histories; writers must not publish `index_version >= 1` on them", and the
advertising manifest bit (1024) is documented but not implemented - section 7. Upstream also
walked back the conflict claim: "FRI does not remove conflicts between overlapping rewrites."

---

## 12. Distributed write and indexing

Lance exposes APIs for distributed work but provides **no scheduler** - the caller drives the
workflow (Ray and Spark integrations exist for the common cases). How a table is handed to a
remote executor is `TableIdentifier` (`protos/table_identifier.proto`) - two modes, see
`ops.md` section 16.

**Substrait.** `lance-datafusion` parses Substrait filter and aggregate expressions
(`parse_substrait` / `parse_substrait_aggregate`, `rust/lance-datafusion/src/substrait.rs:385,473`),
exposed through the Python bindings and the filtered-read exec node. This is the path for
handing Lance a language-agnostic serialized predicate rather than a SQL string - useful when
the planner producing the filter is not itself Rust or Python.

**Two-phase distributed write** (`docs/src/guide/distributed_write.md`):

1. **Parallel writes** - each worker generates `LanceFragment`s in parallel via
   `write_fragments(data, data_uri, schema=)`, returning `FragmentMetadata`.
2. **Commit** - gather all `FragmentMetadata` on one worker, serialize via
   `FragmentMetadata.to_json`/`from_json`, and commit in a single `LanceOperation`
   (`Overwrite`, `Append`, `Merge`, or `Update`) via `lance.LanceDataset.commit(uri, op,
   read_version=)`.

**Distributed indexing** (`docs/src/guide/distributed_indexing.md`). v8 unifies **all** index
builds onto one segment-based lifecycle: workers each build an index **segment** for a
fragment subset via `create_index_builder(...).fragments(...).execute_uncommitted()` (Rust;
`create_index_uncommitted(..., fragment_ids=)` in Python), writing under
`_indices/<segment_uuid>/`. The caller then commits segments as-is via
`commit_existing_index_segments(...)` or groups and merges them via
`merge_existing_index_segments(...)`. Within one commit, segments must have **disjoint
fragment coverage**. Uncommitted staging directories are cleaned by `cleanup_old_versions`.
The standalone `IndexSegmentBuilder` API (and its `build_all()` / `target_segment_bytes`
size-based grouping) was **removed in v8** from Rust, Python, and Java (PR #6997); use the
`execute_uncommitted` path above. Distributed BTree and bitmap builds were folded into this
same framework (PR #7013, #6869) - the old Python Bitmap shard path
(`create_scalar_index(..., fragment_ids=)` + `merge_index_metadata(..., "BITMAP")`) is gone.

**Segment grouping is a separate decision from worker build, and it is yours to make.** The guide
is explicit that the two are not the same knob: "The grouping decision is separate from worker
build. Workers only build segments; Lance applies the segment build policy when it plans physical
segments." Grouping sets commit granularity - how many logical builds land per physical segment -
so it drives both commit count and later merge cost. Full treatment in
`references/docs/guide/distributed_indexing.md` ("Segment Grouping").

**v10 extends the segment lifecycle to the rest of the scalar family.** BLOOMFILTER (PR #7925),
RTREE (PR #7932), NGRAM (PR #7244), and LABEL_LIST (PR #7884) all gained segment-native
build/merge/commit, giving Python this cumulative segment-native set:
`BTREE, BITMAP, INVERTED, FTS, NGRAM, RTREE, ZONEMAP, BLOOMFILTER, LABEL_LIST`
(`python/python/lance/dataset.py:3301`). Consequences:

- **`IndexSegment::new` is BREAKING** - 4 params to 6 (adds `fields` and `dataset_version`) plus
  a second generic; `into_parts` widens from a 4-tuple to a 6-tuple; new accessors
  `fields() -> &[i32]` and `dataset_version() -> u64` (`rust/lance/src/index/api.rs:74,95`).
- Merged segments now inherit the **minimum** source `dataset_version` rather than the current
  manifest version (PR #7925) - so a merged segment's coverage claim stays honest.
- **NGRAM has a hard ordering constraint**: "NGRAM segments built before a deferred compaction
  must be merged before commit so their postings can be rebuilt against current row addresses"
  (`python/python/lance/dataset.py:4305`). Creating an NGram index also now raises a *retryable*
  conflict against a concurrent deferred rewrite over overlapping fragments.
- Segment commit now issues its LIST calls concurrently (`buffered(io_parallelism)`,
  `rust/lance/src/index.rs:319`, PR #7657) - measured ~8x faster against remote storage
  (2837 ms -> 350 ms at 128 segments / 20 ms RTT).

`merge_existing_index_segments(...)` "currently supports vector, inverted, bitmap, BTree, and
zone map segments" (`distributed_indexing.md:109-110`); other scalar families can still be
committed without merging. **Vector model scope**: workers may share one trained IVF/PQ model
*or* use **independent segment models** - "each worker trains the IVF/PQ model for its own
`fragment_ids`. The resulting segments can be committed together as one logical index without
sharing centroids or codebooks" (`distributed_indexing.md:124`, PR #7148). Distributed builds
cover vector indexes, bitmap, segmented btree, segmented inverted (FTS), zone map, and
**LabelList** (added in v9, PR #7223); the
`filtered_read` proto serializes the `FilteredReadExec` scan operator for plan-then-execute
distributed scans.

lance-reference.md 1.1 KB

# Lance v12 reference - section index

The reference is split across five files. Cross-references written as "section N" (in these
files, in `performance.md`, and in `SKILL.md`) use the original 16-section numbering:

| Sections | File |
|----------|------|
| 1 What Lance is, 2 Crate workspace, 3 File format, 4 Data types | `format-file.md` |
| 5 Table format, 6 Schema evolution, 7 Versioning/tags/branches, 8 Row IDs, 9 Transactions, 10 MemWAL | `format-table.md` |
| 11 Indexes, 12 Distributed write and indexing | `indexes.md` |
| 13 Object store, 15 Capability matrix, 16 Source map | `ops.md` |
| 14 What changed (v7 -> v13) | `changelog-v7-v13.md` |

Each file keeps the original section headings and its own table of contents.

Citations are `path:line` relative to the repo root. Build a permalink as
`https://github.com/lance-format/lance/blob/v13.0.0-beta.4/<path>`. Line numbers drift
between tags; treat them as approximate. The authoritative in-repo sources are the format
spec under `docs/src/format/`, the user guide under `docs/src/guide/`, the protobuf schemas
under `protos/`, and the Rust workspace under `rust/`.

maintenance.md 5.8 KB

# Maintaining this skill

Citations across the reference files are `path:line` relative to the `lance-format/lance` repo;
build a permalink as `https://github.com/lance-format/lance/blob/v13.0.0-beta.4/<path>`. Line
numbers drift between tags - treat them as approximate.

To refresh: `git -C <your lance-format/lance clone> fetch --tags`, then read the newest tag (the
major may have jumped again - the release train re-roots on any `breaking-change` label, and it
has now fired on **four consecutive lines**, so sort tags by date rather than assuming the
current major, and do not assume an intermediate `.1` line ever got a final or even a beta tag).
Check for a **final** too: `v12.0.0` shipped the same week as `v13.0.0-beta.4`, so the stable pin
and the tracked frontier can both move in one refresh.

Three traps worth knowing before you start:

- **The `breaking-change` label is a floor, not a ceiling.** It drives the release bot, so it is
  the right query for "did the major re-root" - but it does **not** enumerate what will break a
  consumer. The `v12.0.0-beta.6 -> beta.15` range is the worked example: exactly two labeled PRs,
  while the two changes most likely to bite (`stable` resolving to 2.2, #8657; the IVF_RQ 5-bit
  default, #8936) carried a conventional-commit `!` and no label. **Always also scan the range for
  `!` commits** (`git log --oneline <a>..<b> | grep '!'`) and diff the defaults you already
  document - version enums, `*_DEFAULT_*` consts, and the sizing formulas in
  `docs/src/guide/performance.md`. The rule is a floor in both directions: in the
  `v12.0.0-beta.15 -> v13.0.0-beta.4` window all three `!` commits (#7465, #9192, #9101) *did*
  carry the label, so the two signals agreeing is not evidence that either is complete. The
  defaults diff is what caught `inline_optimization_enabled` flipping `true -> false` (#9180),
  which carried neither marker.
- **The spec pages can run ahead of the code.** `FLAG_FRAGMENT_REUSE_INDEX` (1024) is documented
  in `format/table/versioning.md` as reader/writer `Yes`, with the unknown boundary at 2048,
  while `rust/lance-table/src/feature_flags.rs` declares it *above* `FLAG_UNKNOWN` (512) and
  never reads it again. When a doc table and a `const` disagree, **verify which one the build
  enforces** - here `supported_flags() = FLAG_UNKNOWN - 1` settles it - and record the split
  rather than copying the table.
- **A final is not on `main`.** Finals are cut on stabilization branches, so
  `git merge-base --is-ancestor <final> main` returns false for a perfectly official release.
  Check GitHub Releases / crates.io / PyPI to identify the stable pin, not ancestry.
- **You do not need to move `HEAD` to read a tag.** `git show <tag>:<path>`,
  `git grep <pattern> <tag> -- <path>`, `git diff <tagA> <tagB> -- <path>` and
  `git ls-tree -r <tag>` are all read-only, which matters when the clone is shared. Note that a
  `blob:none` partial clone cannot always produce arbitrary historical diffs (`git log -S` may
  fail on absent blobs); tag-to-tag reads of present files still work.

Then:

1. Re-copy the docs mirror: the `.md` files of `docs/src/{guide,quickstart}`,
   `docs/src/format` (plus the `format/index/*.svg` diagrams), and
   `docs/src/integrations/datafusion.md` into `references/docs/`, preserving the tree.
   Update the directory counts in `SKILL.md` if docs were added or removed. **Three** deviations
   from upstream bytes are expected and enforced by this repo's pre-commit hooks, not drift:
   trailing whitespace is stripped from every file; `end-of-file-fixer` removes the trailing
   blank line that `format/table/layout.md` and `format/table/row_id_lineage.md` carry upstream;
   and the same hook *adds* a trailing newline to **all four** `.drawio.svg` diagrams, each of
   which is therefore one byte larger in the mirror than upstream (`indices-compaction`
   51381 -> 51382, `scalar_index` 8210 -> 8211, `starter-example` 14622 -> 14623,
   `indices-fragment handling` 21856 -> 21857).
   Normalize all three before diffing the mirror against a new tag. In practice both
   normalizations reduce to: strip trailing whitespace from every line, then force exactly one
   trailing newline on every file - applying that to a fresh `git archive` of `docs/src`
   reproduces the committed mirror byte-for-byte.
2. Re-check Part A of `references/performance.md` against the new `guide/performance.md` and the
   other perf-bearing sections it routes to. Part A deliberately does **not** copy that text -
   it points at `references/docs/`, so a mirror refresh updates it automatically; what needs
   hand-editing is the provenance note (which tags the guide is byte-unchanged across) and the
   two source-derived "Performance changes not in the guide" subsections. Part B (field-verified
   practices) is experience-derived - only edit it with new *measured* results, never
   speculation.
3. Re-verify the crate workspace and re-read the format spec for the reference files
   (`format-file.md`, `format-table.md`, `indexes.md`, `ops.md`, `changelog-v7-v13.md`), then
   bump `metadata.upstream` plus every current-tag version reference. `lance-reference.md` is
   only the section-number index - update it if the split changes.

## Reference file layout

`references/lance-reference.md` maps the original 16 section numbers onto five files:

| Sections | File |
|----------|------|
| 1-4 (what Lance is, crates, file format, data types) | `format-file.md` |
| 5-10 (table format, schema evolution, versioning, row IDs, transactions, MemWAL) | `format-table.md` |
| 11-12 (indexes, distributed write/indexing) | `indexes.md` |
| 13, 15, 16 (object store, capability matrix, source map) | `ops.md` |
| 14 (v7 -> v13 delta) | `changelog-v7-v13.md` |

Cross-references inside those files are written as "section N" against the original numbering,
so the index file must stay in sync with any re-split.

ops.md 25.5 KB

# Lance v12 reference - object store, capabilities, source map (sections 13, 15, 16)

Part of the Lance v13 reference (`lance-format/lance@v13.0.0-beta.4`). Citations are `path:line`
relative to the repo root; build a permalink as
`https://github.com/lance-format/lance/blob/v13.0.0-beta.4/<path>`. Line numbers drift between
tags - treat them as approximate. Cross-references written as "section N" use the original
16-section numbering; `lance-reference.md` maps every number to its file.

## Contents

- [13. Object store](#13-object-store) - URI schemes, `storage_options`, per-backend config,
  commit handlers per backend, `LANCE_*` env vars
- [15. Capability matrix](#15-capability-matrix) - what Lance can and cannot do
- [16. Source map](#16-source-map) - where each spec, proto, and crate lives in the repo

Other files: `format-file.md` (1-4), `format-table.md` (5-10), `indexes.md` (11-12),
`changelog-v7-v13.md` (14).

---

## 13. Object store

The object store is chosen by URI scheme (`docs/src/guide/object_store.md`): `s3://`,
`s3+ddb://` (S3 + DynamoDB commits), `gs://`, `az://` / `abfss://`, `oss://` (Alibaba),
`cos://` (Tencent), `tos://` (Volcengine, new in v8), `goosefs://` (feature-gated `goosefs`,
new in v8), `hf://` (Hugging Face, read-oriented, new in v13), `file://`, `file+uring://`,
`memory://`, `shared-memory://` (in-memory, cross-component).

**`hf://` has an opt-in resolve cache with a staleness hazard** (#9236). Setting
`storage_options={"hf_enable_resolve_cache": "true"}` "reuses resolved HTTP download URLs and
XET file metadata across readers"; it "defaults to `"false"`". The caveat is a correctness one,
not a tuning one: "Enable the resolve cache only when existing files will not change. Updates,
including changes behind a moving branch or tag, may remain invisible while cached results are
reused." A moving branch or tag is exactly the common Hugging Face setup, so leave it off unless
you are pinned to an immutable revision.

The underlying stack moved in the `v12.0.0` final: `object_store` 0.13 -> **0.14** and OpenDAL
-> **0.59** (#9123), with OpenDAL reaching 0.59.2 in the v13 line. Anything depending on
`object_store` types transitively needs the matching bump.

`file+uring://` is a **local** store, not a remote one: `is_local()` returns true for both
`file` and `file+uring` (`rust/lance-io/src/object_store.rs:630`), and
`is_uring()` distinguishes it (`:653`). Any rule stated for "the local store" therefore covers
it - a scheme check written as `scheme == "file"` silently excludes uring stores.
Config comes from environment variables or the `storage_options` map passed to
`lance.dataset` / `lance.write_dataset`.

`shared-memory://` is opt-in and distinct from `memory://`: `memory://` mints a fresh
in-memory store per call, while `shared-memory://<authority>` resolves - across object-store
registries, threads, and unrelated components in the same process - to one process-global
`InMemory` backend keyed by the URL authority. The pool is never evicted and grows for the
process lifetime; it is meant for tests and harnesses that coordinate a writer and an
independent reader. Pick distinct authorities for isolation
(`rust/lance-io/src/object_store/providers/shared_memory.rs:16`).

General options: `allow_http` (default false), `connect_timeout` (5s), `timeout` (30s),
`client_max_retries` (3), `download_retry_count` (3), `proxy_url`, `user_agent`.

**`request_timeout` is not a key** - it appears in no Lance source at any recent tag, and #8706
corrected the docs to `timeout`. Anything still passing `request_timeout` is silently setting
nothing. `timeout` bounds the *entire* request, from connection until the response body
finishes, and applies per individual request - so on a large write it must cover one complete
multipart part upload. Raise it alongside `LANCE_INITIAL_UPLOAD_SIZE`.

**Multipart part sizing** - `LANCE_INITIAL_UPLOAD_SIZE` sets the first part's size, clamped to
5 MB..5 GB (values outside the range are clamped with a warning, not rejected;
`rust/lance-io/src/object_writer.rs:55`).

**Bulk copy strategy** (v12, #8770). Set `LANCE_IO_SERVER_SIDE_COPY_ENABLED` to a truthy value
(`1`, `true`, `on`, `yes`, `y`, case-insensitive) to route cloud copies whose source and
destination share the same object-store client into the provider-native server-side copy
operation instead of streaming bytes through the client - the relevant case for index movement
and for copying between two prefixes of one bucket. Deep clone separately bounds non-local file
movement to four concurrent files; `LANCE_DEEP_CLONE_STREAM_CONCURRENCY` overrides that
operation-specific limit (`rust/lance/src/dataset.rs:3352`).

Per-backend highlights:

- **S3** - `aws_region`, `access_key_id` / `secret_access_key` / `session_token`,
  `aws_endpoint` (for S3-compatible stores like MinIO - both region and endpoint required),
  `aws_server_side_encryption` (`AES256` / `aws:kms` / `aws:kms:dsse`) + `aws_sse_kms_key_id`.
  `AWS_PROFILE` is environment-only. New in v11: **`aws_provider_scheme`** (PR #8103) pins a
  dataset to one credential provider instead of the default chain - "useful when two datasets in
  the same process need different AWS auth (for example, one bucket using IRSA and another using
  ECS container credentials)" (`docs/src/guide/object_store.md:118-121`). Exactly three values,
  each failing hard rather than falling back: `token` (static credentials), `ecs` (container
  credentials), and `irsa`, which "Reads `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` from
  the environment" (`:127`). Anything else errors with "Invalid aws_provider_scheme '{}'. Valid
  values are: token, ecs, irsa" (`rust/lance-io/src/object_store/providers/aws.rs:550`).
- **S3-compatible endpoints: set the region explicitly, and know why.** The docs state the
  requirement plainly - `aws_region` "must be specified for S3-compatible stores"
  (`object_store.md:102`) - because SigV4 embeds the region in the signed credential scope even
  when the endpoint is not AWS. But the code does **not** enforce it: `resolve_s3_region` returns
  `Ok(None)` precisely in the endpoint-without-region case (`aws.rs:239,264`), and the builder
  then applies `DEFAULT_REGION = "us-west-2"` (`aws.rs:316`, applied at `:103`). There is no
  error path for a missing region, so an omitted `aws_region` **silently signs with `us-west-2`**
  rather than failing fast - producing signature errors, or worse, quiet misrouting on providers
  that accept any region string. Third-party endpoints also usually need
  `virtual_hosted_style_request` (or `aws_virtual_hosted_style_request`) set explicitly - both
  spellings work because the key is passed straight through to the `object_store` crate
  (`aws.rs:99,535`); it defaults to `False` (`object_store.md:107`).
- **S3 Express** - directory buckets; auto-recognized via the `--x-s3` suffix, or set
  `s3_express: "true"`; reachable only from a same-region EC2 instance. Its listing is not
  lexically ordered, so the `latest_version_hint.json` mechanism accelerates latest-version
  lookup there.
- **GCS** - `GOOGLE_SERVICE_ACCOUNT` (JSON file) or `service_account_key`. Default HTTP/1;
  `HTTP1_ONLY=false` for HTTP/2.
- **Azure** - `account_name` / `account_key`, service principal, SAS tokens, managed
  identity, workload-identity federation.
- **Alibaba OSS** - `oss_endpoint` (required), `oss_access_key_id`, `oss_secret_access_key`.
- **Tencent COS** (`object_store.md:333`) - `cos://bucket/path` with `cos_endpoint`,
  `cos_secret_id`, `cos_secret_key`, and optional `cos_enable_versioning`; env vars are read
  from the `COS_` or `TENCENTCLOUD_` prefixes.
- **Volcengine TOS** (new in v8, `object_store.md:303`) - `tos://bucket/path` with
  `tos_endpoint` required (e.g. `https://tos-cn-beijing.volces.com`), plus `tos_region` and
  access-key options.
- **GooseFS** (new in v8, feature-gated `goosefs`, now documented at `object_store.md:396`) -
  `goosefs://host:port/path`; master address comes from `goosefs_master_addr` (HA-aware:
  `"addr1:port,addr2:port"`), the URL host, or default port `9200`. Optional keys:
  `goosefs_write_type` (`MUST_CACHE` / `CACHE_THROUGH` / `THROUGH` / `ASYNC_THROUGH`),
  `goosefs_auth_type` (`nosasl` / `simple`), `goosefs_auth_username`, `goosefs_block_size`,
  `goosefs_chunk_size` (`rust/lance-io/src/object_store/providers/goosefs.rs:24-61`).

  **`storage_options` keys must be lowercase** (#8940, `v12.0.0-beta.12`) - a wrong-case key is
  now a hard error rather than a silently ignored value: "Uppercase or mixed-case spellings such
  as `GOOSEFS_MASTER_ADDR` are rejected with an explicit error - they are not ignored, and they
  are not treated as the matching environment variable" (`object_store.md:547-549`). A config
  that appeared to work by accident will start failing loudly on upgrade.

  **`goosefs_block_size` / `goosefs_chunk_size` accept unit suffixes** (#8943): "Accepts a raw
  byte count or GooseFS suffixes such as `64MB` (binary units: `1KB = 1024`). Optional."
  (`object_store.md:556`).
  **In v11 GooseFS commits became safe** (PR #8134): manifest commits now use
  `ConditionalPutCommitHandler` (`PutMode::Create` / if-not-exists), "backed by GooseFS master's
  atomic no-replace rename so concurrent writers cannot clobber each other's versioned
  manifests" (`object_store.md:404-407`), replacing `UnsafeCommitHandler`. **Mixed-version
  writers are a data-loss hazard during the rollout**: "The `if-not-exists` guarantee only holds
  when **every** writer for a dataset routes through this new handler. A writer running an older
  Lance release still selects `UnsafeCommitHandler` for `goosefs://` and writes the version path
  unconditionally, which can overwrite a manifest that an upgraded writer has already won"
  (`:382-386`). Upgrade all writers before relying on it.

**Multipart upload retries (v11, PR #8174).** A failed part upload used to be retried by calling
`MultipartUpload::put_part` again, but "Native cloud stores allocate a new part number when that
method is called, so the retry skipped the failed part and completion reported `Missing part`."
Retries now happen inside the HTTP connector, preserving part identity, for native S3/Azure/GCS;
"OpenDAL stores retain their existing behavior and are outside this repair." The
`LANCE_CONN_RESET_RETRIES` env var (default 20) was removed along with the old writer-level
resubmission path.

**`LANCE_*` changes in the v11 range.** One removal (`LANCE_CONN_RESET_RETRIES`, above) and two
additions, both from the AMX-FP16 work in `beta.16` (PR #8540):

| Variable | Scope | Effect |
|----------|-------|--------|
| `LANCE_DISABLE_AMX` | runtime | Kill switch for the AMX-FP16 paths. Also reverts IVF partition assignment to the approximate path, so an index built with it set is **not** equivalent to one built without it |
| `LANCE_AMX_FP16_CC` | build time | Overrides the compiler used to build the AMX kernel (`rust/lance-linalg/build.rs:27`); the kernel needs clang >= 16 or gcc >= 13 |

Grep trap: `LANCE_AMX_CFG_SEARCH`, `LANCE_AMX_CFG_GEMM`, and `LANCE_AMX_TILE_COUNT` look like
env vars in a tree-wide `LANCE_[A-Z_]*` grep but are **C preprocessor macros** in
`rust/lance-linalg/src/simd/amx_fp16.c:107-137`. They are not readable from the environment.

**Base-aware access (v7).** `Dataset::object_store` takes an `Option<u32>` base id - `None`
for the primary store, `Some(base_id)` for an additional base. Caching/instrumentation
wrappers are applied per `store_prefix` and propagate to all base stores.

**Per-base `storage_options` (v9, PR #7608).** For multi-base datasets you can scope a storage
option to one base with a `base_<id>.<key>` key (`docs/src/guide/object_store.md:44-70`): "A
storage option key of the form `base_<id>.<key>` applies `<key>` only to the base path with
that manifest id. Every base inherits the unscoped options; base-scoped entries add to or
override them." Base ids are assigned when bases are registered (`initial_bases` ids "assigned
sequentially starting at 1"); keys that don't match the pattern exactly (e.g. `base_url`) are
treated as regular options. Precedence: an exact per-base parameter map (`base_store_params`,
keyed by base-path URI) beats a `base_<id>.<key>` scoped key.

**v10 object-store and runtime changes.**

- **`memory://` datasets could spuriously fail** with `DatasetNotFound` in optimized builds:
  `ObjectStoreParams` Hash/Eq keyed on a trait-object fat pointer, and "Trait object pointers
  include vtable metadata, which is not stable across codegen units. Cache identity must follow
  the Arc allocation instead" (PR #8068).
- **No more panic on tokio runtime shutdown** mid-read - an in-flight parallel read now returns
  an I/O error, "I/O request was dropped before completion ({} of {} reads delivered)"
  (`rust/lance-io/src/scheduler.rs`, PR #7478).
- **`LANCE_CPU_THREADS` and `LANCE_IO_CORE_RESERVATION` are now validated** instead of
  `.parse().unwrap()`-panicking on garbage (`rust/lance-core/src/utils/tokio.rs:50-70`,
  PR #7856). `LANCE_CPU_THREADS` must be at least 1; `LANCE_IO_CORE_RESERVATION` still allows 0
  (reserve no cores for IO); unset still defaults to 2.
- **Namespace behavior change worth auditing call sites for**: the directory namespace no longer
  collapses storage failures into `TableNotFound`. Upstream's motivation - "During a stress run
  on a popular cloud provider, 503 errors when listing objects failed and the dir namespace
  reported the affected tables as non-existent" - meant a create-or-open caller could **overwrite
  a live table** because a transient listing error read as "does not exist". Throttles and 5xx
  now surface as `Throttling` (21) / `ServiceUnavailable` (17) / `Internal` (PR #7931). Callers
  catching `TableNotFound` (4) to mean "absent" must be updated. Alongside it,
  `create_table_version` enforces strict version CAS (only `latest+1`) and is idempotent on
  retry when the resubmitted manifest content matches; `declare_table` returns
  `TableAlreadyExists` when `.lance-reserved` exists; and directory-namespace `query_table` now
  honors `structured_query` FTS, which was previously **silently ignored** - "a `structured_query`
  was silently ignored, so the scan ran with no FTS filter and returned all rows" (PR #7592).
- **`lance-namespace` 0.8.5 -> 0.11.1 at v12** (PR #8903; `python/pyproject.toml` now pins
  `lance-namespace>=0.11.1,<0.12`, Java moved 0.7.7 -> 0.11.1). Four `LanceNamespace` methods
  return a response object instead of a bare value, and callers must unwrap:
  `count_table_rows` -> `CountTableRowsResponse` (`.count` / `.getCount()`), `query_table` ->
  `QueryTableResponse` (`.data` / `.getData()`), `namespace_exists` -> `NamespaceExistsResponse`,
  `table_exists` -> `TableExistsResponse`. Anyone implementing `LanceNamespace` themselves needs
  the same signature updates. Note the Python side is not type-checked against the ABC -
  `python/lance/namespace.py` is outside the pyright allowlist - so a missed unwrap surfaces at
  runtime, not at check time.
- **Latest-version resolution stopped listing the whole prefix** (v12, PR #8679). It previously
  walked all of `_versions/`: on a ~340k-version table that is ~344 sequential `ListObjectsV2`
  pages, "~25s of pure I/O wait", paid by every `open_table` / `describe` / `merge_insert` that
  resolves the latest version. Heals purely on upgrade.

`latest_version_hint.json` (`{"version": N}` under `_versions/`) gives fast latest-version
lookup on stores where listing is not lexicographically ordered (S3 Express, local FS); it is
purely an optimization, always safe to delete, and skipped where listing is already ordered.
Disable globally with `LANCE_USE_VERSION_HINT=0`.

---

## 15. Capability matrix

What Lance can and cannot do at `v13.0.0-beta.4`.

**Storage and format**

| Capability | Status |
|------------|--------|
| Local FS, S3 (+ S3-compatible), S3 Express, GCS, Azure, Alibaba OSS, Tencent COS, Volcengine TOS | yes |
| GooseFS (`goosefs://`) | yes (feature-gated `goosefs`) |
| In-memory store (`memory://`, `shared-memory://`) | yes |
| Multi-base storage (hot/cold, multi-region, shallow clone) | yes (`FLAG_BASE_PATHS`) |
| File format 2.1, 2.0, legacy 0.1 (read-only) | yes |
| File format 2.2 (Map type, Blob v2) - **the current default** | yes, stable |
| Mixed exact V2 versions within one dataset | yes (`FLAG_MIXED_DATA_FILE_VERSIONS`, v12 final) |
| File format 2.3 sparse structural pages (auto-selected, or forced via `structural-encoding=sparse`) | yes, but `next` / unstable |
| Concurrent writes on plain `s3://` | yes (native conditional PUT) |
| Concurrent writes - GCS / Azure / local | yes |

**Data and schema**

| Capability | Status |
|------------|--------|
| Full Arrow type system, nested structs/lists | yes |
| Map type | yes (file format 2.2) |
| JSON type (JSONB), JSON path filtering and indexing | yes |
| Blob v2 - large binary, lazy `BlobFile` streaming, external URIs | yes (2.2) |
| Zero-copy add/drop/rename column (metadata-only) | yes |
| Cell-level updates without base-file rewrite (data overlay files) | yes, but unstable (env-gated `LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES`; release builds refuse) |
| Type change / cast | yes (rewrites that column; drops its index) |
| Time travel, tags, branches | yes |
| Stable row IDs (opt-in at creation; existing datasets can migrate) | yes |
| Change data feed | yes (stable row IDs only) |

**Indexes and search**

| Capability | Status |
|------------|--------|
| Vector ANN - IVF + FLAT/HNSW + FLAT/PQ/SQ/RQ (RQ multi-bit, `num_bits` 1..=9; `approx_mode` fast/normal/accurate) | yes |
| ACORN-1 prefiltered HNSW traversal | yes, opt-in (`approx_mode="fast"`) |
| Distance metrics L2 / Cosine / Dot / Hamming | yes |
| Scalar - btree, bitmap, label-list, ngram, zonemap, bloom filter, FM-Index | yes |
| FM-Index substring / prefix / regex search on raw bytes | yes (segment-based) |
| Full-text search - BM25, multilingual tokenizers, phrase queries | yes (Lance-native) |
| Geo / RTree spatial index + geo UDFs | yes (`geo` feature) |
| Distributed / segmented index builds (vector, bitmap, btree, FTS, ngram, rtree, zone map, bloom filter, label-list) | yes (no scheduler) |
| Hamming clustering / near-duplicate detection over binary hashes | yes (v9 utility) |
| `COUNT(*)` pushdown | yes (fast path on stable-row-id datasets) |
| SQL over datasets | via DataFusion (`LanceTableProvider`) - projection / filter / limit only, **no vector search**, see below |

**Concurrency and ops**

| Capability | Status |
|------------|--------|
| MVCC + optimistic concurrency, automatic rebase | yes |
| Pluggable commit handlers (conditional-put, DynamoDB, lock) | yes |
| MemWAL high-throughput streaming writes | yes, **experimental** |
| Two-phase distributed write | yes |
| Namespaces (Directory, REST) + DataFusion catalog bridge | yes |
| Compaction, version cleanup, fragment reuse index | yes |

**Not in Lance** - a query-builder API, an embedding registry, rerankers as an API, managed
Cloud/Enterprise tiers (those are LanceDB, a separate product); authentication / user
identity; a built-in cross-dataset join planner (use DuckDB/DataFusion on top); a metrics
dashboard.

**The SQL surface has no vector search.** `LanceTableProvider::scan` pushes down projection,
then filter, then limit, then in-order-ness, and calls `create_plan()`
(`rust/lance/src/datafusion/dataframe.rs:161-164`) - it never calls `Scanner::nearest`, and the
symbol appears nowhere in the DataFusion glue. There is no `ORDER BY vec <-> query LIMIT k`
operator and no vector-distance UDF: `register_functions` adds `contains_tokens`
(`rust/lance-datafusion/src/udf.rs:17`) and the JSON functions, plus geo UDFs
(`st_distance` / `st_area` / `st_intersects`) when the non-default `geo` feature is on
(`udf.rs:30-33,44-53`) - those are geometric, not embedding, distances. **An IVF/HNSW index is
therefore unreachable from `Dataset::sql()`**; use the scanner API (`Scanner::nearest`) for ANN
and reserve SQL for relational work. FTS is the exception - it *is* SQL-reachable through the
registered `fts` table function (`ctx.register_udtf("fts", ...)`,
`rust/lance/src/dataset/udtf.rs:78`), but `Dataset::sql()` itself registers only the table plus
`register_functions`, so there is no vector analogue of that escape hatch.

---

### 15.1 Operational notes that bite

**Migrating an existing dataset to stable row IDs has a prerequisite checklist.** Upstream now
documents the procedure rather than treating creation-time opt-in as the only path, and the
prerequisites are strict: "Before migration, stop all index builds and index commits, drop every
secondary index so no index entry remains in the dataset metadata, and keep index creation
quiesced until migration completes." There is a legacy-manifest pre-step too - run a no-op
`false`-predicate delete first to recompute physical row counts, because "affected releases may
have recorded stale counts, which would produce incomplete row ID sequences." Do not run the
migration directly on such a manifest.

**Lance exposes no dataset-level identity, which quietly breaks external caches.** There is no
dataset UUID, and every candidate identity - fragment ids, row ids, version numbers - **restarts
when a dataset is recreated**. So a cache keyed on any of them still looks current after the
dataset it described was dropped and rebuilt, and silently answers against different data. The
check that actually works needs no new persisted state: sample a few row ids and `take_rows` them
from the dataset; if what comes back disagrees with what the cache recorded, the cache describes
a different dataset - discard and rebuild.

**An index-free copy is not a dataset root minus `_indices`.** `_indices` is part of the root and
manifests can reference index metadata, so copying a dataset directory and deleting the index
subtree produces a root whose manifest points at things that are gone. For an index-free archive,
write fresh dataset roots from scans instead of editing a copied one.

**Net-new surface worth knowing** (v12 final and the v13 line): `Dataset::frag_reuse_index()` is
public (#9112) and returns `None` when the loaded version has no FRI; `FileFragment::write_overlay`
returns a real `OverlayWriter` keyed by `_rowaddr` (#8761, still env-gated behind
`LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES`); Python gained `lance.bitmap.Bitmap` (a real
RoaringBitmap binding, #7837), `deep_clone()` (#9181), `base_paths()` (#9191) and
`update_columns(with_offsets=True)` (#8891); Java gained `DataStorageVersion`, `FileWriteOptions`
and `ScanOptions.indexSegments`. Namespace table listing is finally bounded - #9165 moved
`list_directory_tables` onto the v12 `read_dir_page` so "a bounded caller only pays for what it
asks for", with a 1000-entry page hint.

**A third real `LANCE_*` env var landed**: `LANCE_COMMIT_RETRY_TIMEOUT_SECS` (#9177) overrides
the commit retry timeout, still 30s by default. That makes the full set `LANCE_DISABLE_AMX`,
`LANCE_AMX_FP16_CC` and this one - plus the overlay gate above. The old grep trap still holds:
`LANCE_AMX_CFG_*` and `LANCE_AMX_TILE_COUNT` are C macros, and `LANCE_FACTOR` is a substring of
`BALANCE_FACTOR`.

## 16. Source map

Where to look in `lance-format/lance` at `v13.0.0-beta.4`.

| Topic | Path |
|-------|------|
| Format spec overview | `docs/src/format/index.md` |
| File format | `docs/src/format/file/{index,encoding,versioning}.md` |
| Table format | `docs/src/format/table/{index,layout,schema,transaction,versioning,branch_tag,row_id_lineage,mem_wal}.md` |
| Index spec | `docs/src/format/index/{index.md,vector/,scalar/,system/}` (scalar incl. `scalar/fmindex.md`) |
| User guide | `docs/src/guide/{blob,data_evolution,data_types,json,object_store,read_and_write,performance,tags_and_branches,tokenizer,distributed_write,distributed_indexing,migration}.md` |
| Integrations | `docs/src/integrations/{index,datafusion,pytorch,tensorflow}.md` |
| Protobuf schemas | All 12: `protos/{file,file2,table,transaction,rowids,index,index_old,ann,filtered_read,table_identifier,encodings_v2_0,encodings_v2_1}.proto` (`index_old.proto` is a v9 forward-compat shim; `file.proto` is the legacy v1 container; the two `encodings_v2_*` files hold the per-version encoding messages) |
| Rust workspace | `rust/` (entry point `rust/lance/`) |
| Commit / OCC | `rust/lance/src/io/commit.rs`, `rust/lance-table/src/io/commit.rs` |
| Transactions | `rust/lance-table/src/transaction/` (moved out of `rust/lance/src/dataset/transaction.rs` in v11 by #8053/#8054/#8056; `lance::dataset::transaction` survives as a re-export shim) |
| MemWAL | `rust/lance/src/dataset/mem_wal/` |
| Indexes | `rust/lance-index/src/` |
| Object store | `rust/lance-io/src/object_store/` |
| File-version identity | `rust/lance-file/src/version.rs` (both `LanceFileVersion` and `ConcreteFileVersion`; `lance-encoding/src/version.rs` was deleted in v11) |
| Cache keys / backends | `rust/lance-core/src/cache/` (`key.rs`, `quick.rs`) |
| Data overlay resolution | `rust/lance/src/dataset/overlay.rs` |
| Release train / breaking detection | `ci/publish_beta.sh`, `ci/check_breaking_changes.py` |

**`TableIdentifier` (`protos/table_identifier.proto`)** is how a table is handed to a remote
executor for distributed read/write, and it has two modes the filename alone does not reveal:
"1. uri + serialized_manifest (fast): remote executor skips manifest read. 2. uri + version +
etag (lightweight): remote executor loads manifest from storage" (`:10-11`). Mode 1 trades
message size for a saved round trip - the right default when the manifest is already in hand
and the executor count is modest; mode 2 keeps the message small when fanning out widely, at
one manifest read per executor.

Auto-generated API docs and the language-agnostic namespace spec live in sibling repos under
`github.com/lance-format`. The canonical docs site is `lance.org`. To refresh this reference,
see the maintenance note in `../SKILL.md`.

performance.md 64.7 KB

# Lance performance - combined reference

Everything performance-shaped for Lance (`lance-format/lance@v13.0.0-beta.4`) in one place.
**Part A** routes to the official guidance - which lives verbatim in this skill's
`references/docs/` mirror, so it is pointed at rather than re-copied - and then adds the
performance behavior upstream has *not* documented, derived from source and commit history.
**Part B** is field-verified practice from running Lance against remote object storage.

## Contents

- [Part A: Official guidance](#part-a-official-guidance)
  - [Where the official text lives](#where-the-official-text-lives)
  - [OpenTelemetry metrics](#opentelemetry-metrics-not-in-the-performance-guide)
  - [Performance changes not in the guide (v10, source-derived)](#performance-changes-not-in-the-guide-v10-source-derived)
  - [Performance changes not in the guide (v11, source-derived)](#performance-changes-not-in-the-guide-v11-source-derived)
- [Part B: Field-verified remote-storage practices](#part-b-field-verified-remote-storage-practices)
  - [The governing rule: minimize remote calls first](#the-governing-rule-minimize-remote-calls-first)
  - [Write path](#write-path)
  - [Local-filesystem crash safety and recovery](#local-filesystem-crash-safety-and-recovery)
  - [Index maintenance](#index-maintenance)
  - [Read path and query shaping](#read-path-and-query-shaping)
  - [Version-specific behavior (verify on your exact pin)](#version-specific-behavior-verify-on-your-exact-pin)
  - [Benchmarking traps](#benchmarking-traps)

# Part A: Official guidance

## Where the official text lives

Read these directly - they are byte-verbatim copies of the upstream docs at the tracked tag.

| File in `references/docs/` | Heading | Covers |
|----------------------------|---------|--------|
| `guide/performance.md` | (entire file) | Logging; trace events (File Audit, Dataset Events, Object Store Throttle Events, I/O Events, Execution Events); Threading Model; Memory Requirements (Metadata Cache, Index Cache, Scanning Data, Cloud Store Throttling); Fragment Sizing; Conflict Handling + Fragment Reuse Index; Indexes (BTree, Bitmap, Storage Requirements, Performance, Vector Index sizing formulas) |
| `quickstart/full-text-search.md` | "Performance Tips" | FTS index maintenance (incremental `optimize`, coverage monitoring), index configuration best practices, query optimization |
| `guide/json.md` | "Performance Considerations" | `json_get_*` vs `json_extract`, JSON scalar index on hot paths, nesting depth, strict type conversion, array access |
| `format/table/transaction.md` | "CreateIndex Compatibility" | Why index creation is safe concurrently with appends/updates/deletes, and why unindexed fragments are fine |
| `format/index/scalar/fts.md`, `format/index/vector/index.md` | (whole files) | Per-index storage, memory, and training costs |
| `quickstart/vector-search.md` | (whole file) | ANN build and query tuning walkthrough |
| `guide/observability.md` | (whole file) | Logging, trace events, object-store metrics |

Provenance: `docs/src/guide/performance.md` was byte-unchanged from `v9.1.0-beta.8` through
`v11.0.0-beta.2`, then **changed at `v11.0.0-beta.4`** (#8387), which added the "Tuning remote
scans" section and a `row_id_meta` component to the Row Id Sequence cache key, and **again at
`v11.0.0-beta.16`** (#8540), which appended the "AMX Acceleration" section (+29 lines, no other
edit). It then held byte-unchanged through `v12.0.0-beta.6` and **changed again at
`v12.0.0-beta.12`** (#8936), which rewrote the RQ block for the 5-bit default: new per-row sizing
formulas, a new worked example (~10.8 GiB -> ~47.3 GiB), and new `Fast`-mode guidance. It held
again through `v12.0.0` and **changed at `v13.0.0-beta.1`** (#9112, +8 lines), which documented
`Dataset::frag_reuse_index()` - the first edit to the guide since `v12.0.0-beta.12`.

The mirror is refreshed to `v13.0.0-beta.4`, so every number, default, and recommendation in it
is current as written. **Any RQ sizing figure you remember from an earlier read of this guide is
stale** - re-read the block rather than trusting a cached number. The other perf-bearing sections
above remain byte-unchanged across the range.

## OpenTelemetry metrics (not in the performance guide)

Beyond the trace events in `guide/performance.md`, Lance can export metrics through the
`metrics` crate facade (Rust `metrics` feature). The `pylance` wheels are built with the
`metrics` feature enabled: install the OpenTelemetry extra (`pip install "pylance[otel]"`) and
call `instrument_lance_metrics`, which registers Lance's metrics as observable instruments on
your OpenTelemetry `MeterProvider` (`docs/guide/observability.md`, PR #7537).

## Performance changes not in the guide (v10, source-derived)

These are upstream performance behaviors verified against the `v10.0.0-beta.7` source and
commit history. They are **not** in `docs/src/guide/performance.md` - upstream has not
documented them there - so treat this subsection as source-derived rather than official text.

**The cache backend changed, and it can silently refuse entries.** quick_cache is now the
default for both the index cache and the metadata cache, hard-wired in `Session::new` with no
env var or Cargo feature to opt out (PR #7953, #8013). quick_cache splits its weight budget
evenly across shards with no borrowing and **silently refuses entries heavier than a shard's
share**. Shards are `min(cpus / 2, capacity / 4 GiB)`, floor 1, so on a small-CPU or
small-capacity configuration a large index partition may never cache at all - with no error and
no log line, only a persistent miss. If a partition looks uncacheable, compute the per-shard
share before tuning anything else. Measured FTS gain at concurrency 128: 180.7 -> 1340.6 qps,
710 -> 96 ms, 47% -> 93% CPU.

Cache keys also became opaque 16-byte BLAKE3 digests (`CACHE_KEY_FORMAT = "blake3-128-v1"`,
PR #7878) with **no legacy fallback** - every warm or persisted cache cold-misses once after
the upgrade. Budget for one cold window when rolling v10 out; do not read it as a regression.

**FTS query concurrency.** `LANCE_FTS_SEARCH_CHUNK` (default 16, min 1) sets how many
partitions are searched per CPU-pool task; chunking stops query concurrency from flooding the
pool with one small task per partition. Measured 227 -> 428 qps at concurrency 16. `=1` restores
the old per-partition shape. This is one of the few store-adjacent knobs worth knowing exists -
the default is right in the common case.

**FTS row-id resolution moved after the global top-k merge** (PR #7897): at most `limit` lookups
per query instead of per-partition. 4.1 -> 107.7 qps (26x), 3.9 s -> 148 ms on a 100M-doc
benchmark. The tradeoff is memory: each partition's ROW_ID column is now a separate weighed
index-cache entry at ~8 bytes/doc per partition (~800 MB for a 100M-doc index), and it now
counts against `index_cache_size_bytes`. Size the index cache accordingly.

**Segment commit LISTs concurrently** (PR #7657) - measured ~8x faster against remote storage
(2837 ms -> 350 ms at 128 segments, 20 ms RTT). This is a pure win requiring no action, but it
changes the shape of index-commit latency, so re-baseline before comparing against pre-v10
numbers.

**Compaction can now skip row-address map construction** entirely when no remappable data index
exists - FRI-only and system-index-only datasets included (PR #7778). If compaction was
previously dominated by the `_rowid` scan and RoaringTreemap build on such a dataset, that cost
is gone.

## Performance changes not in the guide (v11, source-derived)

Same caveat as above: verified against the `v11.0.0-beta.16` source and commit history, absent
from `docs/src/guide/performance.md`. The section is unchanged at `v13.0.0-beta.4`. The edits to
`docs/src/guide/performance.md` between `v12.0.0-beta.6` and `v13.0.0-beta.4` are three hunks
from line 483 onward in the RQ sizing block (#8936), plus the FRI-inspection paragraph added at
`v13.0.0-beta.1` (#9112), so the official "Tuning remote scans" numbers and everything else
Part A routes to still stand as written - but the RQ sizing figures do not, and are restated in
`references/indexes.md`.

**Large commits got much cheaper on the manifest side** (PR #7881). Transactions serialized
above 20 MiB are no longer inlined into the manifest and live only in their external
`_transactions/` file. Measured on a large workload: the full-commit manifest shrank from
1576 MiB to ~790 MiB (-50%). There is no configuration to set - it is automatic, and it matters
most on object storage, where manifest size is read on every dataset open.

**Two O(n) fixes on hot paths.** `build_manifest` no longer does `O(n*m)` fragment comparisons
for Update/Delete transactions (PR #8210), which shows up on datasets with many fragments; and
fixed-width decode buffers are now preallocated exactly via a new optional
`decoded_size_bytes` contract on the decompressor traits (PR #8091), cutting **index-cache entry
weight by up to 74%** on IVF_SQ configurations. The latter is a pure accounting improvement -
"This does not change the on-disk format" - but it means the same `index_cache_size_bytes` now
holds substantially more, so re-measure before shrinking the budget.

**FTS conjunctions are cheaper and cold search parallelizes.** Conjunction approximations are
now ordered by iterator cost (PR #8299: 1.50x fewer comparisons, 1.12x speedup, bit-for-bit
identical results and tie order), and doc lengths preload in parallel on the cold deferred
search path (PR #8119).

**The cache backend is no longer a hard-wired choice** (PR #7683), which softens the "no opt-out"
statement in the v10 subsection above. A process-wide registry accepts custom backends, and a
compact URI form (`moka://?capacity=1073741824`) selects one without code. Related: reported
cache sizes shrink after PR #8159, because shared `Arc`/Arrow allocations are no longer charged
once per entry - **recalibrate any alert thresholds keyed on `LanceCache::deep_size_of()`
rather than reading the drop as a regression.**

**Incremental compaction is now expressible** (PR #8116): `compact_files(max_source_fragments=N)`
bounds a run to N source fragments, "allowing compaction to proceed incrementally. Fragments are
processed oldest first." Also settable as the manifest config key
`lance.compaction.max_source_fragments`. This is the clean answer to "compaction takes too long
to ever finish in my maintenance window" - previously the only lever was letting it run to
completion.

**Two more budgets joined it at `beta.8`** (PR #8235, `breaking-change`-labeled):
`max_source_rows: Option<usize>` and `max_source_bytes: Option<u64>`
(`rust/lance/src/dataset/optimize.rs:278,287`), each also settable as
`lance.compaction.max_source_rows` / `lance.compaction.max_source_bytes` (`:363-365`). Prefer
these over `max_source_fragments` when fragments vary widely in size - a fragment count is a
poor proxy for work when one fragment holds 100x the rows of another, and bytes is the closest
proxy to actual IO. And at `beta.14`, `excluded_fragment_ids: Vec<u32>` (`:295`, PR #8532)
keeps named fragments out of planning entirely - the lever for "compact everything except the
partition currently being written".

**AMX-FP16 changes index *results*, not just speed** (PR #8540, `beta.16`). This one is now
documented upstream - the "AMX Acceleration" section of `guide/performance.md` - but it deserves
flagging here because it is the rare performance change that alters what an index contains. On
Linux x86_64 with an AMX-FP16 CPU (Intel Granite Rapids / Xeon 6 and newer), `float16` vector
columns using `dot` route partition assignment through tile instructions. Three gates, all
required, all silently declining the work when unmet: `float16` + `dot`, `dimension >= 32` (one
tile pass covers 32 dimensions), and `num_centroids >= 32` (the GEMM steps its centroid loop by
32 with no partial-tile path). Below those sizes a tile pass costs more than it saves.

Where it does engage, **index build switches from an approximate graph search over the centroids
to comparing every vector against every centroid**. Recall improves and partition assignments
differ from what an older build produced - so a rebuild on new hardware is not a no-op, and an
A/B against an older index is comparing two different indexes. `LANCE_DISABLE_AMX=1` takes the
paths out of service without a rebuild, but because it also reverts assignment to the
approximate path, "an index built with it set is not equivalent to one built without it;
compare recall, not just build time". Availability is also a **build-time** property: the kernel
only exists if the build machine had clang >= 16 or gcc >= 13, with `LANCE_AMX_FP16_CC`
overriding the compiler choice (`rust/lance-linalg/build.rs:27`).

**There is no resident data cache.** A `Session` carries exactly two caches - `index_cache` and
`metadata_cache` (`rust/lance/src/session.rs:49-70`) - holding structural metadata (manifests,
schemas, page tables, row-id maps) and index pages. **Decoded column values are never cached**,
and there is no caching `ObjectStore` wrapper, so repeatedly taking the same rows re-reads their
value buffers from the object store every single call. If a workload does repeated point reads
of a hot row set, the cache to add is your own, above Lance; no amount of `index_cache_size_bytes`
tuning will do it.

What a `Session` *does* buy is sharing. Passing one `Arc<Session>` to several datasets via
`DatasetBuilder::with_session` (`rust/lance/src/dataset/builder.rs:525`) lets them share
index and metadata caches instead of each paying its own cold start - worth doing whenever one
process opens several datasets, which is the normal shape for a namespace of tables.

**Cold first search on remote storage is dominated by paging indexes in**, and the remedy is
`prewarm_index(name)` - or `prewarm_index_segments(name, segment_ids)` to warm only chosen
segments (`rust/lance/src/index/api.rs:194-238`; exposed on the Python `Dataset`). Note what
prewarming does *not* fix: it loads index structures, so a query whose cost is dominated by
**materializing the hit rows' other columns** - scattered point reads, roughly one per
hit-row-fragment - sees little benefit. Diagnose which half you are in before optimizing;
prewarm helps the index half only.

**Two per-operation stats structs that callers routinely discard.** Merge-insert returns
`MergeStats` with `num_inserted_rows` / `num_updated_rows` / `num_deleted_rows` and
`num_attempts` (`rust/lance/src/dataset/write/merge_insert.rs:2739`) - `num_attempts` is the
direct read on how much OCC contention a write is actually hitting, which is otherwise
invisible. And `Scanner::scan_stats_callback` (`rust/lance/src/dataset/scanner.rs:1349`)
delivers `ExecutionSummaryCounts` (see `indexes.md`) per scan, giving `iops`, `requests`, and
`bytes_read` without an `EXPLAIN ANALYZE` round trip - the cheapest way to verify that a
"minimize remote calls" change actually reduced calls.

**Multipart uploads no longer lose parts on retry** (PR #8174). The old path called
`put_part` again on failure, and native cloud stores allocate a fresh part number for that call,
so the retry skipped the failed part and completion failed with `Missing part`. Retries now
happen inside the HTTP connector. If you saw sporadic `Missing part` failures on large writes,
this is the fix; OpenDAL-backed stores were never affected and are unchanged.

## Performance changes not in the guide (v12, source-derived)

Verified against `v13.0.0-beta.4`.

**Latest-version resolution stopped listing the whole `_versions/` prefix** (PR #8679). The
namespace path previously enumerated every historical manifest to find the newest: on a
~340k-version table that is ~344 sequential `ListObjectsV2` pages, "~25s of pure I/O wait",
paid by *every* `open_table` / `describe` / `merge_insert` that resolves the latest version.
Heals purely on upgrade, and it compounds with version bloat - a second reason to keep history
short beyond storage.

**Provider-native bulk copy** (PR #8770). `LANCE_IO_SERVER_SIDE_COPY_ENABLED` routes cloud
copies whose source and destination share an object-store client into the provider's own
server-side copy instead of streaming bytes through the client. The relevant cases are index
movement and copying between prefixes of one bucket. Deep clone separately bounds non-local
file movement to four concurrent files, overridable via `LANCE_DEEP_CLONE_STREAM_CONCURRENCY`.

**`LANCE_DEFAULT_IO_BUFFER_SIZE`** - the input scan for vector shuffling "uses a 2 GiB I/O
readahead buffer by default", configurable through this variable
(`docs/src/guide/performance.md:439`). This is a distinct knob from the per-scan
`io_buffer_size` in the "Tuning remote scans" block, and it is the one that dominates memory
during a large index build.

**Blob v2 materialization got a byte budget** (PR #8919, beta.10), wired through the new
`FilteredReadOptions.materialization_readahead_bytes` (proto field 13). It is "a nonzero upper
bound on bytes reserved by Blob v2 materialization awaiting ordered emission in one scanner
execution"; admission follows output order and "one oversized output batch may exceed the bound
when no other batch is reserved". **If absent, Blob v2 materialization has no independent memory
bound** - which is the state you are in by default, so set it before scanning wide blob columns.
A sibling field 14, `batch_size_bytes`, gives the file reader a byte-based batch boundary
alongside the row-based `batch_size` (PR #8933, also carried through distributed
`FilteredReadOptions`).

**MemWAL memory accounting was wrong in a way that matters for sizing** (PR #7831).
`MemTable::estimated_size` counts buffered batches plus the PK bloom filter - "every in-memory
index is invisible to it." The sharp edge is HNSW: `OnlineHnswBuilder::try_with_capacity`
pre-allocates fixed-size node storage for `max_memtable_rows`, so a vector memtable commits its
**entire** graph on the first insert, not gradually. Measured: 125k rows -> 64.7 MiB, 500k ->
258 MiB, 1M -> 517 MiB, 2M -> 1033 MiB, all charged on row #1. Size `max_memtable_rows` against
that, not against the row count you expect to hold.

---

# Part B: Field-verified remote-storage practices

Everything below comes from production benchmarks of a Lance-based application (2.2M-row
corpus) running against S3-compatible object storage - not from the official docs. Each
practice was measured with a before/after comparison and shipped. Nothing here is
speculative; if an approach was tried and did not clearly win, it is not listed.

## The governing rule: minimize remote calls first

A tool that must work against arbitrary buckets (AWS, Hetzner, R2, MinIO, ...) cannot assume any
particular provider's rate limits or bandwidth. The order-of-magnitude wins in this document all
come from issuing **fewer remote calls** - fewer commits, fewer scans, fewer round trips - not
from tuning the store. Do that first. Explicit per-column compression metadata was tried and
yielded little real-world benefit relative to the effort: reducing what you read beats shrinking
it.

**Tuning the store is a legitimate second move, once call volume is already minimized.** v11
added an official "Tuning remote scans" section (`docs/src/guide/performance.md`, #8387) with a
concrete starting point for bandwidth-constrained access:

| Knob | Suggested start | Why |
|------|-----------------|-----|
| `LANCE_IO_THREADS` | 8 | Cloud stores default to **64**, "intended for high-bandwidth, in-region access and can be too aggressive across regions or over the public internet" |
| `fragment_readahead` | 1 | "Set it to `1` to match the fragment-level I/O pattern, then increase it if the storage connection has spare bandwidth" |
| `batch_readahead` | 2 | Bounds decode-ahead work |
| `io_buffer_size` | 64 MB | Caps in-flight buffered bytes |

These are upstream's numbers for cross-region or public-internet access, not ours - we have not
A/B'd them against the workload behind Part B, and a value tuned for one bucket can misbehave on
another. Treat the table as a documented starting point to measure from, and keep the
benchmark-verified practices below as the primary lever.

Two counter-intuitive caveats from the same section, worth knowing before you tune:

- **`scan_in_order=True` does not serialize fragment reads.** "An ordered dataset scan still
  overlaps I/O from multiple fragments. `scan_in_order=True` controls the order in which batches
  are returned; it does not make fragment reads sequential." This is why a dataset scan issues
  more concurrent requests than scanning one fragment directly.
- **Lowering `batch_size` may not shrink the request.** "Lance reads encoded pages from storage,
  so reducing `batch_size` changes the returned and decoded batch sizes but may not reduce the
  initial range request."

Every practice below is an instance of the governing rule: fewer commits, fewer scans, fewer
round trips.

## Write path

- **Commit count, not row count, is the cost unit.** Each commit is roughly a 1-second
  object-store round trip and rewrites the manifest, which grows with fragment count, and
  every version is retained until cleanup. A full-corpus copy that issued one
  `merge_insert` per batch took 75.7 min / 354 commits; rewritten as a single-commit
  append path the same copy took 18.2 min / 1 commit. A bounded A/B on one delta measured
  3,890 ms (merge) vs 882 ms (append).
- **The anatomy of that cost: a commit is three sequential round trips.** A LIST of `_versions/`
  (the conflict scan - *not* a HEAD of the latest manifest), then an unconditional awaited PUT of
  the `.txn` file, then the conditional PUT of the manifest itself (`PutMode::Create`,
  `rust/lance-table/src/io/commit.rs:1569`). On non-lexically-ordered stores a fourth
  best-effort hint PUT follows. The LIST runs on **every attempt** by design, so a contended
  commit multiplies all three. At 50-100 ms RTT that is 150-300 ms per commit before any data
  moves - which is why commit count dominates.
  Note that inlining a sub-20 MiB transaction into the manifest cuts *read* round trips, not
  write ones: the separate `.txn` file is written either way.
- **Composite `merge_insert` keys are index-accelerated now - re-measure if you avoided them.**
  Lance used to accelerate only a *single-column* merge key, so any composite key fell back to
  scanning the key columns on every merge to locate rows (field-measured at an older pin: 143
  MiB read to write 8 rows, 6.36 s per call). At v12 `indexed_join_keys` probes **each** join
  column that has a scalar index supporting exact equality, ANDs the probes inside one
  `MapIndexExec`, and lets the downstream hash join filter on the full composite key -
  "unindexed columns simply do not prune the candidate set - they are checked by the
  post-filter" (`rust/lance/src/dataset/write/merge_insert.rs:1188`). Fragments outside the
  *intersection* of the participating indexes' bitmaps still scan. The design advice inverts:
  index the key columns rather than collapsing to one synthetic key.
- **`Dataset::versions()` costs O(history) remote round trips, not one.** Manifests are named
  `{u64::MAX - version}.manifest` (`rust/lance-table/src/io/commit.rs:86`) and the call issues a
  `list_manifest_locations` followed by a `read_manifest` **per location**
  (`rust/lance/src/dataset.rs:2608-2610`). On a version-bloated remote store this is a fetch
  storm, with individual reads hitting the 120 s timeout. A second, non-storage reason to keep
  history short - and a reason not to call `versions()` on a hot path at all.
- **Use `Append` for append-shaped data; reserve `merge_insert` for genuine upserts.**
  Merge is commit-latency-bound on object storage; append is bandwidth-bound.
- **`merge_insert` accelerates only when *every* `on` column is indexed.** The v7/v8 rule was
  stricter - exactly one join column - but at v10+ the dispatch requires that all `on` columns
  carry an exact-equality scalar index (`rust/lance/src/dataset/write/merge_insert.rs:1323`),
  plus `use_index == true` and `delete_not_matched_by_source == Keep`. A **partially** indexed
  composite key silently falls through to a full-table join. The cost lands in the read, not the
  write: a measured 8-row update wrote one data file and one deletion vector but **read ~143
  MiB**, scanning the full key columns across 2.1M rows to locate the 8 matches.
- **Manifest *size* grows with fragment count, so `_versions/` can dwarf the data.** Each
  manifest lists every current fragment. Measured on a fragmented store: `_versions/` at 110 MB
  across 178 manifests (~2.3 MB each for the older ones) against a much smaller data footprint;
  on a small-row table, 7.0 MB of data carried 54 MB of `_versions/` and 3.5 MB of
  `_transactions/`. Compaction reduces future manifest size; only cleanup reclaims the old ones.
- **Never commit per item.** A benchmark that committed once per logical unit produced
  3.3 GB of store for 40k tiny rows in ~20 min (manifest churn); the same work batched at
  ~100 units per commit was 17 MB in 1.6 s.
- **When batching is not enough, coalesce the commits themselves.** Lance has a public primitive
  for this that the docs barely surface: write N batches with
  `InsertBuilder::execute_uncommitted()` (`rust/lance/src/dataset/write/insert.rs:133`), which
  writes data files and returns a `Transaction` **without** committing, then publish them all with
  one `CommitBuilder::execute_batch(Vec<Transaction>)`
  (`rust/lance/src/dataset/write/commit.rs:560`) for a single manifest bump. The data-file writes
  can be fanned out concurrently; only the final commit is serialized. **Append-only for now** -
  the API's own warning reads "Only works for append transactions right now. Other kinds of
  transactions will be supported in the future." This is the right shape for a micro-batching
  ingest daemon, where commit count is the binding constraint.
- **Skip no-op merges.** A `merge_insert` where every row matches with
  `WhenMatched::DoNothing` still commits a new (empty) version. Pre-filter to genuinely
  new keys and skip the merge entirely when the set is empty.
- **Compute derived columns before the append.** Embedding-then-merge-back doubles
  commits and rewrites rows; embedding before the append lets the vector ride the row's
  birth commit for free.
- **Append retries are not idempotent.** Unlike `merge_insert` (which no-ops on re-read),
  a retried append after a lost commit ack duplicates rows, and Lance has no unique
  constraint - OCC does not conflict two writers inserting the same new key. Retry only
  on genuine commit-conflict errors, and verify with `COUNT(*)` vs `COUNT(DISTINCT pk)`,
  not just missing-row checks.
- **Match Lance's typed conflict errors before wrapping them.** `CommitConflict`,
  `RetryableCommitConflict`, and `TooMuchWriteContention` are distinct variants
  (`rust/lance-core/src/error.rs:176,193,202`). Erasing them into an opaque application error
  early - `anyhow`, a generic `Storage` variant - makes every exhausted retry indistinguishable
  from a storage outage, and the retry loop can no longer tell "rebase and try again" from
  "give up". Match at the Lance boundary, not after three layers of `?`.
- **`merge_insert` silently changes execution mode with the source schema shape.** When the
  source schema is a strict subset of the target's and unmatched rows are kept, the builder
  routes to `RewriteColumns` (a cheap column update) rather than rewriting whole fragments
  (`rust/lance/src/dataset/write/merge_insert.rs:2232-2260`). Passing a full-schema source for
  what is logically a two-column update therefore costs dramatically more, with no warning.
  Project the source down to the key plus the columns actually being updated.
- **Local-filesystem durability is delegated, not added.** Lance's `file://` store is
  `object_store::LocalFileSystem` (`rust/lance-io/src/object_store/providers/local.rs:26`,
  `object_store 0.13.2`); Lance issues no fsync of its own on the commit path. Whatever
  crash-durability guarantee you get on a local dataset is that crate's, so do not assume a
  returned commit means the manifest bytes survived a power loss. See the next subsection for
  what that costs you in practice.

## Local-filesystem crash safety and recovery

This is the sharpest edge in Part B, because the failure is **permanent and silent** and has no
analogue on object storage. Mechanism, verified at `v11.0.0-beta.2`:

- **Latest-version resolution is "highest number wins", with no fallback.** On a local store
  Lance scans `_versions/` and keeps the maximum
  (`rust/lance-table/src/io/commit.rs:693-694`, `current_manifest_local`, selected first for
  local at `:283-286`). A manifest that exists but is truncated or zero-length is a hard error -
  "Invalid format: file size is smaller than 16 bytes"
  (`rust/lance-table/src/io/manifest.rs:60-63`) or "Invalid format: magic number does not match"
  (`:68`) - propagated straight up by `Dataset::latest_manifest`
  (`rust/lance/src/dataset.rs:1180`). The only error the open path swallows is `NotFound`
  (`rust/lance/src/io/commit.rs:313`), and corruption is not `NotFound`. **There is no
  "fall back to version N-1" anywhere.** So an unclean stop that creates the manifest without
  flushing its bytes leaves a dataset that will not open, even though version N-1 is fully
  intact on disk.
- **Editing `latest_version_hint.json` back does nothing.** On a local store the hint file is
  never read at all - `current_manifest_path` branches on `object_store.is_local()` before the
  hint path is considered (`commit.rs:283-291`). It *is* written on local now (`:79`, gated by
  `version_hint_globally_enabled() && !list_is_lexically_ordered`, `:328`), but writes are
  best-effort and it "never affects correctness (readers verify the hinted version and probe
  upward from there)" (`:340-341`). Rewinding it is inert.
- **The repair is to move the poisoned manifest aside**, not to touch the hint: quarantine
  (never delete) the sub-16-byte `*.manifest` files under `_versions/`, and the next-highest
  intact version becomes latest again. Rename rather than remove, so a misdiagnosis is
  reversible.
- **Then validate with a scan, not with `count_rows`.** `count_rows(None)` sums
  `physical_rows - deletions` straight from fragment metadata and opens no data file when the
  manifest carries the counts (`rust/lance/src/dataset/fragment.rs:1375-1379`, summed at
  `rust/lance/src/dataset.rs:1664-1671`). A zeroed or truncated data file that a surviving
  manifest still references is completely invisible to it - the row count reports fine while
  the data is unreadable. Only a drained scan proves integrity.
- **Object storage is structurally immune to this.** S3-style PUTs are atomic, so a killed
  writer leaves *no object* rather than an empty one, and there is no half-written manifest to
  poison the version sequence. This is a local-store-only failure class.
- If you run Lance on a local filesystem on hardware that can lose power, the mitigation is an
  fsync-on-write `WrappingObjectStore` (file plus parent directory after every put,
  multipart-complete, copy, and rename) installed innermost in the store-wrapper chain and gated
  to local URLs only, plus a `_versions/` walk on open that rename-quarantines undersized
  manifests. A later A/B put a number on the fsync layer's cost, superseding an earlier
  "not detectable" reading: **+5.54 s real on a 130.86 s sync, +4.2%** - and that is with
  macOS's notoriously expensive `F_FULLFSYNC`. It stays small for a structural reason worth
  internalizing: Lance writes **few, large** files, so fsyncs amortize per-file, not per-row.
  Expect the same shape on any workload with that write profile, and expect it to degrade if
  you push Lance into many-small-files territory.
- **Verify recovery with a full-projection scan, not an id-only probe.** A scan that projects
  only the id column reads only that column's data files, so a crash-zeroed data file behind a
  *different* column - the classic case being a column added later, such as an embedding
  vector - stays invisible and the probe passes. Drain a scan that projects every column.
- The local-store rules above cover `file+uring://` as well as `file://`; both report
  `is_local() == true`. A scheme comparison written as `scheme == "file"` silently skips uring
  stores.

## Index maintenance

- **Batch index folds behind a row-count threshold; never fold on every write tick.**
  Folding FTS + vector indexes on each 5-minute sync cost 15-445 s of the sync; deferring
  folds until the unindexed tail reaches ~5,000 rows cut a 80-524 s sync to ~44 s. The
  deferred tail costs only ~50-350 ms extra per query (see next point).
- **A remote fold is close to fixed-cost per pass, not proportional to the delta.** Measured on
  S3-compatible storage, folding a delta of **~200 rows took ~346 s**; the same fold against a
  local store took **2-4 s for a delta of 424k rows**. The dominant cost is neither transfer nor
  verification but the index fold itself on the remote - roughly the same ~100x object-store
  penalty that shows up everywhere else in this document, and essentially a floor you pay to push
  even one new row. This is what makes a row-count threshold non-optional remotely: the amortized
  cost per row falls almost linearly with how much you batch behind it, so the threshold should be
  tuned against fold *frequency*, not against how stale the tail is allowed to get.
- **The mechanism behind that floor is one round trip per IVF partition, so fold latency tracks
  `num_partitions`, not delta size.** Instrumenting an 81-row fold that took 445 s showed **zero
  throttle, retry or 503 responses and zero warnings** - it was not remote-side throttling but
  sequential object-store round trips during the index append, one per partition, exposed to
  variable per-request latency. The "partition N is empty, skipping" lines are the visible trace
  of Lance walking all of them (256 in that run). Two consequences: a smaller `num_partitions`
  directly shortens every fold on remote storage, and a fold that looks throttled is worth
  measuring before you tune retry or concurrency settings, because those are not the bottleneck.
- **An unindexed tail is a latency concern, not a correctness one - if `fast_search` is
  off.** Lance answers FTS and vector queries as a union of the index scan and a flat
  scan of unindexed fragments. `fast_search` skips that flat arm, silently dropping the
  newest rows from results. Only enable it when no unindexed tail exists, and keep a
  tail-recall regression test. On v11, an unindexed tail additionally disqualifies the
  posting-backed compound FTS scorer (section 11.3), so it costs plan quality too.
- **Know exactly what the flat arm does, because it bounds your commit cadence.** Read the
  branch at `rust/lance/src/dataset/scanner.rs:3380-3403` (`v13.0.0-beta.4`). It scans **every**
  unindexed fragment, and two properties make that cost scale badly. First, the filter is applied
  as a post-scan `LanceFilterExec` over the scanned rows rather than through scalar indexes - the
  code says so outright: "we could try and use the scalar indices here to reduce the scope of this
  scan but the most common case is that fragments newer than the vector index are also newer than
  the scalar indices." Second, **limit/offset is not pushed down** - "Can't pushdown limit/offset
  in an ANN search" - so a `k` of 10 does not shrink the scan. A selective filter therefore does
  not save you here the way it does on the indexed arm.

  The practical consequence: every commit adds at least one fragment, and every fragment stays on
  this arm until the next `optimize_indices`. Query cost grows roughly linearly with **commits
  since the last fold**, not with rows. A one-second ingest cadence against hourly folds means
  thousands of fragments flat-scanned per query. For a frequent-append workload this - not commit
  throughput - is usually the first thing to fall over, and the fix is fold frequency, not a
  bigger machine.
- **Measuring the backlog: there is no `count_unindexed_rows()`.** The supported API is
  `Dataset::unindexed_fragments(idx_name)` on the `DatasetIndexInternalExt` trait
  (`rust/lance/src/index.rs:2548`) - public, but carrying "Internal use only. No API stability
  guarantees", so pin your Lance version if you depend on it. `index_statistics()` does surface
  `num_unindexed_rows`, but only inside an untyped JSON string and at the cost of a full
  `count_rows`. Prefer counting rows in the returned fragments.
- **Choosing ngrams over `simple`+stem costs relevance, not just RAM.** On a 111-query
  paraphrase set over the same corpus (Success@3, full corpus): word `simple`+stem scored
  **66/111** against production `ngram(3,5)` at **31/111** - roughly 2x better - while using
  ~5x less RAM (379 MB vs 1,868 MB at 2M rows) and ~4x less disk. ngram posting size scales with
  document *bytes*, not document count: a measured ngram(3,5) index over 161,718 text values
  produced 737 MB of postings, about **4.5 KB of index per document**, because every text emits
  one posting per `(length - n + 1)` substrings at each n in the range. Reach for ngrams only
  when you actually need substring/typo matching, and measure recall before assuming it helps.
- **Gate `cleanup_old_versions` to every Nth commit.** Its cost is O(accumulated
  versions), not O(delta): it consumed 8.8 s (58%) of a 200-row incremental sync and gets
  slower as versions pile up. Gating it on `dataset.version_id() % N` cut cleanup walks
  by ~87% with no behavior change.
- **Cleanup does not have to run on the write path at all.** The official guide documents an
  off-write-path alternative under "Other cleanup strategies"
  (`references/docs/guide/read_and_write.md`): drive cleanup from an external scheduler rather
  than from the writer. The tradeoff is stated plainly - it "keeps cleanup off the write path
  entirely, avoiding any impact to write latency, but requires setting up and maintaining
  additional infrastructure." For a latency-sensitive ingest path where the Nth-commit gate still
  shows up in tail latency, this is the next move.
- **"Pending cleanup" bytes are the retention window, not bloat.** Versions younger than the
  retention window are pinned by design, so an optimize pass over a young store legitimately
  reclaims zero. Know your actual window before calling it a leak: Python's
  `cleanup_old_versions` defaults `older_than` to **14 days** when neither `older_than` nor
  `retain_versions` is given (`python/python/lance/dataset.py:3113`), while automatic cleanup
  uses whatever `lance.auto_cleanup.older_than` the dataset config carries - the docs example
  sets `"3600s"` (`docs/src/guide/read_and_write.md:585`), which is where a "one hour" default
  is easily misremembered from. Never benchmark space reclamation on a store younger than the
  window actually in force.
- **A second, independent floor: unverified files are held for 7 days.**
  `UNVERIFIED_THRESHOLD_DAYS = 7` (`rust/lance/src/dataset/cleanup.rs:319`) is hardcoded. With
  the default `delete_unverified=false`, any file not reachable from a manifest but newer than
  7 days is treated as possibly-in-progress and refused for deletion **regardless of
  `older_than`**. Shortening the retention window does not touch this floor, so a store can sit
  well above its expected size for a week after heavy rewriting and still be behaving
  correctly.
- **A `replace=true` index rebuild roughly doubles store size until a later cleanup.**
  `create_index(replace=True)` writes the new index set and leaves the entire superseded set on
  disk as pending-cleanup - a measured rebuild roughly doubled both store size and object count
  until the next cleanup pass. Provision headroom for a full extra index set before rebuilding,
  and do not schedule a rebuild and a tight retention window against each other.
- **Measure maintenance on the real remote store.** A full FTS consolidation rebuild
  measured ~1 min locally but 4.5-5 min (rewriting ~190 MB) against the remote store -
  round trips dominate. If a periodic rebuild can exceed your scheduler interval, rely on
  Lance OCC (conflicting commit -> retry) and keep rebuild cadence low rather than trying
  to serialize externally.
- **`optimize_indices(append())` never collapses anything - and nothing else will either.**
  `append()` sets `num_indices_to_merge: Some(0)` (`rust/lance-index/src/optimize.rs:79-80`) and
  the merge path short-circuits on zero (`rust/lance/src/index/append.rs:408`). **No automatic
  count- or size-based collapse threshold exists in the codebase**, so a pipeline that only ever
  calls `append()` accumulates delta segments without bound until something else breaks - and
  what breaks first depends on the index family, which makes the symptoms look unrelated. Note
  the *default* `OptimizeOptions` is better behaved than `append()`: with `num_indices_to_merge`
  unset it collapses the trailing segment on every call (`append.rs:400-403`), and
  `retrain: true` merges everything into one. Pick an explicit ladder - collapse at a modest
  segment count, full rebuild at a higher one - and treat those numbers as correctness floors
  rather than tuning preferences.
- **`LANCE_MEM_POOL_SIZE` is the hidden ceiling on large from-scratch scalar index builds, and
  the default is smaller than it looks.** BTree/JSON training scans run with `use_spilling: true`
  (`rust/lance/src/index/scalar.rs:166`, `rust/lance-index/src/scalar/btree.rs:1942`), which
  wraps a DataFusion `FairSpillPool` sized from that variable
  (`rust/lance-datafusion/src/exec.rs:364,378`). The default is
  `DEFAULT_LANCE_MEM_POOL_SIZE_PER_PARTITION = 150 MiB` (`exec.rs:309`) multiplied by
  `target_partition.unwrap_or(1)` (`exec.rs:314,324`) - **so with `target_partition` unset the
  pool is 150 MiB total, not 150 MiB per core.** A large sort that fragments into more spill
  files than its fair-share slice can merge fails with a DataFusion `Resources exhausted ...
  ExternalSorterMerge` error rather than spilling further. Raise `LANCE_MEM_POOL_SIZE` (or set
  `LANCE_BYPASS_SPILLING`, `exec.rs:349`, to disable the pool entirely) before concluding the
  index cannot be built.
- **Auto-cleanup is opt-in and interval-gated - but `skip_auto_cleanup` is Rust-only.** The
  per-commit hook returns immediately unless the dataset config carries
  `lance.auto_cleanup.interval` and the current version is a multiple of it
  (`rust/lance/src/dataset/cleanup.rs:1450-1455`), and `auto_cleanup` defaults to `None`
  (`rust/lance/src/dataset/write.rs:434`); the gate itself is an in-memory config read costing no
  I/O. When it *does* fire it is expensive exactly as documented - it "lists and reads every
  manifest in the dataset even when nothing is old enough to delete" (`write.rs:356-357`,
  listing at `cleanup.rs:680`) - so choose the interval against version-accumulation rate, not
  write rate. Rust callers can also set `skip_auto_cleanup: true` (`write.rs:367`, builder at
  `write/commit.rs:212`); **pylance does not expose it** (only `auto_cleanup_options` /
  `enable_auto_cleanup` / `disable_auto_cleanup`), so from Python the lever is the interval or
  disabling the feature.
- **Cleanup runs outside the OCC protocol, which is *why* the 7-day floor exists.** Cleanup
  writes no manifest - it is a list-and-delete pass (`.remove_stream(paths_to_delete)`,
  `cleanup.rs:752`) - and the module doc states the bind directly: "It is also difficult to
  distinguish between a data/tx/idx file which was leftover from an abandoned transaction and a
  data file which is part of an ongoing operation (both will look like unreferenced data files)"
  (`cleanup.rs:20-22`). Hence `maybe_in_progress` holds anything newer than the threshold
  (`:666-667`). The operational rule that follows: **never run cleanup with
  `delete_unverified=true` while writers are live** - that flag removes the only thing standing
  between a concurrent in-flight write and deletion of its data files.
- **Object Lock / WORM retention must be off on the bucket.** Cleanup issues real per-object
  deletes (`store.delete(&location)`, `rust/lance-io/src/object_store.rs:964`) covering
  unreferenced data files, old manifests, and transaction files; index builds also delete temp
  objects (`rust/lance/src/index/vector/ivf.rs:2489`, `ivf/io.rs:501,520`). WORM retention blocks
  those deletes outright, so maintenance fails and the store grows without bound. (Compaction
  itself issues no deletes - it only writes new fragments and lets cleanup reclaim the old ones -
  so the failure surfaces at cleanup time, not at compaction time.)
- **Turning off stable row IDs is not the whole story on remap cost.** The gate is now
  `!uses_stable_row_ids() && !options.defer_index_remap && has_address_style`
  (`rust/lance/src/dataset/optimize.rs:2060-2061`), and row-address capture is skipped entirely
  when nothing will consume it (`:1653-1657`). So a non-stable-row-id dataset with **no**
  address-style index pays no remap cost at all - the "every compaction rewrites every index
  entry" rule only bites when such an index actually exists.

  The correctness half of that gate is covered in `indexes.md` section 11.5: on a
  **stable-row-id** dataset no remap happens either, so an address-domain index (ZoneMap) has
  its rewritten fragments *dropped* from coverage rather than remapped. Before v11 the bitmap
  was instead advanced onto the new fragment ids, leaving zones pointing at dead ones. If your
  datasets predate v11 and carry a ZoneMap, probe payload-vs-live fragment ids with
  `calculate_included_frags` once and recreate any index that mismatches - upgrading does not
  repair existing damage.
- **Replace the compaction planner rather than post-filtering its output.** `CompactionPlanner`
  is a public trait and `compact_files_with_planner` accepts any implementation
  (`rust/lance/src/dataset/optimize.rs:687,928`). The load-bearing detail is that an empty plan
  short-circuits before any commit - `if compaction_plan.tasks().is_empty() { return
  Ok(CompactionMetrics::default()); }` - so a veto-style planner produces **zero churn and zero
  new versions**, which post-filtering cannot promise. Byte-aware policy needs no extra I/O
  either: `DataFile.file_size_bytes` is already carried in the manifest as `CachedFileSize`.
  Measured on a heavy-row table, deriving `target_rows_per_fragment` from bytes instead of the
  1M-row default took daily rewrites from ~190 GB to under ~100 MB.
- **Compaction bins split at index-coverage boundaries, and that will not be relaxed.** The
  planner "cannot mix 'indexed' and 'non-indexed' fragments"
  (`rust/lance/src/dataset/optimize.rs:849`); the split is load-bearing for correctness, because
  a rewrite group straddling an index bitmap creates Rewrite-vs-CreateIndex conflicts (PR
  #6610), and a commit that mixes them fails later at `load_indices` with "split of indexed and
  non-indexed". The consequence to design around: freshly compacted fragments land outside every
  per-index `fragment_bitmap`, so `unindexed_fragments()` keeps reporting them and a naive
  "compact until clean" loop never terminates. Compaction is an **operator-cadence** verb -
  running it at write cadence is the impedance mismatch.
- **Compaction non-convergence is confined to the reencode path.** Candidacy is purely
  `physical_rows < target_rows_per_fragment` (`rust/lance/src/dataset/optimize.rs:728`) - there
  is no byte term anywhere, including bin splitting, and the in-tree `CompactionOptions` doc
  admits it ("This does not affect which frgamnets need compaction", typo upstream). The
  reencode writer flushes on whichever of the row target or byte cap fires first, so a
  byte-capped task can emit fragments that are *still* under the row target and stay candidates
  forever - one measured incident churned 31 rounds x 980 MiB of net-zero rewrites. Binary copy
  cannot loop: it ignores `max_bytes_per_file` entirely and flushes on the row target at whole-
  file granularity. Two caveats before you reach for it: `compaction_mode` defaults to
  `Reencode`, so this divergence only appears after explicitly opting into
  `TryBinaryCopy`/`ForceBinaryCopy`; and a **single blob column disqualifies binary copy for the
  whole dataset**. `max_bytes_per_file` is also effectively inert when unset - the writer
  default is 90 GB.
- **`file_size_bytes` backfill is cheap on Lance-written stores.** `migrate_manifest` runs at
  every commit and issues one `ObjectStore::size` (HEAD) per data file whose size is unknown
  (`rust/lance/src/io/commit.rs:839`), in parallel. Lance's own writers set the field at write
  time, so on a Lance-written store the cost is zero; it only bites data files adopted from
  elsewhere. The larger hidden cost in the same area is that **one** fragment missing
  `physical_rows` forces `migrate_fragments` across all fragments.

## Read path and query shaping

- **Freshness is poll-only, and polling is cheaper than it looks.** There is **no `subscribe`,
  `watch`, or version-notification API anywhere in the workspace** at `v13.0.0-beta.4` - a reader
  that must see new commits polls, full stop. The good news is the cost model:
  `Dataset::checkout_latest()` (`rust/lance/src/dataset.rs:559`) on an **unchanged** version costs
  a single list/head and does **not** re-read the manifest body, so a ~100 ms poll interval is
  affordable even remotely; you only pay manifest decode when the version actually moved. Budget
  for the changed case, not the steady state. The only alternatives are in-process
  `lance::dataset_events` tracing (same process only) and MemWAL's `WalTailer` (cross-process, but
  only for MemWAL tables - and see the parallel-stack caveat in section 10).

- **A latent timezone smell in scalar-index coercion - worth knowing, not currently a bug.**
  `safe_coerce_scalar`'s same-unit arm is
  `DataType::Timestamp(TimeUnit::Microsecond, _) => Some(value.clone())`
  (`rust/lance-datafusion/src/expr.rs:437`, unchanged at `v13.0.0-beta.4`): when the literal's
  time unit already matches the column's, it returns the literal **unchanged, discarding the
  target timezone**. The other-unit branches clone the timezone correctly. At the pinned
  `datafusion-common` 54.x this is harmless - `ScalarValue::partial_cmp` for two same-unit
  timestamps ignores the timezone entirely, so ZoneMap range pruning compares values correctly.
  It is worth tracking because there are **no timezone tests anywhere under
  `rust/lance-index/src/scalar/`**, and the arm has been untouched since 2024: a DataFusion that
  makes `partial_cmp` timezone-aware turns this into silently-pruned zones.

  **Field report, older pin, not re-verified at v12:** this has been observed firing as a real
  bug rather than a latent one - `ScalarValue::partial_cmp` across mismatched timezones returns
  `None`, so `>=`/`<=` are all false, every zone is pruned, and a tz-aware
  `Timestamp(us, "UTC")` column returns 0 rows in *both* directions. It is unescapable from the
  caller: a naive literal, an explicit `+00:00`, `cast(...)`, and
  `arrow_cast(..., 'Timestamp(us,"UTC")')` all return 0, because DataFusion normalizes the
  literal to the column's unit before pushdown and lands in the same-unit arm. The three ways
  out, in increasing order of pain: **drop the index** (the only one that does not rewrite
  data), migrate the column to `Timestamp(us, None)`, or fork the workspace via
  `[patch.crates-io]`. If you see a tz-aware timestamp predicate under-returning on a
  ZoneMap-indexed column, check this first - and pin your DataFusion.
- **A bare `Ne` predicate hits a slow path; And it with `IsNotNull`.** *(Observed on a Lance
  7-10 pin, not re-verified at v12 - treat as a thing to try, not a documented behavior.)* A
  `!=` filter carrying no `IsNotNull` conjunct ran 59.5 s; And-ing `IsNotNull` on a **narrow**
  column brought the same query to 7.35 s (~8x), and a related query went 130 s -> 12-17 s.
  This is distinct from the known "no per-column null metadata" cost below. The companion lever
  from the same investigation: when two columns are written together, filter on the narrow one
  rather than the fat one - swapping `embedding_model IS NULL` for `vector IS NULL` took a scan
  from 1.2 GB read to 149 KB.
- **Answer metadata questions from the manifest, never from a column scan.**
  `count_rows("col IS NOT NULL")` reads the entire column (Lance keeps no per-column null
  metadata to short-circuit it) - on a wide text column that was ~133 MB of reads per
  call. "Does this dataset have embeddings?" is answered by index presence in the
  manifest, not by `IsNotNull("vector")` (measured 6.8-44 s per call on S3). Cache counts
  that only change on ingest.
- **Scalar and JSON indexes accelerate `WHERE` pushdown only.** A `GROUP BY
  json_get_string(col, 'name')` or a join key extracted from JSON evaluates the
  expression per row with no predicate to push down - the whole fat column ships over the
  network. Materialize hot JSON fields as narrow native columns (indexed if selective):
  this took the flagship analytics queries from >30 s timeouts on S3 to 8.5-24 s, and
  35x/14x faster locally.
- **A fat column co-located with narrow rows defeats late materialization.** Even a
  selective predicate over the narrow columns pays the fat column's page I/O when the
  rows interleave on the same pages. Keep wide payload columns out of tables you scan
  analytically, or split the hot fields out.
- **Substring search over an unindexed column is a full scan** - a BTree cannot serve
  `LIKE '%needle%'`. The official substring answers are the NGRAM index (for
  `contains()`) and the FM-Index (v8+, exact-byte only, segmented, no BM25 ranking).
  Until one is built, narrow the scan with indexed/materialized predicates first. Two
  constraints decide which index is even possible: NGRAM requires a `Utf8`/`LargeUtf8` column
  and rejects `LargeBinary` outright, and NGRAM lowercases and ASCII-folds both sides while
  FM-Index matches raw bytes - so the two return different result sets for the same needle
  (`indexes.md` section 11.2). Do not benchmark one against the other without checking
  that they are answering the same question.
- **Tune a slow vector query before rebuilding the index.** `nprobes` (IVF partitions searched)
  and `refine_factor` (candidates re-ranked against full-precision vectors) are query-time
  parameters that trade latency for recall with no reindex
  (`docs/src/quickstart/vector-search.md:208-210`). Sweep those first; a rebuild with different
  build-time parameters is the expensive last resort. Related trap: `approx_mode="fast"` does
  not reliably mean ACORN ran - it is skipped when the prefilter mask passes all rows or leaves
  under 10%, so a null result from that flag may mean the path was never entered.
- **Scalar-index pushdown does not wait for scale.** The planner emits a `ScalarIndexQuery`
  whenever the index exists, with no row-count or selectivity heuristic anywhere in the
  expression module - the plan for four rows and for two thousand is identical. Useful in both
  directions: a small table does benefit from a scalar index, and an unwanted index scan will
  not "optimize itself away" on small data.
- **`Dataset::versions()` is O(history) remote reads, not a metadata lookup.** It lists
  manifests and then **reads every one** to recover its timestamp
  (`rust/lance/src/dataset.rs:2618-2635`, which carries an upstream
  `// TODO: this API should support pagination`). On a dataset with hundreds of versions over
  remote storage this is a fetch storm - it shows up in access logs as reads of historical
  manifest versions in descending order. Use `version()` / the current manifest for "what
  version am I on"; reserve `versions()` for genuine history browsing, and never put it on a
  hot path or a health check.
- **A bitmap index turns prefix `LIKE` into an error, not a scan fallback.** With a BITMAP
  index on the column, `col LIKE 'prefix%'` fails with "LIKE prefix queries are not supported
  for bitmap indexes" (`rust/lance-index/src/scalar/bitmap.rs:823`) rather than degrading to a
  flat scan. Index choice can therefore *remove* a query shape that worked before the index
  existed - if you need both set membership and prefix matching on one column, a bitmap index
  alone is the wrong choice.
- **A blob column in SQL is a `{position, size}` struct descriptor.** Any cast or text
  operation on it surfaces as an opaque planner error (`Unsupported CAST from Struct(...) to
  Utf8View`) with nothing pointing at blobs as the cause. Read blob payloads through the blob
  APIs (`read_blobs` / `read_blob_ranges` / `take_blobs`) or
  `scanner(blob_handling="all_binary")`, not through SQL projection.

## Version-specific behavior (verify on your exact pin)

Same API, different behavior across majors - each of these was discovered in production,
not in release notes:

- **Lance 7.x: `optimize_indices` with `append()` silently full-rebuilds scalar
  (BTree/bitmap) indexes** - the O(delta) delta-segment path only works on v8+. On 7.x
  every fold rescans the whole indexed column.
- **Lance 7.0.0: incremental inverted-index (FTS) merge is broken twice over** - a
  token-id out-of-bounds panic once 4 delta segments accumulate, and an
  empty-delta-segment codec mismatch (`VarintDelta` vs `Fixed32` defaults) that poisons
  every subsequent merge. Consolidate by full rebuild (`create_index` with
  `replace=true`) instead of merging, and guard folds so an all-null tail never creates
  an empty segment.
- **Lance 7.x: `defer_index_remap=true` (Fragment Reuse Index) panics when combined with
  stable row IDs.** They are alternative solutions to the same problem - pick one. (v9
  rejects the combination cleanly.)
- **Stable-row-id datasets: `COUNT(*)` cannot use count pushdown before v9** (fixed
  upstream in PR #7360) - every count is a full scan plus a WARN. Don't count on the hot
  path; silence the warning with a scoped log directive, not a blanket mute.
- **v8 -> v9 renamed the FM-Index proto message** (PR #7397), making existing FM indexes
  unreadable after a bump. Treat any Lance major bump as an index-lifecycle event:
  re-validate fold, merge, compaction, and count behavior end-to-end on the exact pinned
  version rather than trusting API presence.
- **A Lance major bump does not imply a redesign - or a small blast radius.** The major is
  bumped automatically by `ci/publish_beta.sh` on any `breaking-change`-labeled PR, so v9 -> v10
  is the *same dev line* renamed. Read the labeled PRs, not the version delta: v10's four are
  the blob null-selection change, the cache-key change, the compaction remapper signature, and
  the MemWAL rename.
- **v10 cold-misses every cache once.** Cache keys became opaque BLAKE3 digests with no legacy
  fallback (PR #7878), so the first run after upgrading re-populates warm and persisted caches
  from scratch. Expect one slow window; do not chase it as a regression.
- **v10 changed blob API return types to `Optional`** (PR #7903). Code that zipped blob results
  positionally against its inputs was already wrong whenever a null blob appeared (results were
  omitted, not `None`); after v10 it will not compile in Rust, and in Python it silently starts
  yielding `None` entries. Audit blob call sites as part of the bump.
- **v10 may reject a caller's "table not found" assumption.** The directory namespace no longer
  reports transient storage failures as `TableNotFound` (PR #7931) - a create-or-open path that
  treated that error as "absent" could previously

CHANGELOG.md 55.8 KB

# Changelog

All notable changes to this skill will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/2.0.0/),
and this skill adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.20.0] - 2026-09-17

### Added

- The `v12.0.0` final delta (`beta.15 -> v12.0.0`, 55 commits) and a new **v13 line** section
  (`release-root/13.0.0-beta.N -> v13.0.0-beta.4`, 66 commits, 2 labeled breaking PRs) in
  `references/changelog-v7-v13.md`, plus a "The v13 delta" section in `SKILL.md`.
- #9192 (`WriteParams.file_writer_options`, the single labeled PR that re-rooted the major),
  #7465 (lazy page-metadata init: `StructuralFieldScheduler` signature, the
  `init_ranges`/`init_from_buffers` split, and the `FieldDataCacheKey` -> `PageDataCacheKey`
  change that cold-misses every warm or persisted metadata cache), and #9101 (`json_extract` /
  `json_get` no longer route to JSON indices, with the three wrong-answer bugs it fixed and the
  silent full-scan fallback it costs).
- #9072: the caller-provided Writer / `open_part` flow and `FileFragment::write_columns_from_parts`
  were removed in favour of `Dataset::concat_data_file_parts`, with caller-owned staging
  lifecycle - superseding the skill's #8923 note.
- The FRI versioned on-disk contract (#9136): `versions` -> `legacy_versions`, tagged
  `transitions` at field 2, the OrderedCompaction/StablePartition oneof, the `LSPC` counts-matrix
  physical format, the stable-row-ID exclusion, and the stricter cleanup retention rule.
- The FSL all-null compatibility fence (#9130) in both directions, and the upstream doc bug that
  cites a never-released "Lance >= 11.1.0" for a fix that shipped in `v12.0.0`.
- Rebuild/repair-class fixes #9053 (HNSW stranded nodes, a second trigger alongside #8834),
  #8507, #9204, #8941, #9084 and #8944, plus #9254 as the read-path-only contrast case.
- `hf://` object store and `hf_enable_resolve_cache` with its staleness hazard; `object_store`
  0.14 / OpenDAL 0.59; `LANCE_COMMIT_RETRY_TIMEOUT_SECS` as a third real `LANCE_*` env var.
- Net-new API surface: `Dataset::frag_reuse_index()`, `OverlayWriter`, `lance.bitmap.Bitmap`,
  `deep_clone()`, `base_paths()`, `update_columns(with_offsets=True)`, Java `DataStorageVersion`
  and `FileWriteOptions`, and bounded namespace listing (#9165).
- The stable-row-ID migration procedure with its prerequisite checklist and legacy-manifest
  pre-step, in `references/ops.md`.
- Field-verified practice: FM-Index sizing and residency (~1:1 on disk with the raw column;
  `prewarm_partitions` warms every partition, so `num_segments` cuts build RSS but not query
  RSS); the absent dataset-level identity and the `take_rows` validity test for external caches;
  why an index-free copy cannot be a dataset root minus `_indices`; the per-IVF-partition round
  trip behind remote fold latency; the `lance_arrow::json` vs `lance::arrow::json` import trap;
  time travel not being an archive mechanism; and LanceDB `create_table` being unable to enable
  stable row IDs.

### Changed

- **Breaking:** tracked tag moves to `v13.0.0-beta.4`; the **stable pin moves `v11.0.0` ->
  `v12.0.0`** (released 2026-09-17), with crates.io `lance` and PyPI `pylance` both at `12.0.0`
  and fury.io carrying `pylance-13.0.0b1` through `b4`.
- **Breaking:** `FLAG_MIXED_DATA_FILE_VERSIONS` (256) became a supported capability in the
  `v12.0.0` final and `FLAG_UNKNOWN` moved `1 << 8` -> `1 << 9`; mixed data-file versions shipped
  (#8581-#8585), so "reserved without being spent" and "still 1 of 6" are both superseded. A
  half-set paired manifest is now a hard error.
- **Breaking:** `data_storage_version` is **no longer fixed at dataset creation** - it is the
  write-time default, with per-operation targeting on update, merge-insert and compaction, and
  binary-copy compaction. V1 and V2 still cannot be mixed.
- **Breaking:** `covering_fields` is no longer a trailing suffix of `fields` (#8856); a keyed
  column may also be carried, coverage is per-segment, and V3 IVF writers do now materialize
  carried values - so "no index builder writes carried values yet" is removed.
- The v12 line's final numbers: **225 commits and 7 `breaking-change`-labeled PRs**, not 170
  and 5.
- The release train has now re-rooted on **four consecutive lines**; the 12.1 line took a
  `chore: bump main to 12.1.0-beta.0` commit and was re-rooted to 13 four commits later without
  ever being tagged.
- `lance-namespace`: Java moved to 0.12.0 (#8979), so only Python still holds 0.11.1.
- `inline_optimization_enabled` default flipped `true` -> `false` (#9180), carrying neither a
  `breaking-change` label nor a `!`.
- `references/maintenance.md` records that the `!`-vs-label signals agreed in this window (the
  inverse of the v12 case) and that the spec pages can run ahead of the code.
- Docs mirror refreshed to `v13.0.0-beta.4` (10 mirrored files changed; still 45 markdown + 4
  diagrams, so every SKILL.md directory count stands).
- `references/changelog-v7-v12.md` renamed to `references/changelog-v7-v13.md`.

### Fixed

- **Bit 1024 is documented but not implemented.** `format/table/versioning.md` lists
  `FLAG_FRAGMENT_REUSE_INDEX` as reader/writer `Yes` with the unknown boundary at 2048, while
  `feature_flags.rs` declares it above `FLAG_UNKNOWN` (512) and never reads it again, so
  `supported_flags() = FLAG_UNKNOWN - 1` refuses it. The skill now records the split and says the
  code wins.
- Row-ID sequences: upstream explicitly repudiates the 200KB inline/external threshold the skill
  stated - writers always store inline, and readers cannot load external version sequences.
- `references/ops.md` no longer claims stable row IDs must be enabled at creation, no longer
  calls 2.1 the default, and no longer marks 2.2 unstable.
- The claim that `docs/src/format/file/versioning.md` is byte-identical across the range; it
  gained a 29-line "Compatibility Caveats" section in the `v12.0.0` final.
- `references/performance.md` provenance note now records the `v13.0.0-beta.1` change to
  `guide/performance.md` (#9112), the first since `v12.0.0-beta.12`, and three code citations
  were re-resolved to their current line numbers.
- Dropped a stale usage-derived report: the `RowAddrTreeMap::from_sorted_iter` panic is fixed
  upstream - `FlatIndex::try_new` now sorts by row id before building the bitmap.

### Security

- rustls 0.23.40 -> 0.23.45 for RUSTSEC-2026-0285 (#9212). PyO3 RUSTSEC-2026-0176 and
  RUSTSEC-2026-0177 remain outstanding behind the still-open #8997.

Verified against: lance-format/lance@v13.0.0-beta.4

## [0.19.0] - 2026-09-09

### Added

- v12 delta extended to `v12.0.0-beta.15`: two more `breaking-change`-labeled PRs (#8800
  predecessor-conditioned publication, #8915 namespace merge-insert key list), taking the v12
  line from 3 to 5, plus a full `beta.6 -> beta.15` section in `references/changelog-v7-v12.md`.
- MemWAL `SsTable` accounting fields `in_memory_bytes` / `physical_rows` / `primary_key_bytes`
  (proto fields 3-5, #8981), with the "absent is not zero" reader rule.
- Blob v2 logical Arrow schema: the Minimal and Complete shapes, the row-level invariants, and
  the shape-preservation guarantee across create/append/merge-insert (#8929).
- `FilteredReadOptions` fields 13/14 (`materialization_readahead_bytes`, `batch_size_bytes`) and
  the proto governance rule exempting execution-plan schemas from the format vote.
- Net-new API surface: cleanup of specific versions (#8617), `LanceDataset.slice()` (#8059), six
  scanner options on `LanceFragment` (#8429), restored Python index retraining (#8786),
  `MemTableVisibility` (#8835), namespace-managed clone deprecated to a shim (#8964).
- Correctness fixes needing rebuild or repair rather than an upgrade: #8779 (NGRAM rebuild),
  #8510 (compaction crossed logical columns), #8984 (resurrected index), #8837 (unreopenable
  MemWAL shard), plus #8441, #8935, #8842 and the longer heals-on-upgrade list.
- MemWAL is a parallel stack: `Dataset::scanner()` has no MemWAL integration, no manifest feature
  flag marks a MemWAL table, and no SSTable compactor ships in-tree.
- Field-verified practice: append-commit coalescing via `execute_uncommitted` +
  `CommitBuilder::execute_batch`; `checkout_latest` polling cost and the absence of any
  subscribe/watch API; remote index folding as near fixed-cost per pass (~346 s for a ~200-row
  delta vs 2-4 s for a 424k-row delta locally); what the unindexed-fragment flat arm actually
  does (full scan, post-scan filter not scalar-index-accelerated, no limit/offset pushdown).
- The v2 manifest-path compatibility fence (default on; unreadable by Lance < 0.17.0) and
  `migrate_manifest_paths_v2`.
- Measured `rust-stemmers` -> `frostem` stem drift (~0.2% of English words), which makes a
  v11 binary silently miss forms in a v10-built FTS index.
- Pointers to six concepts that existed only in the docs mirror: SBBF internals, the MemWAL
  bucket-hash transform, MemWAL query planning, distributed-indexing segment grouping, FRI load
  cost and trimming, and off-write-path cleanup strategies.

### Changed

- **Breaking:** tracked tag moves to `v12.0.0-beta.15` (89 commits, 9 beta tags).
- **Breaking:** `stable` now resolves to **2.2**, which is also the enum `#[default]` and the
  default for new datasets (#8657). The skill previously called 2.1 the current default and 2.2
  "the real experimental frontier"; 2.3 remains the only code-unstable version. Upstream did not
  update the docs with this, so the code is the authority.
- **Breaking:** IVF_RQ defaults to **5 bits** per dimension, not 1 (#8936) - roughly a 4.4x
  index-size increase at the default. Per-row sizing is now `dimension/8 + 20` at 1-bit plus a
  separate multi-bit formula.
- `lance-namespace` is no longer one number: the Rust client is 0.12.0 (#8915) while the Java and
  Python pins deliberately stay at 0.11.1.
- GooseFS `storage_options` keys must be lowercase - a wrong-case key is now a hard error rather
  than silently ignored (#8940) - and block/chunk sizes accept `64MB`-style suffixes (#8943).
- `references/maintenance.md` now warns that the `breaking-change` label is a floor, not a
  ceiling, and that a refresh must also scan for `!` commits and diff documented defaults.
- Docs mirror refreshed to `v12.0.0-beta.15` (4 files changed; still 45 markdown + 4 diagrams,
  every per-directory count unchanged).
- `SKILL.md` v11 and v12 delta sections and the version landscape condensed to absorb the new
  material; full detail stays in `references/changelog-v7-v12.md`.

### Removed

- Column slice stitching (#8660). It was reverted at beta.9 (#8926); #8923's caller-managed data
  file parts replace it.

### Fixed

- `is_unstable() = self >= Next` described no recent tag and could not compile - the enum carries
  no `Ord`. The selector delegates to `resolve()`; the concrete version is `matches!(self, V2_3)`.
- Stale workspace metadata in `format-file.md`: `[workspace.package] version` (`11.0.0-beta.2` ->
  `12.0.0-beta.15`), the Python `lance-namespace` pin, the `lance-namespace-reqwest-client`
  version, and the `opendal` / `object_store` / `object_store_opendal` line citations.
- The RQ per-row sizing citation pointed at `guide/performance.md:416`, the KMeans `sample_rate`
  paragraph, rather than `:483`.
- Citation drift from the mirror refresh: `mem_wal.md` cites above line 148 shift +23, `blob.md`
  cites above line 67 shift +19.
- Pre-existing stale `object_store.md` citations (TOS, GooseFS, the GooseFS commit-handler
  quotes, Tencent COS) retargeted against the refreshed mirror.
- `performance.md` claimed `docs/src/guide/performance.md` was byte-identical across the range;
  it changed at beta.12 (#8936), so the provenance note and the RQ figures it implied were wrong.

Verified against: lance-format/lance@v12.0.0-beta.15

## [0.18.1] - 2026-09-09

### Changed
- Description condensed to fit the repo's 250-character limit.

## [0.18.0] - 2026-09-01

### Added

- Covering indexes: `IndexMetadata.covering_fields` (proto field 11), the redefinition of
  `fields` as keyed-then-carried columns, and the widened index-invalidation rule.
- `FLAG_MIXED_DATA_FILE_VERSIONS` (bit 8, reserved *at* the unknown boundary) and the
  `STICKY_PAIRED_FLAGS` carry mechanism.
- A v12 delta section in `SKILL.md` and `references/changelog-v7-v12.md`, plus a
  beta.16 -> v11.0.0-final delta the skill previously stopped short of.
- Object-store surface: `LANCE_IO_SERVER_SIDE_COPY_ENABLED`, `LANCE_DEEP_CLONE_STREAM_CONCURRENCY`,
  `LANCE_INITIAL_UPLOAD_SIZE`, `LANCE_DEFAULT_IO_BUFFER_SIZE`.
- v11-final and v12 API surface: `merge_insert` `write_mode`, `Scanner::with_row_addr_prefilter`,
  `get_deleted_row_ids`, `ObjectStore::read_dir_page`, Python `ObjectStoreProvider` registration.
- Field-verified compaction practice: the `CompactionPlanner` veto pattern, the index-coverage
  bin split, `Dataset::versions()` as an O(history) round-trip cost, and a `Ne`-predicate
  slow path (labelled as observed on an older pin, not re-verified).
- `references/maintenance.md` now warns that finals are not ancestors of `main` and that tags
  can be read without moving `HEAD`.

### Changed

- **Breaking:** tracked tag moves to `v12.0.0-beta.6`; the stable pin becomes `v11.0.0`
  (crates.io `lance` 11.0.0, PyPI `pylance` 11.0.0, GitHub Releases `Latest`).
- **Breaking:** manifest bit 128 is `FLAG_COVERED_INDEX_METADATA`, not
  `FLAG_MEM_WAL_INDEX_CATCHUP` - the MemWAL bit was retired in `v11.0.0` final and the bit
  reclaimed. Builds pinned in `v11.0.0-beta.4`..`beta.17` open a covering dataset instead of
  refusing it.
- MemWAL: a shard absent from `index_catchup` now unconditionally means *unknown*; the
  flag-gated "absence means caught up" reading and the one-way-flag rule are gone.
- Index invalidation keys off any column in `fields`, not only the indexed column.
- `merge_insert` composite keys now index-accelerate per column.
- `ShardManifestStore::read_latest` -> `latest`, `read_latest_uncached` -> `refresh_latest`,
  `write` is crate-private.
- The v11 delta is now stated against the final: 357 commits and 16 breaking PRs, up from
  313 and 14 at `beta.16`.
- The release-train note records **three** consecutive re-rooted lines; the 11.1 line never
  got even one beta tag.
- `references/changelog-v7-v11.md` renamed to `references/changelog-v7-v12.md`.
- Docs mirror refreshed to `v12.0.0-beta.6` (6 files changed; still 45 markdown + 4 diagrams,
  every per-directory count in `SKILL.md` unchanged).

### Removed

- `FLAG_MEM_WAL_INDEX_CATCHUP` and proto field
  `Transaction.UpdateMemWalState.require_index_catchup`.

### Fixed

- `references/ops.md` listed `request_timeout` as a `storage_options` key. It exists in no
  Lance source at any recent tag; the real key is `timeout`.
- Documented the v11 fix whereby address-domain indexes (ZoneMap) lose coverage of rewritten
  fragments under compaction instead of falsely claiming them. New compactions are safe; an
  index damaged under v10 or earlier still needs recreating.
- Corrected the `safe_coerce_scalar` citation (`expr.rs:302` -> `:311`) and added the field
  report plus the three available escape hatches.

Verified against: lance-format/lance@v12.0.0-beta.6

## [0.17.1] - 2026-08-21

### Changed

- Declared ClawHub browse categories (`development, integrations`) and topics in `metadata`, so the release pipeline publishes them instead of leaving the skill in the `other` category.

### Removed

- `skill-card.md`. The ClawHub CLI strips a root `skill-card.md` from every publish and the registry generates its own card, so the authored file never reached ClawHub.

## [0.17.0] - 2026-08-21

### Changed

- Condensed `SKILL.md` from 24,463 to ~18,300 chars by trimming "The v11 delta" from ~10,400
  chars (42% of the file) to a summary plus a pointer. **No information was lost**: all 54 PR
  citations in that section already appeared verbatim in `references/changelog-v7-v11.md`, which
  exists to hold exactly this content. The retained summary keeps the five changes that break
  upgraders, the manifest feature-flag change, the `LANCE_*` grep trap, and the
  heals-on-upgrade vs requires-rewrite split. Rationale: the per-PR delta is needed only by
  someone upgrading across majors, so it satisfies the conditional-loading test for living in a
  reference file, and the entry point was within 550 chars of the 25k recommended budget.
- Corrected the `check_fragment_ids` citation to `rust/lance/src/io/commit.rs:687`.

### Added

- `format-table.md` section 5.3: "Legacy manifests and fragment resolution", documenting that
  `Dataset::get_frags_from_ordered_ids` resolves ids as
  `manifest.fragments[fragment_bitmap.rank(id) - 1]` - correct only when `manifest.fragments` is
  sorted by id, guarded only by a `debug_assert_eq!` that compiles out in release. Its sibling
  `find_fragment` was given a check-and-fall-back guard for exactly these legacy shapes (#8636).
  Covers which legacy shape is reachable (unsorted pre-0.10 manifests pass every commit-time
  check; duplicate-id manifests are largely blocked by `check_fragment_ids`, whose `windows(2)`
  scan only detects *adjacent* duplicates), and how consequences differ by caller -
  `take.rs:305` re-checks the returned id and so drops rows, while `index/scalar.rs:233` does
  not and can train an index on the wrong fragments. **Flagged explicitly as a static finding,
  not a demonstrated defect**: read at `v11.0.0-beta.16`, not reproduced, and not a reported
  upstream issue. Notes that the upstream test which appears to cover this varies only the query
  array order, not the manifest order.

## [0.16.0] - 2026-08-21

### Changed

- Re-grounded from `v11.0.0-beta.6` to `v11.0.0-beta.16` (91 commits, 1 newly
  `breaking-change`-labeled PR: #8235).
- v11 delta figures re-grounded: 222 commits / 13 breaking PRs (accurate at beta.6) ->
  **313 commits / 14 breaking PRs** at beta.16. All other structural invariants re-verified
  and unchanged (26 crates, 16 transaction ops, `num_retries` 20, `next => 2.3` / default 2.1,
  arrow 58, datafusion 54, MSRV 1.91.0, Edition 2024, Python 3.10+, manifest feature flags).
- **Corrected:** "No new `LANCE_*` env vars landed in v11" was false in two places. `LANCE_DISABLE_AMX`
  (runtime kill switch) and `LANCE_AMX_FP16_CC` (build-time compiler override) both landed in
  beta.16. Added the grep trap that `LANCE_AMX_CFG_*` / `LANCE_AMX_TILE_COUNT` are C macros,
  not env vars.
- **Corrected:** the v10 external-manifest note is superseded by #8499 - object storage is now
  authoritative, the external store's put-if-not-exists is a *reservation* rather than the
  commit point, and a stored ETag must be ignored rather than trusted (a retained one makes
  readers reject a good manifest with `Manifest e_tag mismatch`).
- **Corrected:** "stable row IDs cannot be turned on later" is superseded by
  `Dataset::migrate_to_stable_row_ids` (#8521).
- Three citations retargeted: `rust/lance/src/dataset/transaction.rs` was deleted upstream
  (#8053/#8054/#8056); the code now lives in `rust/lance-table/src/transaction/`. Documented
  that the `lance::dataset::transaction` re-export shim survives, so the common surface holds.
- Java SDK doc link fixed: `lance-format.github.io/lance-java-doc` 404s; upstream points at
  javadoc.io. Added the canonical `lance.org` docs domain.
- `references/ops.md` proto list completed (12 protos exist, 9 were listed).
- `references/maintenance.md` mirror-deviation #3 corrected: the trailing-newline hook adds a
  byte to all **four** `.drawio.svg` diagrams, not one; recorded the two-step normalization that
  reproduces the mirror byte-for-byte.
- `references/performance.md` Part A provenance updated - `guide/performance.md` changed again
  at beta.16 (+29 lines, AMX).
- Part B: the fsync-wrapper cost "was not detectable" is superseded by a measured **+4.2%**
  (5.54 s on a 130.86 s sync, macOS `F_FULLFSYNC`), with the per-file amortization that explains it.
- Refreshed the 4 stale files in the `references/docs/` mirror; counts unchanged (45 md + 4 svg).

### Added

- AMX-FP16 IVF acceleration: the three shape gates, the switch from approximate to **exact**
  partition assignment (recall and assignments both change), the `LANCE_DISABLE_AMX` kill switch,
  and the clang >= 16 / gcc >= 13 build requirement.
- `version_refs()` / `VersionRef`; `Dataset::migrate_to_stable_row_ids`; compaction
  `max_source_rows` / `max_source_bytes` / `excluded_fragment_ids`; `FileFragment::write_columns`.
- MemWAL shard pruning; `VersionAuxData`; `IndexSection`; `TableIdentifier`'s two remote
  reconstruction modes; Substrait filter/aggregate parsing; the `DiskAnn` proto stage.
- `file+uring://` added to the object-store scheme list, with the note that `is_local()` covers it.
- Cache and observability guidance: Lance has **no resident data cache**; shared `Arc<Session>`
  via `DatasetBuilder::with_session`; `prewarm_index` for cold search (and what it does not fix);
  `MergeStats` and `Scanner::scan_stats_callback`.
- The `merge_insert` "Ambiguous merge inserts are prohibited" error documented as deterministic
  and **non-retryable** - an OCC retry loop must not swallow it.
- `changelog-v7-v11.md` gained a beta.6 -> beta.16 delta splitting the correctness fixes into
  **heals-on-upgrade** vs **requires rewriting data on disk**, plus the #7966 ZoneMap caveat
  (reads heal, pruning selectivity does not until reindex).

### Security

- Noted #8613: `h2` bumped to 0.4.16 for RUSTSEC-2026-0258.

Verified against: lance-format/lance@v11.0.0-beta.16

## [0.15.0] - 2026-08-12

### Changed

- Re-grounded from `v11.0.0-beta.2` to `v11.0.0-beta.6` (94 commits, 4 newly
  `breaking-change`-labeled PRs: #8027, #8028, #8347, #8360).
- **Breaking (upstream):** the stable pin moved `v9.0.1` -> `v10.0.0`. `v10.0.0` final **was**
  tagged (2026-08-08, annotated, on `release/v10.0`, not an ancestor of `main`); the skill's
  previous claim that it never was is corrected everywhere it appeared. crates.io and PyPI both
  now carry `10.0.0`.
- **Breaking (format):** the v11 delta's "manifest feature flags unchanged" is corrected -
  `FLAG_MEM_WAL_INDEX_CATCHUP` (bit 128) is new and `FLAG_UNKNOWN` moved 128 -> 256. The
  feature-flag table gained bits 32, 64, and 128 (32 and 64 existed before v11 but were
  undocumented).
- v11 delta figures re-grounded: 128 commits / nine breaking PRs (accurate at beta.2) -> 222
  commits / 13 breaking PRs at beta.6.
- Vector-index storage spec: `__ex_codes` -> `__blocked_ex_codes` with a new sizing formula
  (readers still accept the old column); `__pq_code` and `_rabit_codes` sizings corrected; all
  storage columns are now nullable.
- `cos://` no longer routes to `ConditionalPutCommitHandler` - it has a dedicated
  `TencentCosCommitHandler` that fails closed.
- Part B's governing rule softened from "don't tune the store" to "minimize remote calls first",
  presenting upstream's new remote-scan tuning table as a legitimate second move.
- Refreshed the 13 stale files in the `references/docs/` mirror; all directory counts verified
  unchanged (45 md + 4 svg).

### Added

- Scan concurrency controls (`fragment_readahead`, `batch_readahead`, `scan_in_order`,
  `io_buffer_size`) with upstream's suggested starting values and the two counter-intuitive
  caveats (`scan_in_order` does not serialize fragment reads; lowering `batch_size` may not
  shrink the request).
- FTS tokenization surface: `lance.tokenize` / `FtsToken`, the `max_token_length` tri-state,
  `analyze_plan` tokenized-query output, and the conditions under which `CompoundQueryExec`
  abandons the posting-backed scorer.
- Stable row ids in hand-assembled distributed transactions (`row_id_meta`, never minting ids,
  leaving `*_version_meta` as `None`); the Tencent COS `commit_lock` requirement; nested Blob v2
  fields; `preserves_nullability` and its conflict rule.
- A consolidated roundup of v11's eleven silent-corruption and wrong-results fixes, each with its
  triggering condition.
- New changelog section for the `v11.0.0-beta.2 -> v11.0.0-beta.6` delta.
- Field-verified operational findings, each re-grounded at beta.6 before inclusion: the FTS
  `Fixed32` empty-segment merge failure (scoped to `format_version=1`/legacy), bfloat16 rejection
  on both build and query paths, the three-round-trip commit anatomy, `merge_insert` composite-key
  indexing, compaction convergence, the unindexed-backlog API, and ngram relevance/RAM cost.

### Fixed

- `MAX_INLINE_TRANSACTION_BYTES` is gated on `#[cfg(not(test))]`, not "release builds", so every
  non-test build gets 20 MiB.
- Mirror-exclusion note: Spark/Ray/Trino were deleted from the checked-in nav (#8419) rather than
  assembled at build time.
- Part A provenance: `guide/performance.md` is no longer byte-unchanged since `v9.1.0-beta.8` -
  it changed at `v11.0.0-beta.4`.
- `maintenance.md` now documents a third expected pre-commit deviation (a trailing newline added
  to the `.drawio.svg`).
- Corrected a pond-derived lead before publication: the ZoneMap/tz-aware-timestamp issue is a
  latent coercion smell, **not** a "returns 0 rows" bug - the pinned `datafusion-common` 54.x
  `partial_cmp` is timezone-blind, so pruning is currently correct.

Verified against: lance-format/lance@v11.0.0-beta.6

## [0.14.1] - 2026-08-07

### Fixed

- Completeness audit against the pre-split originals rescued three items that the condensation
  would otherwise have dropped: the OpenTelemetry-metrics subsection (the only text unique to
  the embedded performance-guide copy, now its own Part A section); the "authoritative in-repo
  sources" pointer from the old reference preamble (now in the `lance-reference.md` index); and
  the generated per-language SDK docs URL (restored to the ecosystem paragraph in `SKILL.md`).
  Verified by line-level `comm` against `git show HEAD:` originals: zero content lines, zero of
  57 section headings, zero of 271 PR citations, and zero of 48 measured-figure tokens lost.

## [0.14.0] - 2026-08-07

### Changed
- Consolidated the skill's structure. No upstream re-grounding: still `v11.0.0-beta.2`, and no
  benchmark-verified figure was removed - every relocated fact is cited below.
- `SKILL.md` 32,029 -> ~14.6k chars. The four "What's new in v9 / v9.1 / v10 / v11" sections
  collapsed into a `Version landscape` table (one row per major, naming its breaking theme) plus
  a compressed `The v11 delta`; all 66 PR citations were verified present in the reference files
  before cutting. The 45-row docs-mirror file map became one row per directory with a file count
  (the "Not mirrored" paragraph is unchanged, verbatim). The crate-workspace section, the
  release-train prose, and `Navigating the reference` were compressed; the sparse
  auto-selection deep dive was dropped in favour of the fuller treatment in section 3.1.
- `references/lance-reference.md` (170,653 chars) split into five topic files, each with its own
  table of contents: `format-file.md` (sections 1-4), `format-table.md` (5-10), `indexes.md`
  (11-12), `ops.md` (13, 15, 16), `changelog-v7-v11.md` (14). `lance-reference.md` is now a stub
  mapping the original 16 section numbers onto those files, so existing "see section N"
  cross-references still resolve.
- `references/performance.md` 69,077 -> ~38.6k chars, and gained a table of contents. Part A no
  longer re-copies text that already exists byte-identically in `references/docs/`: the embedded
  `guide/performance.md` copy, the full-text-search "Performance Tips", the JSON "Performance
  Considerations", and the transaction "CreateIndex Compatibility" block are replaced by a
  routing table pointing at the mirrored files and headings. The OpenTelemetry-metrics
  subsection, which was the only content unique to the embedded copy, was relocated to its own
  section in Part A. Both "Performance changes not in the guide" subsections and all of Part B
  are unchanged.

### Added
- `references/maintenance.md` - the refresh workflow moved out of `SKILL.md`, extended with the
  reference-file layout table.


## [0.13.0] - 2026-08-07

### Changed
- Re-grounded against upstream `v10.0.0-beta.7` -> `v11.0.0-beta.2` (128 commits, 9
  breaking-labeled PRs). Retitled "Lance v10" -> "Lance v11"; bumped the workspace pin,
  permalink base, and citation tag across `SKILL.md`, both references, and `skill-card.md`.
- **Release-line correction**: `v10.0.0` FINAL was never tagged - `release/v10.0` sits at
  `10.0.0-rc.3` and branched exactly at `v10.0.0-beta.7`. The `10.1.0-beta.*` line was
  re-rooted in place as `11.0.0-beta.*` by the release bot; both release-root tags share
  base `10.0.0-rc.1`. The stable pin is now **`v9.0.1`** (2026-08-06), matching crates.io.
- **Module reorganization** (section 2.1, new): `lance-encoding::version` deleted -
  `LanceFileVersion` and `ConcreteFileVersion` now live in `lance-file::version`. `FileWriter`
  became an enum, `lance_io::encodings` and `lance-encoding::previous` were removed, and
  per-version `versions/v2_{0,1,2,3}` / `array_encoding` trees replaced them.
- **Breaking (format-level)**: fragment ids are now a dataset-lifetime high-water mark -
  overwrite no longer restarts at 0, overwrite fragments carrying a deletion file are
  rejected, and duplicate fragment ids block all commits (#8206).
- Commit-handler routing table corrected: `goosefs` (#8134) plus the already-missing
  `abfss` / `tos` / `shared-memory` all route to `ConditionalPutCommitHandler`.
- Dependency pins: `opendal 0.57 -> 0.58.1`, `object_store_opendal 0.58`; `strum` and
  `goosefs-sdk` dropped from workspace deps; `rust-stemmers -> frostem` (Greek panic fix).
- Doc mirror resynced: 4 changed files (`guide/migration.md`, `guide/object_store.md`,
  `guide/observability.md`, `quickstart/full-text-search.md`). No files added or removed.
- Compressed the v9/v8/v7 history sections in `SKILL.md` to make room for v11 under the
  repo's 500-line cap; stripped stale "(current tag)" labels from historical delta headings.

### Added
- New FTS index axis: `DocumentGranularity` (ROW / LIST_ELEMENT), `posting_format_version`,
  the `_doc_index` column, and `list_element` as a third trigger requiring FTS format v3.
- Zone map `has_null_bitmap` (making `IS NOT NULL` scan-free) and all-type support, with
  null counts only for nested types.
- Compound FTS scoring core - Boolean/Phrase/Boost composition, public `CompoundQueryExec`,
  cost-ordered conjunctions; `AND` clauses are scoring `MUST` clauses that affect `_score`.
- Manifest transaction spilling above 20 MiB; pluggable cache-backend registry (`moka://`);
  `CacheBackend::deep_size_of_entries` and its effect on reported cache sizes.
- Object store: `aws_provider_scheme` (token / ecs / irsa); the GooseFS conditional-put
  migration and its mixed-version overwrite hazard; multipart-retry part-identity fix.
- Query-time vector knobs `nprobes` and `refine_factor`; the `when_not_matched_by_source_*`
  merge-insert family; the FTS 18-language roster and text-vs-json document types;
  jieba/lindera user dictionaries; MemWAL GC and reader-consistency semantics; dense-vs-sparse
  data-overlay shapes; the JSON projection limitation; the Blob v2 rewrite-migration path; and
  the wider ecosystem (Flink, pglance, Lance Graph, named catalog implementations).
- `performance.md`: a new "Local-filesystem crash safety and recovery" subsection (no
  fallback to version N-1 on a corrupt manifest; `latest_version_hint.json` is not read on
  local; `count_rows` cannot validate integrity), plus auto-cleanup gating economics,
  `LANCE_MEM_POOL_SIZE` sizing, `optimize_indices` delta-collapse semantics, WORM/Object Lock
  incompatibility, and `memory://` vs `shared-memory://` test-isolation traps.
- `lance-reference.md`: the SQL/DataFusion surface has no kNN; dataset *creation* is not
  OCC-protected; the FRI is not per-index coverage; NGRAM vs FM-Index matching semantics; the
  benign IVF_PQ empty-partition warning; volume-independent scalar-index pushdown.

### Fixed
- Sparse-writer citation corrected to `encoding.md:373-375`.
- ACORN-1 nuance added: skipped when the prefilter mask passes all rows or leaves under 10%,
  with fallback to `search_basic`.

Verified against: lance-format/lance@v11.0.0-beta.2

## [0.12.0] - 2026-07-30

### Changed
- Re-grounded against upstream `v9.1.0-beta.8` -> `v10.0.0-beta.7` (78 commits, 4 breaking);
  bumped workspace version pin, permalink base, and citation tag. Retitled "Lance v9" ->
  "Lance v10". Resynced the 5 changed doc-mirror files.
- **Release-line correction**: `v9.0.0` FINAL shipped 2026-07-24 and is now the stable pin
  (supersedes `v8.0.0`). The `9.1.0-beta.*` dev line was mechanically re-rooted as
  `10.0.0-beta.*` by CI breaking-change detection, so `v9.1.0` was never tagged. `v9.0.0`
  lives on `release/v9.0` (now `9.0.1-beta.0`) and is not an ancestor of `main`. crates.io
  publishes finals only - newest is `lance 9.0.0`, so beta pins are git dependencies.
- **Crate pins**: `lance-arrow-stats` is also pinned `=58.0.0` (the skill previously named only
  `lance-arrow-scalar`); new workspace dep `blake3 1.8.5`.
- File format 2.3: sparse structural pages are now **auto-selected** by the 2.3 writer under a
  rep/def budget heuristic (PR #7756); `structural-encoding` reworded "Select" -> "Force".
- MemWAL vocabulary overhaul across spec, Rust, Python, Java, and proto (section 10), with a
  full rename map.
- `performance.md`: corrected the "`cleanup_older_than` defaults to ~1 hour" claim - Python
  `cleanup_old_versions` defaults `older_than` to 14 days; the 3600s figure is the docs'
  `lance.auto_cleanup.older_than` example, not a library default.

### Added
- Section 14: `v9.1.0-beta.8 -> v10.0.0-beta.7` delta subsection.
- Section 3.6: `ConcreteFileVersion` exact-identity type, the DataFile encode/decode wire
  table, and the byte-exact writer fixtures.
- Section 3.5: `read_blob_ranges` as the fourth blob read path, plus the null-preservation
  signature table.
- Section 9.4: cache keys and backend - `CACHE_KEY_FORMAT = "blake3-128-v1"`, removed cache
  APIs, `QuickCacheBackend`, and the per-shard admission ceiling that silently refuses
  oversized entries.
- Section 11.1: ACORN-1 prefiltered HNSW traversal (opt-in, `approx_mode="fast"`) with its
  documented recall regression; vector append across heterogeneous segment models.
- Section 11.2/12: segmented index family extended to BLOOMFILTER, RTREE, NGRAM, LABEL_LIST;
  `IndexSegment::new` signature change; the NGRAM merge-before-commit constraint.
- Section 11.3: FTS `total_tokens` metadata key, `LANCE_FTS_SEARCH_CHUNK`, top-k row-id
  resolution, deterministic tie ordering, `bm25_search` removal.
- Sections 5.5/9.1: data-overlay/index correctness work and the proto-rename impact.
- Section 13: `memory://` fix, env-var validation, tokio-shutdown fix, and the namespace
  error-classification change.
- `performance.md` Part A: a source-derived "Performance changes not in the guide" subsection
  (cache admission ceiling, FTS chunking, top-k row-id resolution, concurrent segment commit).
- `performance.md` Part B: `Dataset::versions()` O(history) manifest reads, the 7-day
  unverified-file floor, transient index-set doubling on `replace=true`, typed commit-conflict
  errors, `merge_insert` mode switching on source schema shape, bitmap-index prefix-LIKE
  erroring, blob-column SQL descriptors, and local-FS durability delegation.
- SKILL.md: an explicit note that `docs/src/images/` is not mirrored.

### Changed (breaking, upstream)
- **Blob APIs preserve null selections** (#7903, the PR that triggered the v10 bump): Rust
  `take_blobs*` -> `Vec<Option<BlobFile>>`, `ReadBlob::data` -> `Option<Bytes>`; Python
  `read_blobs -> List[Tuple[int, Optional[bytes]]]`, `take_blobs -> List[Optional[BlobFile]]`;
  Java lists may contain null elements.
- **Cache keys are an opaque 16-byte BLAKE3 digest** (#7878) - all warm/persisted caches
  cold-miss after upgrade, no legacy fallback; `invalidate_prefix`, `LanceCache::keys`, and
  `Session::*_cache_keys` removed.
- **`IndexRemapperOptions::create_remapper` is now async**, returning
  `Result<Option<Box<dyn IndexRemapper>>>` (#7778).
- **MemWAL renames** (#7943, #7957): proto `FlushedGeneration` -> `SsTable`,
  `MergedGeneration` -> `CompactedSsTable`, `flushed_generations` -> `sstables`,
  `merged_generations` -> `compacted_sstables`. Wire-compatible (field numbers unchanged) but
  every generated symbol and binding name changes; no deprecation shims.
- `LanceFileVersion::try_from_major_minor` and `to_numbers` removed (#7879).
- `InvertedPartition::bm25_search` removed (#7863).
- Directory namespace no longer collapses storage failures into `TableNotFound` (#7931).

### Security
- `quinn-proto` 0.11.14 -> 0.11.16 (Dependabot security alert) across the root workspace,
  `/python`, and `/java/lance-jni` (#7983, #7984, #7982).

Verified against: lance-format/lance@v10.0.0-beta.7

## [0.11.1] - 2026-07-22

### Added

- skill-card.md release record following NVIDIA's skill-card format

## [0.11.0] - 2026-07-22

### Changed
- Re-grounded against upstream tag `v9.0.0-beta.18` -> `v9.1.0-beta.8` (127-commit range,
  1 breaking-labeled PR); bumped workspace version pin, permalink base, and citation tag.
  Copied the changed doc-mirror subset (12 files + 1 new) from the new tag.
- **Crate workspace 25 -> 26**: new `lance-index-core` crate (PR #7713).
- **Transaction ops 15 -> 16**: new `DataOverlay` operation (env-gated unstable; release
  builds refuse overlay datasets), sections 9.1 + 5.5.
- **datafusion 53 -> 54** (PR #7793); geodatafusion 0.4 -> 0.5. Build toolchain (not MSRV)
  1.91 -> 1.97 (#7712); MSRV `rust-version` unchanged at 1.91.0. Python min still 3.10
  (3.14 support added, #7728).
- File-format 2.3 is **no longer scaffolding-only**: sparse structural pages shipped
  (PR #7889, `sparse.rs`); `lance-encoding:structural-encoding=sparse` selects it (requires
  2.3). Corrected the "6 refs vs 98, no distinct encodings" claim (now 59 vs 97).

### Added
- Section 5.5: **Data Overlay Files** - cell-level `(row offset, field)` updates without
  rewriting base data files; new `DataOverlay` transaction op, feature flag 64, spec
  `data_overlay_file.md` (unstable, `LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES`) (PR #7535/#7536).
- Section 3.1: sparse structural pages / `sparse` encoding (Lance 2.3, PR #7889).
- Section 11.2: zonemap + bloom-filter indexes now carry a `null_bitmap` -> **exact IS NULL**.
- Section 11.3: FTS configurable posting `block_size` (128/256, 256 experimental, format-v3
  gate) (PR #7466); FTS code-analyzer tokenizer (PR #7681); nested-field FTS (PR #7686);
  bulk MAXSCORE / impact-skip / conjunction paths (#7602/#7603/#7624).
- OpenTelemetry metrics for Python (`instrument_lance_metrics`, `pylance[otel]`, PR #7537),
  noted in `performance.md`.
- Section 14: new `v9.0.0-beta.18 -> v9.1.0-beta.8 delta` subsection.
- `references/docs/format/table/data_overlay_file.md` mirrored; SKILL.md format-specs file map
  gains its row.

### Changed (breaking, upstream)
- **FTS/inverted-index creation takes a `block_size` param** (compressed posting blocks;
  128/256, default 128; 512 rejected). `block_size=256` and the code analyzer require FTS
  on-disk **format v3** (PR #7466, #7866). Section 11.3.

Verified against: lance-format/lance@v9.1.0-beta.8

## [0.10.1] - 2026-07-10

### Changed
- CHANGELOG preamble pinned to Keep a Changelog 2.0.0 (format unchanged; KaC 2.0.0 keeps existing changelogs valid).

## [0.10.0] - 2026-07-08

### Added
- `references/docs/` - a verbatim mirror of the official docs (`docs/src` at the tracked
  tag): all 14 guides, 4 quickstarts, the complete format spec tree (file / table / index,
  including the 4 index-lifecycle SVG diagrams), and `integrations/datafusion.md` -
  48 files, unedited. Ends doc cherry-picking: every official page is now directly
  loadable from the skill.
- `references/performance.md` - all official performance guidance combined in one
  document (the full performance guide incl. the new Fragment Sizing section, the FTS
  quickstart performance tips, JSON performance considerations, and the
  CreateIndex-compatibility passage from the transaction spec), followed by a
  field-verified Part B: benchmark-backed remote-storage practices (commit count as the
  cost unit, append vs merge_insert, index-fold batching, `fast_search` recall rule,
  cleanup gating, manifest-not-scan metadata questions, narrow-column materialization,
  version-gated v7/v8/v9 behavior, benchmarking traps). Part B's governing rule: leave
  store knobs at defaults and optimize by minimizing remote calls.
- SKILL.md: full routing file map for the docs mirror, a "Performance questions" section,
  and a three-layer reference navigation intro.
- Section 14: new `v9.0.0-beta.16 -> v9.0.0-beta.18 delta` subsection (36 commits, no
  breaking changes; pylance prewarm segment selection #7677, object-store metrics #7533,
  RLE v2 widths #7376, FTS/MemWAL/JSON fixes).

### Changed
- Re-grounded against upstream tag `v9.0.0-beta.16` -> `v9.0.0-beta.18`; bumped workspace
  version pin, permalink base, and citation tag. Copied doc files verified identical
  between the tags except `guide/performance.md` (+31 lines, Fragment Sizing),
  `guide/read_and_write.md` (cleanup + auto-cleanup docs), and the new
  `guide/observability.md`.
- Maintenance instructions now cover refreshing the docs mirror and performance.md at
  each version bump.

Verified against: lance-format/lance@v9.0.0-beta.18

## [0.9.0] - 2026-07-06

### Changed
- Re-grounded against upstream tag `v9.0.0-beta.10` -> `v9.0.0-beta.16` (commit
  `78a814b6b`); bumped workspace version pin, permalink base, and citation tag.
  58-commit range, 1 breaking change. All structural invariants reverified
  unchanged: 25 crates, arrow 58, datafusion 53, opendal 0.57, jieba-rs 0.10,
  itertools 0.14, lance-namespace-reqwest-client 0.8.6, rust 1.91.0, resolver 3,
  edition 2024, version enum (`Next => 2.3`, default `V2_1`), 15 transaction ops,
  `CommitConfig num_retries = 20`.
- SKILL.md: `v8.0.0` FINAL shipped 2026-07-01 (was "rc.3, no final tag yet") -
  use `v8.0.0` as the stable pin.
- Section 3.5: blob read APIs reworked in the docs (PR #7530, #7558) - `read_blobs`
  (full payloads, batched through the scheduler) is now the primary path,
  `take_blobs` reserved for streaming/seeking, `scanner(blob_handling="all_binary")`
  for Arrow binary columns. Added Blob v2 auto-tiering defaults (<16 KiB inline /
  mid-size shared `.blob` sidecar / >2 MiB dedicated) and the new
  `lance-encoding:blob-pack-file-size-threshold` field-metadata key (PR #7322).
- Section 13: per-base `storage_options` scoping via `base_<id>.<key>` keys, with
  `initial_bases` id assignment and `base_store_params` precedence (PR #7608).

### Added
- Section 14: new `v9.0.0-beta.10 -> v9.0.0-beta.16 delta` subsection.
- Section 11.2: ZoneMap min/max read without a scan (`zonemap_value_range`, PR #7463);
  BTREE + ZONEMAP scalar indices now accept `large_string`/`LargeUtf8` (PR #7525).
- Section 6: schema evolution now allows adding all-null `Map` columns (PR #7462);
  multi-base merge-insert with target-base routing (PR #7610).
- Section 10: prefiltered LSM vector + FTS search across base/flushed/in-memory
  mem-wal sources (PR #7138).
- DirectoryNamespace now implements `update_table` / `delete_from_table` (PR #6923)
  and `alter_transaction` (PR #6974).

### Changed (breaking, upstream)
- FTS / inverted indexes now default to on-disk **format v2** (PR #7512, 9.0.0
  migration note) - `LANCE_FTS_FORMAT_VERSION` no longer controls new indexes; pass
  `format_version=1` for older-reader compatibility. Existing v1 indexes stay
  queryable and are maintained as v1 (append/optimize/mem-wal flush). Section 11.3.

Verified against: lance-format/lance@v9.0.0-beta.16

## [0.8.0] - 2026-07-01

### Changed
- Re-grounded against upstream tag `v8.0.0-beta.14` -> `v9.0.0-beta.10` (commit
  `e25b71e74`); retitled "Lance v8 reference" -> "Lance v9 reference", bumped the
  workspace version pin, permalink base, and citation tag. 129-commit range, major
  version boundary. All structural invariants reverified unchanged: 25 crates, arrow 58,
  datafusion 53, opendal 0.57, jieba-rs 0.10, rust 1.91.0, resolver 3, edition 2024,
  version enum (`Next => 2.3`, default `V2_1`), 15 transaction ops, `CommitConfig
  num_retries = 20`.
- The v9 major bump is auto-triggered by `ci/check_breaking_changes.py` (GitHub
  `breaking-change` label detection), fired by #7158 and #7345 - not by the FMIndex rename.
- Dep pins: `lance-namespace-reqwest-client` 0.8.4 -> 0.8.6; `itertools` 0.13 -> 0.14.
  pylance runtime `lance-namespace>=0.8.5,<0.9` unchanged.
- Section 3.1: docs version table (`file/versioning.md`) now lists **2.3 as unstable** and
  no longer labels 2.2 unstable (was "docs still list only 2.2").
- Section 3.3: miniblock value chunks now tunable up to 32k via `LANCE_MINIBLOCK_MAX_VALUES`
  (PR #7356; default stays 4096).
- Section 7: `cleanup` / cleanup-explain now exposed to Python and Java (PR #7248).
- Section 6: `alter_columns` now allows Dict <-> value-type casts (PR #7289).

### Added
- Section 14: new "v8.0.0-beta.14 -> v9.0.0-beta.10 delta (major-version boundary)"
  subsection covering the three breaking changes, the `as_vector_index` removal, and
  net-new features.
- SKILL.md: note that v8.0.0 is the concurrent stabilizing release (rc.3) for users who
  need a stable pin instead of the v9 dev betas.
- Section 11.1: hamming clustering / near-dup detection utility over 64-bit binary hashes
  (`pairwise_hamming_distance`, `UnionFind`, `hamming_clustering_for_ivf_partition`,
  PR #7379); COUNT(*) pushdown now works on stable-row-id datasets (PR #7360).
- Section 3.5: per-column blob inline/dedicated thresholds
  (`lance-encoding:blob-inline-size-threshold` / `...-dedicated-size-threshold`, PR #7269).
- Section 11.2: ngram index now accelerates regex and infix LIKE (PR #7139).
- Section 11.3: ICU split tokenizer variant `icu/split` (PR #7474); mixed-language FTS stop
  words (PR #7324).
- Section 12: distributed LabelList scalar index builds (PR #7223).

### Removed
- Section 11.1: `as_vector_index` removed from the public `Index` trait (PR #7392);
  callers downcast via `as_any()`.

### Changed (breaking, upstream)
- FM-Index proto message renamed `FMIndexIndexDetails` -> `FMIndexDetails` (PR #7397) -
  existing FM indexes become unreadable (sections 11.2, 16).
- Python 3.9 dropped; minimum is now 3.10 (PR #7345) - section 2 binding note.
- `alter_columns` cast now fails-fast when the column has an attached index; drop the index
  first (PR #7158) - section 6.

Verified against: lance-format/lance@v9.0.0-beta.10

## [0.7.0] - 2026-06-16

### Changed
- Re-grounded against upstream tag `v8.0.0-beta.9` -> `v8.0.0-beta.14` (commit
  `c188de59f`); bumped the workspace version pin, permalink base, and citation tag.
  31-commit range, 2 breaking changes (both RaBitQ/vector). All structural invariants
  reverified unchanged: 25 crates, arrow 58, datafusion 53, opendal 0.57, jieba-rs 0.10,
  rust 1.91.0, resolver 3, edition 2024, version enum (`Next => 2.3`, default `V2_1`),
  15 transaction ops, `CommitConfig num_retries = 20`.
- Dep pins: `lance-namespace-reqwest-client` 0.8.2 -> 0.8.4; pylance `lance-namespace`
  `>=0.8.0,<0.9` -> `>=0.8.5,<0.9`.
- IVF_RQ default `target_partition_size` is now 4096 (was the generic fallback) (PR #7273).

### Added
- Public vector-search `approx_mode` (`fast` / `normal` / `accurate`) for RaBitQ-backed
  indexes; serialized as `VectorApproxMode approx_mode` in `protos/ann.proto` (PR #7179,
  breaking proto change). Dedicated SIMD kernels for RaBitQ ex-code reranking (PR #7205).
  (Section 11.1)
- Cleanup explain API: `Dataset::cleanup(policy)` with `explain()` / `execute()` returning
  a `CleanupExplanation` (PR #7147). (Section 7)
- Tencent COS and GooseFS object-store config keys now documented in the object-store guide
  (COS: `cos_endpoint` / `cos_secret_id` / `cos_secret_key` / `cos_enable_versioning`,
  `COS_`/`TENCENTCLOUD_` env prefixes; GooseFS: `goosefs_write_type` / `goosefs_auth_type` /
  `goosefs_block_size` / `goosefs_chunk_size`, default port 9200) (PR #7151). (Section 13)
- Python zonemap segment builds exposed (PR #7177); per-query I/O metrics
  (`bytes_read`/`iops`/`requests`) on ANN operators in EXPLAIN ANALYZE (PR #7204);
  branch-aware version ops in Directory/REST namespaces (PR #7166). (Section 14 delta)

### Removed
- Upstream removed `table_version_storage_enabled` and the `__manifest`-backed table-version
  path (version ops now use `_versions/` exclusively, PR #7222); brotli dropped from the
  dependency graph (PR #7270).

### Fixed
- Corrected the reference-file H1 and table-of-contents, which still read "Lance v7" though
  the body is v8 (carryover miss from the 0.6.0 v7->v8 re-grounding).
- Dropped the stale claim that GooseFS is "not in the object-store guide" - it now has a
  full guide section.

Verified against: lance-format/lance@v8.0.0-beta.14

## [0.6.0] - 2026-06-10

### Changed
- Re-grounded against upstream tag `v7.2.0-beta.5` -> `v8.0.0-beta.9` (annotated tag,
  commit `a0664baf1`); bumped every version pin, permalink base, and the title from
  "Lance v7 reference" to "Lance v8 reference". 86-commit range, 6 breaking changes.
- The v8 boundary is the unification of all index builds onto one segment-based
  lifecycle. Bitmap migrated to the segment-based distributed workflow (PR #6869); the
  old public `create_scalar_index(..., fragment_ids=)` + `merge_index_metadata(...,
  "BITMAP")` Bitmap shard path is no longer exposed. Distributed BTree moved to the same
  segmented framework (PR #7013).
- RaBitQ (IVF_RQ) is no longer 1-bit-only: multi-bit shipped (`num_bits` 1..=9). Ex-code
  bits store in `__ex_codes` (+ `__add_factors_ex`/`__scale_factors_ex`); a `query_estimator`
  field selects `residual_query` (legacy default) or `raw_query`; raw-query search adds
  `__error_factors` for lower-bound pruning (PR #7038, #7078). The `dimension/8 + 16`
  per-row formula now holds only for `num_bits=1`.
- Crate workspace 24 -> 25 (see Added). Workspace dep versions: arrow 58, datafusion 53,
  opendal 0.57, jieba-rs 0.10, lance-namespace 8.0.0-beta.9, lance-namespace-reqwest-client
  0.8.2; pylance `lance-namespace>=0.8.0,<0.9`.
- Distributed indexing: `merge_existing_index_segments(...)` now covers vector, inverted,
  bitmap, BTree, and zone map segments (was vector/bitmap/btree/FTS); added independent
  per-worker vector models (each worker trains its own IVF/PQ model) (PR #7148, #7128).
- File/index writers' `finish()` now returns `FileWriteSummary { num_rows, size_bytes }`
  instead of a bare row count (PR #7096). `describe_indices()` reports full nested field
  paths and derives type from index details without opening the index; `list_indices()` is
  now a typed `IndexInformation` wrapper; the `load_indices()` Python binding was removed
  (PR #6903).
- Added a merge-insert (upsert / find-or-create) note to section 6: default
  `SourceDedupeBehavior::Fail` on duplicate source PKs (opt into `FirstSeen`); empty `on`
  keys fall back to the schema PK; `WhenMatched::UpdateAll` rewrites whole fragments.

### Added
- New `lance-derive` crate (PR #6229): `#[derive(DeepSizeOf)]` proc-macro for Arrow-aware
  memory accounting, replacing the external `deepsize` crate (which double-counts Arc-shared
  Arrow buffers). Crate workspace count goes 24 -> 25.
- FM-Index scalar index (Section 11.2): a Ferragina-Manzini / Burrows-Wheeler compressed
  substring index for arbitrary substring, prefix, and regex search on raw bytes. Built on
  the Segmented Index architecture (`num_segments`), normalization-independent, sanitizes
  `\x00`/`\xFF` to space at build time.
- Volcengine TOS object store (`tos://`, `tos_endpoint`/`tos_region`/...) and a feature-gated
  GooseFS provider (`goosefs://`, `goosefs_master_addr` with HA) (Section 13).

### Removed
- `IndexSegmentBuilder` API removed from Rust, Python, and Java (PR #6997); staged segments
  now publish directly via `create_index_uncommitted` / `execute_uncommitted` +
  `merge_existing_index_segments` + `commit_existing_index_segments`. `build_all()` is gone.
  The old builder's `target_segment_bytes` size-based grouping has no direct replacement.

Verified against: lance-format/lance@v8.0.0-beta.9

## [0.5.0] - 2026-06-05

### Changed
- Re-grounded against upstream tag `v7.1.0-beta.2` -> `v7.2.0-beta.5` (annotated tag,
  commit `1506693b`); bumped every version pin, permalink base, and the workspace crate
  version to `7.2.0-beta.5`. No breaking changes, no new crate (still 24), no new
  transaction op (still 15) across the 66-commit range.
- Corrected the file-format `next` alias: in code `next` resolves to **2.3** (a `V2_3`
  enum scaffolding version with no distinct encodings yet, 6 refs vs 98 for 2.2 across
  `lance-encoding`), while 2.2 remains the actual unstable frontier carrying Map / Blob v2 /
  `VariablePackedStruct`. The docs version table still lists only 2.2 as unstable.
- Updated the pylance runtime dependency to `lance-namespace>=0.8.0,<0.9` (was
  `>=0.7.7,<0.8`); noted `lance-namespace-reqwest-client 0.8.0`, `opendal 0.57`,
  `jieba-rs 0.10` workspace-dep bumps.
- Refined the RaBitQ (RQ) note: explicitly 1-bit-only (multi-bit is future work), added the
  `code_dim` metadata field and the `dimension/8 + 16` per-row storage formula.
- Fixed stale "all 23 crates" reference-nav line to 24.

### Added
- ICU FTS base-tokenizer (`base_tokenizer="icu"`, bundled ICU4X segmenter data, no external
  language model). Default tokenizer stays `simple` (an ICU-default PR was reverted).
  (PR #6956, revert #7006)
- Scalar-index fast search: `fast_search=True` routes through scalar/BTREE-indexed fragments
  and skips unindexed ones (not on legacy file version). (PR #6784)
- Batched vector queries via `Scanner::nearest` (no separate API), exposing a synthetic
  0-based `query_index` discriminator column. (PR #6828)
- Streaming IVF k-means training params (`streaming_sample_rate`, `streaming_coreset_rate`,
  `streaming_refine_passes`) for bounded-memory IVF training. (PR #6913)
- Section 14 delta subsection for v7.1.0-beta.2 -> v7.2.0-beta.5, also covering Arrow
  Utf8View/BinaryView encoding (PR #6985), HuggingFace `download_mode` (PR #7022), and
  MemWAL LSM local-scoring FTS (PR #6951).

Verified against: lance-format/lance@v7.2.0-beta.5

## [0.4.0] - 2026-05-25

### Changed
- Re-grounded against upstream tag `v7.1.0-beta.2` (commit `24b8afec`); bumped every
  version pin, permalink base, and the workspace crate version to `7.1.0-beta.2`.
- Removed stale claim that `lance-namespace-datafusion` is pinned at `7.0.0-beta.9` -
  it has used `version.workspace = true` since v7.1.0-beta.1.
- Fixed stale workspace-member count (22 -> 24) and dropped the wrong claim that
  `rust/arrow-stats` is a path dependency rather than an explicit member.

### Added
- New `lance-select` crate (PR #6879): mask code (`RowAddrMask`, `RowIdMask`,
  `IndexExprResult`) extracted from `lance-core` and `lance-index`. Crate workspace
  count goes from 23 to 24.
- v7.1.0-beta.2 delta section: MemWAL correctness fixes - flushed memtables now
  build secondary indexes (PR #6901, fixes invisible vector rows in `fast_search`)
  and a per-source PK-hash block-list post-filter suppresses stale LSM vector reads
  when the fresh row falls out of its source's top-k (PR #6899).
- Section 16: integrations `index.md` landing page (PR #6915).

Verified against: lance-format/lance@v7.1.0-beta.2

## [0.3.0] - 2026-05-21

### Added
- Section 11.1: IVF_PQ build prerequisites - no empty-table build; 256-row floor for
  default 8-bit PQ; IVF k-means needs >= num_partitions rows.
- Section 11 / 11.5: no-index queries flat-scan transparently (vector and FTS); the
  `optimize_indices(&OptimizeOptions)` API (`append` / `merge(N)` / `retrain`).
- Section 13: `shared-memory://` is an opt-in, authority-keyed, never-evicted
  process-global pool intended for tests and harnesses.
- Section 2: protoc build requirement and the `lance-datafusion` feature-cascade gap.

## [0.2.0] - 2026-05-21

### Changed
- Re-grounded against upstream tag `v7.1.0-beta.1` (commit `cffa8cb5`); bumped every
  version pin, permalink base, and the workspace crate version to `7.1.0-beta.1`.
- Updated the `pylance` runtime dependency to `lance-namespace>=0.7.7,<0.8`.

### Added
- Materialized-view namespace API (`create_materialized_view` / `refresh_materialized_view`).
- Typed `VectorIndexDetails` / `HnswParameters` index-details messages (`protos/index.proto`).
- v7.1.0-beta.1 delta section: granular tracing event targets, multi-base `write_fragments`
  bindings, MemWAL primary-key dedup fixes.

Verified against: lance-format/lance@v7.1.0-beta.1

## [0.1.0] - 2026-05-20
- Initial CHANGELOG; tracking established.

LICENSE.txt 8.9 KB

Apache License
Version 2.0, January 2004
https://www.apache.org/licenses/

TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

1. Definitions.

"License" shall mean the terms and conditions for use, reproduction, and
distribution as defined by Sections 1 through 9 of this document.

"Licensor" shall mean the copyright owner or entity authorized by the
copyright owner that is granting the License.

"Legal Entity" shall mean the union of the acting entity and all other
entities that control, are controlled by, or are under common control with
that entity. For the purposes of this definition, "control" means (i) the
power, direct or indirect, to cause the direction or management of such
entity, whether by contract or otherwise, or (ii) ownership of fifty percent
(50%) or more of the outstanding shares, or (iii) beneficial ownership of
such entity.

"You" (or "Your") shall mean an individual or Legal Entity exercising
permissions granted by this License.

"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation source, and
configuration files.

"Object" form shall mean any form resulting from mechanical transformation or
translation of a Source form, including but not limited to compiled object
code, generated documentation, and conversions to other media types.

"Work" shall mean the work of authorship, whether in Source or Object form,
made available under the License, as indicated by a copyright notice that is
included in or attached to the work (an example is provided in the Appendix
below).

"Derivative Works" shall mean any work, whether in Source or Object form,
that is based on (or derived from) the Work and for which the editorial
revisions, annotations, elaborations, or other modifications represent, as a
whole, an original work of authorship. For the purposes of this License,
Derivative Works shall not include works that remain separable from, or
merely link (or bind by name) to the interfaces of, the Work and Derivative
Works thereof.

"Contribution" shall mean any work of authorship, including the original
version of the Work and any modifications or additions to that Work or
Derivative Works thereof, that is intentionally submitted to Licensor for
inclusion in the Work by the copyright owner or by an individual or Legal
Entity authorized to submit on behalf of the copyright owner. For the
purposes of this definition, "submitted" means any form of electronic, verbal,
or written communication sent to the Licensor or its representatives,
including but not limited to communication on electronic mailing lists, source
code control systems, and issue tracking systems that are managed by, or on
behalf of, the Licensor for the purpose of discussing and improving the Work,
but excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."

"Contributor" shall mean Licensor and any individual or Legal Entity on
behalf of whom a Contribution has been received by Licensor and subsequently
incorporated within the Work.

2. Grant of Copyright License. Subject to the terms and conditions of this
License, each Contributor hereby grants to You a perpetual, worldwide,
non-exclusive, no-charge, royalty-free, irrevocable copyright license to
reproduce, prepare Derivative Works of, publicly display, publicly perform,
sublicense, and distribute the Work and such Derivative Works in Source or
Object form.

3. Grant of Patent License. Subject to the terms and conditions of this
License, each Contributor hereby grants to You a perpetual, worldwide,
non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this
section) patent license to make, have made, use, offer to sell, sell, import,
and otherwise transfer the Work, where such license applies only to those
patent claims licensable by such Contributor that are necessarily infringed by
their Contribution(s) alone or by combination of their Contribution(s) with
the Work to which such Contribution(s) was submitted. If You institute patent
litigation against any entity (including a cross-claim or counterclaim in a
lawsuit) alleging that the Work or a Contribution incorporated within the Work
constitutes direct or contributory patent infringement, then any patent
licenses granted to You under this License for that Work shall terminate as of
the date such litigation is filed.

4. Redistribution. You may reproduce and distribute copies of the Work or
Derivative Works thereof in any medium, with or without modifications, and in
Source or Object form, provided that You meet the following conditions:

(a) You must give any other recipients of the Work or Derivative Works a copy
of this License; and

(b) You must cause any modified files to carry prominent notices stating that
You changed the files; and

(c) You must retain, in the Source form of any Derivative Works that You
distribute, all copyright, patent, trademark, and attribution notices from
the Source form of the Work, excluding those notices that do not pertain to
any part of the Derivative Works; and

(d) If the Work includes a "NOTICE" text file as part of its distribution,
then any Derivative Works that You distribute must include a readable copy of
the attribution notices contained within such NOTICE file, excluding those
notices that do not pertain to any part of the Derivative Works, in at least
one of the following places: within a NOTICE text file distributed as part of
the Derivative Works; within the Source form or documentation, if provided
along with the Derivative Works; or, within a display generated by the
Derivative Works, if and wherever such third-party notices normally appear.
The contents of the NOTICE file are for informational purposes only and do not
modify the License. You may add Your own attribution notices within Derivative
Works that You distribute, alongside or as an addendum to the NOTICE text from
the Work, provided that such additional attribution notices cannot be
construed as modifying the License.

You may add Your own copyright statement to Your modifications and may provide
additional or different license terms and conditions for use, reproduction, or
distribution of Your modifications, or for any such Derivative Works as a
whole, provided Your use, reproduction, and distribution of the Work otherwise
complies with the conditions stated in this License.

5. Submission of Contributions. Unless You explicitly state otherwise, any
Contribution intentionally submitted for inclusion in the Work by You to the
Licensor shall be under the terms and conditions of this License, without any
additional terms or conditions. Notwithstanding the above, nothing herein
shall supersede or modify the terms of any separate license agreement you may
have executed with Licensor regarding such Contributions.

6. Trademarks. This License does not grant permission to use the trade names,
trademarks, service marks, or product names of the Licensor, except as
required for reasonable and customary use in describing the origin of the Work
and reproducing the content of the NOTICE file.

7. Disclaimer of Warranty. Unless required by applicable law or agreed to in
writing, Licensor provides the Work (and each Contributor provides its
Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied, including, without limitation, any warranties
or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any risks
associated with Your exercise of permissions under this License.

8. Limitation of Liability. In no event and under no legal theory, whether in
tort (including negligence), contract, or otherwise, unless required by
applicable law (such as deliberate and grossly negligent acts) or agreed to in
writing, shall any Contributor be liable to You for damages, including any
direct, indirect, special, incidental, or consequential damages of any
character arising as a result of this License or out of the use or inability to
use the Work (including but not limited to damages for loss of goodwill, work
stoppage, computer failure or malfunction, or any and all other commercial
damages or losses), even if such Contributor has been advised of the
possibility of such damages.

9. Accepting Warranty or Additional Liability. While redistributing the Work
or Derivative Works thereof, You may choose to offer, and charge a fee for,
acceptance of support, warranty, indemnity, or other liability obligations
and/or rights consistent with this License. However, in accepting such
obligations, You may act only on Your own behalf and on Your sole
responsibility, not on behalf of any other Contributor, and only if You agree
to indemnify, defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason of your
accepting any such warranty or additional liability.

END OF TERMS AND CONDITIONS

SKILL.md 35.1 KB

---
name: lance-format
description: Deep reference for Lance v13 columnar format, its Rust crates, and pylance - file encodings, table format, indexes, schema evolution, time travel. Use when building on the Lance crates or reading .lance datasets, not the LanceDB product.
metadata:
  version: "0.20.0"
  categories: "development, integrations"
  topics: "lance, columnar-format, vector-search, rust, lakehouse"
  upstream: "lance-format/lance@v13.0.0-beta.4"
  openclaw:
    homepage: https://github.com/tenequm/skills/tree/main/skills/lance-format
    emoji: "🗄️"
---

# Lance v13 reference

Lance is an open columnar format for multimodal AI - "a columnar data format that is 100x
faster than Parquet for random access." It is not one format but a stack of interoperating
specs: a **file format**, a **table format**, **index formats**, **catalog specs**, and a
**namespace client spec**. The Rust workspace at `lance-format/lance` implements all of them
plus Python (`pylance`) and Java bindings.

This skill tracks **`v13.0.0-beta.4`** (the `lance-format/lance` git tag), the current
development frontier; **`v12.0.0`** is the stable pin, released 2026-09-17. Pin against tags, not
`main` - Lance ships beta tags every few days and `next`-format encodings can change. Version
landscape below.

Three layers of reference, load what the task needs:

- **The deep reference** - any concrete schema, parameter, proto, or constraint. Split by topic:

  | File in `references/` | Covers | Sections |
  |------|--------|----------|
  | `format-file.md` | What Lance is, the 26 crates, file format, data types | 1-4 |
  | `format-table.md` | Dataset layout, manifests, fragments, schema evolution, versioning/tags/branches, row IDs, transactions + OCC, MemWAL | 5-10 |
  | `indexes.md` | Vector / scalar / FTS / geo indexes, distributed builds | 11-12 |
  | `ops.md` | Object store, capability matrix, source map | 13, 15, 16 |
  | `changelog-v7-v13.md` | The full v7 -> v13 delta | 14 |

  Cross-references written as "section N" resolve through `references/lance-reference.md`.
- `references/performance.md` - ALL performance guidance. Part A routes to the official text and
  adds the source-derived changes upstream has not documented; Part B is field-verified
  remote-storage practice. Load for any performance, tuning, maintenance-cost, or "why is this
  slow" question.
- `references/docs/` - a **verbatim mirror of the official docs** (`docs/src` at the tracked
  tag): every guide, quickstart, and format spec, unedited. Load when you need the full official
  text. Directory map below.

`references/maintenance.md` covers refreshing this skill against a new upstream tag.

## Lance vs LanceDB

These are two different things and conflating them produces wrong answers.

- **Lance** - the format and engine. The `lance-format/lance` repo; the `lance` /`lance-*`
  Rust crates; `pylance`. It gives you datasets, the file/table format, indexes, commits,
  scans. Consumed directly by DuckDB, Polars, Ray, Spark, PyTorch, DataFusion, or your own
  Rust/Python code. **This skill is about Lance.**
- **LanceDB** - a separate database *product* (`lancedb/lancedb`) built on top of Lance. It
  adds a query-builder API, an embedding registry, rerankers-as-API, multi-language SDK
  parity, and managed Cloud / Enterprise tiers. Not covered here.

**The wider ecosystem** (separate repos, own version lines, none covered here): Flink streaming
writes (`lance-flink`), PostgreSQL reads via `pglance`, a Cypher graph engine (`lance-graph`), a
dataset browser (`lance-data-viewer`), agentic context management (`lance-context`), and
namespace catalogs for Hive, Polaris, Gravitino, Unity Catalog, and AWS Glue.
The canonical docs site is **`lance.org`**. Generated per-language SDK docs live at
`lance-format.github.io/lance-python-doc` for Python and
[javadoc.io](https://www.javadoc.io/doc/org.lance/lance-core/latest/index.html) for Java - the
matching `lance-format.github.io/lance-java-doc` path 404s.

Linking the `lance` crate in `Cargo.toml` means you are using Lance directly - use this skill.
For LanceDB internals, the storage layer underneath is still Lance, so this skill remains the
authority for the format itself.

**The wrapper can hide format features.** LanceDB's `create_table` cannot enable stable row IDs;
only pylance's `write_dataset(enable_stable_row_ids=True)` can. If a format-level capability
matters to your design, check whether the wrapper exposes it before assuming the underlying
format settles the question - and reach for `pylance` directly when it does not.

## The crate workspace

26 crate directories under `rust/`. **`lance` is the public entry point** - `Dataset`, scanner,
indexes, commits; everything else (`lance-table`, `lance-file`, `lance-encoding`, `lance-index`,
`lance-io`, `lance-core`, `lance-datafusion`, `lance-linalg`, `lance-namespace*`, ...) is a layer
beneath it. Edition 2024, MSRV 1.91.0, arrow 58, datafusion 54; Python bindings need 3.10+. Full
table with roles, versions, and every workspace dep in `references/format-file.md` section 2.

**If you depend on anything below `lance`, v11 will break you** - PRs #8020-#8026 deleted
`lance-encoding::version` with no re-export (`LanceFileVersion` and `ConcreteFileVersion` both
live in `lance-file::version` now), removed `lance_io::encodings` and the `previous` namespaces,
and gave each current format its own `versions/v2_{0,1,2,3}` module. Section 2.1.

The transaction code moved too (#8053/#8054/#8056): `rust/lance/src/dataset/transaction.rs` is
**deleted**, replaced by a `rust/lance-table/src/transaction/` module tree (`builder`,
`conflicts`, `operation`, `proto`, `manifest_build`, `validate`, `index_maintenance`,
`row_version`, `update_map`). A `lance::dataset::transaction` shim still re-exports `Operation`,
`Transaction`, `TransactionBuilder`, `RewriteGroup`, `UpdateMap` and friends, so the common
surface is unbroken - but a symbol the shim omits, or a citation of the old path, needs
retargeting.

## File format versions

The file format carries a single major.minor version. `data_storage_version` is set per dataset
at creation - but as of `v12.0.0` it is **no longer fixed once the dataset exists**. It is the
*default* for writes that omit a target, not a summary of what the dataset holds: "Create and
overwrite establish this default; append, update, merge-insert, and compaction do not change it."
An existing V2 dataset can take `"2.0"`, `"2.1"`, `"2.2"` or `"2.3"` per operation without
rewriting its other files (#8582-#8585), so one dataset can hold data files at several exact V2
versions. **V1 and V2 still cannot be mixed.** Section 3.

| Version | Status | Notes |
|---------|--------|-------|
| `0.1` (`legacy`) | read-only | Original format; no longer writable |
| `2.0` | stable | Removed row groups; null support for lists/FSL/primitives |
| `2.1` | previous default | Adaptive structural encodings; better integer/string compression; nulls in struct fields; better nested random access. Was the default from Lance 5.0.0 until `v12.0.0-beta.15` |
| `2.2` | **current default** (`stable`) | Map type, Blob v2, `VariablePackedStruct`, larger mini-blocks. Required for Map and Blob v2 |
| `2.3` | unstable (`next`) | The current `next` alias target (`V2_3` in the enum). Ships **sparse structural pages**, which the 2.3 writer now auto-selects under a rep/def budget heuristic |

**`stable` now resolves to 2.2, not 2.1** (#8657, beta.15), and `2.2` is the enum `#[default]`,
so a dataset created without an explicit `data_storage_version` is written as 2.2. The change
reaches new-dataset creation through `DataStorageFormat::default() -> stable_file_version()`,
and Python's `write_dataset` inherits it because its default routes through `stable`. **The docs
were not updated with it** - `format/file/versioning.md` still only says `stable` is an "alias
for the default version", so the code is the authority here. `next` resolves to 2.3. Pin an
explicit number for deterministic behavior across builds.

2.3 is the only version the code flags unstable; 2.2 never was, and is now what you get by
default. The release *selectors* (`LanceFileVersion`) are a type distinct from the persisted
identity (`ConcreteFileVersion`). Details, plus the sparse auto-selection rules, in
`references/format-file.md` sections 3.1 and 3.6.

## Version landscape

The major is bumped by a bot, not a human: `ci/publish_beta.sh` re-roots at `MAJOR+1` whenever
any PR since the release root carries the GitHub `breaking-change` label - the marker is the
**label**, not a conventional-commit `!`. A major bump therefore means "some labeled breaking
change landed", not a redesign, and a `!` without the label bumps nothing. It has now fired on
**four consecutive lines**, which is why **none of `v9.1.0`, `v10.1.0`, `v11.1.0`, or `v12.1.0`
was ever released**. The 12.1 line is the clearest case: `main` took a
`chore: bump main to 12.1.0-beta.0` commit, and four commits later the bot re-rooted to 13, so
`release-root/12.1.0-beta.N` and `release-root/13.0.0-beta.N` are the **same base commit**
(`c3c9632a2`) and no `v12.1.0-beta.*` tag exists.

Three recent lines **did** ship a final: `v10.0.0` (2026-08-08), `v11.0.0` (2026-08-30) and
`v12.0.0` (2026-09-17). Each sits on a stabilization branch that is **not an ancestor of `main`**
- normal for a Lance final, not a sign the release is unofficial.

| Major | Its breaking theme |
|-------|--------------------|
| **v13** (current, `v13.0.0-beta.4`) | `WriteParams` gained `file_writer_options` (#9192 - the one labeled PR that re-rooted the major); lazy page-metadata init changed the `StructuralFieldScheduler` signature and the metadata **cache key shape** (#7465); `json_extract`/`json_get` no longer route to JSON indices (#9101). Delta below |
| **v12** (`v12.0.0`, 2026-09-17) | `WrappingObjectStore` implementors must add `wrap_paginated` (no default); MemWAL `ShardManifestStore` renamed and narrowed; `lance-namespace` returns response objects; external stores gained predecessor-conditioned publication; namespace merge-insert keys became a list; the caller-provided Writer / `open_part` flow was removed (#9072). Unlabeled but bigger: `stable` -> 2.2 and the IVF_RQ 5-bit default. Net-new format capability: mixed data-file versions. Delta below |
| **v11** (`v11.0.0`, 2026-08-30) | Fragment ids became a dataset-lifetime high-water mark; large internal reorganization of `lance-file` / `lance-encoding`; the first new manifest feature flag since v7 - which was then **reallocated before the final**. Net-new: covering indexes, `merge_insert` `write_mode`, row-address prefilter. Delta below |
| **v10** | Blob APIs preserve null selections; cache keys became opaque BLAKE3 digests (every warm or persisted cache cold-misses, no legacy fallback); async `create_remapper`; MemWAL renamed generation -> SSTable, merge -> compaction (wire-compatible, symbol-breaking) |
| **v9.1** (never released; renamed into v10) | FTS/inverted creation took a `block_size` param. Net-new: Data Overlay Files (cell-level updates without base-file rewrite, unstable + env-gated), sparse structural pages, `lance-index-core` |
| **v9** | Python 3.9 dropped; `alter_columns` fails fast when casting an indexed column; FM-Index proto rename made existing FM indexes unreadable; FTS/inverted defaults to on-disk format v2 |
| **v8** | All index builds unified onto one segment-based lifecycle. Net-new: `lance-derive`, FM-Index, multi-bit IVF_RQ, public `approx_mode`, TOS + GooseFS object stores |
| **v7** | MemWAL, branches, the geo/RTree index, the `lance-select` crate, ICU FTS |

**`v12.0.0` is the stable pin** and what GitHub Releases marks `Latest`. crates.io carries
**finals only** (newest `lance 12.0.0`, no 13.x); PyPI `pylance` is likewise at `12.0.0`. So a
beta pin means a git dependency - beta wheels publish to fury.io instead, under the renamed org
(`https://pypi.fury.io/lance-format`), which currently carries `pylance-13.0.0b1` through `b4`.

Full per-tag deltas with every PR citation: `references/changelog-v7-v13.md`.

## The v11 delta

357 commits from `v10.0.0-beta.7` to the `v11.0.0` final, with **16 `breaking-change`-labeled
PRs** (14 through `beta.16`, plus #8407 and #8535 in the final). Most structural invariants held:
**26 crates**, **16 transaction ops**, `CommitConfig.num_retries` **20**, arrow 58 / datafusion 54,
MSRV 1.91.0, Edition 2024, Python 3.10+ - and all of them still hold at `v13.0.0-beta.4`.

**`references/changelog-v7-v13.md` has the full delta** - every PR citation, the per-tag
breakdown from v7 forward, the Python/Java surface, and each correctness fix with its trigger
condition. Load it for any "what changed / will this break me" question. What follows is only
what bites hardest.

**Five things that break you at v11:**

- **Fragment ids are a dataset-lifetime high-water mark** (#8206) - a *format* invariant, not
  just an API. Overwrite no longer restarts ids at 0, an overwrite fragment carrying a deletion
  file is rejected, and any commit producing duplicate ids is rejected - so datasets written by
  Lance 0.16 and earlier may still read but no longer commit. `dataset.get_fragment(0)` after an
  overwrite must read ids from the manifest. Section 5 - which also covers a resolution hazard on
  pre-0.10 unsorted manifests that can make a fragment-filtered index cover the wrong fragments.
- **The file-version types and reader/writer composition moved** (#8020-#8026) -
  `lance-encoding::version` deleted with no re-export; `LanceFileVersion` lost `PartialOrd`/`Ord`
  (#8027, #8028), so `v >= LanceFileVersion::Next` no longer compiles. `FileWriter` is now an
  enum with all constructors removed. Most of these break silently at compile time. Section 3.6.
- **Transaction code moved to `lance-table`** (#8053/#8054/#8056) - see the crate-workspace note
  above; the `lance::dataset::transaction` shim covers the common surface.
- **`Operation::Project` / `Merge` gained `preserves_nullability`** (#8347) - a nullability
  *tightening* must not set it, and such a projection now conflicts with any concurrent
  value-write. This closed a real hole where `alter_columns` could let a racing write land nulls
  unreadable under the tightened schema. Section 9.2.
- **The external-manifest protocol changed** (#8499) - object storage is authoritative, the
  external store's put-if-not-exists is a *reservation*, and a stored ETag must be **ignored**;
  a retained one makes readers reject a good manifest with `Manifest e_tag mismatch`. Section 9.

**The manifest feature flags changed - and bit 128 was reallocated before the final.** v11 added
the first new bit since v7 and moved `FLAG_UNKNOWN` 128 -> 256. But the bit it added,
`FLAG_MEM_WAL_INDEX_CATCHUP`, was **retired again** (#8680) and the reclaimed bit handed to
`FLAG_COVERED_INDEX_METADATA = 128` (#8535) before `v11.0.0` shipped. At the final and at v12
there is no index-catchup flag and no `require_index_catchup` proto field; a shard absent from
`index_catchup` now unconditionally means *unknown*. Both reader and writer must hold bit 128 or
refuse the table. Section 7.

**Do not pin anywhere in `v11.0.0-beta.4` through `beta.17`.** Those builds treat bit 128 as a
MemWAL flag they support, so they *open* a covering-index dataset instead of refusing it - wrong
neighbours, no error. The exposure is inherited by whichever flag takes the bit.

**Covering indexes are the v11 net-new format feature** (#8535), **redefined at v13** (#8856).
`IndexMetadata.covering_fields` (proto field 11) names the columns an index *carries* values for,
so a query projecting only those columns is answered without a base-table take. It is **no longer
a trailing suffix of `fields`**: it "must be a subset of `fields`, in the order the index emits
them. A column is carried if and only if it is named here", including a column the index is also
keyed on - and `fields[0]` remains a keyed column. Index invalidation stays wide: **any** index
whose `fields` include the updated column, "whether the index is keyed on it or merely carries
it".

The old "no index builder writes carried values yet" no longer holds. V3 IVF auxiliary files can
physically carry columns, and "a reader discovers carried columns by exclusion, not by position:
any column in the auxiliary file's schema that is not one of the quantizer's internal columns is
a carried column", bound to source fields by a new `covering_field_ids` metadata key. Coverage is
now per-segment, not per-index: "one logical index may hold values for some of its segments and
not others". `VectorQueryProto.covering_projection` (field 15) reserves the query-side tag, where
absent / present-and-empty / present-and-non-empty are three distinct meanings. Section 11.

**Bit 8 was spent in the `v12.0.0` final.** `FLAG_MIXED_DATA_FILE_VERSIONS = 1 << 8` (256) is no
longer a reservation pinned equal to `FLAG_UNKNOWN`: the assert relaxed to
`FLAG_MIXED_DATA_FILE_VERSIONS < FLAG_UNKNOWN`, `FLAG_UNKNOWN` moved `1 << 8` -> `1 << 9` (512),
and the build now both reads and writes mixed-version datasets. It is still carried by
`STICKY_PAIRED_FLAGS`, and a **half-set** manifest is now a hard error: "Manifest has only one of
the mixed data-file-version reader and writer feature bits set, so its semantics are undefined".
Section 7.

**Bit 1024 is where the docs and the code disagree - trust the code.**
`FLAG_FRAGMENT_REUSE_INDEX = 1 << 10` is declared at `rust/lance-table/src/feature_flags.rs:69`
and, at `v13.0.0-beta.4`, **that declaration is its only occurrence in the entire tree**. It sits
*above* `FLAG_UNKNOWN` (512), and the supported set is computed as `FLAG_UNKNOWN - 1`, so a
manifest setting it is **refused**. The spec page meanwhile lists it as reader `Yes` / writer
`Yes` and puts the unknown boundary at 2048. The docs describe the intended end state; the code
has only reserved the constant. Anything you build against tagged FRI today is building against
prose, not behavior.

**Two `LANCE_*` env vars landed** (from the AMX work, #8540): `LANCE_DISABLE_AMX` (runtime kill
switch) and `LANCE_AMX_FP16_CC` (build-time compiler override). Grep trap: `LANCE_AMX_CFG_*` and
`LANCE_AMX_TILE_COUNT` are **C macros in `amx_fp16.c`, not env vars**, and `LANCE_FACTOR` is a
substring of `BALANCE_FACTOR` - a plain `LANCE_*` grep reports all four as if they were real.

**Worth knowing without reading the full delta:** FTS gained a document-boundary axis
(`DocumentGranularity`, #7788) whose `list_element` mode is a third trigger requiring FTS on-disk
format v3; transactions above **20 MiB** spill out of the manifest entirely (#7881); MemWAL
catch-up became derived rather than declared (#8481); transaction proto field 9 is deprecated for
field 10 (#7432); compaction gained row/byte budgets plus fragment exclusion (#8235, #8532);
`merge_insert` gained `write_mode` (#8423); and Python commit conflicts became
`lance.commit.CommitConflictError`, a subclass of `OSError`, so existing handlers keep working
(#8563). Full list with citations in `references/changelog-v7-v13.md`.

**Address-domain indexes stopped falsely claiming compacted fragments** (v11, `beta.16` or
earlier). A rewrite used to advance *every* index's `fragment_bitmap` onto the new fragment ids -
including ZoneMap, whose stored addresses point into the fragments the rewrite dropped. The
Rewrite path now branches on `results_are_row_addrs()`: an address-domain index gets
`drop_rewritten_fragments` and a full-scan fallback, correct-but-slower instead of stale
addresses. **Heals only for new compactions**: an index already damaged under v10 or earlier must
be recreated, and the damage does not self-heal through routine maintenance, because the
refreshed `fragment_bitmap` also makes incremental folds a no-op. Section 11.

**Correctness fixes split by whether upgrading is enough.** Most are read-path only and heal on
upgrade. These do **not** - they need data rewritten or repaired: #8382, #8669, #8509, #7703,
#8539, #8459, #8378, #8482, #8834 (rebuild HNSW - a persisted graph can hold edges to ids it does
not contain; lost recall stays lost), #8101 (**nullable primary keys silently duplicated rows** on
every repeat `merge_insert`; existing duplicates must be removed by hand), #8511, #8427, #8513,
#8839, #8904. Conditions for each in `references/changelog-v7-v13.md`.

## The v12 delta

**225 commits** from `release-root/12.0.0-beta.N` to the `v12.0.0` final, with **7
`breaking-change`-labeled PRs** - the five visible at beta.15 plus **#9072** and **#9101** in the
run-up to the final. No new index types and no new crates; every structural invariant above still
holds. **The label is a floor, not a ceiling** - the two biggest behavior changes in the line
carry a conventional-commit `!` but no label, so the bot never counted them: the `stable` -> 2.2
move (#8657, above) and the IVF_RQ 5-bit default (below).

- **`WrappingObjectStore` implementors must add `wrap_paginated`** (#8606) - "There is
  deliberately no default: getting this wrong is either a silent loss of speed or a silent loss
  of the wrapper, and neither announces itself." Return `Some` to keep listing pushdown through
  the wrapper, `None` to give it up and fall back through `inner`. One wrapper giving it up gives
  it up for the whole chain. Anything wrapping the object store fails to compile until updated.
- **New paged listing: `ObjectStore::read_dir_page`** (#8606) - one page of a prefix's immediate
  children plus an opaque resume token. The trap: "One page is one request, so a page can hold
  fewer children than `limit` asked for and still be followed by more" - walk until the token is
  `None`, never until a page comes back short.
- **MemWAL `ShardManifestStore` renamed and narrowed** (#8640) - `read_latest` -> `latest`,
  `read_latest_uncached` -> `refresh_latest`, and `write` is now crate-private (reach it through
  `commit_update`, `claim_epoch`, or `initialize_shard`). Existing `commit_update` closures need
  no change. Section 10.
- **`lance-namespace` 0.8.5 -> 0.11.1** (#8903) - four `LanceNamespace` methods now return
  response objects instead of bare values: `count_table_rows` -> `CountTableRowsResponse`,
  `query_table` -> `QueryTableResponse`, `namespace_exists` / `table_exists` -> their own
  response types. Callers unwrap; anyone implementing the trait needs the same signature updates.
- **External manifest stores gained predecessor-conditioned publication** (#8800) -
  `put_if_predecessor` reserves a version only while the predecessor still carries the identity
  the writer observed, and `commit_after` refuses with `PrerequisiteFailed`, "never a conflict".
  The hard compile break is the new `ManifestLocation.identity` field, not the trait methods
  (all default-implemented). No built-in store implements it. Section 9.
- **Namespace merge-insert keys became a list** (#8915) - `on` moves from `Option<String>` to
  `Option<Vec<String>>`, with **arity-dependent NULL semantics**: a single-column key treats NULL
  as equal to NULL, a composite key uses SQL equality, "under which a NULL key matches nothing -
  not even a byte-identical NULL".

**The `lance-namespace` pin is no longer one number.** #8915 moved the Rust client to **0.12.0**
and #8979 moved **Java** to 0.12.0 as well; only **Python** still holds `>=0.11.1,<0.12`, because
its generated models still send `on` as a bare string. Quote a language-specific pin, never one
number for all three - and note this split moved once already, so re-check it rather than
carrying the pairing forward.

**IVF_RQ now defaults to 5 bits per dimension, not 1** (#8936) - roughly a **4.4x index-size
increase** at the default (upstream's 100M x 768d example: ~10.8 GiB -> ~47.3 GiB). `Fast` search
mode "uses only the 1-bit sign code even when the index stores additional bits", so it pays the
storage without using it; set `num_bits=1` explicitly to opt out, at the cost of the multi-bit
distance estimate and some recall. Sizing formulas in `references/indexes.md`.

**Column slice stitching (#8660) was reverted** at beta.9 (#8926) - it "should not ship while the
caller-managed replacement in #8923 is being developed". `rust/lance-file/src/concat.rs` exists
again at beta.15, but holds #8923's caller-managed data file parts, not the reverted stitching.

**Two proto additions.** MemWAL `SsTable` gained `in_memory_bytes`, `physical_rows` and
`primary_key_bytes` (fields 3-5, #8981); all optional, and **absence must not be read as zero**.
`FilteredReadOptions` gained `materialization_readahead_bytes` and `batch_size_bytes` (13, 14) -
not a format change under a new rule in `protos/AGENTS.md`: execution-plan schemas "are wire
contracts, not persisted Lance formats". `transaction.proto` / `ann.proto` / `index.proto` are
untouched.

**Net-new, non-breaking:** provider-native bulk copy and a deep-clone concurrency bound (section
13); Python `ObjectStoreProvider` registration (#8522); `BinaryView` in the packed blob writer
(#8700); caller-managed data file parts (#8923); cleanup of specific versions (#8617);
`LanceDataset.slice()` (#8059) and six more `LanceFragment.scanner` options (#8429); restored
Python index retraining (#8786); namespace-managed clone deprecated to a shim (#8964). Namespace
latest-version resolution no longer lists the whole `_versions/` prefix (#8679) - on a
~340k-version table that was ~344 list pages, "~25s of pure I/O wait", paid by every open.

**Fixes needing a rebuild or rewrite, not just an upgrade:** #8779 (rebuild NGRAM indexes), #8510
(rewrite data compacted from uniformly reordered fragments), #8984 (re-drop a resurrected index),
#8837 (repair a MemWAL shard below ~2.7KB/row - it cannot be reopened). Full per-PR conditions,
plus the much longer list that *does* heal on upgrade, in `references/changelog-v7-v13.md`.

**Mixed data-file versions LANDED** - it is no longer "1 of 6". #8581-#8584 shipped in `v12.0.0`
(validation, per-operation V2 write targets, propagation across dataset operations, compaction
targeting) and #8585 exposed it in the bindings in the v13 line. The proto changed with it:
`DataStorageFormat.version` is now "the default format version used when writing data files",
and "each DataFile's version is authoritative for decoding" once the capability is set.

**In flight, not landed - do not treat as shipped:** generic block v5 compression is **still 1 of
10** PRs merged (#8324; #8325-#8333 all remain open). Next big dependency break in the queue:
#8997, "upgrade to arrow 59, DataFusion 55, and pyo3 0.29" - **still open** at `v13.0.0-beta.4`,
so arrow 58 / datafusion 54 still hold. It also gates two outstanding PyO3 advisories
(RUSTSEC-2026-0176/0177); rustls was separately patched to 0.23.45 for RUSTSEC-2026-0285 (#9212).

## The v13 delta

**66 commits** from `release-root/13.0.0-beta.N` to `v13.0.0-beta.4`, with **2
`breaking-change`-labeled PRs**. No new crates and no new index types; 26 crates, 16 transaction
ops, `CommitConfig.num_retries` 20, arrow 58 / datafusion 54, MSRV 1.91.0, Edition 2024 and
Python 3.10+ all still hold.

**The `!`-vs-label rule inverted in this window.** All three conventional-commit `!` commits
(#7465, #9192, #9101) *do* carry the `breaking-change` label. Keep treating the label as a floor
rather than a ceiling - but this window is the counter-example, not more evidence for the gap.

- **`WriteParams` gained `file_writer_options`** (#9192) - the single labeled PR that re-rooted
  the major. `FileWriterOptions { data_cache_bytes, max_page_bytes, keep_original_array }` is now
  reachable from the dataset write APIs in Rust, Python and Java. A zero `max_page_bytes` is
  rejected before encoder construction rather than misbehaving later. Anything constructing
  `WriteParams` by struct literal fails to compile.
- **Page metadata is initialized lazily, and the metadata cache key changed shape** (#7465).
  `StructuralFieldScheduler::initialize` now takes `requested_ranges`, and the page-scheduler
  `initialize` splits into `init_ranges()` and `init_from_buffers(buffers, io)` - any external
  implementor fails to compile. The public `DecodeBatchScheduler::try_new` kept its signature;
  the range-aware entry point is the crate-private `try_new_with_ranges`. The part that bites
  without a compile error: caching moved from a per-column `FieldDataCacheKey` to a per-page
  `PageDataCacheKey { column_index, page_index, view_tag }`, so **every warm or persisted
  metadata cache cold-misses** across this upgrade. The payoff is real - "a cold point/range
  read's metadata IO is invariant to the column's total page count".
- **`json_extract` and `json_get` no longer route to JSON indices** (#9101). Only the four typed
  accessors (`json_get_int` / `_float` / `_bool` / `_string`) reach the index; everything else
  falls back to a full scan. This fixes three real wrong-answer bugs - a quoted-key mismatch that
  "searched for a quoted key and matched nothing", a `Utf8` literal driving an `Int64` btree into
  a panic, and an unsound range because "quoting is not order-preserving (`ab` < `ab!` but
  `"ab"` > `"ab!"`)". **The cost is silent**: a `json_extract` workload that used to hit an index
  now scans, with no error and no plan warning. Rewrite those predicates onto the typed
  accessors.

**The Fragment Reuse Index gained a versioned on-disk contract** (#9136). `InlineContent` field 1
was renamed `versions` -> `legacy_versions` and a tagged `transitions` list added at field 2,
gated on `index_version >= 1`; mappings are now a oneof of `OrderedCompaction` or
`StablePartition`. A stable partition "assigns source rows to destination fragments while
preserving their relative source order within each destination", which lets FRI reuse existing
indices after **reclustering** - a second use case the v0 model had no concept of. Its physical
form is an immutable row-map Lance file with `uint16` labels and an `LSPC`-magic counts matrix.
Two hard rules: **stable row IDs and tagged FRI are mutually exclusive** ("writers must not
publish `index_version >= 1` on them"), and cleanup "must retain intermediate transitions still
needed to translate old addresses". Upstream also softened the old claim - "FRI does not remove
conflicts between overlapping rewrites". Section 11.

**Net-new, non-breaking:** `Dataset::frag_reuse_index()` is public (#9112) and documented in the
performance guide; `FileFragment::write_overlay` returns a real `OverlayWriter` (#8761, still
env-gated); an `hf://` object store with `hf_enable_resolve_cache` (#9236); Python
`lance.bitmap.Bitmap`, `deep_clone()` (#9181), `base_paths()` (#9191) and
`update_columns(with_offsets=True)` (#8891); Java `DataStorageVersion`, `FileWriteOptions` and
`ScanOptions.indexSegments`; and namespace table listing finally bounded by `read_dir_page`
(#9165). `inline_optimization_enabled` **flipped `true` -> `false`** (#9180), which upstream
justifies with a -49% write-p50 measurement at 1M entries.

## Performance questions

For anything performance-shaped - slow scans or searches, remote/object-storage cost, index
maintenance cost, memory sizing, version bloat, benchmarking - load
`references/performance.md` first. Part A routes to the official guidance plus the undocumented
source-derived changes; Part B is field-verified practice against S3-compatible storage. The
governing rule stays **minimize remote calls** - fewer commits, fewer scans, fewer round trips -
because that is where the order-of-magnitude wins are. The official **"Tuning remote scans"**
section (v11, unchanged at v12) gives a starting point for cross-region or public-internet
access, where the cloud default of 64 concurrent requests is too aggressive: `LANCE_IO_THREADS=8`,
`fragment_readahead=1`, `batch_readahead=2`, `io_buffer_size=64MB`. It is a legitimate second
move once call volume is already minimized.

**AMX-FP16** (#8540, beta.16) is the one v11 performance change that alters *results*, not just
speed: where it engages, IVF partition assignment becomes **exact instead of approximate**, so
recall improves *and* assignments differ from an older build. It is shape-gated (`float16` +
`dot`, `dimension >= 32`, `num_centroids >= 32`); everything else keeps the previous path.
`LANCE_DISABLE_AMX=1` disables it, but reverts assignment to the approximate path too - so an
index built with it set is not equivalent to one built without it.

Two cache facts to know before tuning anything remote: Lance has **no resident data cache** (a
`Session` holds only index and metadata caches, never decoded values, so repeated point reads
re-pay object-store IO), and one `Arc<Session>` shared via `DatasetBuilder::with_session` lets
datasets share it. Cold first search is dominated by paging indexes in - `prewarm_index` is the
remedy. Note that **#7465 changes the metadata cache key shape**, so the first run after a v13
upgrade re-pays that paging even against a warm or persisted cache. Details and build-time
requirements in `references/performance.md`.

**Time travel is not an archive mechanism.** Versions look like free history, but the default
cleanup reclaims anything older than 7 days and cleanup is part of routine optimize - so a design
that treats old versions as the durable record loses it on the first maintenance pass. Keep an
explicit archive if you need one.

## Official docs mirror

`references/docs/` mirrors `docs/src` of `lance-format/lance` at the tracked tag, verbatim -
45 markdown files plus 4 diagrams, all directly readable.

| Directory | Files | Covers |
|-----------|-------|--------|
| `guide/` | 14 | CRUD, performance, object store, distributed write + indexing, JSON, tokenizers, data types, data evolution, blob, arrays, tags/branches, migration, observability |
| `quickstart/` | 4 | First dataset, vector search, full-text search, versioning |
| `format/` | 1 | Spec-stack overview |
| `format/file/` | 3 | Container spec, structural encodings + compression, format versions |
| `format/table/` | 9 | Layout, schema, transactions (**conflict-resolution matrix**), versioning, row-id lineage, branch/tag, MemWAL, data overlay files |
| `format/index/` | 1 + 4 svg | Index lifecycle, fragment coverage, compaction interplay |
| `format/index/scalar/` | 9 | fts, fmindex, ngram, btree, bitmap, bloom_filter, label_list (`array_has_any/all`), zonemap, rtree |
| `format/index/vector/` | 1 | IVF / PQ / SQ / RQ / HNSW concepts and storage layout |
| `format/index/system/` | 2 | Fragment reuse index, MemWAL system index |
| `integrations/` | 1 | DataFusion SQL over Lance, incl. JSON functions |

**Not mirrored:** `docs/src/images/` (PNG/GIF assets), so image links in the mirrored pages do
not resolve - the prose is self-contained, and the four `.drawio.svg` diagrams *are* mirrored.
Also out by design: `community/`, `examples/`, `integrations/{index,pytorch,tensorflow}.md`; and
the landing stubs and contributor files (`format/AGENTS.md`, `format/CLAUDE.md`).

**A whole tier of docs is not in this repo at all**, so it cannot be mirrored and cannot be
enumerated from a clone. `docs/make-full-website.sh` assembles `format/catalog`,
`format/namespace`, and the `integrations/{duckdb,huggingface,spark,ray,trino,context}` sections
at build time from six sibling repos with their own version lines - the checked-in
`integrations/index.md` links `spark/`, `duckdb` and `trino` as if they were local, but those
paths do not exist in the tree. **Lance Context** and the **HuggingFace** integration docs are
whole nav sections that exist only on the built site. For any of those, read `lance.org` rather
than this mirror. Protobuf message bodies are likewise expanded at build time from `protos/` by
`mkdocs_protobuf`, so the mirrored spec pages show `%%% proto.message.X %%%` placeholders where
the site shows a rendered schema.

Comments (0)

No comments yet.

Reviews (0)

No reviews yet.

Claude Prompt

Tighten-my-prose editing prompt

Claude Skill

Adversarial code-review skill for Claude

Claude Workflow

Deep-research workflow (fan-out + verify)

Claude Skill

Conventional-commit writer for Claude