This is the advanced tier. The foundationdb-keys-and-layers skill (key encoding, subspaces, the IFdbLayer<TState> pattern) and foundationdb-transactions skill (retry loop, idempotency, atomics, watches) are prerequisites — this skill assumes them and explains the why underneath, plus the patterns for performant, distributed, multi-node layers.
A fully worked, compile-checked reference for everything in §5–§6 lives in samples/SkillValidation/BookStore.cs + BookStore.ChangeFeed.cs.
(FoundationDB's published architecture; the constraints below fall out of it directly.)
\n| Role | \nResponsibility | \n
|---|---|
| Coordinators | \nSmall Paxos group; elect the cluster controller, hold the cluster file. Clients bootstrap here. | \n
| Cluster Controller | \nSingleton; recruits/monitors all other roles, drives recovery. | \n
| Master / Sequencer | \nHands out monotonically increasing versions — read versions and commit versions. The global logical clock. | \n
| GRV proxies | \nServe get-read-version: ask the master for the latest committed version, confirm the tlogs are still live (so a read version is never stale after a recovery); throttled by Ratekeeper. | \n
| Commit proxies | \nDrive commits: get a commit version from the master, send conflict ranges to resolvers, make mutations durable on the tlogs. | \n
| Resolvers | \nHold the last ~5 s of committed writes in memory; compare a committing tx's read-conflict ranges against them → this is where conflicts (not_committed, 1020) are decided. | \n
| Transaction Logs (tlogs) | \nDurable, replicated WAL; receive mutations in version order and only ack once fsync'd on a quorum. | \n
| Storage servers | \nHold the sharded, replicated data; keep ~5 s of mutations in memory + on-disk data \"as of 5 s ago\"; serve reads via MVCC. | \n
| Ratekeeper / Data Distributor | \nSingletons: throttle transaction start rate near saturation / keep shards balanced across storage servers. | \n
Lifecycle of a read-write transaction:
\nWhy the rules you already follow exist:
\ntransaction_too_old (1007). It's also why a recovery \"fast-forwards 90 s\" and aborts in-flight transactions. → keep transactions short; page long scans across many transactions.The native client pipelines concurrent requests. The enemy of latency is a serial data dependency — code that reads, inspects the result, then reads again. Each such hop is a full client↔cluster round-trip that cannot be hidden.
\nBatch independent reads — never await them in a loop:
// ❌ N round-trips (each await blocks on the previous)\nforeach (var id in ids) results.Add(await tr.GetAsync(subspace.Key(id)));\n\n// ✅ one batched multi-read\nSlice[] values = await tr.GetValuesAsync(ids.Select(id => subspace.Key(id))); // GetValuesAsync<TKey>(...)\n\n// ✅ or issue concurrently and let them pipeline into ~one round-trip\nSlice[] vs = await Task.WhenAll(tr.GetAsync(k1), tr.GetAsync(k2), tr.GetAsync(k3));\ntr.GetValuesAsync(keys) reads many independent keys in one logical batch (this is what a document store's metadata fetch uses). For ranges, GetRangeAsync(range, options) returns a page per round-trip — tune FdbRangeOptions (WantAll, WithLimit, streaming mode) to your access pattern.
Collapse read→decide→read dependencies. If you find yourself reading key A only to decide whether/how to read B, ask whether the information can be encoded so a single read carries it. (The change-feed in §5 does exactly this: instead of \"read the trim marker, then range-read the feed,\" the trim signal is a tombstone inside the feed, so one GetRange returns both the data and the eviction signal — see §5.4.) If you genuinely can't, issue both in parallel with Task.WhenAll and discard the wasted one in the rare case.
Other levers:
\ntr.Snapshot.GetAsync/GetRange) skip read-conflict tracking — cheaper and conflict-free; use when a slightly stale read is acceptable.Fdb.Bulk.* (it manages batching and the 5-second window for you).Conflicts are resolver verdicts on read-conflict ranges. A key that many transactions read-then-write serializes there. Avoid it:
\nAtomicAdd64, AtomicIncrement32/64, AtomicMax/Min, AtomicOr/And/Xor) — they don't read, so they create no read-conflict and never conflict with each other. Counters, statistics, signal keys.AddConflictRange) only when your reads/writes don't already imply the semantics you need.The sequencer is the only source of \"now\" that every node agrees on. Use it; never use node-local wall clocks for cross-node decisions.
\ntr.GetReadVersionAsync() → the read version: a monotonic, cluster-wide logical clock, identical regardless of which node reads it. Use it for leases / liveness, ordering, \"as-of\" reasoning.tr.CreateVersionStamp() + SetVersionStampedKey/Value → the commit version, assigned atomically at commit. Globally ordered, collision-free → the backbone of queues, logs, and change feeds (§5).GetVersionStampAsync() / GetCommittedVersion() → recover the version a transaction committed at.\n\n⚠️ Two clock traps (both real, both bite):
\n\n
\n- Local wall clocks have no shared \"now.\" Comparing a timestamp minted on node A against node B's
\nDateTime.UtcNowis meaningless (skew, drift, NTP steps, VM pauses) — like comparing times across relativistic frames. Cross-node liveness must use the database clock.- The version tick-rate is not constant (~1e6/s but it drifts; idle clusters advance slower). So do not convert a version delta into a duration (
\nnow - lease > N_versionsis unsound). Instead, store a DB-sourced token and test it for change (equality), and measure elapsed time only as the gap between an observer's own consecutive local reads (§5.3).
A shared clock removes skew, but not the fundamental failure-detector impossibility: you still cannot distinguish \"slow\" from \"dead.\" So liveness is always a policy (a threshold) backed by evict-and-resync, never a proof.
\nA change feed lets other nodes observe a stream of changes and maintain an in-memory view. It composes every primitive above. Full compile-checked code: BookStore.ChangeFeed.cs.
Each mutation appends a change under a commit-ordered VersionStamp and bumps a single watched signal key — all in the same transaction as the data write, so the feed can never disagree with the data:
\nvar stamp = tr.CreateUniqueVersionStamp(); // distinct per change, even several per tx\ntr.SetVersionStampedKey(subspace.Key(SUBSPACE_FEED, stamp), FdbValue.ToJson(change));\ntr.AtomicIncrement64(subspace.Key(SUBSPACE_SIGNAL)); // wake every subscriber; conflict-free\nThe consumer reads pages after its cursor; when caught up, it watches the signal key (outer token, not tr.Cancellation), awaits outside the transaction, then re-reads. Expose it as IAsyncEnumerable<T> and wrap thinly as a Channel<T> or a callback. The VersionStamp of the last entry is the resume cursor.
A version-stamped log grows forever, so a GC must trim it. Trim everything consumed by all live subscribers (up to the slowest live cursor); if nobody's live, drop the backlog. \"Live\" is decided without comparing clocks:
\nunchanged for N polls ≈ N × the observer's own local delay) — equality-check only, never version→time, never cross-node timestamp comparison;A subscriber frozen long enough gets evicted and the GC reclaims past its cursor → it missed changes and its view is untrustworthy. It must be told. The efficient signal is a tombstone: when the GC reclaims (·, horizon], it leaves one empty-value entry at the horizon's versionstamp.
GetRange — an empty value deserializes to null (a real change is always non-null JSON), so it's detected with no extra read / no serial dependency.ChangeFeedOutOfSyncException that propagates through the enumerable / channel / callback; the consumer catches it, reloads current state, and re-subscribes from \"now.\"This is the same contract as Kafka's OffsetOutOfRange / DynamoDB Streams' TrimmedDataAccessException: you can't prevent a too-slow consumer from missing data — you detect it cleanly and force a resync.
GetValuesAsync), parallel (Task.WhenAll), or encoded into one read (tombstone-style)?await-ed in a loop?Fdb.Bulk.*?State confined to the transaction?