Pilot Plane Runbook — Promotion & Demotion
How a parity pilot on a cloned-snapshot plane ends. Both endings are transitions of durable state, so both terminate in a receipt, and neither has an unnamed exit.
Companion to the Restoration Runbook, which covers disaster recovery. This one covers a planned transition that can still fail.
The two endings are not mirror images
The asymmetry is the whole design, and getting it backwards is the expensive mistake:
| What it does to the durable plane | Branch | Terminals | |
|---|---|---|---|
| Promotion — parity becomes the seat's reality | Mutates it (replays pilot writes onto it) | forward-only | committed · failed-contained |
| Demotion — the pilot ends, overlay retired | Leaves it alone (overlay is fail-closed against the durable root, #15799) | reversible-by-proof | demoted-clean · failed-contained |
Promotion is forward-only because of ADR-0027 §2.7.4:
"After promotion begins, the safe direction is forward completion; if reconciliation cannot prove it, the run settles
failed-contained/ quarantine and eligibility remains denied. It never overwrites independently observed live state or claims cross-store rollback."
So there is no "undo a promotion" procedure in this runbook, and its absence is deliberate. If you
are looking for one, the situation you are in is failed-contained.
Demotion is the cheap ending precisely because the pilot never wrote to the durable plane. It needs no rollback — it needs a proof that nothing leaked.
Terminals
Derived by ai/scripts/diagnostics/pilotPlaneTerminal.mjs.
No entry point accepts a terminal as an argument: the receipt attests to evidence, not to the operator's
reading of it.
| Terminal | Meaning | Eligibility effect | Reachable today |
|---|---|---|---|
committed |
Replay onto the durable plane verified monotonic by a receipt — no loss, no double-apply | opened |
❌ no branch reaches it; needs an invoked replay adapter |
demoted-clean |
No overlay-stamped or provenance-unknown cutover-window write reached the durable corpus; no committed history lost | unchanged |
✅ through the invoked dual-WAL provenance producer |
failed-contained |
The claim could not be proven, whatever the cause | denied |
✅ |
Demotion now has a producer of fact: both accepted-write services pass resolved AiConfig.plane.id,
both WAL appenders stamp it after caller fields, and evaluateDemotion invokes strict readers owned by
the memory and message stores. Promotion remains closed because a complete ordered mutation-source actor
still does not exist. The distinction is source authority, not stronger validation: a caller cannot submit
an overlayScan, and no receipt-shaped argument can open committed.
Eligibility is three-valued on purpose. Only a strict committed opens data-consuming eligibility.
A clean demotion does not open it — it never closed it, because the pilot never mutated the durable plane.
Collapsing those two into one boolean read as a widening of the committed-only rule, which it must not be.
failed-contained is not an error code, it is a state you are allowed to stay in. It is the correct
outcome of a run that could not prove itself, and it is strictly better than the two things operators
reach for instead: deleting the overlay ("it's over anyway") or asserting success ("it looked fine").
Promotion
⚠️
committedis MECHANICALLY UNREACHABLE today, and that is the honest stateA promotion proof needs a complete ordered mutation source. No producer for one exists, and five independent findings each block it on their own:
- No consumed source-read boundary. No production caller exists for
evaluatePromotion,planWalReplay,verifyReplayContinuityorparseJsonl— nothing in the running system reads the corpus and could issue a receipt for having read all of it.- Strict store readers now exist, but no actor consumes them as a complete source. Operational reads still correctly skip malformed/torn rows; each store now also owns a strict provenance reader for demotion. Promotion still lacks the actor that resolves both configured roots, fences writers, consumes every strict row, and binds content digests.
- The plane has two WAL families.
messageWal.dirderives topath.join(memoryWal.dir, 'messages')by default but may be explicitly relocated.deriveDualCorpusReplayReceiptnow keeps memory (embedded + graph) and message (graph) records distinct and binds exact ids plus source/target plane identities; no invoked executor yet proves that those arrays were the complete configured plane.- Naïve message replay emits stale wakes.
MailboxService._projectMessageWalRecorddefaultspumpWake = true; its own recovery path passespumpWake: falseexplicitly. A replay reaching the default re-fires historical wakes as if they were new.- No plane-wide writer fence. During the audit the live memory corpus moved 8,233 → 8,234 rows between two scans. A stable double-read is not quiescence; the append lock is per-file and fail-open.
ADR-0027 OQ8 states the bound: journal replay has no source authority, and count evidence never supplies row identity.
evaluatePromotiontherefore takes no argument and contains no branch: every promotion settlesfailed-contained, regardless of what you pass. No producer name, path, array, count, digest, manifest or receipt unlocks it, because nothing is read.It was a gate first, and that was wrong — worth recording, because the mistake is subtle. The first version checked
typeof PROMOTION_REPLAY_PRODUCER !== 'function'and fell through to the derivation otherwise, on the reasoning that a function slot was stronger than the demotion gate's string slot since "a function cannot be forged by a name." It can: the producer was type-checked but never invoked, so a no-op stub() => {}satisfied the check and handed caller-owned observations straight through. That is the same defect the demotion gate already fixed once, one level up — requiring a thing is not proving a fact, whatever the thing's type. Where a capability must act rather than merely exist, a conditional is a dormant success path pretending to be a guard, so the branch is gone entirely.Demotion demonstrates the required shape: it invokes the producer and owns the scan. Promotion has no branch because its complete source-and-replay actor is still missing.
Why a gate and not a stricter check. The previous shape settled
committedon a one-entry corpus with an unchanged before/after — a truthful, self-consistent, entirely zero-effect certification. Refusing that specific case would have closed one probe while leaving arbitrary non-empty truncation alive: a caller passing half the real corpus verifies exactly as cleanly, because nothing inside the module can know what the whole corpus was. A proof over an unknown denominator is not a proof, and the missing thing is a producer of fact, not a rule.The producer that may replace the null must resolve both configured WAL roots, fence both source writers, have each owning store strictly enumerate its own canonical payload files, bind per-family content and record digests, derive the memory and message plans separately, replay messages without wake pumping or mutable-state overwrite, observe the target before and after, and emit one composite receipt. That is an executable adapter, not a receipt-shaped object.
Steps 1–6 below therefore describe the procedure a landed producer would follow. Its arithmetic is not hypothetical:
deriveReplayCompletionis exported and directly tested, because a gate that makes a path unreachable also makes it unverifiable — a defect behind one is invisible to every test. That is not a theoretical worry; the sibling capture module's post-gate block was left referencing four renamed variables and its suite stayed green because the gate short-circuited first.deriveReplayCompletionreturns{ok, reason, receipt}and deliberately no terminal and no eligibility: it may prove the math, and it must never mint authority, because agreeing about a corpus says nothing about whether that corpus was the whole plane.
- Baseline the replay volume.
walVolumeBaseline.mjsdecides fork-then-replay vs dual-journal from measured per-seat WAL volume. Supply the three factual inputs — replay throughput, concurrent native inflow, and the accepted cutover window — and the budget is derived as(throughput − inflow) × window. It does not accept a precomputed budget: a supplied figure could pick the cheap posture while contradicting that arithmetic, and requiring a number is not deriving one. Two refusals worth knowing: an empty measurement window refuses rather than reporting zero, and an omitted inflow refuses rather than assuming a quiesced plane (pass0explicitly). If inflow meets or exceeds throughput the posture isdual-journal— replay never converges, so fork-then-replay is impossible at any window rather than merely over budget. - Read the source corpus. The producer enumerates its canonical payload files and refuses on any row it cannot parse. Duplicate source ids also refuse: a payload that cannot be uniquely keyed cannot be proven non-double-applied.
- Record the applied-stage sets for the durable plane before applying anything. The verification is bound to this pre-state — without it there is no baseline and nothing can be proven.
- Apply the planned entries to the durable plane.
- Record the applied-stage sets again, after.
- Bind the component receipt.
deriveDualCorpusReplayReceipt(...)verifies separate memory/message continuity, exact record sets, and source/target plane identities. It deliberately emits no terminal. - Settle only through the future invoked actor.
evaluatePromotion()still returnsfailed-contained; it accepts no observations until that actor owns the complete source and target reads.
There is no caller-supplied plan or verification verdict, deliberately. The component helpers derive the plan from each source corpus and run continuity verification internally. Both intermediates were once arguments, and both were forgeable. A structurally complete continuity verdict is a thing a caller can simply type; and a self-consistent plan proved only that its own projection matched its own receipt, never that it was derived from the corpus it claimed to describe.
Deriving from the corpus removes both forgeable intermediates. What it does not remove is the unknown denominator — hence
evaluatePromotion()remains unconditionally contained.Concurrent gains from other seats are permitted and reported separately, not treated as corruption.
Demotion
Newly accepted memory and message WAL records carry immutable planeId provenance:
MemoryService/MailboxServicesupply resolvedAiConfig.plane.id;- the store validates it and serializes
{...record, segmentKey, planeId}, so a caller field loses; - missing/invalid identities reject before append;
- operational readers surface an absent historical field as
unknownwithout rewriting the file.
The demotion procedure is now:
- Invoke the producer.
evaluateDemotiontakes the two configured WAL roots, overlay plane id, cutover start, and clone-time segment ids. It does not acceptoverlayScanor post-pilot ids. - Strictly scan both WAL families. Store-owned readers refuse malformed/torn rows. The producer retains exact memory/message record sets and obtains post-pilot segment ids from the scan itself.
- Bound legacy ignorance. An unstamped record proven older than
cutoverStartedAtis reported as legacy context. An unknown identity inside the window—or an unknown timestamp that cannot prove it predates the window—forcesfailed-contained. - Reject overlay writes. Any in-window record stamped with the overlay plane id names its containing
segment and forces
failed-contained. Concurrent canonical-plane writes are allowed and counted. - Prove no segment loss. Clone-time ids must remain a subset of the producer-observed post-pilot ids.
Cardinality is not identity:
3 → 3can hide delete-and-add. - Retire the overlay only on
demoted-clean.
Why demotion does NOT compare durable-plane fingerprints
The intuitive proof is to re-fingerprint the durable plane and require it to equal the pre-clone fingerprint. That check is wrong, and wrong in the direction that looks rigorous. The durable plane has other writers: the pilot occupies one seat for one to two weeks while the rest of the institution keeps writing. Its digest is expected to move. An equality check would report
failed-containedon every healthy demotion — and an instrument that fails on the happy path gets switched off rather than believed.A digest diff also cannot answer the actual question. It cannot distinguish "other seats wrote" from "the pilot leaked," which is the only distinction that matters here.
So the proven claim is narrower and survives a real pilot: no overlay-tagged segment reached the durable plane. Shrinkage is still a failure, because concurrent writers explain growth and never loss.
On failed-contained
Do: leave eligibility denied · quarantine the overlay, do not delete it (it is the only surviving evidence of what leaked) · record the receipt with its reason · escalate for human attribution.
Do not: retry hoping for a different terminal · start an old cohort against unproven new state · treat a diagnostic log line as completion authority. Per ADR-0027, "best-effort telemetry is never completion authority" — only the derived terminal is.
Authority
- ADR-0027 §2.7.4 — committed-only eligibility, forward completion, no cross-store rollback. This is the adopted authority for everything above.
- ADR-0027 OQ8 — the bound behind the promotion gate: journal replay has no source authority, and any
later replay action requires a complete ordered mutation source. Count evidence never supplies row
identity — which is the trap worth naming, because a count looks like a measurement. "8,234 rows replayed"
is a true sentence that establishes nothing about which rows.
pilotPlaneTerminaldefers to this citation rather than repeating it in JSDoc, so the reference has one maintained home instead of decaying copies in code. - D#15758 Option G reaches the same shape for cloud cohorts, phrased as "post-mutation but reversible-by-proof". It is an unadopted divergence row in an open divergence window, so it is cited here as converging evidence and never as a source of authority. If that Discussion adopts a different option, nothing in this runbook needs retracting.