LearnNewsExamplesServices
Frontmatter
id16169
titleStamp WAL records with plane provenance for clean demotion
stateClosed
labels
enhancementaitestingarchitecture
assigneesneo-gpt
createdAtJul 30, 2026, 1:40 PM
updatedAtJul 30, 2026, 3:27 PM
githubUrlhttps://github.com/neomjs/neo/issues/16169
authorneo-gpt
commentsCount0
parentIssue15798
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 30, 2026, 3:27 PM

Stamp WAL records with plane provenance for clean demotion

Closed Backlog/active-chunk-11 enhancementaitestingarchitecture
neo-gpt
neo-gpt commented on Jul 30, 2026, 1:40 PM

Context

Native sub-issue of #15798. The parity pilot shipped snapshot, replay, promotion/demotion, and evidence evaluators in #15806 / PR #16037, but one honest terminal remains unreachable: demoted-clean requires proof that no overlay write reached the durable plane.

That proof cannot be reconstructed from paths, fingerprints, counts, or timestamps. The write-ahead records themselves need immutable plane provenance from the process that accepted the write.

The Problem

Exact-head inspection at dev@65092deb422cec749e5817937a351b106cb27d79 confirms the missing producer on both durable intent streams:

  • ai/services/memory-core/helpers/memoryWalStore.mjs:119-138 writes JSON.stringify({...record, segmentKey}).
  • ai/services/memory-core/helpers/messageWalStore.mjs:79-95 writes the same shape for messages.
  • MemoryService.mjs:462-474 supplies memory/document/projection data but no plane identity.
  • MailboxService.mjs:1719-1743 builds and appends the message record but no plane identity.

The Tier-1 authority already exists as AiConfig.plane.id; the append paths simply do not persist it.

Without that field, the pilot cannot distinguish an overlay-written segment from legitimate writes by other canonical-plane seats. Fingerprint equality is wrong because the durable plane is expected to change under concurrent writers. Segment counts are weaker still: equal cardinality does not prove equal identity. The evaluator therefore settles failed-contained, correctly refusing a clean claim, even when the overlay did not leak.

The Architectural Reality

  • AiConfig.plane.id is the opaque identity authority from ADR 0019. Path comparison is explicitly not a same-plane predicate.
  • MemoryService and MailboxService are the accepted-write boundaries. They already consume the resolved Memory Core config and must supply the resolved plane identity to the WAL append operation.
  • appendWalMemory and appendWalMessage are the durable serialization choke points. They must stamp provenance after the caller record is spread so a payload cannot override it.
  • The memory and message WALs are one parity continuity subject. Fixing only one would allow a “clean” demotion while the other corpus remained unattributable.
  • Existing pre-provenance records are historical facts. They remain readable/drainable but are unknown, never retroactively guessed or rewritten as canonical.
  • The pilot evaluator from #15806 is the first consumer. Drain, projection-marker, pruning, recency, and replay consumers must preserve behavior unless they explicitly consume the new field.

The Fix — one coherent PR

Extend both append contracts to require the resolved planeId alongside dir, then serialize server-stamped provenance on every newly accepted memory and message WAL record. Validate the identifier at the append boundary and ensure record payload fields cannot replace it.

Thread aiConfig.plane.id from MemoryService and MailboxService into those append calls. Update read/selection helpers and receipts to retain the field without changing pending/drained semantics. Classify legacy rows with no field as unknown.

Connect the real producer to the parity demotion scanner: a clean demotion proves that every inspected post-cutover durable-plane write has known provenance and none carries the overlay plane id. Concurrent canonical-plane writes are permitted and reported. Unknown rows inside the bounded cutover window block demoted-clean; rows proven to predate that window remain legacy context, not fabricated overlay evidence.

Exercise the same provenance contract through the fork-then-replay path for both memory and message corpora. The continuity receipt binds source plane, target plane, selected record ids, and post-replay result without treating replayed origin identity as the target write identity.

Contract Ledger

Target Surface Source of Authority Proposed Behavior Fallback / Edge Case Docs Evidence
appendWalMemory(record, options) AiConfig.plane.id resolved by MemoryService Require options.planeId; persist it after caller fields Missing/invalid id rejects before append; record-supplied id cannot override JSDoc Store unit matrix
appendWalMessage(record, options) AiConfig.plane.id resolved by MailboxService Same immutable stamp for message intents Same fail-closed behavior JSDoc Store + mailbox tests
WAL read helpers accepted JSONL record Preserve planeId; surface absent legacy provenance as unknown No path/cwd/default inference JSDoc Legacy/current fixtures
Pilot demotion scan #15806 terminal evaluator + stamped WAL records Allow unrelated canonical writes; reject overlay or unknown in the cutover window Insufficient evidence remains failed-contained Pilot runbook Concurrent-writer integration fixture
Fork-then-replay receipt #15806 replay plan + both WAL corpora Bind source/target planes and exact selected records Partial/ambiguous corpus fails contained Pilot runbook Dual-corpus replay fixture

Decision Record impact

Aligned with ADR 0019; no amendment required. This consumes the existing opaque plane identity at the write boundary and introduces no new config leaf, env read, fallback, or same-plane predicate.

Acceptance Criteria

  • Every newly appended memory WAL record carries an immutable, validated planeId sourced from the resolved server config.
  • Every newly appended message WAL record carries the same provenance contract.
  • A caller-provided record.planeId cannot spoof or override the server-stamped value; missing/invalid resolved identity fails before the JSONL append.
  • Current readers, drains, projection markers, pruning, recency, and mailbox behavior remain green while preserving the field.
  • Legacy records without planeId remain readable and drainable but surface as unknown; no migration invents provenance for them.
  • The demotion evaluator reaches demoted-clean in a fixture with concurrent canonical writers and zero overlay writes.
  • One overlay-stamped durable write or one provenance-unknown write inside the bounded cutover window forces failed-contained with a named reason.
  • A dual-corpus fork-then-replay fixture binds source/target plane ids and exact memory/message record sets, proving no loss and no double-apply.
  • Focused unit coverage includes spoof attempts, legacy rows, mixed planes, both WAL stores, and the evaluator/replay integration.

Out of Scope

  • Rewriting historical WAL files to add guessed provenance.
  • Using paths, checkout names, Compose project names, or fingerprints as plane identity.
  • Changing the elected fork-then-replay posture to dual journaling.
  • General event-sourcing schema versioning.
  • The operator-run maintainer-machine cutover owned by sibling #16167.

Avoided Traps

  • Fingerprint equality: reports failure on healthy concurrent canonical writes and cannot attribute the writer.
  • Segment counts: cardinality is not identity or continuity.
  • Caller-authored planeId: makes the proof spoofable at the exact boundary it is meant to certify.
  • Memory-only provenance: permits message leakage to hide behind a clean memory result.
  • Legacy backfill by current config: rewrites history and turns ignorance into false certainty.
  • Stamping only receipts: a receipt cannot prove what the durable record itself omitted.

Related

Parent #15798 · pilot tooling #15806 · plane identity #15799 · in-process dual WAL drains #15802 · machine dogfood #16167.

Origin Session ID: 71ff2f5e-17d6-47b2-90db-82bc1773b0a0

Retrieval hints: query_raw_memories("WAL plane provenance demoted-clean parity pilot 15806"); Memory 4530a08a-aefb-49a0-bc08-6bb6a7a092de; exact source anchors memoryWalStore.mjs:119-138 and messageWalStore.mjs:79-95 at 65092deb422cec749e5817937a351b106cb27d79.

Live duplicate sweep: checked the latest 20 open issues by creation time, recent 30 A2A messages, live GitHub search, Knowledge Base tickets, and issue/discussion corpus at 2026-07-30T11:39:20Z. No equivalent open ticket or in-flight claim exists; #15806 records the missing producer but is closed and did not implement it.

tobiu referenced in commit 8849ca1 - "feat(ai): stamp WAL plane provenance for clean demotion (#16169) (#16175)" on Jul 30, 2026, 3:27 PM
tobiu closed this issue on Jul 30, 2026, 3:27 PM