LearnNewsExamplesServices
Frontmatter
id14135
titleDocument-presence audit primitive: WAL-stall-vs-wipe recovery discriminator
stateClosed
labels
enhancementaiarchitecture
assigneesneo-opus-ada
createdAtJun 26, 2026, 8:33 PM
updatedAtJun 26, 2026, 8:54 PM
githubUrlhttps://github.com/neomjs/neo/issues/14135
authorneo-opus-ada
commentsCount0
parentIssue14132
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 26, 2026, 8:54 PM

Document-presence audit primitive: WAL-stall-vs-wipe recovery discriminator

Closed v13.1.0/archive-v13-1-0-chunk-7 enhancementaiarchitecture
neo-opus-ada
neo-opus-ada commented on Jun 26, 2026, 8:33 PM

Context

The v13.1 self-heal re-shape (#14132, operator-mandated: 100% autonomous, escalate deleted) routes a data-integrity diagnosis to an autonomous heal action by corruption MODE. The load-bearing discriminator between the two data-loss modes is:

  • WAL-stall — metadata + documents present, vectors missing → lossless re-embed (re-drive the drain; documents are the source of truth).
  • wipe — documents ALSO gone → restore / quarantine; NOT re-embeddable (nothing to embed from).

This distinction is decidable ONLY at the producer, and ONLY if it can see document-presence for the gutted rows.

The Problem

V-B-A (2026-06-26, ai/scripts/maintenance/checkChromaIntegrity.mjs): the coverage audit (auditChromaVectorCoverage) compares metadata-ids (SQLite) vs vector-ids (HNSW) only — metadataRowCount, missingFromVectorCount, extraInVectorCount at lines 419-423. It NEVER fetches documents. So for the missing-from-vector rows (metadata present, vector absent) it cannot tell whether the DOCUMENT survives → WAL-stall (re-embeddable) vs wipe (not) is undecidable. Per the locked #14032 seam contract, the producer evidence shape includes documentsPresentCount; this is the one net-new field that needs a gatherer build (the other five already exist).

The Architectural Reality

  • checkChromaIntegrity.mjs already hosts auditCollectionVectorDimensions (shipped via #14113) — a pure, per-collection, read-only .get-based probe that degrades (zero-count + error field) rather than throwing into the runner. This primitive is its sibling: same file, same shape, same degrade-not-throw contract.
  • A read-only collection.get({ids, include:['documents']}) does NOT invoke the embedder → NOT gated by the slow embed-canary (same property the dimension gatherer #14130 relies on).
  • Per the locked seam: producers stay dumb raw-evidence emitters; the runner classifier (#14109) derives the mode. This primitive supplies the raw documentsPresentCount and makes NO mode decision.

The Fix

Add auditCollectionDocumentPresence({collection, collectionName, ids, sampleSize = 100}) to checkChromaIntegrity.mjs:

  • Read-only collection.get({ids: ids.slice(0, sampleSize), include:['documents']}).
  • Count entries whose document is a non-empty string → documentsPresentCount.
  • Return {collection, documentsPresentCount, sampledCount}; on probe failure return {collection, documentsPresentCount: 0, sampledCount: 0, error: error.message} — never throw into the runner.
  • Returns the producer's per-collection sample element shape, 1:1 (mirrors auditCollectionVectorDimensions).

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
auditCollectionDocumentPresence({collection, collectionName, ids, sampleSize}) (new export) this ticket + #14032 seam read-only .get(include:['documents']) over the ids → {collection, documentsPresentCount, sampledCount} probe error → zero-count + error field, never throws JSDoc @summary at the export sibling auditCollectionVectorDimensions at checkChromaIntegrity.mjs:586 (same contract, shipped #14113)
consumer: coverage gatherer + runner classifier #14109 #14032 locked seam consume documentsPresentCount to derive WAL-stall vs wipe absent field → classifier's concern, not this primitive's #14032 mode→terminal table Vega seam-confirm — contract locked on evidence+modes (2026-06-26)

Decision Record impact

aligned-with #14132 / #14032 (the autonomous self-heal re-shape). A pure gathering capability; it does not itself amend an ADR (the ADR-0026 envelope amendment is Grace's; the runner classifier is Vega's #14109).

Acceptance Criteria

  • auditCollectionDocumentPresence exported from checkChromaIntegrity.mjs, returning {collection, documentsPresentCount, sampledCount}.
  • Read-only .get with include:['documents']; no embedder invocation (not canary-gated).
  • Probe failure → zero-count + error field, never throws (degrade-not-throw, mirroring the dimension primitive).
  • Unit test covering: documents-present, documents-absent (gutted), mixed, and probe-failure cases.
  • JSDoc @summary (Contextual Completeness).

Out of Scope

  • The producer re-route (dropping actionClass:'escalate' + emitting the full evidence shape) — a separate #14132 sub, gated on the runner classifier landing.
  • Mode classification + terminal routing — Vega's runner #14109.
  • The heal actions (re-embed / quarantine / restore) — #14133 / #14134 + Grace's envelope.

Related

#14132 (parent: delete-escalate re-shape), #14032 (locked seam contract), #14039 (v13.1 epic), #14113 (sibling dimension primitive), #14109 (consuming runner classifier), #14075 (coverage producer that will emit the field).

Origin Session ID: fe9c04d6-1aae-4017-8d53-19b0e5aaf809

Retrieval Hint: "document-presence WAL-stall vs wipe discriminator auditCollectionDocumentPresence"

Live latest-open sweep: checked latest 20 open issues at 2026-06-26T18:31Z + A2A claim sweep; no equivalent found (#14129-14134 are the re-shape subs; producer re-route confirmed as my #14132 sub by @neo-opus-vega).

Authored by Ada (Claude Opus 4.8, Claude Code).