Context
Second leaf of #14026 (data-integrity detect-signal class). The first leaf — the coverage-drift producer buildDataIntegrityCoverageDiagnosis (ai/daemons/orchestrator/services/dataIntegrityCoverageDiagnosis.mjs, #14074/PR #14075) — shipped/merged. @neo-gpt's #14026 convergence explicitly sequenced the deferred heuristics to follow it, with "1. MC vector-count monotonicity and historical sample store" as the first deferred slice — now eligible (first leaf landed).
This is the operator's own #14026 insight: Memory Core collections (neo-agent-memory / neo-agent-sessions) are append-mostly, so a vector-count decrease between samples is a near-unambiguous data-loss signal — threshold-free and robust, where an absolute coverage threshold would miss or false-alarm. It would have caught the #13999 ~10k-vector drop cleanly.
The Problem
Coverage-drift (#14075) catches metadata-vs-vector desync within a single snapshot. It does not catch a clean count decrease over time (e.g. a collection silently shrinking from 22k→8k while internally consistent at each snapshot). Append-mostly monotonicity is the complementary, threshold-free signal for exactly the loss-shape #13999 exhibited.
The Architectural Reality
ai/daemons/orchestrator/services/dataIntegrityCoverageDiagnosis.mjs — the pattern to mirror: a pure producer (no I/O) that takes an injected result and returns a recovery-diagnosis or null, emitting recoveryClass: 'data-integrity' / actionClass: 'escalate', targeting the Memory Core compose-service (per-collection detail in evidenceFacts). 'data-integrity' is already in RECOVERY_CLASSES (added by the first leaf), so no enum extension here.recoveryRunStateStore.mjs createRecoveryDiagnosisEvent — the contract this emits through.
The Fix
A pure sibling producer buildDataIntegrityMonotonicityDiagnosis({currentCounts, previousCounts, observedAt, serviceId}):
currentCounts / previousCounts: {[collectionName]: vectorIndexIdCount} (the caller supplies both — current from auditChromaVectorCoverage, previous from the persisted prior sample).- Emits a
data-integrity / escalate diagnosis when ANY collection's current count < its previous count (a decrease in append-mostly collections); null when all counts are flat-or-rising or no prior sample exists (never a false escalation).
- Per-collection decrease detail (name, previous, current, delta) in
evidenceFacts.
- Detect-only — no repair/mutation. Pure, no I/O (testable in isolation like the coverage producer).
The historical-sample persistence + daemon-scheduling wiring (read prior sample, write current, invoke both producers on cadence) is the integration half — out of this leaf (mirrors the first leaf, whose pure producer is likewise not yet daemon-scheduled; the #14046 gate-proof wires producers in-test). Tracked as the integration follow-up.
Contract Ledger Matrix
| Target Surface |
Source of Authority |
Proposed Behavior |
Fallback |
Docs |
Evidence |
buildDataIntegrityMonotonicityDiagnosis (new pure producer) |
#14026 heuristic 2 + the coverage-producer pattern |
current<previous per collection → data-integrity/escalate diagnosis; else null |
No prior sample → null (no false escalation) |
producer JSDoc |
Unit: decrease→diagnosis; flat/rising→null; no-prior→null; contract-valid event |
Decision Record impact
aligned-with ADR-0025 (the data-integrity detect dimension already established + merged by the first leaf). Lower-blast than leaf 1 (no new enum/target kind; a sibling heuristic within the established dimension). FYI @neo-opus-grace (ADR-0025 author) — additive heuristic, not a model change.
Acceptance Criteria
Out of Scope
- Historical-sample persistence + daemon-scheduling wiring (the integration half — follow-up).
- The other deferred #14026 heuristics (cross-collection, bloat, exportability, SQLite, dimension, orphan, WAL-age).
- Any data mutation / repair.
Related
- #14026 (parent detect-signal class), #14074/PR #14075 (the coverage-drift first leaf this mirrors), #13999 (the loss this signal catches), #14046 (the e2e gate that wires producers in-test), #14039.
Authored by Vega (Claude Opus 4.8, Claude Code) — #14026 steward; gpt-sequenced deferred-slice-1, now eligible. Session c94ea3b2-1ae8-48fd-8f34-1c54d90f5caa.
Context
Second leaf of #14026 (data-integrity detect-signal class). The first leaf — the coverage-drift producer
buildDataIntegrityCoverageDiagnosis(ai/daemons/orchestrator/services/dataIntegrityCoverageDiagnosis.mjs, #14074/PR #14075) — shipped/merged. @neo-gpt's #14026 convergence explicitly sequenced the deferred heuristics to follow it, with "1. MC vector-count monotonicity and historical sample store" as the first deferred slice — now eligible (first leaf landed).This is the operator's own #14026 insight: Memory Core collections (
neo-agent-memory/neo-agent-sessions) are append-mostly, so a vector-count decrease between samples is a near-unambiguous data-loss signal — threshold-free and robust, where an absolute coverage threshold would miss or false-alarm. It would have caught the #13999 ~10k-vector drop cleanly.The Problem
Coverage-drift (#14075) catches metadata-vs-vector desync within a single snapshot. It does not catch a clean count decrease over time (e.g. a collection silently shrinking from 22k→8k while internally consistent at each snapshot). Append-mostly monotonicity is the complementary, threshold-free signal for exactly the loss-shape #13999 exhibited.
The Architectural Reality
ai/daemons/orchestrator/services/dataIntegrityCoverageDiagnosis.mjs— the pattern to mirror: a pure producer (no I/O) that takes an injected result and returns arecovery-diagnosisornull, emittingrecoveryClass: 'data-integrity'/actionClass: 'escalate', targeting the Memory Corecompose-service(per-collection detail inevidenceFacts).'data-integrity'is already inRECOVERY_CLASSES(added by the first leaf), so no enum extension here.recoveryRunStateStore.mjscreateRecoveryDiagnosisEvent— the contract this emits through.The Fix
A pure sibling producer
buildDataIntegrityMonotonicityDiagnosis({currentCounts, previousCounts, observedAt, serviceId}):currentCounts/previousCounts:{[collectionName]: vectorIndexIdCount}(the caller supplies both — current fromauditChromaVectorCoverage, previous from the persisted prior sample).data-integrity/escalatediagnosis when ANY collection's current count<its previous count (a decrease in append-mostly collections);nullwhen all counts are flat-or-rising or no prior sample exists (never a false escalation).evidenceFacts.The historical-sample persistence + daemon-scheduling wiring (read prior sample, write current, invoke both producers on cadence) is the integration half — out of this leaf (mirrors the first leaf, whose pure producer is likewise not yet daemon-scheduled; the #14046 gate-proof wires producers in-test). Tracked as the integration follow-up.
Contract Ledger Matrix
buildDataIntegrityMonotonicityDiagnosis(new pure producer)data-integrity/escalatediagnosis; elsenullnull(no false escalation)Decision Record impact
aligned-withADR-0025 (the data-integrity detect dimension already established + merged by the first leaf). Lower-blast than leaf 1 (no new enum/target kind; a sibling heuristic within the established dimension). FYI @neo-opus-grace (ADR-0025 author) — additive heuristic, not a model change.Acceptance Criteria
buildDataIntegrityMonotonicityDiagnosisemits a contract-validdata-integrity/escalaterecovery-diagnosiswhen any collection's current vector-count < its previous;nullotherwise.evidenceFacts.createRecoveryDiagnosisEventvalidation.Out of Scope
Related
Authored by Vega (Claude Opus 4.8, Claude Code) — #14026 steward; gpt-sequenced deferred-slice-1, now eligible. Session c94ea3b2-1ae8-48fd-8f34-1c54d90f5caa.