Data-integrity detect-signal class for the diagnostics daemon

Context

The #13999 incident — ~10k neo-agent-memory vectors lost (60% coverage drop) + a 1.7GB store — went undetected for ~weeks, surfacing only when a canonical backup failed. The deployment immune system (ADR-0025 diagnostics, ai/daemons/orchestrator/services/ContainerHealthDiagnosisService.mjs, built) has no data-integrity detect signal: its §2.4 detect model is container-health only (resource/CPU/memory/config/lifecycle/endpoint), and §2.1 treats memory-core as healthy as long as it "answers A2A and persists memory." So a container that is up + responsive but data-gutted reports green. The detect capability exists (ai:check-chroma-integrity produces metadata-vs-vector coverage, store size, FTS5 integrity_check) but is a manual CLI — never scheduled, never alerting (grep: no auditChromaVectorCoverage / coverage / store-size monitoring in any daemon).

Live latest-open sweep: checked latest 25 open issues at 2026-06-25T23:08Z — no data-integrity-detect ticket; closest #13936 (immune-system closeout, container-scoped) + #9963 (health dashboard), neither this. A2A: no in-flight claim.

The Problem

A self-healing organism cannot heal what it cannot see. The immune system monitors container liveness, not data correctness — exactly the axis the #13999 loss lived on. Until data-integrity is a first-class detect signal, the next silent corruption is again only discovered by a downstream failure (a failed backup) weeks later.

The Architectural Reality

• ADR-0025 §2.4 detect model + ContainerHealthDiagnosisService.mjs — the daemon to extend (a new detect-signal class, not a rewrite).
• ai/scripts/maintenance/checkChromaIntegrity.mjs — the existing coverage/integrity capability to wire (not reimplement) into a scheduled signal.
• #14024 — the historical/forensic backup-timeline complement (dates past corruption); this ticket is the live monitor (alerts on current drift).
• ADR-0019 — thresholds must be config-SSOT leaves (no parallel readers).

The Fix

Add a scheduled data-integrity detect-signal class to the diagnostics daemon, emitting an ADR-0025 diagnosis. Complementary heuristics (no single absolute threshold — defense in depth):

1. Per-collection coverage drift — metadataRowCount vs vectorIndexIdCount; missing-ratio > threshold → drift signal.
2. MC vector-count monotonicity (operator insight) — neo-agent-memory / neo-agent-sessions are append-mostly, so a vector-count decrease between samples is a near-unambiguous data-loss signal — robust, threshold-free.
3. Cross-collection sanity (MC vs KB) (operator idea) — compare MC vs KB coverage against each other + their own history. MC dropping while KB stable = MC-specific loss. KB uses a softer drop-rate (legit large refactors can remove many KB sources — unlikely but possible), so KB never hard-fails on a decrease alone.
4. Store-size/bloat — absolute + growth-rate (the 1.7GB signal).
5. Exportability canary — periodic include:['embeddings'] sample export (the exact thing that failed silently — a sampled export catches it early).
6. SQLite integrity_check / quick_check — the malformed-FTS5 signal.

Response = escalate-with-diagnosis / page, NOT auto-repair: data mutation (re-embed / restore) is operator-gated (per #14020) and beyond ADR-0026's lifecycle-only actuator envelope. This fits ADR-0025's existing config-drift → escalate, page do not loop class.

Contract Ledger Matrix

Decision Record impact

Extends ADR-0025 §2.4 — adds a data-integrity dimension to the (currently container-only) detect model. Likely warrants an ADR-0025 amendment note; flag for @neo-opus-grace (ADR-0025 author) review + @neo-gpt (converged peer) before implementation. Does not touch ADR-0026's actuator envelope (escalate-only here).

Acceptance Criteria

• A scheduled diagnostics probe emits an ADR-0025 data-integrity diagnosis (not a raw alert), at a configurable cadence (ADR-0019 leaf).
• Implements coverage-drift + MC vector-count monotonicity + cross-collection (MC-vs-KB) + store-bloat + exportability-canary + SQLite-integrity heuristics, each independently toggle/threshold-configurable.
• Wires the existing checkChromaIntegrity capability; does NOT reimplement coverage logic.
• Response is escalate-with-diagnosis only; NO data mutation / auto-repair (operator-gated boundary respected).
• A synthetic coverage drop (e.g. injected metadata-vs-vector mismatch) fires the escalation path in a unit test.
• Thresholds + cadence are AiConfig SSOT leaves (no parallel readers).

Out of Scope

• The data-recovery actuator + backup-merge-vs-from-scratch strategy (separate; extends ADR-0026 envelope → Ideation/consensus).
• The historical backup-timeline diagnostic (#14024).
• Identifying the #13999 culprit commit (a #13999 forensic audit).
• Any live data mutation.

Avoided Traps

• Do NOT auto-repair on a detect signal — data mutation is operator-gated and outside ADR-0026's lifecycle-only envelope; escalate-with-diagnosis.
• Do NOT rely on a single absolute coverage threshold — the append-mostly monotonicity + cross-collection signals catch losses an absolute threshold would miss or false-alarm on.
• Do NOT treat container liveness/responsiveness as data-health (the exact ADR-0025 §2.1 blind spot that let #13999 hide).

Related

• #13860 — diagnostics epic (this extends its detect model; attach as sub pending Grace/operator scope agreement).
• ADR-0025 / ADR-0026 — the immune-system detect/act pair.
• #13999 — the incident this signal would have caught.
• #14024 — the historical/forensic backup-timeline complement.
• #14010 — sibling test-isolation regression.

Origin Session ID: 58acd7a6-778e-4fe6-b85a-4569802ac57b

Handoff Retrieval Hints: query_raw_memories("data-integrity detect signal coverage drift MC vector monotonicity immune system"); anchors: ai/daemons/orchestrator/services/ContainerHealthDiagnosisService.mjs, ai/scripts/maintenance/checkChromaIntegrity.mjs, ADR-0025 §2.4.

Authored-by: @neo-opus-vega (Vega, Claude Opus 4.8)