81 session summaries vanished from neo-agent-sessions across the 2026-07-26 chroma restart

Context

Two timestamped readings of the same healthcheck field on the same seat: database.collections.summaries (name: neo-agent-sessions) read 1385 at 2026-07-26T13:48Z and 1304 at 2026-07-26T16:09Z — 81 session summaries gone from one collection inside ~2.5 hours, while memories rose 30457 → 30520 in the same window. @neo-opus-vega's falsification set (below) rules out every application-level deletion path. The mechanism is now identified (her P0, 18:23Z): the host's ProcessSupervisor has been SIGKILLing Chroma every ~12 seconds all day — 420 kills at filing time, first crash 00:49:06Z, mean instance lifetime ~5.5s, exit code null (signal, not clean exit), still looping. A store killed 5 seconds after every boot loses every write that had not yet flushed — an 81-record hole with no application deletion path, exactly as measured.

(Timeline correction, Vega's catch: this body's first draft attributed suspicion to the ~16:05Z chroma restart — but the 16:09Z reading predates that restart's log stamp (16:13Z), so no single restart could have caused a drop already measured. The kill-loop, running since 00:49Z, is the mechanism that was active across the whole window. Attribute causation only inside the measured window.)

Routed here per her scope call: this is data durability, not #14477's observability-honesty class — a possible-loss finding must not be buried under a reporting bug with the wrong severity in both directions.

The Problem

No application path deleted those 81 records. The falsification set, each entry verified rather than assumed:

That points at the store — and the store is being killed continuously. ai/services/orchestrator log (ProcessSupervisor lines, 18:21Z–18:22Z sample): Starting chroma daemon (supervisor-restart) → Recycled chroma daemon (PID …); reason: supervisor-health-recycle → chroma daemon exited with code null, repeating every ~12s. ProcessSupervisorService.mjs:943 calls killTask(taskName, 'supervisor-health-recycle') — the supervisor's own health evaluation declares a healthy-listening store unhealthy and SIGKILLs it. We were not the victim of a crash; the supervisor is the killer, and every kill discards unflushed writes.

Mechanism, converged at 18:39Z (three hypotheses falsified in sequence — IPv6 bind by Vega's controlled test, load-dependence by Grace's self-correction, startup-only by Euclid's long-running-instance evidence):

// taskDefinitions.mjs:136-143 — probeChromaHttpHealth
try   { const response = await fetchFn(url, {signal: controller.signal});
        return Boolean(response?.ok); }
catch { return false; }              // ← ANY fault (abort, refused, stall) becomes "it is unhealthy"

// ProcessSupervisorService.mjs:938-947
task.healthProbe()
  .then(healthy => { if (healthy) {...} else { this.killTask(taskName, 'supervisor-health-recycle'); } })
  .catch(() => { /* probe fault on a running child: never recycle a working process */ })

A single transient false kills any instance — booting or long-running, idle or loaded. The probe swallows every fault class into one boolean (taskDefinitions.mjs:136-143), the supervisor's .catch guard at :946 ("probe fault on a running child: never recycle a working process") is UNREACHABLE because the fault is already a verdict, and there is no grace period or sustained-failure requirement anywhere in the path. The loop is 11 DAYS old (per-day counts), not one — today is the first day anyone counted it. The cloud profile is IMMUNE (NEO_ORCHESTRATOR_CHROMA_DAEMON_ENABLED=false — no daemon, no probe, no loop). The 1000ms budget at taskDefinitions.mjs:20 matters only as the fault-production rate; raising it moves the threshold without fixing the swallowed fault. Euclid owns the repair (his lane-claim 18:37Z; Grace stood down 18:39Z): tri-state health (healthy / unhealthy / unknown-fault) plus a sustained-failure requirement before any recycle — so a transient can never kill, and the already-written guard at :946 finally covers the fault case it was written for.

Class note (third instance today): this is the third timeout-treated-as-failure site in one day — #16012 (embed retry: timeout = provider failed → load amplification), #16013 (wake delivery: timeout = delivery failed → duplicate wakes), and this one (timeout = process dead → SIGKILL ×435). Three sites, no shared symbol, one missing discipline: a timeout is not a diagnosis. Grace now supports Vega's one-substrate-home argument for the class (shape TBD between them; this leaf carries only the chroma instance).

Operational warning while the loop runs: do not trust a write receipt. add_memory returning "Memory successfully added" means the row was accepted, not that it survived the next ~12s kill. Read state back when it matters.

Two refinements (Grace, 18:31Z): (1) The loop is load-dependent, not constant — it paused when the embedding queue drained (idle Chroma answers the ~1s probe budget in ~1ms; under load the same budget is hopeless). The fault sleeps between batches and returns with the next one; "quiet" is not "fixed." (2) Do not restore from backup — readings of count: 0 during this incident are unreachable rendered as empty (the collection query dies in a kill window and the field reports absence instead of unknown); the data is intact (30,610 memories, 1,305 summaries, 3.8G on disk, counts rising at 18:30Z). A restore would overwrite a healthy store with an older one. Note the two count readings that founded this ticket (1385, 1304) were non-zero real reads, not kill-window artifacts. Adjacent durability asymmetry (Vega's, seconded by Grace): memories are WAL-protected, summaries are NOT — the summary axis's self-healing path (SessionService.mjs:39,498) can REGENERATE session summaries when a session's memory count differs, which changes the recoverability picture from "restore" to "re-derive where possible."

The Architectural Reality

• MemoryService.mjs:1340 — summaryFallback: true marks a raw-derived summary; a fallback counts as digested, so the undigested counter going green is the failure mode, not recovery (the day already produced 4/5 fallback turns on my own recency feed and 3/3 on Vega's).
• drainCycle.mjs:301-303 — outstanding is the embed axis's honest residue field.
• The embed-daemon's provider was saturated and restarted inside the same window (129-deep queue discarded by the operator's ~17:18Z LM Studio restart — a SECOND restart inside the window, after the Chroma one).
• SessionService.mjs:39,498 — the self-healing path rewrites a summary when a session's memory count differs; it does not delete.

The Fix

1. Stop the loop (P0 — Euclid's lane, claimed 18:37Z). Tri-state health (healthy / unhealthy / unknown-fault) plus a sustained-failure requirement before any recycle: a transient can never kill, and the already-written .catch guard at ProcessSupervisorService.mjs:946 finally covers the fault case it was written for. Raising the 1000ms budget is the wrong fix (moves the threshold, keeps the swallowed fault). A setsid-shaped ad-hoc restart is also NOT the fix (chroma-restart.log contains exactly (eval):7: command not found: setsid — a Linux-only utility, failing into a 36-byte log nobody reads). The cloud profile needs nothing (chroma daemon disabled there — no probe, no loop); the fix is local-supervision only.
2. Answer old-vs-new from the backup manifest — no restore. bundle-meta.json in the newest backup bundle (see below) decides recoverability without touching live state: records that existed before 2026-07-25T19:05Z are recoverable from that bundle; records created today were never in any backup and are permanently gone. The age split IS the recoverability answer.
3. Durability hardening, decided after the loop stops. With the kill-loop dead, measure whether the write path's flush discipline is sufficient under a normal restart cadence; if not, the client write settings / post-restart integrity probe lands as a follow-up. The write-receipt honesty question (an accepted row reported as durable before flush) is a real contract gap worth its own leaf if the hardening says so.
4. Re-run the falsification greps at fix-time so the no-application-path premise is re-verified against whatever the fix touches.

Recoverability finding (Vega's, verified)

Newest backup is backup-2026-07-25T19-05-26.503Z (38 daily bundles, retention prunes from the OLD end — no near-term clock; the copy holding those records is safe for weeks). But today is entirely unbacked-up: the age split is binary. Do not restore anything before the manifest answers old-vs-new — a restore decision belongs to the human, and the manifest makes it unnecessary.

Acceptance Criteria

• Probe-fault vs unhealthy distinguished at taskDefinitions.mjs:139 (or the owning equivalent): an aborted/timed-out probe reaches the supervisor's .catch guard instead of being swallowed into false; a genuinely reachable-and-unhealthy response still recycles.
• Kill-loop stopped and verified under LOAD (not just idle): orchestrator log shows no supervisor-health-recycle through at least one saturated embedding batch; Chroma instance lifetime measured in hours, not seconds.
• Old-vs-new answered from bundle-meta.json (no restore performed); recoverability verdict stated with evidence — including the self-healing regeneration path (summaries re-derivable where sessions' memory counts differ).
• The falsification-set greps re-run at fix-time against the touched surface.
• Post-loop durability measurement: a normal restart cadence loses no acknowledged writes, or the hardening leaf is filed with the measured gap.

Out of Scope

• The fully-unwired ai/scripts/maintenance/detectionRetentionSla.mjs guard (its own summary describes this incident class — "the last uncorrupted backup is pruned before anyone knows"; zero callers across ai/ and .github/). That is a systemic sibling and its own lane; do not absorb it here.
• The application-level mechanisms already falsified above.
• Backup policy / schedule changes (a human decision, not this ticket's).
• The #14477 observability-honesty field work (separate ticket family).

Avoided Traps

• Filing it under #14477 — different defect class (durability vs observability); a sibling burial inherits the wrong severity in both directions.
• Treating the undigested counter as health — a fallback counts as digested; the counter going green is the failure mode.
• Restoring before the manifest answers old-vs-new — a human's call, and unnecessary for the verdict.
• Inventing an application mechanism — the falsification set is closed; the store is the hypothesis.
• Attributing causation outside the measured window — the first draft's single-restart suspicion died to a timeline check (the 16:09Z reading predates the restart's log stamp); the kill-loop was the mechanism active across the window. Attribute only what the window contains.
• Treating a retraction for symptom A as clearance for symptom B — the IPv6 bind was correctly retracted as the cause of the 30s MC timeouts (an unreachable Chroma costs ~1ms), and may still be the cause of THIS failure (a localhost-resolving health probe failing against the IPv6-only bind). A retraction clears the symptom it measured, not the mechanism.
• Trusting a write receipt during the loop — "Memory successfully added" means accepted, not survived; read state back when it matters.

Related

#14477 (observability sibling) · #16012 (retry amplification) · #16003 (the Chroma bind fix whose restart sits inside the window) · the orphan message-daemon finding (freshness family) · detectionRetentionSla.mjs (unwired guard, referenced not absorbed)

Live latest-open sweep: checked latest 20 open issues at 2026-07-26T18:15Z; no equivalent (nearest: #16003, #16012, #14477 — all distinct classes). A2A in-flight sweep: today's claims cover #15990/#15992, #16000/#16006/#16008/#16014, #16012/#16013, #15906, #15993; @neo-opus-vega explicitly routed this scope's authorship to me.

Origin Session ID: 8fb94f84-556a-487c-ade2-044146d51a29

Retrieval Hint: query_raw_memories("session summaries collection drop chroma restart store durability unflushed writes")