LearnNewsExamplesServices
Frontmatter
id14084
titleMemory Core repair: distinguish accepted-loss recovery from failure
stateClosed
labels
enhancementaiarchitecture
assigneesneo-gpt
createdAtJun 26, 2026, 2:24 PM
updatedAt3:22 PM
githubUrlhttps://github.com/neomjs/neo/issues/14084
authorneo-opus-vega
commentsCount6
parentIssuenull
subIssues
14106 Accepted-loss residue classifier + terminal-reason taxonomy (#14084 leaf 1)
14110 Accepted-loss ack record constructor (#14084 leaf 2)
subIssuesCompleted2
subIssuesTotal2
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 27, 2026, 4:17 PM

Memory Core repair: distinguish accepted-loss recovery from failure

Closed v13.1.0/archive-v13-1-0-chunk-6 enhancementaiarchitecture
neo-opus-vega
neo-opus-vega commented on Jun 26, 2026, 2:24 PM

Context

Surfaced during the operator-directed 2026-06-26 #13999 recovery. The Memory Core repair-defrag durably promoted 22636/22637 (neo-agent-memory), 1408/1408 (neo-agent-sessions), and 941/942 (neo-native-graph) rows — an operationally-complete recovery — yet the run ended:

❌ Memory Core repair non-clean for neo-agent-memory, neo-native-graph ... NOT a successful repair.
   (exit 1)

The two un-promoted rows (aa8ecd3f… 45538 tok, issue-11187 35502 tok) are genuinely unembeddable — their documents exceed the embedding model's 32768 context, so no amount of re-running recovers them.

The Problem — and the regression this ticket originally encoded

#14066 correctly made the data partial-promote (recovered rows durable, residue parked). But the outcome status is binary: any unrecoverable row ⇒ anyRepairNonClean()exit 1 ⇒ the immune system (#14061) pages the operator. Because the 2 residue rows can never be embedded, this is permanent: every future repair re-promotes the (already-promoted) corpus and exits 1, and every orchestrator cycle re-escalates "repair failed" — for a system that is, in fact, fully recovered.

The original fix framed the settlement as operator-acknowledged — and that is the regression. In a cloud deployment there is no operator to page or to acknowledge (the canonical ai/deploy / klarso docker-mcp stack: orchestrator + MC + KB + chroma, no human in the loop) — and operatorless self-healing is the entire point of v13.1 (#14039). An operator-gated accepted-loss settlement leaves a cloud deployment escalating into a void forever. The detect→classify→escalate-to-human shape is a smoke detector; v13.1 needs the fire extinguisher — and the actuator already holds it: the orchestrator runs with RUNTIME_ACCESS_LIFECYCLE_OPERATIONS=restart, the docker socket, and a backups volume. The agency exists; it was wired to escalate because acting is scarier than alarming.

The Architectural Reality

  • defragChromaDB.mjsanyRepairNonClean() treats partial-promoted as non-clean → CLI process.exit(1) (per #14066).
  • dataIntegrityCoverageDiagnosis.mjs:63 (and every sibling producer) hardcodes actionClass: 'escalate' — a page, never an act.
  • #14061 escalateDiagnosis composes with the non-zero exit to page — correct for a recoverable failure or an unbounded fault, wrong for accepted, bounded, deterministically-unembeddable loss in an operatorless deployment.
  • The durable unrecoverableByCollection manifest (reasons/ids) + the #14106 terminality classifier + the #14110 ack fingerprint already exist — the missing piece is autonomous settlement, not an operator gate.

The Fix — DECIDED 2026-06-26 (autonomous-cloud-settlement; re-scoped from operator-gated)

The recovery system self-settles bounded, deterministic-terminal loss without a human, and records it durably + observably for asynchronous audit:

  1. Autonomous settle (the default). When every residual unrecoverable row carries a deterministic-terminal reason (embedding-context-exceeded / document-absent — facts from the embed attempt, not judgments) AND the residue is bounded (below a systemic-fault threshold — absolute count + ratio of the collection), the system writes a durable auto-accepted-loss audit record (ids + reasons + the #14106 strategy/provider/context fingerprint) and reports a clean, non-escalating outcome. No operator, no ack. The fingerprint means a later embedding-capability change (chunking, larger context) auto-reopens the residue.

  2. Escalate only the genuinely-unbounded. Residue above the systemic-fault bound (mass terminal-residue = a misconfigured embedder, not bounded accepted loss — the same false-storm class as a bad expectedDimension), OR any transient/unknown reason, still escalates. In cloud, "escalate" is a durable async record + the safest bounded action — never a blocking page into a void.

  3. Operator-ack → override/audit, not gate. The durable accepted-loss ack (#14110) is retained as (a) a dev/operator override to settle an ambiguous residue a human chooses to accept, and (b) the audit surface to review auto-settled losses. It is no longer the precondition for settlement.

The #14106 classifier + #14110 ack constructor are unchanged as machinery: the classifier's terminal-vs-transient decision is the autonomous-settle gate; the fingerprint is the audit record + the auto-reopen key. Only the default flips — bounded-deterministic-terminal self-settles instead of waiting for a human.

This discharges the "silent success on loss" trap WITHOUT an operator — auto-settlement is bounded (mass loss escalates), deterministic (terminal reasons are facts, not guesses), observable (durable audit record), and self-reopening (the strategy fingerprint). Not silent — async-auditable.

Acceptance Criteria

  • Bounded, deterministic-terminal residue settles autonomously (no operator): a durable auto-accepted-loss audit record + a clean, non-escalating outcome; the run exits 0.
  • Residue above the systemic-fault bound (absolute count + ratio) escalates — a mass terminal-residue spike is a systemic fault, not accepted loss.
  • Transient/unknown unrecoverable reasons still escalate (no silent acceptance of recoverable loss).
  • In cloud, escalation is a durable async record + the safest bounded action — never a blocking page that needs a human to clear.
  • The operator-ack (#14110) is retained as an override (settle an ambiguous residue) + audit surface (review auto-settled losses), not the settlement gate.
  • Clean / failed / partial-promoted / auto-accepted-loss / escalated remain distinguishable in CLI output, state marker, and the escalation path.

Decision Record impact

The two-worlds boundary holds; the gap is narrower than a blanket "act not escalate." #14039 deliberately scopes recovery as operator-gated escalate-with-plan, not silent auto-repair (the ADR-0026 two-worlds boundary) — and that is correct for genuinely-dangerous data mutations (restore-from-backup / backup-merge / re-embed): those stay operator-gated. The accepted-loss settle is NOT such a mutation — it records irreducible-terminal residue and exits clean, mutating nothing — so it can resolve autonomously without breaching the boundary. The genuine operatorless-cloud gap @tobiu surfaced is narrower: escalate-with-plan assumes an operator receives the plan, which is a no-op when there is none. So in cloud the gated-escalate must degrade to a durable async record (an operator reviews it when present), never a blocking page into a void — while the dangerous mutations stay gated. Whether this cloud-resolution refinement is v13.1 scope or v13.2 is a premise question under reconciliation on #14039; this re-scope is the autonomous-settle exemplar only, NOT a mandate to autonomise dangerous recovery actions. (Correcting my own earlier over-reach in this section, which had listed restore-from-backup as an autonomous act — it is not.)

Out of Scope

  • Making the oversized docs embeddable (the prevention lane — see Related).
  • Reverting #14066's partial-promotion (this builds on it).
  • Auto-deleting parked/residue collections (a separate bloat/cleanup decision; auto-settle records + retains, it does not delete).

Avoided Traps

  • Silent success on loss. Auto-settlement is never blind: bounded (mass loss escalates) + deterministic-terminal-only + a durable observable audit record + self-reopening on capability change. The audit record is the operator's async review surface — acceptance is recorded and reversible, not invisible.
  • Treating all unrecoverables as terminal. A transient provider failure must still escalate — the #14106 reason taxonomy gates this.
  • Auto-settling a systemic fault. A misconfigured embedder reporting everything terminal is mass loss, not accepted loss — the systemic-fault bound escalates it (mirrors the #14115 expectedDimension false-storm hazard).

Related

  • #14066 (partial-promotion — the contract this refines), #14106/PR #14108 (terminality classifier — the autonomous-settle gate), #14110/PR #14111 (ack fingerprint — now the audit/override machinery), #14068 (parking lifecycle), #14061 (escalation that currently over-fires), #14109 (the runner whose escalate-terminal inherits the same act-vs-escalate correction), #14039 (v13.1 self-healing epic), #13999 (the recovery that surfaced this). Prevention sibling: the Memory Core oversized-doc chunking ticket.

Handoff Retrieval Hints

  • query_raw_memories("Memory Core repair autonomous accepted-loss cloud self-healing terminal unrecoverable exit-1 escalation")
  • Exact anchors: anyRepairNonClean, actionClass: 'escalate', unrecoverableByCollection (defragChromaDB.mjs), classifyRepairResidue (#14106), escalateDiagnosis (#14061).

Re-scope note (2026-06-26, Grace). Original problem-framing + the #13999 anchor authored by Vega (surfaced by @tobiu: "the entire recovery failed with 1 faulty item"). Re-scoped from operator-acknowledged (Q1 "data loss acceptance is operator authority") to autonomous-cloud-settlement after @tobiu's correction: cloud ⇒ no operator ⇒ self-healing is the entire point of v13.1. The original three "needs-design" open questions are now decided — Q1: the system settles autonomously for bounded-deterministic-terminal, operator-ack is override-only; Q2: the terminality taxonomy shipped in #14106; Q3: chunking shrinks the residue (unchanged). Dropping needs-design / not-code-ready.