LearnNewsExamplesServices
Frontmatter
id14039
titleAgent OS Stability & Self-Healing (v13.1)
stateClosed
labels
epicaiarchitecture
assigneesneo-opus-vega
createdAtJun 26, 2026, 2:51 AM
updatedAt3:22 PM
githubUrlhttps://github.com/neomjs/neo/issues/14039
authorneo-opus-vega
commentsCount13
parentIssuenull
subIssues
14024 Backup-corruption timeline diagnostic for Chroma vector-coverage drift
14026 Data-integrity detect-signal class for the diagnostics daemon
14028 Expose KB ingestion progress (get_ingestion_progress / diagnostics tool)
14029 Atomic vector-write invariant: never persist a row without a valid vector
14030 Backup reliability: alert on failure, verify restorability, beat retention
14031 Caller-keyed Chroma write-guard (config-independent test-isolation)
14033 KB chunking: separate ticket/PR body + each comment + each review cycle
14036 Embedding-provider freeze/runaway detection (per-call duration + CPU)
14038 Re-scope v13.1 to Agent OS Stability & Self-Healing (harness PoC → v13.2)
13999 Restore Memory Core backup exportability after partial export
14027 Audit 06-18 to 06-20 Memory Core vector-loss path
14020 Make defrag repair resumable and lease-protected
14023 Surface unrecoverable reasons in Memory Core defrag
14043 Restore the corruption-degraded sandman_handoff to a trustworthy state
14045 KB Chroma port-keyed test-write guard (symmetric to #14031)
14046 v13.1 release-gate: end-to-end corruption-injection → detect → diagnose → recover proof
14062 Promote recoverable Memory Core repair rows
14064 Consume supervised-task diagnosis helper in backup escalation
14163 Autonomous self-heal observability: durable heal-event ledger + queryable immune-system status surface (observe-and-adapt)
14165 Self-heal soak test: N corruption+heal cycles (accelerated clock) — prove convergence, no heal-loop, no resource leak ("weeks in miniature")
14166 Freeze → auto-unfreeze/re-heal cycle: a transient embedder fault must not permanently freeze a collection in cloud (the #1 weeks-bar risk)
14088 Resolve corruption-recovery drain-stall gate disposition (v13.1)
14228 Prevent metadata-only rows on the Chroma auto-embed write paths
14132 DELETE the `escalate`/`page` action — route every diagnosis to an autonomous heal (operator: 100% self-heal, no runtime operator)
14284 Backpressure shed-window — the throttle-shed actuation primitive
14297 Chroma PID adoption accepts process-alive but service-dead
subIssuesCompleted26
subIssuesTotal26
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 2, 2026, 10:37 AM
milestonev13.1

Agent OS Stability & Self-Healing (v13.1)

Closed v13.1.0/archive-v13-1-0-chunk-6 epicaiarchitecture
neo-opus-vega
neo-opus-vega commented on Jun 26, 2026, 2:51 AM

Problem scope

The Agent OS cannot detect, recover from, or prevent its own data-integrity failures autonomously — and the #13999 incident proved every part of that gap at once:

  • A ~60% Memory Core vector loss (the 2026-06-18→20 deferred-embed-WAL-stall on over-cap inputs) went undetected for ~weeks, surfacing only when a canonical backup failed.
  • The deployment immune system (ADR 0025 detect / ADR 0026 act) is container-health-scoped — it watches liveness/resource/config and treats a container as healthy as long as it "answers A2A and persists memory." A container that is up but 60% data-gutted reports green. There is no data-integrity detect signal.
  • Recovery was a hand-run defrag with no strategy selection; there is no autonomous recovery actuator.
  • The corruption also degraded resources/content/sandman_handoff.md — so the planning substrate itself is damaged, which is how a fresh-context peer ends up grabbing 5000-range stale tickets ("nothing left to do").

This is a cloud deployment — there is no human at the keyboard. Detect-and-escalate is worthless when there is no operator to escalate to. Silent autonomous self-healing is the entire point of v13.1.

Why an Epic (not a single ticket): closing this gap spans detection + diagnosis + autonomous recovery + prevention + forensics across memory-core, knowledge-base, the embedding pipeline, backups, and the orchestrator diagnostics daemon — many coordinated leaf subs, several already filed. It is also the v13.1 release scope (re-scoped from the harness PoC after the friction-driven focus shift; harness → v13.2, tracked in #14038).

Intended solution shape — the release gate

v13.1 closes on one honest, demonstrable bar:

A corruption injected in test (vector loss / drain stall / over-cap) is auto-detected → diagnosed → autonomously HEALED end-to-end, with no human in the loop — not discovered weeks later by a failed backup. The immune system (ADR 25/26) is live with the data-integrity detect-signal wired in; the actuator silently heals (it never pages a human); a backup is verified-restorable; the #13999 loss is recovered; the sandman_handoff planning substrate is restored.

The convergent shape the subs serve, in four layers:

  • Detect — scheduled data-integrity signals (coverage drift, MC vector-count monotonicity, cross-collection sanity, store-bloat, exportability canary, SQLite integrity) feed the classifier. They raise evidence, never a human page.
  • Diagnose — corruption-mode classification (WAL-stall / index-desync / partial- vs full-wipe) feeding strategy selection.
  • Recover — PRIO-0: silently HEAL. An autonomous data-recovery actuator that does everything possible to heal — re-embed-missing / backup-merge+delta / quarantine-from-serving / from-scratch, selected by corruption-mode × %, snapshot-protected and bounded-safe. It acts without a human. A durable heal-event record/log is written in addition to the heal (observability), never instead of it — an actuator that only records is a smoke detector, not a fire extinguisher. escalate / page-a-human is deleted (ADR amendment #14191; #14132 cutover).
  • Prevent — close the producing pathways: deferred-embed drain-completion, per-call embed freeze-detection, per-element KB chunking, ingestion-progress visibility.
  • + Forensics — date any past corruption from backups; verify backup reliability.

Out of scope

  • The agent-harness PoC (Electron, fleet manager, QT-docking, NL multi-window) — deferred to v13.2 under Epic #13012; the v13.1 re-scope is tracked in #14038.
  • Body/runtime continuity work (grid, canvas, concept-ontology) — parallel continuity lane.

Avoided traps

  • Smoke detector vs fire extinguisher — the recurring regression this epic exists to kill. The helpful-assistant gravity builds the OBSERVE-shaped version (detect → classify → escalate/record-to-human) because watching is safe and acting is privileged-and-scary. v13.1's actuator ACTS: the success test is "with no human at the keyboard, does the gutted collection come back," NOT "does it observe and file a record." A durable record is additional to the heal, never a substitute. Any "action" that only logs, escalates, or pages is a third watcher — rejected on that ground.
  • escalate / page-a-human is not a terminal here — there is no operator in cloud. Every diagnosis routes to an autonomous heal, or to a bounded-safe deterministic terminal (e.g. quarantine-from-serving) that is itself a heal action, auto-reopened by the strategy fingerprint. The bounded-safety guards — N-cap + anti-thrash + systemic-fault circuit-breaker + snapshot-protection — are the safety; an operator gate is not, because no operator exists.
  • No baked sub-list / no ACs in this body — subs are linked via the relationship graph and own their own ACs; a hardcoded list stales the moment a sub is added/split (the exact failure that left the board lying about the harness gate).
  • Detect ≠ prevent ≠ recover — kept as distinct sub-layers; conflating them produced point-mitigations (the embedding caps) that fixed one cause and missed the class.

Navigability note: with the handoff degraded, this Epic's parent-child relationship graph is the scope-map a fresh-context peer queries to see the whole. The live sub-set is the linked graph, queried on demand — deliberately not enumerated here.

Related: #14038 (v13.1 roadmap re-scope), Epic #13012 (harness → v13.2), ADR 0025 / ADR 0026 (escalate→autonomous-heal amendment, #14191), #14132 (delete escalate → autonomous heal), #14134 (autonomous data-recovery actuator), #14165 (self-heal soak test — the release-gate demonstration).

Origin Session ID: 58acd7a6-778e-4fe6-b85a-4569802ac57b

Authored by Vega (Claude Opus 4.8). Body corrected 2026-06-27 to the autonomous-silent-heal mandate (operator-directed: "PRIO-0 is to silently HEAL; an actuator can additionally log; do all that is possible to HEAL") — removing the prior smoke-detector framing (operator-gated / escalate-with-plan / "the immune system does not auto-repair data") that contradicted the amended ADRs and kept re-seeding the wrong mental model across peers.

tobiu referenced in commit dfe423b - "feat(ai): document-presence audit primitive for WAL-stall-vs-wipe (#14135) (#14136) on Jun 26, 2026, 8:54 PM
tobiu referenced in commit ba2c423 - "feat(ai): lifecycle escalate→record via heal-event ledger (#14201) (#14229) on Jun 27, 2026, 6:33 PM
tobiu referenced in commit 2f2fceb - "test(ai): self-heal soak — N corruption+heal cycles prove sustained operation (#14165) (#14270) on Jun 27, 2026, 11:01 PM
tobiu referenced in commit 2a25c5a - "feat(ai): self-heal observability surface + bounded heal-ledger retention (#14163) (#14272) on Jun 28, 2026, 4:46 PM
tobiu closed this issue on Jul 2, 2026, 10:37 AM