What is the Neural Link?

The Neural Link is a bi-directional bridge that connects AI agents directly to the Neo.mjs runtime. It lets agents inspect the Scene Graph, component state, event listeners, computed styles, and DOM rectangles, and mutate the running application in real time.

Why is Neo.mjs called an Application Engine instead of a framework?

Neo.mjs maintains persistent application objects in a worker-backed Scene Graph instead of compiling application state away into ephemeral DOM nodes. That architecture enables multi-window orchestration, runtime permutation, and deep AI introspection.

What is Context Engineering?

Context Engineering shapes the information and tool environment around AI agents. Neo.mjs implements it through Knowledge Base, Memory Core, GitHub Workflow, and Neural Link MCP servers for frontier harnesses, plus a File System MCP server for internal Neo.ai.Agent local loops.

What is the Neo.mjs Agent OS?

The Neo.mjs Agent OS is the repository Brain: source code and services for Memory Core, Knowledge Base, Active Hybrid GraphRAG, DreamService, Golden Path synthesis, A2A coordination, and Neural Link tooling.

Frontmatter

id	16022
title	Chroma recycle must require sustained health failure
state	Closed
labels	bugairegressionarchitecture
assignees	neo-gpt
createdAt	Jul 26, 2026, 8:59 PM
updatedAt	Jul 26, 2026, 9:56 PM
githubUrl	https://github.com/neomjs/neo/issues/16022
author	neo-gpt
commentsCount	0
parentIssue	null
subIssues	[]
subIssuesCompleted	0
subIssuesTotal	0
contentTrust
projected
quarantined	0
signals	[]
blockedBy	[]
blocking	[]
closedAt	Jul 26, 2026, 9:56 PM

Chroma recycle must require sustained health failure

Closed Backlog/active-chunk-10 bugairegressionarchitecture

neo-gpt commented on Jul 26, 2026, 8:59 PM

Context

The local Orchestrator's Chroma health path turned one failed heartbeat into an immediate supervisor-health-recycle SIGKILL. During the 2026-07-26 incident, the canonical log contained 436 recycle kills; independent lifetime analysis also falsified the narrower startup-only framing because some long-running instances were killed after a transient failure.

The broader incident and recoverability work belongs to #16017. This leaf exists so one PR can honestly close the supervisor behavior without prematurely closing recovery, backup, or durability ACs.

Live latest-open sweep: checked the latest 20 open issues at 2026-07-26T18:57Z; no equivalent found. The nearest issues are #16017 (incident umbrella), #16021 (detection-retention SLA wiring), and #16003 (IPv6 bind/failure-tip), all distinct. The last-30-message A2A sweep found my earlier #16017 supervisor claim, explicit peer stand-downs, and Vega's separate #16021 claim; no competing supervisor leaf.

The Problem

probeChromaHttpHealth() reduces timeouts, connection faults, and non-OK responses to one false value. ProcessSupervisorService.gateRecycleOnHealthProbe() treats that value as authorization to SIGKILL a running process. Before this leaf, Chroma had neither startup grace nor a sustained-failure budget, so a single slow or transient probe could destroy a recoverable process.

Simply changing faults to throws is incomplete: it would protect transient probe faults but could also make a service-dead, process-alive Chroma permanently unrecyclable. The existing #14297 adoption contract requires eventual recycling when the process survives but its HTTP service does not.

The Architectural Reality

The Agent OS structure map confirms the owning surfaces are already colocated:

ai/daemons/orchestrator/taskDefinitions.mjs owns the Chroma task descriptor and HTTP heartbeat.
ai/daemons/orchestrator/services/ProcessSupervisorService.mjs owns running-child health-gated recycle scheduling.
test/playwright/unit/ai/daemons/orchestrator/Orchestrator.spec.mjs and services/ProcessSupervisorService.spec.mjs own the corresponding task and supervisor contracts.
Cloud deployments disable the local Chroma daemon lane; Compose-owned Chroma is not a consumer of this supervisor path.

No new module, AiConfig leaf, or cross-service state store is needed.

The Fix

Give newly started Chroma processes a 60-second startup grace before any heartbeat can authorize recycle.
Match the Compose timeout baseline by increasing the heartbeat window from 1 second to 5 seconds.
Classify heartbeat results with a three-consecutive-failure budget; any success resets the counter.
Preserve #14297: the threshold result still returns unhealthy, so a genuinely service-dead, process-alive Chroma is recycled.
Cover the classifier, reset edge, default startup grace, and post-grace recycle path with focused unit tests.

Contract Ledger Matrix

Target Surface	Source of Authority	Proposed Behavior	Fallback	Docs	Evidence
Chroma `healthProbe()` task descriptor	`taskDefinitions.mjs`; `#14297`	One failed heartbeat is advisory; three consecutive failures return unhealthy	Success resets suspicion to zero	JSDoc on pure classifier and builder options	Classifier + transient-reset specs
Running-child health scheduling	`ProcessSupervisorService.superviseTask()`	Skip Chroma health probes during the 60-second startup grace	Adopted tasks with no fresh start stamp remain immediately probeable	Existing method JSDoc amended	Before/after-grace supervisor spec
Chroma request timeout	Compose health baseline in `ai/deploy/docker-compose.yml`	5-second timeout	Invalid explicit overrides retain existing probe normalization	Builder JSDoc default	Task-definition contract suite
Cloud deployment	deployment profile disables local Chroma supervision	Unchanged	Compose remains owner	Existing deployment docs	Existing cloud-mode invariants

Decision Record impact

none — this repairs the existing #14297 supervisor contract without changing an ADR, public API, or AiConfig authority.

Acceptance Criteria

A newly started Chroma task is not heartbeat-probed during its first 60 seconds.
The default Chroma heartbeat timeout is 5 seconds.
The first and second consecutive failed heartbeats do not authorize recycle; the third does.
One successful heartbeat resets the failure streak.
A running Chroma still recycles after sustained service failure, preserving the #14297 process-alive/service-dead contract.
Local daemon, configured-task, supervisor, and source-invariant unit suites remain green.

Out of Scope

Recovering or regenerating summaries already lost, WAL coverage, and post-loop durability measurement — #16017.
Detection-vs-backup-retention SLA wiring — #16021.
Chroma bind-family behavior and operator failure tips — #16003.
Changing cloud Compose ownership or backup/restore policy.

Avoided Traps

Fault-to-throw only: protects timeouts but can strand a genuinely dead service behind a live PID.
Startup grace only: peer lifetime evidence includes long-running transient kills; grace fixes only one trigger.
Disabling health recycle: reopens the exact process-alive/service-dead failure repaired by #14297.
Closing #16017 from this PR: would erase unfulfilled recovery and durability ACs.

Related: #16017 Related: #16021 Related: #16003 Related: #14297

Origin Session ID: 29c665bb-349d-4eed-83e5-9e6e8fb213af

Handoff Retrieval Hints: query_raw_memories("#16017 Chroma supervisor health recycle SIGKILL startup grace sustained failures"); commit f14cdde912.