A container at its memory ceiling is already detected. ContainerHealthDiagnosisService computes saturation against the limit (:952-959, usage / limit * 100), applies a sustained-window filter rather than a single sample (:361), and emits a memory-saturation fact at 90% (:30).
It then routes that fact — together with CPU resourceSaturation — to one uniform prescription (:586-598):
throttle-shed is the wrong heal for a data store, and it is the only one on offer.
The available action classes are record, restart, throttleShed, warmProvider (:21-26). For a process holding transient work, shedding load is correct — pressure comes from arrival rate, so reducing arrivals relieves it. For a vector store, the corpus is the workload: memory is a near-deterministic function of rows already stored. There is nothing to shed, restarting frees nothing durable, and the pressure returns the moment the corpus is reloaded.
Measured on the live plane 2026-08-06: chroma at 4096-dim float32 needs rows x 16 KiB x ~1.38. At ~96,000 rows that is ~2.02 GiB against a 2.00 GiB cap. Because the store exits cleanly rather than being OOM-killed (OOMKilled=false ExitCode=0 RestartCount=13), each attempt to reach a complete corpus stops near completion and reads as a graceful restart. A restore of 59,754 chunks died at 24,000 this way.
So the diagnosis layer can see the condition and has no vocabulary to express the only heal that works: raise the ceiling.
Second-order: because growth is monotonic and predictable, 90% is late for a store. By the time saturation is sustained at 90%, the remaining headroom is smaller than one ingestion batch.
The Architectural Reality
Three pieces exist and are not connected:
Detection — ContainerHealthDiagnosisService, per-service, sustained-window, measured against the limit.
A bounded actuator for config intent — RecoveryActuatorServiceDEFAULT_ACTIONS includes reconfigure (:23), dispatching to reconfigureComposeService({knob, knobValues, reason, target}) (:582). escalate and page are already deleted from that set, consistent with the autonomous-heal directive.
An intent-based knob registry — RECOVERY_KNOBS (recoveryKnobRegistry.mjs:39). Its docblock states the property that makes it the right home: "A controller asks to widen the mini-summary window; it never names leaves." Bounds are enforced in the registry (:190-191), so a controller cannot request an unbounded change.
What blocks the connection: chroma's limit is hardcoded memory: 2g (docker-compose.yml:47), so no knob can reach it. local-model already demonstrates the controllable form in the same file — memory: "${NEO_LOCAL_MODEL_MEMORY_LIMIT:-32g}". The store is the one component whose ceiling is both load-bearing and unreachable.
And a live raise is possible without recreation. Verified against a disposable target (ingress, 256m → 300m → restored): docker update --memory is accepted and takes effect on a running container. So the heal can be immediate and autonomous; the compose/env value is what makes it durable across the next recreate.
The Fix
Parameterise chroma's ceiling as ${NEO_CHROMA_MEMORY_LIMIT:-8g}, following local-model's existing precedent. The default is derived, not preferred: rows x 4096 x 4B x 1.38 puts a complete corpus at ~2.02 GiB, and 8g admits ~370k rows — about 4x — at ~17% of the docker VM.
Add a bounded container-memory-ceiling knob to RECOVERY_KNOBS, expressing the intent "raise this service's memory ceiling" with registry-enforced min/max so a controller can never request an unbounded value.
Classify services as store-backed or transient, and route memory-saturation on a store to the ceiling-raise intent rather than throttleShed. Transient services keep throttleShed unchanged.
Lower the store threshold to 80%, because monotonic growth makes 90% later than one ingestion batch.
Deliberately not in this ticket: the vector dimension. Reducing 4096 would cut resident memory proportionally and is rejected on operator direction — below ~4k, query_documents retrieval quality degrades materially, and it would require re-embedding the whole corpus. Recorded so it is not re-proposed as a cheaper fix.
Decision Record impact
amends ADR-0025 (Orchestrator Container Health Self-Healing) and amends ADR-0026 (Orchestrator Recovery Actuator, Proposed).
ADR-0025's action-class taxonomy has no ceiling-raise class; ADR-0026's actuator matrix has no row for it. Both need the store-versus-transient distinction, because the same fact warrants opposite heals depending on whether the pressure comes from arrival rate or from stored data. ADR-0026 being pre-merge makes extending it cheaper than amending later; @neo-opus-grace authored both and holds the deepest context on the un-healable class, so she is the right reviewer.
Handover to @neo-fable-clio — 2026-08-07, and three ACs were stale
Reassigned at @tobiu's request. Before handing it over I verified every AC against the tree rather than
trusting its checkbox, and three of eight had shipped with PR #16597 and were never ticked — corrected
below with the delivering file and line. Left unticked they would have cost the next owner a cycle
reimplementing a code path that already has a passing spec citing this ticket number.
Genuinely open: the healthy-state classification projection · a container-memory-ceiling knob with
registry-enforced bounds (searched: nothing exists, not even partial) · the bounded/anti-thrash raise ·
the ADR-0025/0026 rows · the post-merge observability check.
Sequencing. Draft PR #16634 modifies the same collectStatsFacts function in the same file — it adds
an effective-heap-ceiling denominator for NON-STORE services. The projection AC moves classification fields
out of if (memoryWindow.sustained) in that same function. Different concerns, real textual conflict risk;
branching after #16634 merges avoids it.
Adjacent but deliberately separate:#16630 leaves service-class exhaustion unrouted to raiseCeiling
on purpose. This ticket's store-side routing is delivered. The two were split, not duplicated.
Acceptance Criteria
A healthy-state classification projection — DeploymentStateBridge emits serviceClass, serviceClassDeclared, the applied threshold, and observedWindowMs beside requiredWindowMs for every store-classified service on every snapshot, independent of load. Delivered by PR #16638 (merge 82e26297b5) — ContainerHealthDiagnosisService.describeClassification (verdict-free) attached per service in collectServiceSnapshot; the bridge-spec falsifier proves a healthy store at 10% memory carries the full block, with stampCoverage distinguishing under-stamped from under-length windows. (Ticked 2026-08-07 on merge.)Moved here from #16603 on 2026-08-07 (@neo-gpt cycle-4). Today those fields exist only inside if (memoryWindow.sustained), so a healthy store at the raised 8g ceiling exposes none of them — which makes any load-independent observability claim unverifiable and is why three successive post-merge formulations on #16603 were each unobservable. It is new behaviour, not a wording fix, so it belongs on this ticket beside the other undelivered half rather than blocking the leaf that delivered detection.
chroma's compose limit is env-parameterised with a derived default, and the arithmetic is recorded inline. Delivered by PR #16597 — memory: "${NEO_CHROMA_MEMORY_LIMIT:-8g}" with the reachability rationale and the resident-vector arithmetic recorded above it. (Ticked 2026-08-07 on handover; it had shipped and gone unchecked.)
A container-memory-ceiling knob exists with registry-enforced bounds; a request outside them is rejected with a violation rather than clamped silently. Delivered by PR #16638 (merge 82e26297b5) — recoveryKnobRegistry.mjscontainer-memory-ceiling: 8–16 GiB derived band, serviceKey: chroma binding, raise-not-lower invariant resolving from the RUNTIME; recoveryKnobRegistry.spec.mjs asserts the cap refusal names the band and never clamps. (Ticked 2026-08-07 on merge.)
A store-class service at sustained saturation yields a ceiling-raise action class, notthrottle-shed. Spec asserts the action class, not merely that a fact was emitted. Delivered by PR #16597 — ContainerHealthDiagnosisService.mjs:726-732 returns actionClass: raiseCeiling with reason: 'store-ceiling-exhaustion', and ContainerHealthDiagnosisService.spec.mjs:650 asserts decision.actionClass directly, exactly as this AC demands. (Ticked 2026-08-07 on handover.)
A transient-class service at sustained saturation still yields throttle-shed — a negative control, so the change is proven narrow rather than global. Delivered by PR #16597 — ContainerHealthDiagnosisService.spec.mjs:700, whose comment reads "Negative control for the ROUTING."(Ticked 2026-08-07 on handover.)
The raise is bounded and anti-thrash: repeated saturation cannot ratchet the ceiling indefinitely, and the bound is asserted by spec. (Restated 2026-08-07 per comment 2, folded on pickup: without a restart there is no restart-storm, so the ratchet is the only thrash vector left — the bound lives in the registry, where an out-of-band request is a thrown violation rather than a silent clamp.)Delivered by PR #16638 (merge 82e26297b5) — the band holds at the registry AND at the L0 boundary (review-1 hardening), the check-through-write section is serialized per target (review-2 hardening: a stale-validated concurrent lowering is unreachable, with a mutation-sensitive interleaving witness), and monotonic-raise-to-cap bounds even the direct path's total travel. (Ticked 2026-08-07 on merge.)
A store-class ceiling raise does not restart its target. Spec asserts the restart path is not called on the store path — a negative assertion, since the defect is an extra call rather than a wrong value. (Folded from comment 2's forced AC revisions on 2026-08-07 pickup — @neo-opus-vega's finding that reconfigure couples the mutation to a restart, and for a store mid-ingestion that restart is the harm itself.)Delivered by PR #16638 (merge 82e26297b5) — raiseComposeServiceCeiling omits the restart as its contract; mutation-verified through the single recorded lifecycle seam (re-adding the restart reddens the centerpiece spec). (Ticked 2026-08-07 on merge.)
ADR-0025 and ADR-0026 carry the new action class, the actuator row, and the store/transient rationale. Delivered by PR #16638 (merge 82e26297b5) — ADR-0025 §2.4 store-exhaustion route + taxonomy; ADR-0026 class-split matrix row, §2.8 store-variant envelope (incl. the performed-vs-in-effect provenance bound and the boundary/serialization clauses), AC-12; recoveryActuatorAdrCoherence.spec.mjs pins code↔ADR agreement. (Ticked 2026-08-07 on merge.)
[L3-deferred — operator handoff needed; Residual-Owner: #13936] A deployed store crossing 80% produces a raise attempt with a recorded reason, observable without reading logs.
Out of Scope
Applying a live docker update raise. Verified feasible, but wiring the runtime call belongs with the actuator's ops layer; this ticket delivers the detection, vocabulary, and bounded intent.(Superseded 2026-08-07 by comment 2, recorded on pickup: reading "the actuator's ops layer" as reuse of reconfigure would have carried the restart in with it, and the restart is the harm. The live raise IS the store path's activation step and ships with the actuator half — the no-restart AC above is its guard.)
The chroma recreate that activates a new default on the running plane — operator-owned.
Any other service's ceiling, and revisiting local-model's 32g.
Chunk-identity/tenant-slug divergence, a separate deletion-class failure — not this ticket.
Adding a watchdog. The obvious reading of "we have no pre-cliff signal" is to build a monitor. The monitor exists, measures the right ratio, and applies a sustained window. Building a second one would have duplicated working code and left the actual defect — a store being told to shed a corpus — in place. The gap is vocabulary and routing, not observation.
Raising the number alone. A fixed cap on a store whose size grows with the repo re-creates the cliff further out. The ceiling has to be reachable by a controller, or the next corpus doubling repeats this incident.
Related
#16595 — the measurement that produced the sizing; this supersedes its config-only shape.
#16452 — activation kernel as the only mutation path; a durable ceiling change is a config mutation, so the knob-override path must compose with it rather than bypass it.
tobiu referenced in commit ea5b305 - "A store at its memory ceiling is diagnosed and told to raise it (#16596) (#16597) on Aug 7, 2026, 8:42 AM
tobiu referenced in commit 82e2629 - "feat: A store's ceiling is raisable — bounded knob, live update, no restart (#16637) (#16638) on Aug 7, 2026, 7:21 PM
tobiu referenced in commit 5f11053 - "The declared heap ceiling becomes observable, and the envelope guard names its resource (#16636) (#16697) on Aug 8, 2026, 4:21 PM
tobiu referenced in commit b6ec735 - "feat(ai): actuate bounded store ceiling raises (#16596) (#17011)" on Aug 12, 2026, 1:41 PM
Context
A container at its memory ceiling is already detected.
ContainerHealthDiagnosisServicecomputes saturation against the limit (:952-959,usage / limit * 100), applies a sustained-window filter rather than a single sample (:361), and emits amemory-saturationfact at 90% (:30).It then routes that fact — together with CPU
resourceSaturation— to one uniform prescription (:586-598):recoveryClass: 'exhaustion', actionClass : CONTAINER_HEALTH_ACTION_CLASSES.throttleShed, reason : 'resource-exhaustion'The Problem
throttle-shedis the wrong heal for a data store, and it is the only one on offer.The available action classes are
record,restart,throttleShed,warmProvider(:21-26). For a process holding transient work, shedding load is correct — pressure comes from arrival rate, so reducing arrivals relieves it. For a vector store, the corpus is the workload: memory is a near-deterministic function of rows already stored. There is nothing to shed, restarting frees nothing durable, and the pressure returns the moment the corpus is reloaded.Measured on the live plane 2026-08-06: chroma at 4096-dim float32 needs
rows x 16 KiB x ~1.38. At ~96,000 rows that is ~2.02 GiB against a 2.00 GiB cap. Because the store exits cleanly rather than being OOM-killed (OOMKilled=false ExitCode=0 RestartCount=13), each attempt to reach a complete corpus stops near completion and reads as a graceful restart. A restore of 59,754 chunks died at 24,000 this way.So the diagnosis layer can see the condition and has no vocabulary to express the only heal that works: raise the ceiling.
Second-order: because growth is monotonic and predictable, 90% is late for a store. By the time saturation is sustained at 90%, the remaining headroom is smaller than one ingestion batch.
The Architectural Reality
Three pieces exist and are not connected:
ContainerHealthDiagnosisService, per-service, sustained-window, measured against the limit.RecoveryActuatorServiceDEFAULT_ACTIONSincludesreconfigure(:23), dispatching toreconfigureComposeService({knob, knobValues, reason, target})(:582).escalateandpageare already deleted from that set, consistent with the autonomous-heal directive.RECOVERY_KNOBS(recoveryKnobRegistry.mjs:39). Its docblock states the property that makes it the right home: "A controller asks to widen the mini-summary window; it never names leaves." Bounds are enforced in the registry (:190-191), so a controller cannot request an unbounded change.What blocks the connection: chroma's limit is hardcoded
memory: 2g(docker-compose.yml:47), so no knob can reach it.local-modelalready demonstrates the controllable form in the same file —memory: "${NEO_LOCAL_MODEL_MEMORY_LIMIT:-32g}". The store is the one component whose ceiling is both load-bearing and unreachable.And a live raise is possible without recreation. Verified against a disposable target (ingress, 256m → 300m → restored):
docker update --memoryis accepted and takes effect on a running container. So the heal can be immediate and autonomous; the compose/env value is what makes it durable across the next recreate.The Fix
${NEO_CHROMA_MEMORY_LIMIT:-8g}, followinglocal-model's existing precedent. The default is derived, not preferred:rows x 4096 x 4B x 1.38puts a complete corpus at ~2.02 GiB, and 8g admits ~370k rows — about 4x — at ~17% of the docker VM.container-memory-ceilingknob toRECOVERY_KNOBS, expressing the intent "raise this service's memory ceiling" with registry-enforced min/max so a controller can never request an unbounded value.memory-saturationon a store to the ceiling-raise intent rather thanthrottleShed. Transient services keepthrottleShedunchanged.Deliberately not in this ticket: the vector dimension. Reducing 4096 would cut resident memory proportionally and is rejected on operator direction — below ~4k,
query_documentsretrieval quality degrades materially, and it would require re-embedding the whole corpus. Recorded so it is not re-proposed as a cheaper fix.Decision Record impact
amends ADR-0025(Orchestrator Container Health Self-Healing) andamends ADR-0026(Orchestrator Recovery Actuator,Proposed).ADR-0025's action-class taxonomy has no ceiling-raise class; ADR-0026's actuator matrix has no row for it. Both need the store-versus-transient distinction, because the same fact warrants opposite heals depending on whether the pressure comes from arrival rate or from stored data. ADR-0026 being pre-merge makes extending it cheaper than amending later; @neo-opus-grace authored both and holds the deepest context on the un-healable class, so she is the right reviewer.
Handover to @neo-fable-clio — 2026-08-07, and three ACs were stale
Reassigned at @tobiu's request. Before handing it over I verified every AC against the tree rather than trusting its checkbox, and three of eight had shipped with PR #16597 and were never ticked — corrected below with the delivering file and line. Left unticked they would have cost the next owner a cycle reimplementing a code path that already has a passing spec citing this ticket number.
Genuinely open: the healthy-state classification projection · a
container-memory-ceilingknob with registry-enforced bounds (searched: nothing exists, not even partial) · the bounded/anti-thrash raise · the ADR-0025/0026 rows · the post-merge observability check.Sequencing. Draft PR #16634 modifies the same
collectStatsFactsfunction in the same file — it adds an effective-heap-ceiling denominator for NON-STORE services. The projection AC moves classification fields out ofif (memoryWindow.sustained)in that same function. Different concerns, real textual conflict risk; branching after #16634 merges avoids it.Adjacent but deliberately separate: #16630 leaves service-class exhaustion unrouted to
raiseCeilingon purpose. This ticket's store-side routing is delivered. The two were split, not duplicated.Acceptance Criteria
DeploymentStateBridgeemitsserviceClass,serviceClassDeclared, the applied threshold, andobservedWindowMsbesiderequiredWindowMsfor every store-classified service on every snapshot, independent of load. Delivered by PR #16638 (merge 82e26297b5) —ContainerHealthDiagnosisService.describeClassification(verdict-free) attached per service incollectServiceSnapshot; the bridge-spec falsifier proves a healthy store at 10% memory carries the full block, withstampCoveragedistinguishing under-stamped from under-length windows. (Ticked 2026-08-07 on merge.) Moved here from #16603 on 2026-08-07 (@neo-gptcycle-4). Today those fields exist only insideif (memoryWindow.sustained), so a healthy store at the raised8gceiling exposes none of them — which makes any load-independent observability claim unverifiable and is why three successive post-merge formulations on #16603 were each unobservable. It is new behaviour, not a wording fix, so it belongs on this ticket beside the other undelivered half rather than blocking the leaf that delivered detection.memory: "${NEO_CHROMA_MEMORY_LIMIT:-8g}"with the reachability rationale and the resident-vector arithmetic recorded above it. (Ticked 2026-08-07 on handover; it had shipped and gone unchecked.)container-memory-ceilingknob exists with registry-enforced bounds; a request outside them is rejected with a violation rather than clamped silently. Delivered by PR #16638 (merge 82e26297b5) —recoveryKnobRegistry.mjscontainer-memory-ceiling: 8–16 GiB derived band,serviceKey: chromabinding,raise-not-lowerinvariant resolving from the RUNTIME;recoveryKnobRegistry.spec.mjsasserts the cap refusal names the band and never clamps. (Ticked 2026-08-07 on merge.)throttle-shed. Spec asserts the action class, not merely that a fact was emitted. Delivered by PR #16597 —ContainerHealthDiagnosisService.mjs:726-732returnsactionClass: raiseCeilingwithreason: 'store-ceiling-exhaustion', andContainerHealthDiagnosisService.spec.mjs:650assertsdecision.actionClassdirectly, exactly as this AC demands. (Ticked 2026-08-07 on handover.)throttle-shed— a negative control, so the change is proven narrow rather than global. Delivered by PR #16597 —ContainerHealthDiagnosisService.spec.mjs:700, whose comment reads "Negative control for the ROUTING." (Ticked 2026-08-07 on handover.)@neo-opus-vega's finding thatreconfigurecouples the mutation to a restart, and for a store mid-ingestion that restart is the harm itself.) Delivered by PR #16638 (merge 82e26297b5) —raiseComposeServiceCeilingomits the restart as its contract; mutation-verified through the single recorded lifecycle seam (re-adding the restart reddens the centerpiece spec). (Ticked 2026-08-07 on merge.)recoveryActuatorAdrCoherence.spec.mjspins code↔ADR agreement. (Ticked 2026-08-07 on merge.)Out of Scope
Applying a live(Superseded 2026-08-07 by comment 2, recorded on pickup: reading "the actuator's ops layer" as reuse ofdocker updateraise. Verified feasible, but wiring the runtime call belongs with the actuator's ops layer; this ticket delivers the detection, vocabulary, and bounded intent.reconfigurewould have carried the restart in with it, and the restart is the harm. The live raise IS the store path's activation step and ships with the actuator half — the no-restart AC above is its guard.)local-model's 32g.Avoided Traps
Adding a watchdog. The obvious reading of "we have no pre-cliff signal" is to build a monitor. The monitor exists, measures the right ratio, and applies a sustained window. Building a second one would have duplicated working code and left the actual defect — a store being told to shed a corpus — in place. The gap is vocabulary and routing, not observation.
Raising the number alone. A fixed cap on a store whose size grows with the repo re-creates the cliff further out. The ceiling has to be reachable by a controller, or the next corpus doubling repeats this incident.
Related
Origin Session ID: 6004a4aa-2089-4b14-b73f-b58c08cf53d9
Retrieval Hint:
query_raw_memories("store memory saturation throttle-shed wrong heal raise ceiling knob")·ContainerHealthDiagnosisService.mjs:586-598Authored by @neo-opus-vega (Claude Opus 5).