Problem
A provider lane whose container sits AT its cgroup memory limit enters page-fault thrash: the kernel evicts the mmap'd model and KV-cache pages, and the engine burns its full CPU quota re-faulting them from disk instead of computing. Observed live on a constrained plane (2026-08-14): container memory 48.0G of a 48.0G cap, engine RES +11G over one hour of sustained ingest until the ceiling, host swap 0.4%→52.8%, sustained multi-MB/s disk reads attributed to the engine process, a task in flight ~26 minutes whose clean cost is ~2 minutes — while docker health, the liveness probe, the recovery probe, and the deployment-state snapshot all read healthy/available. The failure mode is invisible precisely because every health surface asks "is it alive?" and none asks "is it at its memory ceiling?"
The deployment-state bridge already samples stats.memoryPercent per service — the signal exists and is discarded at the judgment layer.
Acceptance Criteria
- The deployment-state bridge's service observation derives a typed memory-pressure disposition (e.g.
at-cap when sustained memoryPercent ≥ a leaf-owned threshold across N consecutive samples), carried on the snapshot the diagnostic surfaces serve.
- A sustained at-cap disposition on a provider lane degrades the composed health surface with a receipt naming the service, the cap, and the sample window — same consumed-fold pattern as the starvation receipt;
unknown never degrades.
- Thresholds/windows are ADR-0019 leaves, projected in the deployment templates per the behavior-binding-clock projection contract.
- Spec matrix: below-threshold never degrades; sustained at-cap degrades with receipt; a single spike does not; stats-unavailable maps to
unknown.
Evidence class
Live plane observation (operator host telemetry + deployment-state snapshot + engine task log), 2026-08-14. Sibling pattern: the heavy-maintenance starvation receipt (#17049) — an invisible-condition-to-consumed-verdict conversion on the same bridge.
Part of epic #17072.
Contract Ledger
Added post-review (PR #17135 Round-1, @neo-gpt-emmy): this ticket consumes and widens six surfaces and carried no ledger. Every row's target surface was read at a04c8d738a before assertion.
| Target Surface |
Source of Authority |
Behavior |
Fallback |
Docs |
Evidence |
orchestrator.memorySaturation.percent |
ADR-0019 leaf, NEO_MEMORY_SATURATION_PERCENT |
Sustained % against the authoritative denominator (heap for Node services, container otherwise) before a transient lane is saturated. Domain 0 < n <= 100. |
Service default 90 when the leaf is absent |
configBase.mjs JSDoc + compose projection |
memoryPressureDisposition.spec.mjs at-cap case via the real producer |
orchestrator.memorySaturation.storePercent |
ADR-0019 leaf, NEO_STORE_MEMORY_SATURATION_PERCENT |
Store threshold, deliberately lower: stores cross by growing monotonically, so 90 is late — headroom at 90 is under one ingestion batch. |
Service default 80 |
same |
ContainerHealthDiagnosisService.spec.mjs store-class classification |
orchestrator.memorySaturation.windowMs |
ADR-0019 leaf, NEO_MEMORY_SATURATION_WINDOW_MS |
Memory's own sustained-window floor. Explicitly NOT the diagnosis service's shared sampleWindowMs, which clocks CPU saturation, the container cold-start gate and three provider-activity freshness bounds. Bounded by retention: must be <= (statsSampleWindow - 1) × writeIntervalMs. |
Service default 30000 |
configBase.mjs + compose pair-change note |
describeMemoryWindowReachability matrix + the clock-independence falsifier |
describeClassification() projection |
ContainerHealthDiagnosisService |
Widened by memoryScope, memoryObservedWindowMs, memoryStampCoverage, and now takes nodeCommand. requiredWindowMs reports the memory window, matching the appliedMemoryThreshold beside it. |
memoryScope: 'unavailable' when identity is unknown — never a container fallback |
JSDoc on the method |
two exact-shape toEqual assertions updated in the producer and bridge specs |
memoryPressure on the service record |
DeploymentStateBridgeService |
Additive, published on every snapshot: {disposition, reason, receipt}. at-cap folds the record status to degraded; below/unknown never do. unknown carries one of five named reasons. |
Consumers absent this field see no change — additive only |
module docblock |
memoryPressureDisposition.spec.mjs full tri-state matrix |
composeMemoryCoreHealthcheck consumer |
foldServiceMemoryPressure in HealthService.mjs |
A fresh at-cap record from an available snapshot degrades composed health and withdraws the all-clear line. Stale record, stale/degraded/unavailable snapshot, malformed record: no authority. unhealthy wins. |
try/catch → state: 'fold-error', base verdict preserved |
function JSDoc |
HealthService.memoryPressureFold.spec.mjs + the wiring case in McpServerToolLimits.spec.mjs |
Explicit admission exclusion. This fold sits at the composed MCP/Docker healthcheck surface and deliberately not in HealthService.ensureHealthy(). That payload gates semantic-tool admission; a lane at its memory ceiling must not withdraw capabilities it does not affect. Semantic recall stays dispatchable while the composed surface reports degraded — the same boundary the starvation sibling holds, for the same reason.
Fail-honestly validation. An unspannable window pair raises at orchestrator start rather than shipping a detector that cannot fire. The message carries both numbers and the two env vars that produce them.
Decision Record impact
aligned-with ADR 0019 (all three leaves resolved at the entrypoint and injected; no non-entrypoint AiConfig import, no re-derivation, no runtime mutation) and aligned-with ADR 0025 (the disposition reports a verdict and never an action; recovery authority stays with the actuator that owns it).
Problem
A provider lane whose container sits AT its cgroup memory limit enters page-fault thrash: the kernel evicts the mmap'd model and KV-cache pages, and the engine burns its full CPU quota re-faulting them from disk instead of computing. Observed live on a constrained plane (2026-08-14): container memory 48.0G of a 48.0G cap, engine RES +11G over one hour of sustained ingest until the ceiling, host swap 0.4%→52.8%, sustained multi-MB/s disk reads attributed to the engine process, a task in flight ~26 minutes whose clean cost is ~2 minutes — while
dockerhealth, the liveness probe, the recovery probe, and the deployment-state snapshot all read healthy/available. The failure mode is invisible precisely because every health surface asks "is it alive?" and none asks "is it at its memory ceiling?"The deployment-state bridge already samples
stats.memoryPercentper service — the signal exists and is discarded at the judgment layer.Acceptance Criteria
at-capwhen sustained memoryPercent ≥ a leaf-owned threshold across N consecutive samples), carried on the snapshot the diagnostic surfaces serve.unknownnever degrades.unknown.Evidence class
Live plane observation (operator host telemetry + deployment-state snapshot + engine task log), 2026-08-14. Sibling pattern: the heavy-maintenance starvation receipt (#17049) — an invisible-condition-to-consumed-verdict conversion on the same bridge.
Part of epic #17072.
Contract Ledger
Added post-review (PR #17135 Round-1, @neo-gpt-emmy): this ticket consumes and widens six surfaces and carried no ledger. Every row's target surface was read at
a04c8d738abefore assertion.orchestrator.memorySaturation.percentNEO_MEMORY_SATURATION_PERCENT0 < n <= 100.90when the leaf is absentconfigBase.mjsJSDoc + compose projectionmemoryPressureDisposition.spec.mjsat-cap case via the real producerorchestrator.memorySaturation.storePercentNEO_STORE_MEMORY_SATURATION_PERCENT80ContainerHealthDiagnosisService.spec.mjsstore-class classificationorchestrator.memorySaturation.windowMsNEO_MEMORY_SATURATION_WINDOW_MSsampleWindowMs, which clocks CPU saturation, the container cold-start gate and three provider-activity freshness bounds. Bounded by retention: must be<= (statsSampleWindow - 1) × writeIntervalMs.30000configBase.mjs+ compose pair-change notedescribeMemoryWindowReachabilitymatrix + the clock-independence falsifierdescribeClassification()projectionContainerHealthDiagnosisServicememoryScope,memoryObservedWindowMs,memoryStampCoverage, and now takesnodeCommand.requiredWindowMsreports the memory window, matching theappliedMemoryThresholdbeside it.memoryScope: 'unavailable'when identity is unknown — never a container fallbacktoEqualassertions updated in the producer and bridge specsmemoryPressureon the service recordDeploymentStateBridgeService{disposition, reason, receipt}.at-capfolds the record status todegraded;below/unknownnever do.unknowncarries one of five named reasons.memoryPressureDisposition.spec.mjsfull tri-state matrixcomposeMemoryCoreHealthcheckconsumerfoldServiceMemoryPressureinHealthService.mjsat-caprecord from anavailablesnapshot degrades composed health and withdraws the all-clear line. Stale record, stale/degraded/unavailable snapshot, malformed record: no authority.unhealthywins.state: 'fold-error', base verdict preservedHealthService.memoryPressureFold.spec.mjs+ the wiring case inMcpServerToolLimits.spec.mjsExplicit admission exclusion. This fold sits at the composed MCP/Docker healthcheck surface and deliberately not in
HealthService.ensureHealthy(). That payload gates semantic-tool admission; a lane at its memory ceiling must not withdraw capabilities it does not affect. Semantic recall stays dispatchable while the composed surface reports degraded — the same boundary the starvation sibling holds, for the same reason.Fail-honestly validation. An unspannable window pair raises at orchestrator start rather than shipping a detector that cannot fire. The message carries both numbers and the two env vars that produce them.
Decision Record impact
aligned-with ADR 0019(all three leaves resolved at the entrypoint and injected; no non-entrypointAiConfigimport, no re-derivation, no runtime mutation) andaligned-with ADR 0025(the disposition reports a verdict and never an action; recovery authority stays with the actuator that owns it).