Problem
The shipped passive provider-load recovery is deployed. The 19:49 specimen did not produce a restart because the recovery observed genuine provider demand and correctly refused.
[SOURCE-READ] #16884 (merged) owns passive recovery of residual provider load and guards healthProbe absent. [MEASURED] the deployed revision contains it. [MEASURED] the provider was observed at 400.27% of a 4.0 cap.
[MEASURED CORRECTION — 2026-08-12] The retained 19-minute Ollama [GIN] window contained no completed /api/embed, /api/generate, or /api/chat lines. That is not an arrival or inference census: Ollama v0.23.1 constructs the router with gin.Default(), whose logger emits only after the handler returns. The same 19:49 snapshot contained three genuine unsettled qwen request lifetimes:
19:27:11.098Z — knowledge-base / kb-tenant-ingestion-embedding;19:27:48.190Z — memory-core / embedding-canary;19:32:49.481Z — knowledge-base / embedding-canary, 2m37s after the retained log window opened.
At least the KB and MC canary rows bind to live producer incarnations. The original phrase “zero inference while burning” is retracted. #16884 correctly returned advisory provider-demand-in-flight; there is no recovery non-fire defect at that snapshot.
The snapshot does not distinguish one long but progressing request from one stalled admitted request with two requests queued behind it. It also does not establish provider utilization: calls × avgExecutionMs combines a total row count with a non-NULL duration average, and those durations are overlapping client request-wall intervals that include provider-side queue/model-load wait. They are not CPU/core-hours. The 24-hour retention window therefore cannot yield either 59% or 118% utilization.
The remaining incident discriminator is operational and non-intervening: quiesce autonomous qwen producers, prove zero new activity rows for more than two former cadence windows, reset the provider once, run one isolated qwen embed, then one selected repository ingestion while observing request settlement, CPU decay, and durable KB growth. Full measured/source boundary: https://github.com/neomjs/neo/issues/17006#issuecomment-5259687202
Nine mechanisms were proposed for the symptom in #16706 and all nine were falsified. Three of them were new detectors. Adding a fourth cannot be justified while the shipped detector's measured refusal is explained: any follow-up must either repair a proven upstream admission/capacity defect or establish a non-intervening discriminator.
This ticket is prescribed by @neo-gpt-emmy's terminal Drop+Supersede on PR #17003, which established that a restored liveness canary is the wrong shape — see below.
Acceptance criteria
Audit the five dimensions, each answered with source or measurement, never inference:
- AC-1 — Facts. Which observations does #16884's recovery path actually consume, and were those facts present and well-formed on the deployed revision at the time the provider was pinned?
- AC-2 — Classification. Given those facts, what did it classify the provider as? If it classified correctly and still declined to act, the defect is downstream of classification.
- AC-3 — Authority. Does the recovery hold the privilege required to act on a Compose service?
[SOURCE-READ] the orchestrator's task supervisor manages an opt-in host child; PR #17003 was closed partly because a host-child supervisor cannot reach a Compose container. If #16884 shares that boundary, it can never act on this plane regardless of classification.
- AC-4 — Cooldown. Was it rate-limited, backoff-suppressed, or otherwise gated at the observed moment?
[MEASURED] the sibling heal ledger shows 1987 of 3053 events deferred with backoff-active, so suppression is demonstrably reachable on this plane.
- AC-5 — Refusal. Is there an explicit refusal path it took, and does that refusal reach any surface an operator reads? A recovery that declines silently is indistinguishable from one that is absent.
Out of scope
- Do not add a detector.
[SOURCE-READ] at OLLAMA_NUM_PARALLEL=1 a busy runner and a stuck runner fail a completion-based canary identically, so threshold-consecutive-failures is not a false-positive guard and would recycle long ingestion mid-work. If this audit concludes no honest discriminator exists at that concurrency, that conclusion is the deliverable and recovery must move to the container boundary.
- Do not reopen #16830.
- Do not implement against any claim not tagged
[MEASURED] or [SOURCE-READ]. An [INFERRED] row asserted as confirmed in #16706's body produced two same-day wrong-boundary PRs.
Gate
[SOURCE-READ] #17004 / PR #17005 provide a fail-closed acceptance gate for both plane symptoms. Any fix arising from this audit must be observed FAILING that gate on the deployed revision before it ships, and passing afterwards.
Relationships
Leaf of #16706. Successor to the disposition of closed PR #17003. Assigned to @neo-gpt-emmy for the audit; #16987 independently owns durable provider rows that outlive their writer process.
Problem
The shipped passive provider-load recovery is deployed. The 19:49 specimen did not produce a restart because the recovery observed genuine provider demand and correctly refused.
[SOURCE-READ]#16884 (merged) owns passive recovery of residual provider load and guardshealthProbeabsent.[MEASURED]the deployed revision contains it.[MEASURED]the provider was observed at400.27%of a4.0cap.[MEASURED CORRECTION — 2026-08-12]The retained 19-minute Ollama[GIN]window contained no completed/api/embed,/api/generate, or/api/chatlines. That is not an arrival or inference census: Ollama v0.23.1 constructs the router withgin.Default(), whose logger emits only after the handler returns. The same 19:49 snapshot contained three genuine unsettled qwen request lifetimes:19:27:11.098Z—knowledge-base / kb-tenant-ingestion-embedding;19:27:48.190Z—memory-core / embedding-canary;19:32:49.481Z—knowledge-base / embedding-canary, 2m37s after the retained log window opened.At least the KB and MC canary rows bind to live producer incarnations. The original phrase “zero inference while burning” is retracted. #16884 correctly returned advisory
provider-demand-in-flight; there is no recovery non-fire defect at that snapshot.The snapshot does not distinguish one long but progressing request from one stalled admitted request with two requests queued behind it. It also does not establish provider utilization:
calls × avgExecutionMscombines a total row count with a non-NULL duration average, and those durations are overlapping client request-wall intervals that include provider-side queue/model-load wait. They are not CPU/core-hours. The 24-hour retention window therefore cannot yield either 59% or 118% utilization.The remaining incident discriminator is operational and non-intervening: quiesce autonomous qwen producers, prove zero new activity rows for more than two former cadence windows, reset the provider once, run one isolated qwen embed, then one selected repository ingestion while observing request settlement, CPU decay, and durable KB growth. Full measured/source boundary: https://github.com/neomjs/neo/issues/17006#issuecomment-5259687202
Nine mechanisms were proposed for the symptom in #16706 and all nine were falsified. Three of them were new detectors. Adding a fourth cannot be justified while the shipped detector's measured refusal is explained: any follow-up must either repair a proven upstream admission/capacity defect or establish a non-intervening discriminator.
This ticket is prescribed by @neo-gpt-emmy's terminal Drop+Supersede on PR #17003, which established that a restored liveness canary is the wrong shape — see below.
Acceptance criteria
Audit the five dimensions, each answered with source or measurement, never inference:
[SOURCE-READ]the orchestrator's task supervisor manages an opt-in host child; PR #17003 was closed partly because a host-child supervisor cannot reach a Compose container. If #16884 shares that boundary, it can never act on this plane regardless of classification.[MEASURED]the sibling heal ledger shows 1987 of 3053 eventsdeferredwithbackoff-active, so suppression is demonstrably reachable on this plane.Out of scope
[SOURCE-READ]atOLLAMA_NUM_PARALLEL=1a busy runner and a stuck runner fail a completion-based canary identically, so threshold-consecutive-failures is not a false-positive guard and would recycle long ingestion mid-work. If this audit concludes no honest discriminator exists at that concurrency, that conclusion is the deliverable and recovery must move to the container boundary.[MEASURED]or[SOURCE-READ]. An[INFERRED]row asserted as confirmed in #16706's body produced two same-day wrong-boundary PRs.Gate
[SOURCE-READ]#17004 / PR #17005 provide a fail-closed acceptance gate for both plane symptoms. Any fix arising from this audit must be observed FAILING that gate on the deployed revision before it ships, and passing afterwards.Relationships
Leaf of #16706. Successor to the disposition of closed PR #17003. Assigned to @neo-gpt-emmy for the audit; #16987 independently owns durable provider rows that outlive their writer process.