#16850 fixed the NEO_OLLAMA_* family: five of six leaves never reached the containers, so a containerized plane could not name its own provider. The same defect exists one config block over, on the leaves that decide when Neo gives up on that provider — and there it is worse, because giving up early is now a measured way to break the provider rather than merely to fail.
Surfaced while inventorying every deadline that can abort in-flight work, after the operator observed that a slower plane needs larger timeouts. He is right, and the answer turned out to be that the most dangerous one cannot be changed at all.
The Problem
⚠️ SCOPE CORRECTED 2026-08-10 09:0xZ — it is seven leaves, not three, and @neo-gpt caught it on intake
I scoped this ticket to the three leaves I had personally traced to an abort. The providerReadiness block declares seven env leaves, every one orchestrator-consumed and every one absent from every profile. The body said three while AC-1 said "each readiness leaf" — internally inconsistent, and the narrower number is the wrong one.
I committed the exact error this ticket's Avoided Traps warns about, inside the ticket that names it:"a guard narrowed to the instance that motivated it will not catch the next instance." I wrote that about #16850's family-scoped guard and then scoped this one to my own three findings. Repairing a family one traced instance at a time is what produced a second ticket in the first place; doing it again would produce a third.
Adopting @neo-gpt's root-shape recommendation: pass all seven to orchestrator and derive the guard from both provider-readiness prefixes rather than an enumerated list, so the next leaf is caught mechanically. Defaults unchanged, no behaviour touched.
⚠️ CONSUMER MATRIX CORRECTED 09:2xZ — kb-server and mc-server are NOT non-consumers, and I said they were
Both the correction above and my message adopting it asserted that every readiness leaf is orchestrator-consumed and that KB/MC take non-consumer dispositions. That is false. Verified at origin/dev@70d45b before rewriting, rather than taking the report on trust:
ai/services/memory-core/TextEmbeddingService.mjs:721 reads aiConfig.orchestrator.providerReadiness.timeoutMs — and throws a TypeError at :728 if it is not a positive number. TES is imported by KB's QueryService (:2), VectorService and HealthService, and by MC's HealthService.
ai/services/memory-core/lifecycle/InferenceLifecycleService.mjs:64,66 reads timeoutMsandroutineCacheTtlMs, and MC's Server.mjs:305-306 starts it as a declared dependency.
Shipping the non-consumer version would have left KB and MC unable to raise a timeout they actually read and hard-fail on — strictly worse than the defect, because it would have looked deliberate.
This is the third narrowing error on this one ticket (three leaves → seven; orchestrator-only → three services), and the pattern is mine: I keep asserting a consumer set from the call path I personally traced instead of from the full read set. Earlier today the opposite instrument — an import-graph scan — over-approximated the same question. Neither the traced path nor the module graph is the read set; only the reads are.
Seven leaves govern provider readiness. None of them appears in any deployment profile. Verified at origin/dev across all six variants in ai/deploy/ (docker-compose.yml, .dev, .local-agent-os, .parity-capture, .parity-ci, .test):
STUCK_RUNNER_ENABLED deserves its own line: an operator on a plane where the canary is the hazard cannot turn it off. The one escape hatch that would let a deployment opt out of an intervention it did not ask for is itself unreachable.
The first one is not a passive deadline. It arms an in-flight abort on real inference.
endpoint is role === 'embedding' ? '/api/embed' : '/api/chat' (:807). DreamService.mjs:1169 calls ensureOllamaModelsReady with timeoutMs: readinessConfig.timeoutMs, resolved from AiConfig.orchestrator.providerReadiness — 3 seconds.
Three seconds is not marginal on the hardware this is aimed at. A measured CPU-only in-container embed took 10.8 s; a cold chat warm-up 22.7 s. And it fires on the warm path, which by construction runs precisely when a model is cold or evicted — the slowest moment it will ever have.
Why an unreachable deadline is worse here than in #16850. There, an unset leaf meant the provider could not be named: a connection refused, loud and quick to diagnose. Here, #16853 measured that an early in-flight abort leaves the runner at ~400 % CPU with zero established sockets, persisting after every client is stopped. So the deadline does not merely fail the request — it can strand the provider for everyone, and attempts: 30 with delayMs: 1000 means a single readiness pass can issue up to thirty of them.
The scope boundary that makes this its own ticket rather than more of #16853.#16853 repairs what the code does at the deadline — the correct fix, and the one that removes the hazard. This ticket is about who is allowed to choose the deadline. They are independent: a plane on a build that predates #16853, or any future deadline added to this block, still needs an operator escape hatch. Reachability is the mitigation that survives the repair slipping.
ai/services/graph/providerReadinessHelper.mjs:826 — the in-flight abort on real inference; :807 selects /api/embed vs /api/chat.
ai/daemons/orchestrator/services/DreamService.mjs:1169 — the container-reachable caller; gated on roles.length > 0, not on orchestrator.ollama.enabled and not on supervised-child ownership.
ai/deploy/docker-compose.yml — carries the full NEO_OLLAMA_* set as of #16850, and none of the readiness leaves.
test/playwright/unit/ai/deploy/OllamaProviderEnvCoordinates.spec.mjs — the per-(service, leaf) coordinate guard from #16850. Its REQUIRED table is ollama-family-scoped, so it does not look at these leaves at all — the guard that would have caught this was deliberately narrowed to the family being fixed.
The Fix
Pass all seven readiness leaves to orchestrator, operator-overridable, exactly as #16850 established: ${VAR:-} interpolation of the leaf's own variable, orchestrator-scoped since that is where the consumers live.
Derive the guard from the provider-readiness PREFIXES, not an enumerated list — both NEO_ORCHESTRATOR_PROVIDER_READY_* and NEO_ORCHESTRATOR_STUCK_RUNNER_*. An enumerated table is what let this ticket ship at three-sevenths of its own family; a prefix-derived one reddens when the eighth leaf is declared.
Classify the new keys in ai/scripts/lint/config-leaf-parity.json and move the census, so an unclassified addition fails loud rather than passing silently.
Deliberately not in scope: changing any default. 3000 may well be wrong for CPU-only hardware, but that is a judgement about values and this ticket is about whether anyone downstream is permitted to make it.
Contract Ledger Matrix
Target Surface
Source of Authority
Proposed Behavior
Fallback
Docs
Evidence
orchestrator env
docker-compose.ymlanddocker-compose.dev.yml
all seven readiness leaves pass through as ${VAR:-}
TextEmbeddingService:721 reads it and :728 throws on a bad value
mc-server env
both root profiles
PROVIDER_READY_TIMEOUT_MS + PROVIDER_READY_ROUTINE_CACHE_TTL_MS pass through
unset ⇒ leaf default
inline rationale
InferenceLifecycleService:64,66, started by Server.mjs:305-306
non-consumed (service, leaf) pairs
—
deliberately absent, with a written disposition each
n/a
—
a coordinate with no reader must not ship, or the profile advertises a knob that does nothing
config-leaf-parity.json
the lint census
new keys classified; census moved
unclassified ⇒ lint fails
—
ai:lint-config-template-ssot green
the coordinate guard
OllamaProviderEnvCoordinates.spec.mjs
covers deadline leaves that arm an abort
—
spec header
a mutation removing one from the profile reddens by name
Decision Record impact:aligned-with ADR 0019 — the leaves already own env-override-with-default. This is the deployment layer failing to deliver the env, again.
Acceptance Criteria
Each readiness leaf reaches every service that consumes it, in operator-overridable form, asserted per (service, leaf) coordinate rather than as a name set. The matrix is kb={TIMEOUT_MS}, mc={TIMEOUT_MS, ROUTINE_CACHE_TTL_MS}, orchestrator={all 7}, in both root profiles — derived from the reads, not from the service that owns the config namespace.
A mutation removing one from the profile reddens the guard by name; a mutation hardcoding a value reddens it as not-overridable. Both directions, in-suite, against a cloned compose document.
Defaults unchanged — a deployment setting none of them behaves exactly as today.
The coordinate guard no longer scopes itself to the NEO_OLLAMA_* family alone, so the next deadline leaf that arms an abort cannot ship unreachable. This is the AC that prevents the third instance; the first two were #16850 and this ticket.
Post-merge / plane-level: on a CPU-only plane, raising NEO_ORCHESTRATOR_PROVIDER_READY_TIMEOUT_MS above the measured warm duration is observed to take effect. [L3-deferred — needs a running plane]
Out of Scope
Changing the deadline values. Values are a judgement; this is about reachability.
#16853 — what the code does at the deadline. That repair removes the hazard; this one removes the operator's inability to work around it, and neither substitutes for the other.
#16856 — whether a probe or warm declares itself an intervention.
The model server's own knobs (OLLAMA_NUM_PARALLEL, CPU limits). Deployment-side configuration, not Neo leaves.
Avoided Traps
Treating this as #16853 leftovers. They repair different things and either can land first. A build predating #16853 still needs the escape hatch.
Assuming the #16850 guard covers it. It does not, and for a defensible reason — its REQUIRED table was scoped to the family under repair. A guard narrowed to the instance that motivated it will not catch the next instance, which is exactly what happened here, one config block away and within hours.
Filing it as a values ticket. "3 s is too short" invites an argument about the right number. "No deployment can change it" does not.
Claiming this explains any specific plane's burning cores. The code path is verified; incidence on any given deployment is not, and that deployment's orchestrator would have to be running for this path to fire at all.
Related
#16850 — the same defect in the NEO_OLLAMA_* family, closed; this is its sibling one block over · #16853 — the repair for what happens at the deadline · #16856 — probe/warm intervention classification · #16830 — consumes the canary · #16706 — deployment-stability epic. Governing decision: ADR-0019.
Live latest-open sweep: latest 8 open issues plus a scoped title sweep for providerReadiness/readiness/deadline/timeout at 2026-08-10T08:2xZ; nearest are #16853 (behaviour at the deadline) and #16849 (a request with no deadline at all). No equivalent found. A2A sweep over the last 14 messages: no in-flight claim — offered to @neo-gpt-emmy as an either-or on #16853 and filing rather than blocking on the answer.
Structure map: N/A — modifies ai/deploy/docker-compose.yml, ai/scripts/lint/config-leaf-parity.json and an existing spec in place.
tobiu referenced in commit 73eb465 - "fix(deploy): the provider-readiness deadlines reach no service, so no plane can raise them (#16860) (#16916) on Aug 11, 2026, 2:01 AM
Context
#16850fixed theNEO_OLLAMA_*family: five of six leaves never reached the containers, so a containerized plane could not name its own provider. The same defect exists one config block over, on the leaves that decide when Neo gives up on that provider — and there it is worse, because giving up early is now a measured way to break the provider rather than merely to fail.Surfaced while inventorying every deadline that can abort in-flight work, after the operator observed that a slower plane needs larger timeouts. He is right, and the answer turned out to be that the most dangerous one cannot be changed at all.
The Problem
Seven leaves govern provider readiness. None of them appears in any deployment profile. Verified at
origin/devacross all six variants inai/deploy/(docker-compose.yml,.dev,.local-agent-os,.parity-capture,.parity-ci,.test):NEO_ORCHESTRATOR_PROVIDER_READY_TIMEOUT_MSNEO_ORCHESTRATOR_PROVIDER_READY_ATTEMPTSNEO_ORCHESTRATOR_PROVIDER_READY_DELAY_MSNEO_ORCHESTRATOR_PROVIDER_READY_ROUTINE_CACHE_TTL_MSNEO_ORCHESTRATOR_STUCK_RUNNER_ENABLEDNEO_ORCHESTRATOR_STUCK_RUNNER_CONSECUTIVE_FAILURESNEO_ORCHESTRATOR_STUCK_RUNNER_CANARY_TIMEOUT_MSSTUCK_RUNNER_ENABLEDdeserves its own line: an operator on a plane where the canary is the hazard cannot turn it off. The one escape hatch that would let a deployment opt out of an intervention it did not ask for is itself unreachable.The first one is not a passive deadline. It arms an in-flight abort on real inference.
// ai/services/graph/providerReadinessHelper.mjs:826 — warmOllamaRoleModel const response = await fetchFn(new URL(endpoint, host).toString(), { …, signal : AbortSignal.timeout(timeoutMs) });endpointisrole === 'embedding' ? '/api/embed' : '/api/chat'(:807).DreamService.mjs:1169callsensureOllamaModelsReadywithtimeoutMs: readinessConfig.timeoutMs, resolved fromAiConfig.orchestrator.providerReadiness— 3 seconds.Three seconds is not marginal on the hardware this is aimed at. A measured CPU-only in-container embed took 10.8 s; a cold chat warm-up 22.7 s. And it fires on the warm path, which by construction runs precisely when a model is cold or evicted — the slowest moment it will ever have.
Why an unreachable deadline is worse here than in
#16850. There, an unset leaf meant the provider could not be named: a connection refused, loud and quick to diagnose. Here,#16853measured that an early in-flight abort leaves the runner at ~400 % CPU with zero established sockets, persisting after every client is stopped. So the deadline does not merely fail the request — it can strand the provider for everyone, andattempts: 30withdelayMs: 1000means a single readiness pass can issue up to thirty of them.The scope boundary that makes this its own ticket rather than more of
#16853.#16853repairs what the code does at the deadline — the correct fix, and the one that removes the hazard. This ticket is about who is allowed to choose the deadline. They are independent: a plane on a build that predates#16853, or any future deadline added to this block, still needs an operator escape hatch. Reachability is the mitigation that survives the repair slipping.The Architectural Reality
ai/configBase.mjs:1121-1124—providerReadiness:attempts: leaf(30, …),delayMs: leaf(1000, …),timeoutMs: leaf(3000, …).ai/configBase.mjs:1139—stuckRunner.canaryTimeoutMs: leaf(10000, …).ai/services/graph/providerReadinessHelper.mjs:826— the in-flight abort on real inference;:807selects/api/embedvs/api/chat.ai/daemons/orchestrator/services/DreamService.mjs:1169— the container-reachable caller; gated onroles.length > 0, not onorchestrator.ollama.enabledand not on supervised-child ownership.ai/deploy/docker-compose.yml— carries the fullNEO_OLLAMA_*set as of#16850, and none of the readiness leaves.test/playwright/unit/ai/deploy/OllamaProviderEnvCoordinates.spec.mjs— the per-(service, leaf)coordinate guard from#16850. ItsREQUIREDtable is ollama-family-scoped, so it does not look at these leaves at all — the guard that would have caught this was deliberately narrowed to the family being fixed.The Fix
orchestrator, operator-overridable, exactly as#16850established:${VAR:-}interpolation of the leaf's own variable, orchestrator-scoped since that is where the consumers live.NEO_ORCHESTRATOR_PROVIDER_READY_*andNEO_ORCHESTRATOR_STUCK_RUNNER_*. An enumerated table is what let this ticket ship at three-sevenths of its own family; a prefix-derived one reddens when the eighth leaf is declared.ai/scripts/lint/config-leaf-parity.jsonand move the census, so an unclassified addition fails loud rather than passing silently.Deliberately not in scope: changing any default.
3000may well be wrong for CPU-only hardware, but that is a judgement about values and this ticket is about whether anyone downstream is permitted to make it.Contract Ledger Matrix
orchestratorenvdocker-compose.ymlanddocker-compose.dev.yml${VAR:-}(service, leaf)coordinate assertion, mutation-convictedkb-serverenvPROVIDER_READY_TIMEOUT_MSpasses throughTextEmbeddingService:721reads it and:728throws on a bad valuemc-serverenvPROVIDER_READY_TIMEOUT_MS+PROVIDER_READY_ROUTINE_CACHE_TTL_MSpass throughInferenceLifecycleService:64,66, started byServer.mjs:305-306(service, leaf)pairsconfig-leaf-parity.jsonai:lint-config-template-ssotgreenOllamaProviderEnvCoordinates.spec.mjsDecision Record impact:
aligned-with ADR 0019— the leaves already own env-override-with-default. This is the deployment layer failing to deliver the env, again.Acceptance Criteria
(service, leaf)coordinate rather than as a name set. The matrix iskb={TIMEOUT_MS},mc={TIMEOUT_MS, ROUTINE_CACHE_TTL_MS},orchestrator={all 7}, in both root profiles — derived from the reads, not from the service that owns the config namespace.NEO_OLLAMA_*family alone, so the next deadline leaf that arms an abort cannot ship unreachable. This is the AC that prevents the third instance; the first two were#16850and this ticket.NEO_ORCHESTRATOR_PROVIDER_READY_TIMEOUT_MSabove the measured warm duration is observed to take effect.[L3-deferred — needs a running plane]Out of Scope
#16853— what the code does at the deadline. That repair removes the hazard; this one removes the operator's inability to work around it, and neither substitutes for the other.#16856— whether a probe or warm declares itself an intervention.OLLAMA_NUM_PARALLEL, CPU limits). Deployment-side configuration, not Neo leaves.Avoided Traps
#16853leftovers. They repair different things and either can land first. A build predating#16853still needs the escape hatch.#16850guard covers it. It does not, and for a defensible reason — itsREQUIREDtable was scoped to the family under repair. A guard narrowed to the instance that motivated it will not catch the next instance, which is exactly what happened here, one config block away and within hours.Related
#16850— the same defect in theNEO_OLLAMA_*family, closed; this is its sibling one block over ·#16853— the repair for what happens at the deadline ·#16856— probe/warm intervention classification ·#16830— consumes the canary ·#16706— deployment-stability epic. Governing decision: ADR-0019.Live latest-open sweep: latest 8 open issues plus a scoped title sweep for
providerReadiness/readiness/deadline/timeoutat 2026-08-10T08:2xZ; nearest are#16853(behaviour at the deadline) and#16849(a request with no deadline at all). No equivalent found. A2A sweep over the last 14 messages: no in-flight claim — offered to @neo-gpt-emmy as an either-or on#16853and filing rather than blocking on the answer.Structure map: N/A — modifies
ai/deploy/docker-compose.yml,ai/scripts/lint/config-leaf-parity.jsonand an existing spec in place.Origin Session ID: 3c27118d-2de2-4579-bb42-1062c34cb895
Retrieval Hint:
query_raw_memories("providerReadiness timeout leaves never reach compose AbortSignal.timeout warm path unreachable deadline")