LearnNewsExamplesServices
Frontmatter
id16860
titleA slow plane cannot raise the deadlines that abort its own inference
stateClosed
labels
bugaiarchitecture
assigneesneo-opus-grace
createdAtAug 10, 2026, 10:27 AM
updatedAtAug 11, 2026, 2:01 AM
githubUrlhttps://github.com/neomjs/neo/issues/16860
authorneo-opus-grace
commentsCount4
parentIssue16706
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 11, 2026, 2:01 AM

A slow plane cannot raise the deadlines that abort its own inference

Closed Backlog/active-chunk-14 bugaiarchitecture
neo-opus-grace
neo-opus-grace commented on Aug 10, 2026, 10:27 AM

Context

#16850 fixed the NEO_OLLAMA_* family: five of six leaves never reached the containers, so a containerized plane could not name its own provider. The same defect exists one config block over, on the leaves that decide when Neo gives up on that provider — and there it is worse, because giving up early is now a measured way to break the provider rather than merely to fail.

Surfaced while inventorying every deadline that can abort in-flight work, after the operator observed that a slower plane needs larger timeouts. He is right, and the answer turned out to be that the most dangerous one cannot be changed at all.

The Problem

⚠️ SCOPE CORRECTED 2026-08-10 09:0xZ — it is seven leaves, not three, and @neo-gpt caught it on intake

I scoped this ticket to the three leaves I had personally traced to an abort. The providerReadiness block declares seven env leaves, every one orchestrator-consumed and every one absent from every profile. The body said three while AC-1 said "each readiness leaf" — internally inconsistent, and the narrower number is the wrong one.

I committed the exact error this ticket's Avoided Traps warns about, inside the ticket that names it: "a guard narrowed to the instance that motivated it will not catch the next instance." I wrote that about #16850's family-scoped guard and then scoped this one to my own three findings. Repairing a family one traced instance at a time is what produced a second ticket in the first place; doing it again would produce a third.

Adopting @neo-gpt's root-shape recommendation: pass all seven to orchestrator and derive the guard from both provider-readiness prefixes rather than an enumerated list, so the next leaf is caught mechanically. Defaults unchanged, no behaviour touched.

⚠️ CONSUMER MATRIX CORRECTED 09:2xZ — kb-server and mc-server are NOT non-consumers, and I said they were

Both the correction above and my message adopting it asserted that every readiness leaf is orchestrator-consumed and that KB/MC take non-consumer dispositions. That is false. Verified at origin/dev@70d45b before rewriting, rather than taking the report on trust:

  • ai/services/memory-core/TextEmbeddingService.mjs:721 reads aiConfig.orchestrator.providerReadiness.timeoutMs — and throws a TypeError at :728 if it is not a positive number. TES is imported by KB's QueryService (:2), VectorService and HealthService, and by MC's HealthService.
  • ai/services/memory-core/lifecycle/InferenceLifecycleService.mjs:64,66 reads timeoutMs and routineCacheTtlMs, and MC's Server.mjs:305-306 starts it as a declared dependency.

The required matrix, for both root profiles:

service leaves it consumes
kb-server NEO_ORCHESTRATOR_PROVIDER_READY_TIMEOUT_MS
mc-server NEO_ORCHESTRATOR_PROVIDER_READY_TIMEOUT_MS, NEO_ORCHESTRATOR_PROVIDER_READY_ROUTINE_CACHE_TTL_MS
orchestrator all seven

Shipping the non-consumer version would have left KB and MC unable to raise a timeout they actually read and hard-fail on — strictly worse than the defect, because it would have looked deliberate.

This is the third narrowing error on this one ticket (three leaves → seven; orchestrator-only → three services), and the pattern is mine: I keep asserting a consumer set from the call path I personally traced instead of from the full read set. Earlier today the opposite instrument — an import-graph scan — over-approximated the same question. Neither the traced path nor the module graph is the read set; only the reads are.

Seven leaves govern provider readiness. None of them appears in any deployment profile. Verified at origin/dev across all six variants in ai/deploy/ (docker-compose.yml, .dev, .local-agent-os, .parity-capture, .parity-ci, .test):

leaf default occurrences across all six profiles
NEO_ORCHESTRATOR_PROVIDER_READY_TIMEOUT_MS 3000 0
NEO_ORCHESTRATOR_PROVIDER_READY_ATTEMPTS 30 0
NEO_ORCHESTRATOR_PROVIDER_READY_DELAY_MS 1000 0
NEO_ORCHESTRATOR_PROVIDER_READY_ROUTINE_CACHE_TTL_MS 1000 0
NEO_ORCHESTRATOR_STUCK_RUNNER_ENABLED true 0
NEO_ORCHESTRATOR_STUCK_RUNNER_CONSECUTIVE_FAILURES 3 0
NEO_ORCHESTRATOR_STUCK_RUNNER_CANARY_TIMEOUT_MS 10000 0

STUCK_RUNNER_ENABLED deserves its own line: an operator on a plane where the canary is the hazard cannot turn it off. The one escape hatch that would let a deployment opt out of an intervention it did not ask for is itself unreachable.

The first one is not a passive deadline. It arms an in-flight abort on real inference.

// ai/services/graph/providerReadinessHelper.mjs:826 — warmOllamaRoleModel
const response = await fetchFn(new URL(endpoint, host).toString(), {
    …,
    signal : AbortSignal.timeout(timeoutMs)
});

endpoint is role === 'embedding' ? '/api/embed' : '/api/chat' (:807). DreamService.mjs:1169 calls ensureOllamaModelsReady with timeoutMs: readinessConfig.timeoutMs, resolved from AiConfig.orchestrator.providerReadiness3 seconds.

Three seconds is not marginal on the hardware this is aimed at. A measured CPU-only in-container embed took 10.8 s; a cold chat warm-up 22.7 s. And it fires on the warm path, which by construction runs precisely when a model is cold or evicted — the slowest moment it will ever have.

Why an unreachable deadline is worse here than in #16850. There, an unset leaf meant the provider could not be named: a connection refused, loud and quick to diagnose. Here, #16853 measured that an early in-flight abort leaves the runner at ~400 % CPU with zero established sockets, persisting after every client is stopped. So the deadline does not merely fail the request — it can strand the provider for everyone, and attempts: 30 with delayMs: 1000 means a single readiness pass can issue up to thirty of them.

The scope boundary that makes this its own ticket rather than more of #16853. #16853 repairs what the code does at the deadline — the correct fix, and the one that removes the hazard. This ticket is about who is allowed to choose the deadline. They are independent: a plane on a build that predates #16853, or any future deadline added to this block, still needs an operator escape hatch. Reachability is the mitigation that survives the repair slipping.

The Architectural Reality

  • ai/configBase.mjs:1121-1124providerReadiness: attempts: leaf(30, …), delayMs: leaf(1000, …), timeoutMs: leaf(3000, …).
  • ai/configBase.mjs:1139stuckRunner.canaryTimeoutMs: leaf(10000, …).
  • ai/services/graph/providerReadinessHelper.mjs:826 — the in-flight abort on real inference; :807 selects /api/embed vs /api/chat.
  • ai/daemons/orchestrator/services/DreamService.mjs:1169 — the container-reachable caller; gated on roles.length > 0, not on orchestrator.ollama.enabled and not on supervised-child ownership.
  • ai/deploy/docker-compose.yml — carries the full NEO_OLLAMA_* set as of #16850, and none of the readiness leaves.
  • test/playwright/unit/ai/deploy/OllamaProviderEnvCoordinates.spec.mjs — the per-(service, leaf) coordinate guard from #16850. Its REQUIRED table is ollama-family-scoped, so it does not look at these leaves at all — the guard that would have caught this was deliberately narrowed to the family being fixed.

The Fix

  1. Pass all seven readiness leaves to orchestrator, operator-overridable, exactly as #16850 established: ${VAR:-} interpolation of the leaf's own variable, orchestrator-scoped since that is where the consumers live.
  2. Derive the guard from the provider-readiness PREFIXES, not an enumerated list — both NEO_ORCHESTRATOR_PROVIDER_READY_* and NEO_ORCHESTRATOR_STUCK_RUNNER_*. An enumerated table is what let this ticket ship at three-sevenths of its own family; a prefix-derived one reddens when the eighth leaf is declared.
  3. Classify the new keys in ai/scripts/lint/config-leaf-parity.json and move the census, so an unclassified addition fails loud rather than passing silently.

Deliberately not in scope: changing any default. 3000 may well be wrong for CPU-only hardware, but that is a judgement about values and this ticket is about whether anyone downstream is permitted to make it.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
orchestrator env docker-compose.yml and docker-compose.dev.yml all seven readiness leaves pass through as ${VAR:-} unset ⇒ leaf default, identical to today inline rationale beside the block per-(service, leaf) coordinate assertion, mutation-convicted
kb-server env both root profiles PROVIDER_READY_TIMEOUT_MS passes through unset ⇒ leaf default inline rationale TextEmbeddingService:721 reads it and :728 throws on a bad value
mc-server env both root profiles PROVIDER_READY_TIMEOUT_MS + PROVIDER_READY_ROUTINE_CACHE_TTL_MS pass through unset ⇒ leaf default inline rationale InferenceLifecycleService:64,66, started by Server.mjs:305-306
non-consumed (service, leaf) pairs deliberately absent, with a written disposition each n/a a coordinate with no reader must not ship, or the profile advertises a knob that does nothing
config-leaf-parity.json the lint census new keys classified; census moved unclassified ⇒ lint fails ai:lint-config-template-ssot green
the coordinate guard OllamaProviderEnvCoordinates.spec.mjs covers deadline leaves that arm an abort spec header a mutation removing one from the profile reddens by name

Decision Record impact: aligned-with ADR 0019 — the leaves already own env-override-with-default. This is the deployment layer failing to deliver the env, again.

Acceptance Criteria

  • Each readiness leaf reaches every service that consumes it, in operator-overridable form, asserted per (service, leaf) coordinate rather than as a name set. The matrix is kb={TIMEOUT_MS}, mc={TIMEOUT_MS, ROUTINE_CACHE_TTL_MS}, orchestrator={all 7}, in both root profiles — derived from the reads, not from the service that owns the config namespace.
  • A mutation removing one from the profile reddens the guard by name; a mutation hardcoding a value reddens it as not-overridable. Both directions, in-suite, against a cloned compose document.
  • Defaults unchanged — a deployment setting none of them behaves exactly as today.
  • The coordinate guard no longer scopes itself to the NEO_OLLAMA_* family alone, so the next deadline leaf that arms an abort cannot ship unreachable. This is the AC that prevents the third instance; the first two were #16850 and this ticket.
  • Post-merge / plane-level: on a CPU-only plane, raising NEO_ORCHESTRATOR_PROVIDER_READY_TIMEOUT_MS above the measured warm duration is observed to take effect. [L3-deferred — needs a running plane]

Out of Scope

  • Changing the deadline values. Values are a judgement; this is about reachability.
  • #16853 — what the code does at the deadline. That repair removes the hazard; this one removes the operator's inability to work around it, and neither substitutes for the other.
  • #16856 — whether a probe or warm declares itself an intervention.
  • The model server's own knobs (OLLAMA_NUM_PARALLEL, CPU limits). Deployment-side configuration, not Neo leaves.

Avoided Traps

  • Treating this as #16853 leftovers. They repair different things and either can land first. A build predating #16853 still needs the escape hatch.
  • Assuming the #16850 guard covers it. It does not, and for a defensible reason — its REQUIRED table was scoped to the family under repair. A guard narrowed to the instance that motivated it will not catch the next instance, which is exactly what happened here, one config block away and within hours.
  • Filing it as a values ticket. "3 s is too short" invites an argument about the right number. "No deployment can change it" does not.
  • Claiming this explains any specific plane's burning cores. The code path is verified; incidence on any given deployment is not, and that deployment's orchestrator would have to be running for this path to fire at all.

Related

#16850 — the same defect in the NEO_OLLAMA_* family, closed; this is its sibling one block over · #16853 — the repair for what happens at the deadline · #16856 — probe/warm intervention classification · #16830 — consumes the canary · #16706 — deployment-stability epic. Governing decision: ADR-0019.

Live latest-open sweep: latest 8 open issues plus a scoped title sweep for providerReadiness/readiness/deadline/timeout at 2026-08-10T08:2xZ; nearest are #16853 (behaviour at the deadline) and #16849 (a request with no deadline at all). No equivalent found. A2A sweep over the last 14 messages: no in-flight claim — offered to @neo-gpt-emmy as an either-or on #16853 and filing rather than blocking on the answer.

Structure map: N/A — modifies ai/deploy/docker-compose.yml, ai/scripts/lint/config-leaf-parity.json and an existing spec in place.

Origin Session ID: 3c27118d-2de2-4579-bb42-1062c34cb895

Retrieval Hint: query_raw_memories("providerReadiness timeout leaves never reach compose AbortSignal.timeout warm path unreachable deadline")

tobiu referenced in commit 73eb465 - "fix(deploy): the provider-readiness deadlines reach no service, so no plane can raise them (#16860) (#16916) on Aug 11, 2026, 2:01 AM
tobiu closed this issue on Aug 11, 2026, 2:01 AM