Early Ollama abort can strand a four-core embedding runner

Context

On 2026-08-10, a controlled CPU-only Docker reproduction against Neo's real native-Ollama embedding path separated normal timeout from early disconnect using the same admitted request:

The forced arm used TextEmbeddingService.embedTexts() with one 65,536-byte ASCII input (21,846 estimated tokens against Neo's 28,672-token safe band), qwen3-embedding:latest, and a four-core allocation. After every Neo client was stopped and established Ollama sockets were zero, the runner still sampled 397.41% -> 399.48% -> 400.20% CPU for more than 60 seconds. Restarting only the model cleared the burn. Full public receipt: #16830 comment 5235152342.

This is a controlled positive witness with a natural-timeout control, an early-abort arm, socket isolation, client isolation, and restoration receipts. It does not generalize the behavior to every model, OS, endpoint, or abort timing.

The Problem

Neo currently proves client-local cancellation, then can treat the resulting abort as if no provider work remains. That conclusion is not valid for native Ollama.

The measured defect is broader than one embedding consumer. Three production paths dispatch native-Ollama inference and can then terminate the client side on a Neo-owned deadline:

1. Ollama.embed() receives a caller signal from the native branch of TextEmbeddingService.
2. probeOllamaServing() starts a real /api/chat inference and aborts it at the stuck-runner canary deadline.
3. warmOllamaRoleModel() starts /api/embed or /api/chat and gives the request an AbortSignal.timeout(timeoutMs); Dream reaches this readiness path on container planes.

The embedding reproduction proves that an early client disconnect can leave server work behind. It does not prove that every chat or warm abort wedges, but it falsifies their shared safety premise: aborting dispatched native-Ollama inference is not a read-only observation and cannot be represented as completed server cancellation without evidence.

The readiness attempts setting retries /api/ps; it does not issue 30 warm calls in one pass. Each missing role is warmed once per readiness call, while later Dream/readiness cycles can re-enter the same aborting path.

Architectural Reality

Verified against current origin/dev before this body revision:

• ai/provider/Ollama.mjs passes a caller signal into the native request and destroys the client request on abort/timeout; local promise settlement does not prove provider settlement.
• ai/services/memory-core/TextEmbeddingService.mjs threads caller cancellation and the provider deadline into Ollama.embed().
• ai/services/graph/ollamaStuckRunnerLiveness.mjs dispatches /api/chat, aborts at its deadline, and converts the failure to false.
• ai/services/graph/providerReadinessHelper.mjs dispatches /api/embed or /api/chat from warmOllamaRoleModel() with AbortSignal.timeout(timeoutMs).
• ai/services/graph/DreamService.mjs reaches native-Ollama readiness on container planes.
• ai/services/shared/providerActivityLedger.mjs already owns bounded provider-work lifecycle evidence, including in-flight work.
• AiConfig owns all deadlines at the reactive use site under ADR 0019. A repair must not re-read env, alias, pass config through layers, mutate AiConfig, or introduce shadow defaults.

The Fix

Make every Neo-owned native-Ollama inference boundary outcome-honest:

1. Separate cancellation stages at the production composition. TextEmbeddingService still refuses an already-aborted caller before dispatch. Once native embedding inference is dispatched, it stops forwarding that caller signal into Ollama.embed() and ends only the caller wait. The direct provider API remains available to independent callers whose contract genuinely is transport cancellation.
2. Keep post-dispatch embedding work accounted. The provider-activity operation remains observed and handled until the provider response or existing provider timeout determines its disposition. Return the caller's exact local abort reason without mutating caller-owned errors.
3. Disarm intervention-shaped probes. probeOllamaServing() must not manufacture another inference and abort it to classify the runner. A control-plane or residency read may be retained only under its truthful name; /api/ps is not proof that inference serves.
4. Contain readiness warmups. warmOllamaRoleModel() must obey the same post-dispatch rule for both embedding and chat roles. A short local deadline returns an explicit pending disposition while the signal-free provider request remains handled and coalesced by exact host/role/model/context/keep-alive shape.
5. Preserve authority boundaries. This ticket corrects provider/work evidence. Any recycle uses the already-admitted runtime actuator, threshold, and cooldown owned by #16830; no new action class is introduced.
6. Keep provider-specific semantics. OpenAI-compatible and Gemini behavior remains unchanged absent an independent witness.

Contract Ledger

Decision Record impact

None. This changes no recovery action class or privilege boundary. It corrects provider/probe evidence semantics beneath ADR 0026 and leaves #16830 as the owner of container-plane detection and actuation.

Acceptance Criteria

• An already-aborted signal opens no native Ollama request. — PR #16869 production-bound pre-abort control; 133/133 focused controls green at f5c1afe4.
• After dispatch, caller abort can settle the caller promptly with the exact caller-owned reason, without mutating that error and without claiming provider settlement. — PR #16869 scalar + batch deferred-provider controls preserve the frozen caller error exactly.
• The provider operation remains bounded, handled, and visible as in-flight/uncertain until its response, existing provider timeout, or admitted recovery settles the record; no late unhandled rejection occurs. — PR #16869 late-settlement and provider-activity controls.
• [L4-deferred — operator handoff needed] The exact CPU-only natural-timeout control returns the runner to idle.
• [L4-deferred — operator handoff needed] The exact CPU-only early-abort control can no longer leave an unaccounted approximately four-core runner with zero clients and zero established sockets. It either settles in the bounded provider window or hands one explicit residual to #16830's admitted recovery contract.
• probeOllamaServing() no longer dispatches and early-aborts inference as a supposedly harmless read. /api/ps, if retained, is named only as control-plane/residency evidence and cannot clear a stuck-runner suspicion by itself. — PR #16869 retires the canary/classifier and mechanically removes the task healthProbe.
• warmOllamaRoleModel() obeys the same post-dispatch contract for both /api/embed and /api/chat; its short deadline cannot create silently abandoned provider work. — PR #16869 role-paired no-signal, coalescing, explicit-pending controls.
• Unit controls are production-bound and mutation-sensitive: restoring direct caller-signal transport abort, settling the provider ledger at caller abort, restoring the aborting canary, or restoring aborting warmup behavior makes a named test fail. — PR #16869 exact suite 133/133; independent re-falsification approved all three final repair gates.
• Provider-error-before-caller-abort remains the provider error; later signal state cannot relabel an earlier provider failure. — PR #16869 provider-first causality control.
• [L4-deferred — operator handoff needed] The operator-gated witness records model, input bytes/token estimate, caller settlement, provider settlement, socket count, runner CPU, and restoration. CI does not pretend to supply external-process evidence.
• The reusable local reproduction branch codex/16830-cpu-ollama-repro remains available until the live witness is re-run against the candidate repair. — retained locally at 55219f40d8d1766ba44bd5b40708a3e0c8f2fec4.
• JSDoc and timeout/friction prose distinguish client-local settlement from provider settlement. — PR #16869 source comments and bounded caller-aborted/provider-pending log phase.
• AiConfig remains the reactive SSOT under ADR 0019. — no config pass-along/env re-read; 17/17 config authority controls and AiConfig lints green.

Out of Scope

• Container-plane detector ownership, role coverage, runtime-access wiring, and cooldown behavior; #16830 owns those surfaces and is blocked by this ticket.
• Deployment reach for readiness/canary deadline leaves (#16860).
• The observation-versus-intervention declaration guard (#16856).
• Ollama provider env passthrough (#16850 / PR #16851).
• The unbounded Ollama.stream() path (#16849).
• Proving the origin of any external deployment's historical CPU burn.
• Patching Ollama upstream or generalizing beyond the measured native-Ollama behavior.

Avoided Traps

• Calling AbortError completed cancellation. It proves only that the client stopped waiting.
• Waiting for provider settlement before returning caller cancellation. That destroys the caller's bounded cancellation contract; caller settlement and provider accounting are separate.
• Mutating the caller's Error with disposition fields. It may be frozen and is caller-owned.
• Fixing this with a larger timeout. Moving the cutoff does not establish settlement.
• Using another aborting inference as a detector. The instrument can manufacture or amplify the state.
• Calling /api/ps a serving probe. Residency/control-plane liveness is not inference liveness.
• Restarting on one slow request. Sustained evidence and cooldown remain the recovery boundary.

Related

• Measured container detection and recovery successor: #16830
• Observation/intervention contract: #16856
• Timeout deployment reach: #16860
• Provider lineage: #15694 / PR #15742
• Canary lineage: #13882 / PR #13900
• Historical continuous-burn investigation: #13928

Origin Session ID: 878f05af-2c4e-4da2-a5c2-9e4af666fcb8

Retrieval Hint: query_raw_memories("native Ollama early disconnect residual CPU work")

Commit anchors: cfa64163f0e15b26f307254f00d25840cc6b433c and f1a8013b01488e05445888470ad21a439704b230.