Containerized Ollama runner lacks safe stuck detection and bounded recovery

Sub of #16706. Blocked by #16853.

Context

Neo has a real stuck-runner detector for a supervised Ollama child, but canonical container planes run Ollama as a Compose service. The supervisor neither spawns nor owns that service, so its child-task healthProbe is structurally unreachable there.

A controlled CPU-only reproduction on 2026-08-10 then falsified the detector's original instrument. With the same admitted native embedding request:

Restarting only the model cleared the burn. Receipt: comment 5235152342. This proves an early disconnect can leave native-Ollama embedding work behind; it does not prove the origin of any historical deployment incident.

The Problem

There are two compositional defects:

1. Topology gap. ConfiguredTaskDefinitionsService attaches stuck-runner health only to tasks.ollama, a process-supervisor child. A Compose-owned model has no child PID/task, so no detector or admitted recovery is reached.
2. Instrument gap. The existing detector dispatches real /api/chat inference and aborts it at a short deadline. The controlled embedding result proves that client-local abort is not inherently safe provider cancellation. A detector must not be able to manufacture the work it claims to observe.

The old body prescribed “probe every role over the network.” That is withdrawn. /api/ps can establish control-plane responsiveness and residency, but it cannot prove that inference serves; another aborting inference canary is unsafe until #16853 closes the post-dispatch work contract.

Architectural Reality

Verified against current origin/dev before this body revision:

• ai/services/graph/ollamaStuckRunnerLiveness.mjs owns a pure sustained-failure classifier plus an aborting /api/chat canary.
• ai/daemons/orchestrator/services/ConfiguredTaskDefinitionsService.mjs calls that canary only from the supervised tasks.ollama child.
• ai/services/graph/providerReadinessHelper.mjs can read /api/ps; this is residency/control-plane evidence, not serving evidence.
• ai/services/shared/providerActivityLedger.mjs owns bounded in-flight/recent provider-work evidence.
• Container/process CPU and lifecycle observation belongs at the runtime-access boundary; Compose restart is already in ADR 0026's closed recovery-action set.
• #16853 owns post-dispatch outcome honesty for Ollama.embed(), the old canary, and readiness warmups. #16856 owns the general observation-versus-intervention declaration rule. #16860 owns deployment reach for timeout leaves, not cancellation semantics.

The Fix

Give containerized Ollama a non-intervening, outcome-honest diagnosis and route only sustained high-confidence cases to the existing restart actuator:

1. Sequence behind #16853. No detector may classify a caller abort as provider settlement or create a fresh aborting inference. Provider work that survived caller settlement remains accounted until bounded settlement/recovery.
2. Establish one canonical observation owner. Compose/runtime access supplies service/runner CPU and lifecycle identity; provider-activity projections supply in-flight/recent Neo work; /api/ps may contribute control-plane/residency identity. Preserve source and timestamp for every fact.
3. Classify by conjunction over time, not one probe. A candidate is actionable only when high runner CPU is sustained, the expected model/runner remains resident, and there is no live accounted Neo provider demand capable of explaining the burn. Missing, stale, contradictory, or unsupported evidence yields unknown/advisory, not restart.
4. Keep role truth. Attribute the resident model to configured chat/embedding roles without issuing work to each role. An embedding-only runner must not be invisible merely because chat is healthy.
5. Act through existing authority. The diagnosis carries evidence into the ADR-0026 runtime-access restart path. Preserve the sustained threshold, restart cooldown, authority checks, and append-only provenance. No new action class or direct Docker privilege is introduced.
6. Retire or rebind the child canary. The supervised-child path must consume the same safe evidence contract; it must not retain the short aborting inference as an alternate detector.

Contract Ledger

Decision Record Impact

Aligned with ADR 0026. Restarting a Compose service is already admitted. This ticket adds no action or privilege class; it repairs evidence reach and composition. The diagnosis remains non-privileged evidence, and the runtime-access holder owns the final authority/cooldown check.

Acceptance Criteria

• #16853's post-dispatch native-Ollama contract is delivered first; no path in this ticket treats local abort as proof of provider settlement.
• Container-plane detection dispatches zero inference requests. Mutation-restoring /api/chat or /api/embed as a detector makes a named test fail.
• /api/ps, if used, is named and tested only as control-plane/residency evidence; green residency alone neither proves serving nor clears a stuck suspicion.
• A CPU-only replay of the measured residual—sustained approximately four-core runner, resident model, no established client socket, and no live accounted provider demand after the bounded provider window—reaches the sustained diagnosis on a Compose plane.
• A legitimately active long-running request with matching in-flight provider activity is not restarted, regardless of high CPU. This is the load-bearing negative control.
• A cold-start/model-load interval, an idle resident model, stale/missing telemetry, contradictory role/model identity, and one high-CPU sample are each non-actionable.
• Every configured model role is attributable from source-owned configuration plus observed model identity; an embedding runner is detectable while chat remains healthy without issuing role canaries.
• Restart goes through the existing ADR-0026 runtime-access actuator against the exact Compose service identity. No direct Docker call or new privilege path is introduced.
• Sustained threshold and restart cooldown remain effective; repeated observations inside one cooldown produce at most one restart.
• Lost/contended runtime authority at the last owned boundary produces no restart and no owner-authoritative success record.
• The supervised-child path no longer uses the aborting inference canary and remains compatible with the safe diagnosis contract.
• The diagnosis/provenance record includes observation source, subject identity, model/role, timestamps/age, CPU evidence, provider-work evidence, threshold state, actuator disposition, and recovery outcome without payloads or secrets.
• Mutation controls fail when container reach is removed, provider activity is ignored, stale evidence is treated as fresh, identity is misbound, threshold is bypassed, or cooldown/authority is bypassed.
• The retained local branch codex/16830-cpu-ollama-repro is used for the operator-gated candidate replay and kept until natural, residual-detection, and model-recovery receipts are filed.

Out of Scope

• Preventing/containing post-dispatch native-Ollama aborts; #16853 owns the root repair.
• General observation-versus-intervention substrate; #16856 owns that guard.
• Deployment reach for readiness/canary timeout leaves; #16860 owns that surface.
• Why Ollama can continue computing after client disconnect, or an upstream Ollama patch.
• OLLAMA_NUM_PARALLEL tuning.
• Proving the cause of any historical external-plane burn or empty corpus.
• Treating /api/ps as a serving test.

Avoided Traps

• Running another inference canary. It is an intervention and may create/amplify the wedge.
• Calling zero sockets zero work. The measured residual had no established client socket; combine it with provider accounting and sustained runtime evidence.
• Calling /api/ps serving evidence. A stuck runner can remain resident and control-plane responsive.
• Treating missing telemetry as idle. Missing/stale evidence is unknown and cannot authorize restart.
• Restarting on high CPU alone. Healthy inference and model load legitimately consume cores.
• Restarting on a single sample. Sustained multi-fact evidence plus cooldown is the safety case.
• Letting the diagnosis own Docker. Evidence and action authority stay separate under ADR 0026.
• Preserving the supervised canary as a fallback. Two topologies must not retain two contradictory safety contracts.

Related

• Parent: #16706
• Root post-dispatch repair and blocker: #16853
• Observation/intervention declaration: #16856
• Timeout deployment reach: #16860
• Provider activity observability: #16780
• Residency/eviction distinction: #14154
• Original child canary: #13882 / PR #13900

Origin Session ID: d8332b13-5d97-4839-ac11-d2de4602a989

Reproduction Session ID: 878f05af-2c4e-4da2-a5c2-9e4af666fcb8

Retrieval Hint: query_raw_memories("native Ollama early disconnect residual CPU work")

Retrieval Hint: query_raw_memories("stuck ollama runner undetectable containerized plane healthProbe supervisor child only")