LearnNewsExamplesServices
Frontmatter
id13882
title[recovery-daemon] Detect + restart a stuck (resident-but-not-serving) Ollama runner — privilege-free
stateClosed
labels
enhancementai
assigneesneo-opus-grace
createdAtJun 22, 2026, 11:58 PM
updatedAtJun 23, 2026, 8:49 AM
githubUrlhttps://github.com/neomjs/neo/issues/13882
authorneo-opus-grace
commentsCount1
parentIssue13874
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[ ] 13884 [recovery-daemon] Gated privileged docker/deploy actuator — B1 + Rung-3, [ ] 13883 [recovery-daemon] Rung-2 via B0 — supervised-process restart (privilege-free)
closedAtJun 23, 2026, 8:49 AM

[recovery-daemon] Detect + restart a stuck (resident-but-not-serving) Ollama runner — privilege-free

neo-opus-grace
neo-opus-grace commented on Jun 22, 2026, 11:58 PM

Parent: Epic #13874. Graduated from Discussion #13871 as the fire-relief stuck-runner recovery slice.

Problem / scope

An Ollama runner can be resident yet not serving: /api/tags still answers and residency probes pass, while one pathological request pegs the CPU and queues all later work behind OLLAMA_NUM_PARALLEL=1. The local supervisor already owns the ollama serve child, so this ticket is the privilege-free stuck-runner recovery slice: detect sustained non-serving behavior on the running child and recycle that supervised child after false-positive guards are satisfied.

Out of scope for this leaf: the broader recovery-daemon policy table, Rung-0 shed-load, page terminal, persisted daemon-level anti-thrash, and full recovery observability from parent #13874. Those remain parent/other-leaf territory.

Acceptance Criteria

  • Add a stuck-runner detector that distinguishes sustained non-serving canary failures from a single advisory failure.
  • Treat any completed canary HTTP response, including a fast non-2xx model/config error, as serving; only timeout/abort/no-response counts toward stuck-runner recovery.
  • Add orchestrator.providerReadiness.stuckRunner config leaves for enablement, consecutive failures, and canary timeout.
  • Attach the stuck-runner detector as a running-child healthProbe for the configured Ollama task while preserving the plain residency/down path as livenessProbe.
  • Update ProcessSupervisorService so a running task with sustained-unhealthy healthProbe can be killed and respawned through the existing supervisor/cooldown path.
  • Cover the classifier, canary false-positive boundary, running-child recycle branch, and config snapshot in focused right-hemisphere unit tests.
  • Post-merge live validation confirms a stuck resident-but-not-serving Ollama runner recycles unattended without recycling a legitimately long request or a fast model/config error.

Contract Ledger

  • Adds: stuck-runner healthProbe detection and supervisor recycle for the already-supervised Ollama child.
  • Reuses: existing process supervision, cooldown/anti-thrash behavior, configured task definitions, and AiConfig provider-readiness leaves.
  • Does not add: a general recovery daemon, policy-table dispatch, Rung-0 shed-load, page terminal, or new privileged actuator.

Sequencing

Fire-relief priority for the live resident-but-not-serving Ollama saturation symptom. This leaf can merge independently of the broader recovery-daemon shape as long as parent #13874 continues to own the remaining recovery topology.

tobiu referenced in commit cfa6416 - "fix(ai): detect + restart a stuck (resident-but-not-serving) Ollama runner (#13900) on Jun 23, 2026, 8:49 AM
tobiu closed this issue on Jun 23, 2026, 8:49 AM