Orchestrator lms lane restart-loops — needs HTTP-probe liveness

Context

With orchestrator.lms.enabled=true (operator-enabled 2026-05-31, after the unified-Chroma migration), the orchestrator's lms server (LM Studio CLI) lane spins in a restart-loop. Operator flagged it ("this does not look right") with the spec: "like for chroma, there has to be a check. running ⇒ all good, no logs. not running ⇒ start."

Empirically reproduced in the operator's orchestrator log (PID 1666, four cycles 16:56:03–16:56:50): each cycle logs Starting lms server (supervisor-restart) → stderr Success! Server is now running on port 1234 → completed successfully, then re-spawns ~15s later.

The Problem

#11986 (CLOSED 2026-05-25) added the lms lane to the orchestrator, mirroring the mlx task. Critically, #11986's body specified the correct idempotency primitive:

"Idempotency primitive: the existing InferenceLifecycleService.isInferenceRunning() HTTP probe to /v1/models is the canonical collision-defense check. New orchestrator task should reuse the same probe shape rather than re-implementing."

The shipped implementation did not wire that HTTP probe into the supervision restart-decision. Instead the lms lane is supervised by process-liveness (expectedCommand: 'lms server' match, like chroma/mlx). But lms server start is fire-and-exit: it wakes LM Studio's background service (~5s) and exits 0. LM Studio's actual long-lived server process is not named lms server, so the supervisor's expectedCommand match never finds the lane "running" → it re-spawns lms server start every poll/backoff cycle (~15s).

Non-fatal (LM Studio stays up on :1234; embedding works; re-runs are idempotent) but it produces continuous log noise (incl. the success banner on stderr logged at ERROR) and re-fires the postSpawn model-readiness check every cycle.

Contrast: chroma doesn't loop because the chroma daemon stays foreground (its process matches expectedCommand: 'chroma'); mlx's mlx_lm.server likewise stays foreground. The lms launcher is the lone fire-and-exit case — the exact distinction #11986's mirror-of-mlx missed.

The Architectural Reality

• ai/daemons/orchestrator/TaskDefinitions.mjs (lms task, ~158): command: 'lms', args: ['server','start','--port',lmsPort], expectedCommand: 'lms server', pidFileName: 'lms.pid', postSpawn: ensureLmsModelsLoaded(...).
• ai/daemons/orchestrator/services/ProcessSupervisorService.mjs: the restart-decision uses command.includes(task.expectedCommand) process-match for liveness (~143, ~452) + restart-when-not-running. The LM-Studio nuance is partially noted (~257: "the HTTP server can listen while no chat/embedding model is resident" → hence the postSpawn readiness hook) — but the launcher's fire-and-exit shape is not handled for liveness.
• ai/services/memory-core/lifecycle/InferenceLifecycleService.mjs: isInferenceRunning() — the HTTP probe to /v1/models that #11986 cited as canonical.
• ai/services/graph/ProviderReadinessHelper.mjs: ensureLmsModelsLoaded already HTTP-probes the OpenAI-compatible endpoint (host:port) — reusable readiness primitive.
• Liveness target: AiConfig.openAiCompatible.host + AiConfig.orchestrator.lms.port (:1234), GET /v1/models.

The Fix

Gate the lms lane's supervision restart-decision on an HTTP liveness probe of the OpenAI-compatible endpoint (host:port GET /v1/models, via isInferenceRunning() / ProviderReadinessHelper), not the expectedCommand process-match — i.e., wire in the idempotency primitive #11986 already specified.

• Server up (HTTP probe OK) → no-op, NO logs (quiet steady-state — operator's explicit requirement).
• Server down (probe fails) → lms server start + ensureLmsModelsLoaded (start + pre-warm).
• The exiting launcher's PID/expectedCommand is not the liveness signal for this lane; the HTTP endpoint is.

Acceptance Criteria

• The lms lane's liveness/restart-decision uses an HTTP probe of the configured OpenAI-compatible host:port (GET /v1/models), not the lms server process-match.
• Server up → orchestrator does NOT re-run lms server start and emits NO per-cycle logs (quiet steady-state; the ~15s loop is gone in a live orchestrator run with LM Studio up).
• Server down → orchestrator runs lms server start + ensureLmsModelsLoaded to (re)start + pre-warm the embedding model.
• postSpawn model-readiness fires only on an actual (re)start, not every cycle.
• mlx lane audited for the same fire-and-exit class — fixed if it shares it, or documented-as-unaffected (mlx_lm.server foreground) inline.
• Reuses the existing HTTP-probe primitive (isInferenceRunning / ProviderReadinessHelper) rather than re-implementing.

Out of Scope

• The embedding provider/model choice (openAiCompatible + qwen3-embedding-8b stays).
• The chroma/mlx lanes except for the mlx same-class audit above.
• orchestrator.lms.enabled config semantics (already correct; this is the supervision shape).

Avoided Traps

• Reopening #11986 — it's CLOSED/shipped (it added the lane); this is a distinct implementation defect, filed fresh citing it.
• Re-introducing a retired autoStartInference per-MCP-server flag (see ai/scripts/diagnostics/check-retired-primitives.mjs RETIRED_CONFIG_FLAGS) — the orchestrator owns inference lifecycle via this lane; the fix is the lane's liveness model, not a new flag.
• Disabling the lane (loses orchestrator-managed LM Studio) — rejected; the operator wants the orchestrator to own it.

Decision Record impact

none — orchestrator supervision bug-fix; no ADR change.

Related

• #11986 (origin of the lms lane; CLOSED — this fixes the supervision-liveness gap in its implementation)
• #11093 / PR #11096 (centralized daemon supervision — the regression lineage)
• #9833 (original LM Studio auto-boot + the isInferenceRunning /v1/models probe)
• #12072 (Provider-readiness substrate / ProviderReadinessHelper)

Origin Session ID: 48e040d3-c6b0-471c-b4f5-95e991f6e3b6

Retrieval Hint: "orchestrator lms lane restart-loop fire-and-exit HTTP-probe liveness isInferenceRunning #11986"