Restore lms server lifecycle supervision in orchestrator (regression #11093)

Context

Operator surfaced 2026-05-25 during the cloud-deployment-trial sprint that the embedding endpoint at 127.0.0.1:1234 (LM Studio CLI lms server start) was supposed to be started/restarted as an orchestrator task, "and we have VERY similar logic in there" — referencing the existing mlx task in TaskDefinitions.mjs.

Live forensic confirmed:

• Last successful kbSync: 2026-05-25T12:14:27.076Z (per orchestrator-state.json)
• First ECONNREFUSED 127.0.0.1:1234: 2026-05-25T13:00:50.735Z (orchestrator.log:45546)
• After the embedding endpoint died, no orchestrator-side recovery occurred. Operator had to restart lms server start manually.

Origin Session ID: 8f1a91ee-3ee4-4e4b-9865-b5810f6be353

The Problem

#9833 (closed 2026-04-09, commit 0a3812c4c) originally shipped LM Studio auto-boot via ai/services/memory-core/lifecycle/InferenceLifecycleService.mjs::startInferenceServer(). That implementation natively invoked lms server start and used isInferenceRunning() (HTTP probe to /v1/models) for collision defense.

#11093 / PR #11096 ("Centralize daemon supervision", merged commit 7e596385e) refactored daemon supervision into the orchestrator. The MC-side spawn logic was deleted but the orchestrator-side counterpart was never landed. The remaining InferenceLifecycleService.mjs is gutted:

// ai/services/memory-core/lifecycle/InferenceLifecycleService.mjs:85
logger.warn('[InferenceLifecycleService] Local inference server is offline. AgentOrchestrator should be managing it.');
return { status: 'offline', detail: 'Local inference server is offline.' };

It logs a warning, returns offline, does not spawn. Meanwhile ai/daemons/orchestrator/TaskDefinitions.mjs only supervises mlx_lm.server (conditional on mlxEnabled: false default, port 11435) — no lms task exists. lms server start lifecycle is currently no-one's job.

The Architectural Reality

• Existing prior art (mirror target): ai/daemons/orchestrator/TaskDefinitions.mjs:108-116 — mlx task launches mlx_lm.server via Python venv binary + --port flag, conditional on mlxEnabled from AiConfig.orchestrator.mlx.enabled. Resolved by daemon.mjs::resolveMlxConfig({orchestratorConfig, env}) (#11957) with env-var override (NEO_ORCHESTRATOR_MLX_{ENABLED,MODEL,PORT}).
• lms CLI shape: lms server start --port <N> (default 1234). LM Studio's CLI binary handles the server lifecycle, port binding, OpenAI-compatible API.
• Model-loading distinct from server-up: per operator — "ensuring the embedding model is loaded" — the lms server being up does NOT mean the configured embedding model is loaded. lms ls enumerates loaded models; lms load <model-identifier> loads if needed. The orchestrator task should ensure both server-up AND model-ready before declaring kbSync / add_memory ready.
• Idempotency primitive: the existing InferenceLifecycleService.isInferenceRunning() HTTP probe to /v1/models is the canonical collision-defense check. New orchestrator task should reuse the same probe shape rather than re-implementing.

The Fix

Add a lms task to ai/daemons/orchestrator/TaskDefinitions.mjs mirroring the mlx shape, plus the supporting config + resolver scaffolding.

Code changes

1. ai/config.template.mjs::orchestrator.lms — new namespace mirroring orchestrator.mlx:

      lms: {
       enabled: false,                                            // default off; operator-overlay opts in
       model  : 'qwen3-embedding-8b',                             // matches current operator-overlay embedding model
       port   : '1234'                                            // LM Studio CLI default
   }

2. ai/daemons/orchestrator/daemon.mjs::resolveLmsConfig — new helper, exact mirror of resolveMlxConfig. Reads env-vars NEO_ORCHESTRATOR_LMS_{ENABLED,MODEL,PORT}, falls back to AiConfig.orchestrator.lms.
3. ai/daemons/orchestrator/TaskDefinitions.mjs::buildTaskDefinitions — accept lmsEnabled / lmsModel / lmsPort, append conditional task:

      if (lmsEnabled) {
       tasks.lms = {
           label          : 'lms server (LM Studio CLI)',
           command        : 'lms',
           args           : ['server', 'start', '--port', String(lmsPort)],
           pidFileName    : 'lms.pid',
           expectedCommand: 'lms server'
       };
   }

4. Model-load probe — new ProcessSupervisor hook (or task chain) that calls lms load <model> after server-up, blocking the lms task lastSuccessAt until model is in the /v1/models enumeration.
5. Orchestrator.mjs::start — pass lmsEnabled / lmsModel / lmsPort through to buildTaskDefinitions (mirror of how mlx* props are passed).

Test changes

• test/playwright/unit/ai/daemons/orchestrator/daemon.spec.mjs — add 4 tests mirroring the MLX coverage from #11957 (purity of buildTaskDefinitions with lmsEnabled: false, no env-var lookups inside buildTaskDefinitions, resolveLmsConfig env-var precedence, AiConfig defaults).
• test/playwright/unit/ai/daemons/orchestrator/Orchestrator.spec.mjs — lmsEnabled pass-through assertion.

Healthcheck honesty companion

The MC healthcheck (HealthService.buildEmbeddingProviderBlock) currently projects config without probing the endpoint. After this ticket lands, the healthcheck should also do an active isInferenceRunning() probe and surface reachable: true/false in the providers.embedding block. Tracked separately as a follow-up (see Related), not bundled here.

Contract Ledger

Decision Record Impact

aligned-with the #11093 daemon-supervision centralization (this ticket completes the migration that #11093 / PR #11096 left half-done). No ADR conflict; this restores the lifecycle that #9833 shipped and #11093 inadvertently dropped.

Acceptance Criteria

• AC1 — AiConfig.orchestrator.lms.{enabled, model, port} namespace added to ai/config.template.mjs with documented defaults
• AC2 — daemon.mjs::resolveLmsConfig helper implemented as mirror of resolveMlxConfig
• AC3 — TaskDefinitions.mjs::buildTaskDefinitions accepts lmsEnabled / lmsModel / lmsPort; emits conditional tasks.lms mirroring tasks.mlx
• AC4 — Orchestrator.mjs::start passes lmsEnabled / lmsModel / lmsPort through
• AC5 — Model-load probe wired so task lastSuccessAt is only set when /v1/models enumerates the configured embedding model
• AC6 — Unit coverage on the 4 mirror tests + the model-load probe path
• AC7 — Cross-family review per pull-request §6.1
• AC8 — Post-merge: operator confirms NEO_ORCHESTRATOR_LMS_ENABLED=true actually restarts lms server start when the process dies (live restart-cycle test)

Out Of Scope

• Healthcheck active embedding-endpoint probe — companion ticket; this ticket lands the lifecycle, not the observability surface
• Ollama parallel restoration — #9833 also shipped Ollama (port 11434) but Ollama supervision was migrated cleanly per the orchestrator's other tasks; this ticket targets only the lms regression
• Replacing mlx_lm.server task with lms — the two are parallel choices, not mutually exclusive. Operators pick one via the enabled flag in each namespace
• Model auto-download — if lms load <model> fails because the model isn't installed locally, the task records the error; auto-downloading the model is out of scope (operator-side concern)
• Client cloud-deployment env vars — client-specific deployment will likely use NEO_OPENAI_COMPATIBLE_HOST pointing at their own embedding endpoint (not localhost lms), so this ticket is local-development-trial substrate. Documenting that boundary explicitly is its own concern.

Avoided Traps

• ❌ Don't put lms lifecycle in InferenceLifecycleService — that's MC-substrate; daemon supervision lives in the Orchestrator per the #11093 architecture. Re-gutting MC would re-introduce the boundary violation.
• ❌ Don't bundle the healthcheck honesty fix — different substrate (MC HealthService vs Orchestrator TaskDefinitions). Separate concern; would force this PR to touch MC code unnecessarily.
• ❌ Don't reinvent the HTTP probe — InferenceLifecycleService::isInferenceRunning() already implements the canonical /v1/models probe shape with 3s timeout. Reuse / extract; don't re-implement.
• ❌ Don't conflate lms with mlx_lm.server — operator clarified explicitly that they are different tools. The mlx task launches mlx_lm.server via Python venv; the new lms task launches the LM Studio CLI binary. They have parallel namespaces, not nested.

Related

• Prior art: #9833 (closed 2026-04-09) — original LM Studio auto-boot in InferenceLifecycleService
• Regression source: #11093 / PR #11096 — "Centralize daemon supervision" deleted MC-side spawn without orchestrator-side completion
• Mirror reference (mlx task): #11075 / PR #11957 — MLX config migration established the resolveXxxConfig pattern this ticket mirrors
• Companion (observability): TBD healthcheck active embedding-endpoint probe ticket
• Adjacent test substrate: #11402 / PR #11616 — confirmed lms server start works through the OpenAI-compatible provider path (no separate dispatch needed)

Handoff Retrieval Hints

• Semantic: query_raw_memories({query: "lms server start orchestrator task supervision regression #11093"})
• Empirical: git log --all --oneline -p -- ai/services/memory-core/lifecycle/InferenceLifecycleService.mjs
• Live: check orchestrator-state.json for absence of lms task entry pre-fix vs presence post-fix