LearnNewsExamplesServices
Frontmatter
id13700
titlelms chat-model loads at parallel-4 (lms default) — quadruples the 128K KV cache (~29.5GB) under a lease-serialized workload
stateClosed
labels
bugaiarchitecture
assigneesneo-gpt
createdAtJun 21, 2026, 4:40 AM
updatedAtJun 24, 2026, 11:43 AM
githubUrlhttps://github.com/neomjs/neo/issues/13700
authorneo-opus-grace
commentsCount7
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 24, 2026, 11:43 AM

lms chat-model loads at parallel-4 (lms default) — quadruples the 128K KV cache (~29.5GB) under a lease-serialized workload

Closed v13.1.0/archive-v13-1-0-chunk-4 bugaiarchitecture
neo-opus-grace
neo-opus-grace commented on Jun 21, 2026, 4:40 AM

Context

Surfaced during the 2026-06-21 heavy-maintenance incident (#13624). Operator + Euclid host probes: top 123G used / only 3.7G unused, swap engaged; ~50 GiB across two LM Studio workers — llmworker.js (gemma-4-31b chat) ~29.5 GiB + embeddingworker.js (qwen3-embedding-8b) ~20.6 GiB. The Neo syncGithubWorkflow.mjs process was negligible (236 MiB), Chroma ~67 MiB. The machine memory pressure is LM Studio, not Neo.

Root cause (V-B-A'd to source)

loadLmsModel (ai/services/graph/providerReadinessHelper.mjs:208) invokes lms load <model> --context-length <N> but does not pass --parallel. So lms defaults the chat model to parallel 4 (confirmed via lms ps: gemma PARALLEL 4 @ context 131072; qwen3 PARALLEL -). Each of the 4 slots holds a 131072-token (128K) KV cache → the ~29.5 GiB chat worker.

The chat workload (graph extraction, session summaries, miniSummary) is lease-serialized — the orchestrator runs heavy tasks one-at-a-time via the exclusive-heavy lease, never 4-concurrent. So parallel-4 is pure KV-cache waste.

Not the cause (falsified): the sync does NOT spawn an additional model. syncGithubWorkflow.mjs has no in-process model (236 MiB); its Stage-2 graph ingestion embeds via the shared lms endpoint, reusing the already-loaded qwen3. The two workers are the two models held by requireParallelModels: 2 — by design (the "hold both like ollama" intent), not an extra spawn.

Fix

Thread a --parallel <N> arg through loadLmsModel (config-driven; default 1 for the chat model). Reclaims ~15–22 GiB of idle KV cache. requireParallelModels: 2 stays unchanged — both models remain resident (no reload, the operator's "hold both / no context-regen" intent preserved); only the chat model's parallel-slot count drops.

Acceptance Criteria

  • loadLmsModel accepts + appends --parallel <N> (config-driven via aiConfig.localModels.chat/orchestrator lms config).
  • The chat model loads at parallel 1 (verify lms ps shows PARALLEL 1), cutting its RSS proportionally.
  • requireParallelModels: 2 unchanged (both models still held; no reload churn).

Test Evidence

Evidence: L1 — config/launch-arg change; the AC is observable via lms ps (PARALLEL column) + host RSS, not a unit assertion. A unit test can pin that loadLmsModel appends --parallel when configured.

Deltas

  • New --parallel knob on loadLmsModel; default 1 for chat (was: unset → lms-default 4).

Post-Merge Validation

  • On the live host after reload, lms ps shows the chat model at PARALLEL 1 and the llmworker RSS drops from ~29.5 GiB toward ~7–10 GiB.

Refs #13539 (qwen unload / memory-saving), #13624 (incident).

tobiu referenced in commit 2113bb7 - "fix(ai): thread per-model lms --parallel; default chat to 1 to cut idle KV-cache RAM (#13700) (#13706) on Jun 21, 2026, 6:07 AM
tobiu closed this issue on Jun 21, 2026, 6:07 AM
tobiu referenced in commit 4974090 - "fix(ai): replace stale lms resident models (#13700) (#13941)" on Jun 24, 2026, 11:43 AM
tobiu closed this issue on Jun 24, 2026, 11:43 AM