LearnNewsExamplesServices
Frontmatter
id16982
titleThe native-Ollama embedding role inherits a chat-sized context, so its parallelism is pinned to 1 by RAM it never needed
stateClosed
labels
bugai
assigneesneo-opus-vega
createdAtAug 11, 2026, 4:48 PM
updatedAtAug 11, 2026, 5:40 PM
githubUrlhttps://github.com/neomjs/neo/issues/16982
authorneo-opus-vega
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 11, 2026, 5:40 PM

The native-Ollama embedding role inherits a chat-sized context, so its parallelism is pinned to 1 by RAM it never needed

neo-opus-vega
neo-opus-vega commented on Aug 11, 2026, 4:48 PM

Context

Operator-raised 2026-08-11, then source-confirmed by @neo-opus-ada and independently by me. Separate lane from #16972 (halve on timeout) and #16973 / PR #16978 (stop in-cycle retries) per @neo-gpt — those operate inside one serialized slot; this decides whether embedding can run more than one request at all.

The Problem

Neo already declares what the embedding role needs, and already has a per-model parallelism knob — but only for LM Studio.

ai/configBase.mjs:828-835:

embedding: {
    contextLimitTokens       : leaf(32768, 'NEO_LOCAL_MODELS_EMBEDDING_CONTEXT_LIMIT_TOKENS'),
    safeProcessingLimitTokens: leaf(28672, ...),
    // lms `--parallel` request-slot count for the embedding model. Same primitive as
    // `localModels.chat.parallel`: each slot carries its own KV cache...
    parallel                 : leaf(1, 'NEO_LOCAL_MODELS_EMBEDDING_PARALLEL')
}

The comment says lms --parallelLM Studio only. The native-Ollama path has no per-model parallelism surface.

So on a native-Ollama deployment the embedding role inherits the daemon-wide OLLAMA_CONTEXT_LENGTH, which is sized for the chat model's REM prompts. Measured on an external plane: OLLAMA_CONTEXT_LENGTH=1310724× the 32768 Neo itself declares the embedder needs.

Ollama RAM scales with NUM_PARALLEL × CONTEXT_LENGTH (their compose states this). At 128K, raising parallelism multiplies a 128K KV cache, and that plane is already at 72% memory (37G/51G) — so OLLAMA_NUM_PARALLEL=1 is a rational response to an inflation the embedder never asked for.

Consequence, measured: every embedding consumer — tenant ingestion, both healthcheck canaries, the WAL drain, and the readiness warm loop — serializes behind one slot. The KB canary's 30-second probe cannot acquire it (EMBEDDING_PROBE_TIMEOUT streak 27), which health-gates the KB out of its own tool surface.

The plumbing already exists: providerReadinessHelper.mjs:903 sends num_ctx per request, and the live orchestrator log shows Warming 'qwen3-embedding' with num_ctx 32768. So the daemon value is only a default, and the embedder's context is already controllable per call — for warming. Ingestion embeds do not carry it.

The Fix

Give the native-Ollama path the per-model surface the LM Studio path already has:

  1. Ingestion/canary/WAL embeds send the embedding role's declared contextLimitTokens, rather than inheriting the daemon default.
  2. Expose embedding parallelism for native Ollama as localModels.embedding.parallel does for lms.

Acceptance Criteria

  • A native-Ollama embedding request carries the embedding role's declared contextLimitTokens, not the daemon default. Asserted at the request payload, not at config.
  • Non-vacuity: a chat request still carries the CHAT role's context — the fix must not narrow chat.
  • Embedding parallelism is expressible for native Ollama; the LM Studio path is unchanged.
  • Mutation conviction: removing the per-request context reverts the payload to the daemon default, and the run output is quoted.
  • MEASURED before/after on a live plane: memoryUsageBytes / memoryLimitBytes and per-stage providerActivity latency, from the read-only deployment snapshot. RAM at higher embedding parallelism is currently UNMEASURED and this AC is the falsifier.

Out of Scope

  • #16972 (halve on timeout), #16973 / PR #16978 (in-cycle retry), #16963 (closed).
  • OLLAMA_MAX_QUEUE and other operator-side compose values.
  • Changing the chat role's 128K context — it is needed for REM prompts.

Avoided Traps

  • Lowering OLLAMA_CONTEXT_LENGTH globally. It would truncate REM chat prompts — their compose comment says extraction silently returns 0 entities when it does.
  • Raising parallelism without bounding per-request context. That is the RAM blowup NUM_PARALLEL=1 exists to prevent, and it would make the plane worse.
  • Asserting the win at config level. The claim is about what reaches the provider; only the request payload proves it.

Origin Session ID: 1d4a0d9f-87fb-4d68-827c-ffe16f80035e