LearnNewsExamplesServices
Frontmatter
id12286
titleRetune local embedding context defaults for Qwen3 ingestion
stateClosed
labels
bugaiarchitectureperformance
assigneesneo-gpt
createdAtJun 1, 2026, 6:43 AM
updatedAtJun 1, 2026, 10:29 AM
githubUrlhttps://github.com/neomjs/neo/issues/12286
authorneo-gpt
commentsCount0
parentIssue12264
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 1, 2026, 10:29 AM

Retune local embedding context defaults for Qwen3 ingestion

Closed v13.0.0/archive-v13-0-0-chunk-15 bugaiarchitectureperformance
neo-gpt
neo-gpt commented on Jun 1, 2026, 6:43 AM

Context

The operator flagged that localModels.embedding.contextLimitTokens = 8192 and safeProcessingLimitTokens = 6144 are too small for real file-scale embedding workloads, especially when a raw source file can be thousands of lines. PR #12274 was merged on 2026-06-01 but explicitly did not retune these config defaults.

Duplicate sweep:

  • #12264 is open and now has a comment preserving that embedding defaults are unresolved config-SSOT scope.
  • The open issue search for localModels embedding contextLimitTokens 8192 6144 only returned #12264.
  • No current open PR in the live queue fixes the embedding defaults.

This ticket extracts the embedding-default work into its own implementable child so the intent is not buried under the chat/LMS preload title.

The Problem

The configured OpenAI-compatible embedding model default is text-embedding-qwen3-embedding-8b, while localModels.embedding.* still carries conservative placeholder values:

  • contextLimitTokens: 8192
  • safeProcessingLimitTokens: 6144

The config comment itself says Qwen3-8B embedding typically supports 32K context and that 8K is only a conservative floor pending validation. The same block still says no active consumer reads these values, but #12274 introduced role-keyed LMS context loading that can consume the embedding context length. Separately, KB/Memory Core embedding pipelines are real production consumers of embedding capacity, and raw external repo ingestion can send file-scale chunks through the embedding path.

A 6144-token safe band is therefore not an acceptable settled default for the current local embedding role. It can preemptively reject or underconfigure valid file-scale ingestion, and it encourages operators to debug model/provider failures that are actually config-default drift.

The Architectural Reality

  • ai/config.template.mjs owns the role-keyed local-model context defaults under localModels.chat and localModels.embedding.
  • openAiCompatible.embeddingModel defaults to text-embedding-qwen3-embedding-8b with vectorDimension: 4096.
  • ProviderReadinessHelper.buildLmsContextLengthsMap() maps chat and embedding model IDs to role-specific context lengths for local OpenAI-compatible loading.
  • TextEmbeddingService and KB ingestion are embedding-role consumers; they must not inherit chat-sized assumptions or obsolete placeholder comments.

The Fix

Validate the actual configured local embedding model capability and retune the embedding-role defaults to a safe, documented operational band. The likely target is a 32K-class embedding context with an explicit safe-processing band, but the implementation must let the validation result determine the final values.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
localModels.embedding.contextLimitTokens ai/config.template.mjs Default reflects the validated capability of the configured Qwen3 embedding model, not the old 8K placeholder. Operators can still override via NEO_LOCAL_MODELS_EMBEDDING_CONTEXT_LIMIT_TOKENS. Config comment. Test asserts the default and env override behavior.
localModels.embedding.safeProcessingLimitTokens ai/config.template.mjs Explicit headroom band matches the embedding role and is large enough for file-scale ingestion. Operators can override via NEO_LOCAL_MODELS_EMBEDDING_SAFE_PROCESSING_LIMIT_TOKENS. Config comment + any affected guardrail docs. Test asserts value and no implicit 0.75 drift.
Embedding role consumers KB / Memory Core embedding paths Consumers that read embedding context defaults see the retuned embedding band, not chat defaults or stale placeholder values. If a consumer cannot yet read the value, docs state the gap and ticket/PR body names it. JSDoc/comments near the consuming helper. Unit coverage around context map / guardrail consumption.

Decision Record Impact

None. This is a config-SSOT correction aligned with #12264 and #12090; it does not amend an ADR.

Acceptance Criteria

  • V-B-A confirms the context capability of the configured default embedding model (text-embedding-qwen3-embedding-8b) from the serving stack or primary model docs before changing the defaults.
  • localModels.embedding.contextLimitTokens is retuned from 8192 to the validated embedding-role capacity.
  • localModels.embedding.safeProcessingLimitTokens is retuned from 6144 to an explicit documented headroom band appropriate for that capacity.
  • The obsolete comment claim that no active consumer reads these values is removed or replaced with accurate consumer mapping.
  • Tests cover default values, env overrides, and at least one role-aware context map/guardrail consumer so the embedding role cannot silently regress to a tiny band.
  • PR evidence includes a file-scale sample rationale: a multi-thousand-line source file must not be rejected solely by the old 6144-token placeholder band when the configured embedding model supports a larger window.

Out of Scope

  • Changing chat model/context defaults.
  • Changing vector dimensions.
  • Changing provider selection or forcing all deployments onto OpenAI-compatible.
  • Implementing native Ollama startup parity; tracked separately.

Avoided Traps

  • Do not leave a magic small default just because operators can override it. Defaults are the path most agents and dev environments will exercise.
  • Do not retune chat and embedding together unless evidence shows the same model-role requirement; these are separate roles.
  • Do not cite Qwen3's likely 32K capacity without validating the active served model and server behavior.

Related

  • Parent: #12264
  • Delivered mandate: #12090
  • Triggering merged PR that left this out of scope: #12274
  • Sibling follow-up: #12285

Origin Session ID: d9b4a887-57aa-43aa-8cec-a260ec35ce12

Handoff Retrieval Hints: localModels embedding contextLimitTokens 8192 safeProcessingLimitTokens 6144 Qwen3 embedding defaults; #12264 embedding role config-SSOT comment; buildLmsContextLengthsMap embeddingContextLength

tobiu referenced in commit 031a3c0 - "fix(ai): retune embedding context defaults (#12286) (#12293) on Jun 1, 2026, 10:29 AM
tobiu closed this issue on Jun 1, 2026, 10:29 AM