Derive LMS embedding SEP suffix from GGUF metadata

Context

PR #14014 closed #14009 by appending <|im_end|> to LM Studio GGUF/Qwen3 embedding requests. The operator restarted the Memory Core repair-defrag after pulling dev, but LM Studio still emitted the GGUF warning:

[WARNING] At least one last token in strings embedded is not SEP. 'tokenizer.ggml.add_eos_token' should be set to 'true' in the GGUF header

The follow-up investigation found that the requests were in fact ending with <|im_end|>:

{
  "model": "text-embedding-qwen3-embedding-8b",
  "input": [
    "...<|im_end|>",
    "...<|im_end|>"
  ]
}

That falsifies the earlier assumption that <|im_end|> satisfies this GGUF's required final token.

The Problem

The suffix logic in #14014 was metadata-scoped to LMS/Qwen, but the suffix value itself was still hard-coded from the architecture name. For the installed Qwen3-Embedding-8B-Q4_K_M.gguf, direct GGUF header inspection shows:

tokenizer.ggml.eos_token_id = 151643 -> <|endoftext|>
tokenizer.ggml.eot_token_id = 151645 -> <|im_end|>
tokenizer.ggml.add_eos_token = true

So <|im_end|> is the chat EOT token, not the EOS/SEP token this embedding runtime is warning about. Appending EOT leaves the final-token warning alive during the missing-vector re-embed phase.

The Architectural Reality

• ai/services/memory-core/TextEmbeddingService.mjs owns regular Memory Core / Knowledge Base embedding dispatch.
• ai/services/graph/providerReadinessHelper.mjs owns direct embedding-serving canary requests.
• openAiCompatible is a shared transport surface for LM Studio, MLX, vLLM, llama.cpp, Ollama compatibility, and fixtures. The fix must not turn the provider name into an LMS/Qwen switch and must not add a suffix config that would break Ollama/native or generic OpenAI-compatible deployments.
• The reliable authority for this case is LMS GGUF metadata: the loaded model row identifies a GGUF path, and the GGUF header maps eos_token_id to the actual token string.

The Fix

1. Add a pure shared vector helper that resolves the LMS embedding-input suffix from GGUF tokenizer metadata.
2. For GGUF models, read tokenizer.ggml.eos_token_id and tokenizer.ggml.tokens[eos_token_id]; append that token when absent.
3. Keep non-GGUF, non-LMS, and unknown-metadata paths unchanged.
4. Use the helper from both TextEmbeddingService and the embedding-serving canary path.
5. Add unit coverage proving <|im_end|> alone is not treated as already sufficient when GGUF EOS resolves to <|endoftext|>.

Contract Ledger Matrix

Decision Record impact

Aligned with ADR 0019. This must remain metadata-derived at the provider request boundary and must not introduce a config suffix leaf or OpenAI-compatible-wide default.

Acceptance Criteria

• GGUF tokenizer metadata is read to resolve the actual EOS token text for LMS embedding requests.
• TextEmbeddingService appends <|endoftext|> for the installed Qwen3 embedding GGUF rather than <|im_end|>.
• Inputs already ending with <|im_end|> still receive the GGUF EOS suffix when EOS is different.
• Generic OpenAI-compatible, non-GGUF, and unknown-metadata paths remain unchanged.
• The embedding-serving canary can apply the same suffix when LMS metadata is available.
• No config leaf is added for the suffix.
• Focused unit tests cover the helper, TextEmbeddingService request body, and canary request body.

Out of Scope

• Reopening #14009 or #14014.
• Changing Ollama native embedding behavior.
• Adding a user-configurable suffix.
• Repairing Memory Core vector corruption or defrag progress logging.

Avoided Traps

• Do not make openAiCompatible itself mean LMS.
• Do not suffix every OpenAI-compatible embedding request.
• Do not hard-code <|im_end|> from Qwen architecture; the GGUF header is the authority.

Related

Related: #14009 Related: #14014

Live latest-open sweep: checked latest 20 open issues at 2026-06-25T18:20Z; no equivalent open ticket found. Targeted open search for LMS SEP embedding canary Qwen3 warning returned no open issues. A2A in-flight claim sweep: checked latest 30 all-status messages at the same time; no overlapping [lane-claim] or [lane-intent] found.

Origin Session ID: 9280140f-8b54-4462-9342-49cca7e226f4

Handoff Retrieval Hints: LMS SEP GGUF eos_token_id endoftext im_end, tokenizer.ggml.add_eos_token #14009 #14014, TextEmbeddingService lmsEmbeddingInputSuffix