Normalize LM Studio embedding inputs for SEP-token warnings

Context

During the 2026-06-25 Agent OS stability repair, LM Studio emitted repeated embedding warnings while the Memory Core repair-defrag dry run was re-embedding through the local OpenAI-compatible embedding lane:

[WARNING] At least one last token in strings embedded is not SEP. 'tokenizer.ggml.add_eos_token' should be set to 'true' in the GGUF header

The operator called this out as noisy enough to warrant a cleanup ticket. The current repair lane is #13999; this ticket captures the narrower request-shape/model-header warning so it does not disappear inside the backup-repair incident.

Release classification: boardless operational hardening; not proven release-blocking unless investigation shows vectors are malformed or truncated.

Live latest-open sweep: checked latest 20 open issues at 2026-06-25T14:31:08Z; no equivalent SEP/EOS/LM Studio embedding warning ticket found. A2A in-flight sweep: checked latest 30 all-status messages at 2026-06-25T14:31:08Z; no competing [lane-claim] or [lane-intent] for this warning cleanup found. Semantic/exact sweep: ask_knowledge_base found no existing ticket/discussion specifically about this warning; rg -i "add_eos_token|last token|not SEP|SEP token|tokenizer.ggml|eos_token" resources/content/issues resources/content/discussions ai test learn .agents found no equivalent ticket or code path.

The Problem

The warning is generated by the local embedding provider while Neo sends embedding strings through the OpenAI-compatible /v1/embeddings path. It may be harmless GGUF metadata noise, or it may indicate Neo is sending embedding inputs without the provider's expected terminal token. Either way, repeated warnings during bulk repair/sync make the Agent OS stability lane harder to inspect and can hide more serious provider failures.

The ticket should not assume that appending a separator is automatically correct. The implementation must first determine whether the root cause is:

• Neo request-shape behavior for input strings.
• LM Studio/GGUF model metadata (tokenizer.ggml.add_eos_token).
• A provider-specific warning that should be documented or readiness-checked rather than transformed away.

The Architectural Reality

The shared deployment docs state that the OpenAI-compatible embedding path is currently implemented in ai/services/memory-core/TextEmbeddingService.mjs#embedText[s], posting {model, input} to ${host}/v1/embeddings rather than routing through the chat OpenAiCompatible provider class (learn/agentos/SharedDeployment.md, embedding-provider section).

Current code confirms that TextEmbeddingService writes the request body directly as JSON.stringify({ model: embeddingModel, input: inputData }) in #postOpenAiCompatible (ai/services/memory-core/TextEmbeddingService.mjs). That makes this ticket a provider-boundary cleanup, not a Memory Core storage or Chroma defrag task.

The Fix

Investigate and harden the OpenAI-compatible embedding request boundary so LM Studio does not spam SEP-token warnings during Memory Core / Knowledge Base bulk embedding work.

Candidate outcomes, in priority order:

1. If Neo is omitting a required request-level/input-level terminal marker, normalize embedding input at the provider request boundary without mutating stored documents or Memory Core records.
2. If LM Studio exposes a request option or model-load flag that controls EOS/SEP behavior, document and/or readiness-check the supported setting.
3. If the warning is purely a GGUF artifact outside Neo's control, add a focused operator-facing note so future repair runs distinguish it from vector corruption or context truncation.

Contract Ledger Matrix

Decision Record impact

None for filing. If implementation introduces or mutates aiConfig leaves, ADR-0019 must be activated before code changes and the PR must cite the AiConfig Provider SSOT boundary.

Acceptance Criteria

• Reproduce or fixture the LM Studio warning path for OpenAI-compatible embeddings using the configured Qwen3 embedding model lane.
• Determine whether the correct fix belongs in Neo request shaping, LM Studio model-load/provider configuration, or documentation/readiness diagnostics.
• If Neo request shaping is responsible, add a scoped implementation that avoids mutating persisted content and preserves batch/interactive embedding behavior.
• If provider/model configuration is responsible, add an actionable warning or deployment note that names the required operator setting/model constraint.
• Add focused test coverage for the chosen behavior; do not rely on a full build-all or generic Playwright invocation.
• Verify the change does not weaken the existing context-limit guard, oversized-source splitting, or Memory Core backup repair path.

Out of Scope

• Running or promoting Memory Core defrag/backup repair; that remains under #13999.
• Reopening #14003 or #14008.
• Swapping the team's embedding model without operator approval.
• Changing vector dimensions or Chroma collection schemas.

Avoided Traps

• Do not blindly append printable text to documents before embedding; that would change the semantic input and could perturb vectors.
• Do not bury the fix in ad-hoc environment defaults that bypass the AiConfig Provider SSOT.
• Do not treat provider warning cleanup as proof that backup exportability is repaired.

Related

• #13999
• #14008
• Runtime observation: LM Studio warning at 2026-06-25 16:23:33 local time during Memory Core repair-defrag dry run.

Handoff Retrieval Hints: query_raw_memories("LM Studio SEP token add_eos_token embedding warnings TextEmbeddingService"); query_raw_memories("Memory Core repair defrag LM Studio embedding warning"); exact search add_eos_token|not SEP|tokenizer.ggml.