The embedding split band is 1.75× the engine's slot, and measured in a different unit

Context

A deployment running the openAiCompatible embedding lane (llama.cpp, qwen3-embedding-0.6b, 4 slots × 16,384 tokens) cannot complete ingestion of two repositories. Its knowledge base climbs — 152 → 680 over ~60 minutes — but two repos hold lastIngestedRev: null with consecutiveFailures at 62 and 76, and the sweep reports errors=50 per cycle.

The engine's refusal is precise and self-describing:

HTTP 400 {"error":{"code":400,"message":"request (18832 tokens) exceeds the available context size
(16384 tokens), try increasing it","type":"exceed_context_size_error",
"n_prompt_tokens":18832,"n_ctx":16384}}

Neo already has the mechanism that should have prevented this. #14000 / #14007 / #14085 shipped splitOversizedEmbeddingChunk, and #17155 made the guard measure every provider — its own JSDoc is emphatic: "The band is provider-independent, so the guard measures EVERY provider — recognized is a diagnostic flag for skip receipts, never a licence to skip the measurement."

The splitter never fires. Two independent faults stack in the same three lines.

The Problem

VectorService.measureEmbeddingInput:

measureEmbeddingInput({text, guardrail}) {
    const inputBytes          = Buffer.byteLength(text || '', 'utf8'),
          inputTokensEstimate = bytesToTokens(inputBytes),          // bytes/3 ESTIMATE
          band                = Number(guardrail.safeProcessingLimitTokens);

Fault 1 — the band is the wrong leaf, and 1.75× too high. guardrail.safeProcessingLimitTokens resolves to EMBEDDING_SAFE_PROCESSING_LIMIT_TOKENS = 28672 (ai/embeddingSafeBand.mjs:23). The engine's per-slot ceiling on that deployment is 16,384. A chunk has to reach 28,672 estimated tokens before the splitter touches it, so everything between 16,384 and 28,672 is dispatched whole and refused.

The deployment does declare the real ceiling: it sets NEO_LOCAL_MODELS_EMBEDDING_CONTEXT_LIMIT_TOKENS=16384. That binds localModels.embedding.contextLimitTokens — a leaf resolveEmbeddingInputGuardrail also returns, and which the split decision does not read. The operator configured the truth and the splitter consulted a different field.

Fault 2 — estimate compared against a real-token ceiling. bytesToTokens is Math.ceil(bytes / 3). The engine counts with the model's tokenizer. Those are different units, and the drift is content-dependent: measured against the real Qwen3 tokenizer over this corpus, actual / estimate ranges 0.80 – 1.28. At the top of that range a band-legal 16,384-estimate chunk is ~20,900 actual tokens.

Either fault alone would leak; together the guard is ~2.2× off in the unsafe direction.

Evidence

Ran the deployment's own parser over the affected repository locally (1,394 files, 86,847 chunks, 0 parse failures) and tokenized the largest non-vendor chunks against the same GGUF the deployment serves:

Both refused chunks sit under the 28,672 band and under a 16,384 band measured in estimated tokens. Only a real-token measurement against the real ceiling rejects them.

Population split, same run: 25,759 vendor chunks (50 over band, 200k–1.6M estimated tokens — hopeless by any band, and a source/include-manifest question tracked on #11735) and 61,088 non-vendor chunks, of which 2 exceed the slot. 50 + 2 = 52 reconciles the deployment's observed errors=50.

Worth recording because it is not the defect: the bytes/3 heuristic is accurate within ±25% on this corpus. An earlier hypothesis of mine that it under-counted ~2× was fitted to a single production data point and does not survive measurement — the estimator is serviceable, the comparison is not.

The Architectural Reality

• VectorService.mjs:751 — measureEmbeddingInput, the estimate-vs-band comparison.
• VectorService.mjs:535-551 — the split decision that consumes it; evaluation.skip false ⇒ chunk passes whole.
• VectorService.mjs:595 — splitOversizedEmbeddingChunk, the mechanism that works and is never reached.
• VectorService.mjs:450 / IngestionService.mjs:1455 — resolveEmbeddingInputGuardrail, which returns both contextLimitTokens and safeProcessingLimitTokens; the split path reads only the second.
• ai/embeddingSafeBand.mjs:23 — EMBEDDING_SAFE_PROCESSING_LIMIT_TOKENS = 28672.
• consumerFrictionHelper.mjs:142 — bytesToTokens, BYTES_PER_TOKEN_HEURISTIC = 3.

The safe band and the engine ceiling answer different questions — one is a processing-cost guard, the other a hard admission limit — so this is not a request to merge them. It is that the admission decision must key on the admission limit.

The Fix

1. Derive the split threshold from the engine's admission ceiling (localModels.embedding.contextLimitTokens), not from safeProcessingLimitTokens. Keep the safe band for what it is for.
2. Close the unit gap. Either measure real tokens at the admission boundary, or keep the estimate and apply a margin no smaller than the measured worst-case ratio — with the margin stated as a named constant carrying the measurement, not an unexplained fudge.
3. When both leaves are present, the smaller of the two governs admission; a deployment that lowers contextLimitTokens must not have it silently ignored.
4. Emit the measured/estimated pair on a split or skip receipt, so the next drift is visible rather than inferred from an HTTP 400.

Contract Ledger Matrix

Decision Record impact

none — restores the intent #14000/#17155 established.

Acceptance Criteria

• A chunk whose real token count exceeds the configured admission ceiling is split before dispatch, on every provider.
• A deployment setting NEO_LOCAL_MODELS_EMBEDDING_CONTEXT_LIMIT_TOKENS has that value govern admission; the tracked safe band cannot override it upward.
• The margin (or real-token measurement) is sufficient for a measured actual/estimate ratio of 1.28, with the measurement cited at the constant's definition.
• Split/skip receipts carry both the estimated and the effective token figures.
• Red-proof: a fixture chunk of ~17,200 real tokens against a 16,384 ceiling must be dispatched whole on main (reproducing exceed_context_size_error) and split after the change. A fixture above 28,672 would split on main already and proves nothing — the band gap between the two ceilings is the region under test.
• safeProcessingLimitTokens retains its current behaviour for the processing-cost path it owns.

Out of Scope

• Vendor chunks of 200k–1.6M tokens: no band saves them and no parser can cut them; that is the per-tenant include manifest on #11735.
• Accepting truncated embeddings. A vector that claims to represent a document while covering part of it retrieves confidently wrong; if partial embedding is ever wanted it needs an explicit truncated marker so retrieval can disclose it, which is a separate design.
• Retry-at-identical-size on a deterministic refusal — #16972.
• Graduating a chunk that cannot be delivered at all — #17336.
• The parser's unit shapes. Its ts-enum units are semantically correct and only 2 of 61,088 non-vendor chunks exceed the slot, by 5% and 2%.

Avoided Traps

• Raising the engine ceiling instead. Peak memory for one non-causal embedding request is quadratic in its token count — measured 7.69 GiB idle, 24.14 GiB for a single 13,980-token input. Admitting 18,832 tokens needs ~37 GiB and the next larger chunk needs more. The ceiling cannot chase the corpus; admission has to be bounded.
• "Just lower the band to 16384." Still an estimate against a real-token limit, so a 1.28-ratio chunk keeps leaking. The unit is the defect, not only the number.
• Deleting safeProcessingLimitTokens. It guards a real and different concern; conflating the two is what produced this.
• Blaming the estimator. Measured accurate within ±25%. Replacing it would be effort spent on the component that is behaving.

Related

#14000 · #14007 · #14085 (the splitter) · #17155 (guard measures every provider) · #16972 (identical-size retry on the same refusal) · #17336 (undeliverable graduation) · #17337 (the probe that reports healthy through this) · #11735 (never-ingest set for vendor trees) · #17296

Live latest-open sweep: latest 20 open checked 2026-08-18T11:07:04Z, plus a six-term state=all title sweep (transport, safeProcessingLimit, split band, per-server, context limit, quarantine expiry) and an A2A recency scan over the last 12 messages — no equivalent ticket, no in-flight claim on this scope.

Origin Session ID: 9ccc2fa1-8843-4796-8e85-5e151c0392d2

Retrieval Hint: query_raw_memories("embedding split band safeProcessingLimitTokens 28672 vs 16384 slot estimate real tokens exceed_context_size_error")