Problem
The oversize-input guard that keeps a document under the embedding context band is conditional on the provider name, and it fails open.
IngestionService.mjs:74 defines the entire enabled set:
const LOCAL_EMBEDDING_PROVIDERS = new Set(['openAiCompatible', 'ollama']);
resolveEmbeddingInputGuardrail() returns enabled: LOCAL_EMBEDDING_PROVIDERS.has(embeddingProvider), and when that is false:
IngestionService.filterEmbeddingInputBudget() returns every chunk unfiltered, and
VectorService.measureEmbeddingInput() returns {skip: false, inputBytes: 0, inputTokensEstimate: 0} — without measuring anything.
So on an unrecognised provider a 100,000-token document is sent whole, and the two zeros it reports downstream are indistinguishable from a genuinely tiny input. The guard does not merely stop protecting; it produces a confident-looking measurement it never took.
This is reachable, not theoretical. gemini is a fully implemented embedding provider (TextEmbeddingService.mjs:310, 532, 711, 1993), and the selector carries no domain at all:
embeddingProvider: leaf('openAiCompatible', 'NEO_EMBEDDING_PROVIDER', 'string')'string' accepts anything. NEO_EMBEDDING_PROVIDER=gemini disables the guard; so does NEO_EMBEDDING_PROVIDER=openaicompatible, or any typo, silently and with no diagnostic.
The inversion is the sharp part: the providers where the guard is off are the ones whose input limits are tighter than our 28,672-token band, not looser. A guard scoped to "local" providers protects exactly the deployments that need it least.
The Architectural Reality
- #17070 closed the sibling hole — a provider that truncates silently — by minting
EMBEDDING_INPUT_TRUNCATED from either Neo's bounded LM Studio estimate or an OpenAI-compatible provider's structured exceed_context_size_error. That typed handling is scoped to the same two provider shapes as the pre-check, so a third provider is uncovered on both the pre-check and the refusal-classification paths.
- The
'string' domain is the same defect class @neo-gpt caught in backoffBaseMs on #17126 today: a loose type where the value is an enum. There, 'number' accepted a negative delay; here, 'string' accepts a provider that does not exist.
- The band itself (
EMBEDDING_SAFE_PROCESSING_LIMIT_TOKENS = 28672) and the splitter are sound and mutually consistent — the splitter budgets safeProcessingLimitTokens * 3 bytes, the exact inverse of bytesToTokens. Nothing below is about the band being wrong; it is about the band not being consulted.
The Fix
- Fail closed on an unrecognised provider. Measure regardless, and treat "provider not classified" as a refusal to send rather than a licence to send unmeasured. An unknown provider is precisely the case where we know least about the limit.
- Give
embeddingProvider a domain. The set of implemented providers is knowable at config time; a typo should fail loudly at boot rather than silently disable a safety guard at ingestion.
- Stop reporting zeros for an unmeasured input.
{inputBytes: 0, inputTokensEstimate: 0} is a measurement claim. An unmeasured input needs a distinguishable shape so diagnostics cannot read "not checked" as "checked, tiny".
Out of Scope
- Recalibrating
BYTES_PER_TOKEN_HEURISTIC. That axis was measured against a live tokenizer on 2026-06-23 (dense agent prose at 2.72 chars/token, code at ~4.0 — code is not the dense case) and mitigated from 4 to 3. The residual margin question belongs with that measurement, not here.
- Changing the band, the splitter, or the drop-with-receipt behaviour. Those work: oversized documents are split and any residue is dropped, never truncated —
TextEmbeddingService.mjs:98, "A truncated embedding must never be stored."
Acceptance Criteria
Evidence class
Source-verified 2026-08-15 at dev: IngestionService.mjs:74 provider set, :1400-1406 guardrail resolution, :1433-1486 unfiltered return, VectorService.mjs:744-756 unmeasured zeros, configBase.mjs:722 'string' selector, TextEmbeddingService.mjs gemini branches. Prior art read and deliberately not duplicated: #17070 (closed, typed overflow), #13929 (closed, the local-provider guard), #13918 (closed, heuristic calibration), #17113 (open, slot-fit admission).
Authored by Grace (Claude Opus 5, Claude Code). Session b17338dd-b474-494f-b08c-683044de2ddb.
Problem
The oversize-input guard that keeps a document under the embedding context band is conditional on the provider name, and it fails open.
IngestionService.mjs:74defines the entire enabled set:const LOCAL_EMBEDDING_PROVIDERS = new Set(['openAiCompatible', 'ollama']);resolveEmbeddingInputGuardrail()returnsenabled: LOCAL_EMBEDDING_PROVIDERS.has(embeddingProvider), and when that is false:IngestionService.filterEmbeddingInputBudget()returns every chunk unfiltered, andVectorService.measureEmbeddingInput()returns{skip: false, inputBytes: 0, inputTokensEstimate: 0}— without measuring anything.So on an unrecognised provider a 100,000-token document is sent whole, and the two zeros it reports downstream are indistinguishable from a genuinely tiny input. The guard does not merely stop protecting; it produces a confident-looking measurement it never took.
This is reachable, not theoretical.
geminiis a fully implemented embedding provider (TextEmbeddingService.mjs:310, 532, 711, 1993), and the selector carries no domain at all:embeddingProvider: leaf('openAiCompatible', 'NEO_EMBEDDING_PROVIDER', 'string')'string'accepts anything.NEO_EMBEDDING_PROVIDER=geminidisables the guard; so doesNEO_EMBEDDING_PROVIDER=openaicompatible, or any typo, silently and with no diagnostic.The inversion is the sharp part: the providers where the guard is off are the ones whose input limits are tighter than our 28,672-token band, not looser. A guard scoped to "local" providers protects exactly the deployments that need it least.
The Architectural Reality
EMBEDDING_INPUT_TRUNCATEDfrom either Neo's bounded LM Studio estimate or an OpenAI-compatible provider's structuredexceed_context_size_error. That typed handling is scoped to the same two provider shapes as the pre-check, so a third provider is uncovered on both the pre-check and the refusal-classification paths.'string'domain is the same defect class @neo-gpt caught inbackoffBaseMson #17126 today: a loose type where the value is an enum. There,'number'accepted a negative delay; here,'string'accepts a provider that does not exist.EMBEDDING_SAFE_PROCESSING_LIMIT_TOKENS = 28672) and the splitter are sound and mutually consistent — the splitter budgetssafeProcessingLimitTokens * 3bytes, the exact inverse ofbytesToTokens. Nothing below is about the band being wrong; it is about the band not being consulted.The Fix
embeddingProvidera domain. The set of implemented providers is knowable at config time; a typo should fail loudly at boot rather than silently disable a safety guard at ingestion.{inputBytes: 0, inputTokensEstimate: 0}is a measurement claim. An unmeasured input needs a distinguishable shape so diagnostics cannot read "not checked" as "checked, tiny".Out of Scope
BYTES_PER_TOKEN_HEURISTIC. That axis was measured against a live tokenizer on 2026-06-23 (dense agent prose at 2.72 chars/token, code at ~4.0 — code is not the dense case) and mitigated from 4 to 3. The residual margin question belongs with that measurement, not here.TextEmbeddingService.mjs:98, "A truncated embedding must never be stored."Acceptance Criteria
measureEmbeddingInput, with a control asserting the two shapes differ.embeddingProvidercarries a domain over the implemented providers; an unknown value fails at config resolution with a named diagnostic rather than silently disabling the guard.geminipath specifically is covered by whichever branch the fix chooses (guarded, or explicitly refused as unsupported for ingestion), since it is implemented and reachable today.Evidence class
Source-verified 2026-08-15 at
dev:IngestionService.mjs:74provider set,:1400-1406guardrail resolution,:1433-1486unfiltered return,VectorService.mjs:744-756unmeasured zeros,configBase.mjs:722'string'selector,TextEmbeddingService.mjsgemini branches. Prior art read and deliberately not duplicated: #17070 (closed, typed overflow), #13929 (closed, the local-provider guard), #13918 (closed, heuristic calibration), #17113 (open, slot-fit admission).Authored by Grace (Claude Opus 5, Claude Code). Session b17338dd-b474-494f-b08c-683044de2ddb.