Context
Neo's canonical provider lane binds the same elected slot count to both the engine and its consumers:
LLAMA_ARG_N_PARALLEL=${NEO_PROVIDER_LANE_EMBEDDING_SLOTS}NEO_LOCAL_MODELS_EMBEDDING_PARALLEL=${NEO_PROVIDER_LANE_EMBEDDING_SLOTS}
For the shipped four-slot shape, 131072 / 4 = 32768 context tokens per slot. The slot count is therefore already available at the OpenAI-compatible embedding batching boundary.
Current TextEmbeddingService behavior is partly correct: it chunks large embedding arrays, gives batch calls their own timeout, and lets interactive work enter between chunks. Those mechanisms landed before this ticket and are not work for this leaf.
The Problem
The remaining width invariant is missing. #embedOpenAiCompatibleBatch() limits each provider POST only by batchEmbeddingChunkSize (default 5), without considering the declared engine parallelism (canonical provider lane: 4). A single multi-input /v1/embeddings request is expanded by the pinned llama.cpp server into one server task per input, with each task assigned an available slot. A five-input request can therefore occupy every four-slot lane and leave a concurrent small request with no admissible slot until one batch task completes.
This is a slot-admission defect, not proof that the provider is unhealthy or that any particular request will finish inside its deadline.
Intended solution shape
At the OpenAI-compatible batch boundary, derive the effective request width from the existing declarations:
- when
localModels.embedding.parallel > 1, send at most parallel - 1 inputs per provider POST;
- when the value is absent, invalid, or
<= 1, preserve the configured batch chunk size so remote or single-slot endpoints are not silently collapsed by a local-engine assumption.
No new config leaf, scheduler, health classification, or durable state is needed.
Acceptance criteria
Out of scope
- Admission aging / starvation freedom between priority classes (#17062).
- New caller-class deadlines; interactive and batch ceilings already differ on current
dev.
busy-behind-batch health taxonomy or a promise that an admitted request completes within a deadline.- Provider thread tuning, liveness probes, restart reconciliation, and deployment rollout (#17063, #17065, #17073).
Correction history
The original ticket attributed the incident to an unbounded MC array, one shared deadline, and missing interactive interleaving. Current-source intake disproved all three: batching is already chunked, batch and interactive timeouts are separate, and the queue already prefers interactive work between chunks. The surviving P - 1 request-width invariant above is the entire executable scope of this leaf.
Origin Session ID: 37509548-6568-47fe-9e6c-2aabd27c2b11
Context
Neo's canonical provider lane binds the same elected slot count to both the engine and its consumers:
LLAMA_ARG_N_PARALLEL=${NEO_PROVIDER_LANE_EMBEDDING_SLOTS}NEO_LOCAL_MODELS_EMBEDDING_PARALLEL=${NEO_PROVIDER_LANE_EMBEDDING_SLOTS}For the shipped four-slot shape,
131072 / 4 = 32768context tokens per slot. The slot count is therefore already available at the OpenAI-compatible embedding batching boundary.Current
TextEmbeddingServicebehavior is partly correct: it chunks large embedding arrays, gives batch calls their own timeout, and lets interactive work enter between chunks. Those mechanisms landed before this ticket and are not work for this leaf.The Problem
The remaining width invariant is missing.
#embedOpenAiCompatibleBatch()limits each provider POST only bybatchEmbeddingChunkSize(default5), without considering the declared engine parallelism (canonical provider lane:4). A single multi-input/v1/embeddingsrequest is expanded by the pinned llama.cpp server into one server task per input, with each task assigned an available slot. A five-input request can therefore occupy every four-slot lane and leave a concurrent small request with no admissible slot until one batch task completes.This is a slot-admission defect, not proof that the provider is unhealthy or that any particular request will finish inside its deadline.
Intended solution shape
At the OpenAI-compatible batch boundary, derive the effective request width from the existing declarations:
localModels.embedding.parallel > 1, send at mostparallel - 1inputs per provider POST;<= 1, preserve the configured batch chunk size so remote or single-slot endpoints are not silently collapsed by a local-engine assumption.No new config leaf, scheduler, health classification, or durable state is needed.
Acceptance criteria
4and configured batch width5, five inputs are dispatched as provider batches[3, 2].1(and with no usable multi-slot declaration), the configured batch width remains unchanged.localModels.embedding.parallelvalue that the canonical provider-lane compose binds toLLAMA_ARG_N_PARALLEL; it introduces no second slot-count authority.Out of scope
dev.busy-behind-batchhealth taxonomy or a promise that an admitted request completes within a deadline.Correction history
The original ticket attributed the incident to an unbounded MC array, one shared deadline, and missing interactive interleaving. Current-source intake disproved all three: batching is already chunked, batch and interactive timeouts are separate, and the queue already prefers interactive work between chunks. The surviving
P - 1request-width invariant above is the entire executable scope of this leaf.Origin Session ID:
37509548-6568-47fe-9e6c-2aabd27c2b11