Context
Surfaced during a live heavy-maintenance incident (Refs #13624): the add_memory WAL-drain embedder logged Batch embed exhausted 6 attempts (openAiCompatible request timed out after 15000ms) repeatedly while the github-workflow-sync held the exclusive-heavy lease, starving the single local embedder. V-B-A traced the 15000ms to its source.
Root cause (V-B-A'd to source)
ai/services/shared/vector/chromaClientPrimitives.mjs:53 — the Chroma dynamic embedding-function behind every collection.add():
generate: async (texts) =>
Promise.all(texts.map(text => TextEmbeddingService.embedText(text, provider)))fans every batch out as parallel embedText calls — the interactive path, which uses contentionTimeoutMs (15000, config:186) as requestTimeoutMs (TextEmbeddingService.mjs:394, error thrown at :266). It bypasses embedTexts → #embedOpenAiCompatibleBatch, the batch path built for exactly this (1-hour timeout via DEFAULT_OPENAI_COMPATIBLE_REQUEST_TIMEOUT_MS, sequential chunks of batchEmbeddingChunkSize, with a yield).
Two compounding faults in that one line:
- Wrong timeout — bulk Chroma adds get the 15s interactive cap, not the 1-hour batch cap.
embedTexts' own JSDoc (TextEmbeddingService.mjs:423) states batches must "not be cut short by the interactive contention timeout" — but the Chroma path circumvents embedTexts entirely.
- Self-inflicted contention storm —
Promise.all hits the single local embedder with the whole batch at once → the requests contend with each other → the 15s contention timeout trips. The batch path walks chunks sequentially → no storm.
Confirmed chain: WAL drain (drainCycle.mjs:89 collection.add) → Chroma generate → embedText (15s). Almost certainly also hits the github-sync Stage-2 graph ingestion (same Chroma path).
Fix
Route the batch through the existing batch path:
generate: async (texts) => TextEmbeddingService.embedTexts(texts, provider)
embedTexts returns ordered number[][] — exactly Chroma's generate contract (verified: #embedOpenAiCompatibleBatch returns data.sort(byIndex).map(d => d.embedding); embedText returns result.data[0].embedding, so both yield number[][]). The WAL drain's per-record isolation pass (drainCycle.mjs:107) still surfaces poison records.
Acceptance Criteria
Out of scope
Raising contentionTimeoutMs — the path-swap removes the contention that made 15s fail; revisit only if interactive single-embeds still contend after this lands.
Refs #13624 (heavy-maintenance incident), #13498 (who_is_online canary on the same interactive path — leaf-4, may co-land separately).
Context
Surfaced during a live heavy-maintenance incident (Refs #13624): the
add_memoryWAL-drain embedder loggedBatch embed exhausted 6 attempts (openAiCompatible request timed out after 15000ms)repeatedly while the github-workflow-sync held theexclusive-heavylease, starving the single local embedder. V-B-A traced the 15000ms to its source.Root cause (V-B-A'd to source)
ai/services/shared/vector/chromaClientPrimitives.mjs:53— the Chroma dynamic embedding-function behind everycollection.add():generate: async (texts) => Promise.all(texts.map(text => TextEmbeddingService.embedText(text, provider)))fans every batch out as parallel
embedTextcalls — the interactive path, which usescontentionTimeoutMs(15000,config:186) asrequestTimeoutMs(TextEmbeddingService.mjs:394, error thrown at:266). It bypassesembedTexts→#embedOpenAiCompatibleBatch, the batch path built for exactly this (1-hour timeout viaDEFAULT_OPENAI_COMPATIBLE_REQUEST_TIMEOUT_MS, sequential chunks ofbatchEmbeddingChunkSize, with a yield).Two compounding faults in that one line:
embedTexts' own JSDoc (TextEmbeddingService.mjs:423) states batches must "not be cut short by the interactive contention timeout" — but the Chroma path circumventsembedTextsentirely.Promise.allhits the single local embedder with the whole batch at once → the requests contend with each other → the 15s contention timeout trips. The batch path walks chunks sequentially → no storm.Confirmed chain: WAL drain (
drainCycle.mjs:89collection.add) → Chromagenerate→embedText(15s). Almost certainly also hits the github-sync Stage-2 graph ingestion (same Chroma path).Fix
Route the batch through the existing batch path:
generate: async (texts) => TextEmbeddingService.embedTexts(texts, provider)embedTextsreturns orderednumber[][]— exactly Chroma'sgeneratecontract (verified:#embedOpenAiCompatibleBatchreturnsdata.sort(byIndex).map(d => d.embedding);embedTextreturnsresult.data[0].embedding, so both yieldnumber[][]). The WAL drain's per-record isolation pass (drainCycle.mjs:107) still surfaces poison records.Acceptance Criteria
createDynamicTextEmbeddingFunction.generateroutes batches throughembedTexts(1-hour batch path, sequential chunks), notPromise.all(map(embedText)).generate()exercises the batch path / does NOT apply the interactivecontentionTimeoutMs(assert via the timeout or an injected seam).embedText15s path is unchanged (single latency-sensitive embeds keep it).Out of scope
Raising
contentionTimeoutMs— the path-swap removes the contention that made 15s fail; revisit only if interactive single-embeds still contend after this lands.Refs #13624 (heavy-maintenance incident), #13498 (who_is_online canary on the same interactive path — leaf-4, may co-land separately).