A slow embedding provider fails the whole KB ingest run

Context

Measured on an external deployment, 2026-08-08, via its live /kb/mcp and /mc/mcp Streamable-HTTP surfaces. Its Knowledge Base has never ingested — count: 0, checkpointStatus: uninitialized, lastIngestedRev: null on every configured repo.

The provider is not down. It is slow. That distinction is the whole ticket.

tenantRepoSync   4/4 repos  backoff-suppressed
                 consecutiveFailures 13, backoffMultiplier 8192, cadence at the 2h cap
                 lastErrorCode       KB_TENANT_REPO_SYNC_SYNC_FAILED
                 lastSourceErrorCode KB_VECTOR_EMBED_FAILED   (3 of 4; the 4th null)
accessReadiness  ready 4/4          -> not auth, not access
local-model      399.36% CPU, 14.3G/48G, one ollama serving a 26B chat model

Independent probe: one query_documents embedding of a three-word string did not complete in 120 s (keepalives, no result). I measured that it exceeds 120 s — not what it would eventually be.

The decisive observation is the asymmetry, and it is the operator's, not mine: on this same box add_memory worked, because Memory Core embeds later. The same provider, the same models, the same contention — one write path survived and the other never ran.

Design correction during implementation (2026-08-08)

The observed failure code is the UNCLASSIFIED sentinel, so deferral cannot be keyed on recognising a transient fault.

KB_VECTOR_EMBED_FAILED — the lastSourceErrorCode on all four repos — is not a distinct failure. embedFailureClassification.mjs:67 declares it as KB_VECTOR_EMBED_UNCLASSIFIED: the embed stage failed and the provider's code matched no entry in either vocabulary. None of the timeout classes (KB_VECTOR_EMBED_TIMEOUT, KB_VECTOR_EMBED_PROVIDER_TIMEOUT) fired for the real specimen.

The obvious implementation — defer on a recognised-transient allow-list — therefore rejects precisely the case it was written to survive, and does so silently, because an unrecognised code looks like a decision rather than a gap. Deferral is the default; rejection is the closed set of our own deliberate refusals (KB_EMBEDDING_INPUT_SIZE_EXCEEDED, KB_SYNC_VOLUME_EXCEEDED, KB_TENANT_SPOOF_REJECTED) — the codes whose permanence we actually know because we mint them.

The asymmetry justifies the default: wrongly deferring a permanent failure costs bounded retries whose backlog is observable as a pending depth that never falls; wrongly failing a transient one costs the corpus, which is the measured outcome. This is the same "unrecognised degrades in the safe direction" discipline the module already applies at the durable-state boundary, pointed at the corpus instead.

Shipped as classifyEmbedDisposition in the existing helper (no new module — same concern, same owner). Mutation-proved in both directions: a transient-allow-list implementation fails the specimen, composition, non-vacuity and totality tests; an unconditional-defer implementation fails the rejected-set and non-vacuity tests.

Scope reduction (2026-08-08) — no durable chunk store is needed after all

This ticket originally prescribed a write-ahead store for parsed chunks, mirroring memoryWalStore. That is not necessary, and the evidence is one branch.

The fork was: does re-running an ingest after a partial failure re-embed everything, or only the gap? If everything, a slow provider makes retries useless and durable partial-progress is mandatory. If only the gap, retries converge on their own.

VectorService.mjs:1103-1106, on the path deleteStale: false resolves to (STALE_STRATEGY_SKIP):

expandedKnowledgeBase.forEach(chunk => {
    const chunkId = chunk.id;
    allIds.add(chunkId);

    if (!existingIds.has(chunkId) && !processedIds.has(chunkId)) {
        chunksToProcess.push(chunk);
        processedIds.add(chunkId);
    }
});

existingIds is read from the collection scoped to this corpus (buildOwnedScopeFilter), and chunk ids are content-derived, so a chunk already embedded is never re-embedded. The ingest path is incremental by construction. Successive runs against a recovering provider make real forward progress, and the already-embedded work is already durable — in Chroma, which is where it belongs.

So the durable store this ticket asked for would have re-implemented, in a new WAL, a property the vector store already provides. What remains is much smaller: a run must stop being all-or-nothing.

(Method note, because it is the point: VectorService.mjs:146 asserts this behaviour in a docblock and I declined to build on it, having had three ticket premises falsified today for citing docblocks instead of tracing them. The line above is the trace. The docblock was right — which is not the same as it having been evidence.)

The actual defect, precisely

TenantRepoSyncService.mjs:247 — assertErrorFreeIngestionSummary throws when summary.errors.length > 0. Any error fails the whole run, so one deferrable embed failure discards the checkpoint for every chunk that did embed, and the repo takes a backoff step toward the 2 h cap. Line 779 confirms the intent: "persisted checkpoints remain unchanged until each replay completes without summary errors."

The fix is a third outcome, not a new store:

• Any rejected error → throw exactly as today. #16647's multi-code receipt behaviour is untouched.
• All errors deferrable → do not throw. The run is incomplete, not failed: the checkpoint does not advance, consecutiveFailures does not increment, and the lane stays at base cadence instead of climbing the backoff.
• The distinction is already shipped: classifyEmbedDisposition (e6da5555e2), where deferral is the default because the observed production code was the unclassified sentinel.

The Problem

IngestionService embeds inline, inside the ingest run:

// ai/services/knowledge-base/IngestionService.mjs:249
await this.embedChunkGroups({chunks: embeddableChunks, ...});
// -> :382  await this.vectorService.embed(tempFile, {...})

A provider slower than the caller's patience therefore does not degrade the run — it fails it. The failure propagates as KB_VECTOR_EMBED_FAILED -> KB_TENANT_REPO_SYNC_SYNC_FAILED, the repo takes a backoff step, and the corpus stays at zero. Thirteen consecutive times, in this case, on a box whose provider can demonstrably embed — just not within an awaited call competing with a resident 26B model.

The parse, chunk, and dedup work of every one of those runs is discarded with it. Nothing durable survives an ingest whose embedding was merely late.

The Architectural Reality

Memory Core already solved this, in-tree:

• ai/services/memory-core/helpers/memoryWalStore.mjs — the write lands durably first.
• ai/services/memory-core/MemoryService.mjs:662 — describeDrainState(), allWritesSemanticallyQueryable: pending.length === 0.
• ai/mcp/server/memory-core/toolService.mjs:197-211 — surfaced as memoryWalDrain, with a stalled verdict once oldestPendingAgeMs exceeds stallThresholdMs.

That is a complete deferred-embedding discipline: durable-first write, asynchronous drain, and an honest observable for "is my write queryable yet". The Knowledge Base ingest path has none of the three. The two services embed against the same provider with opposite tolerance for latency, and only one of them survived contact with a slow one.

Note what this ticket does not claim: that the provider contention is itself a defect. gemma4:26b is legitimately the chat model (ask, session summarization, graph processing). A KB that only ingests when no one is using the chat model is the defect.

The Fix

Give KB ingestion the durable-first, drain-later shape Memory Core already has.

• Land parsed chunks durably before embedding, so a late embedding never discards completed parse/chunk work.
• Drain embeddings asynchronously against the provider, at the provider's pace.
• Report drain state on the KB surface the way memoryWalDrain does for Memory Core, so count: 0 is distinguishable from count: 0, 40k pending.
• An embedding that is late must leave the repo's sync outcome un-failed; only an embedding that is rejected (see helpers/embedFailureClassification.mjs) fails the run.

Owning substrate per ai:structure-map: ai/services/knowledge-base/ with helpers/; the WAL primitive precedent sits in ai/services/memory-core/helpers/. Whether the store is lifted to ai/services/shared/ or duplicated is an implementation call for the claimer — boundedRetryGate.mjs is the in-tree precedent for lifting a twice-discovered notion to shared/.

AC disposition after the #16717 split — 2026-08-08 (@neo-opus-grace, ticket owner)

The five ACs below were written around a write-ahead store that implementation falsified. Rather than leave them pending — which would invite someone to build a pending-depth counter for a queue that does not exist — each is dispositioned against the shipped design. Two are delivered, two are retired, one survives and is the live scope.

Traced, not inferred, at dev:

Why AC-3 survives the design change intact

The WAL is gone, but the number AC-3 asked for is real, is computed on every run, and is discarded. VectorService.mjs:1114-1117 derives workVolume from chunksToProcess.length against existingIds, and line 645 carries the genuine backlog on the lease-yield path. The live plane logs it right now:

Found 64209 existing documents in this corpus.
868 chunks to add or update.

That is exactly the discriminator AC-3 named — empty corpus mid-progress vs empty corpus at rest — and on a starved provider it is the difference between a deployment that looks dead and one that is visibly converging. What changed is the mechanism, not the observable: it is a derived outstanding-chunk count, not a queue depth, and its age analogue is when the outstanding set last shrank, not an oldest-pending-write timestamp.

Surface correction — 2026-08-08 17:5xZ (@neo-opus-grace, ticket owner)

I named the wrong surface in the ACs below, and caught it before building on it. The original restated AC said "the KB surface reports…". That is falsified for the deployment case this ticket exists for.

IngestionService.mjs:36-48 declares its own scope, and it rules the KB server out by construction:

INGESTION_PROGRESS_OBSERVED_SCOPE    = 'this-process-only'
INGESTION_PROGRESS_CROSS_PROCESS_HINT = 'Pull-mode tenant-repo ingestion runs in the orchestrator
                                         process and is NOT reflected here; read the deployment-state
                                         snapshot for that lane.'

The measured failure on the external plane is the tenant-repo lane. So the KB server's progress and health surfaces can never answer for it — not as a gap to fill, but as a declared scope boundary. Building the observable there would have produced a number that is correct about the wrong process, which is worse than no number.

The correct surface is the orchestrator's per-repo tenant-sync state — the same durable record that already carries consecutiveFailures, lastSourceErrorCode and (since #16717 / PR #16713) recoveryState and status: deferred, published through the deployment-state snapshot's tenantRepoSync.repos[]. That record is already durable, so no new store is needed — the second thing the original prescription got wrong, after the WAL.

And the numbers already flow. IngestionService summaries already carry totalChunks and embeddedChunks (:686, :695), which are exactly deriveOutstanding's total and embedded. Nothing needs threading out of the KB service; the pair the observable needs is already on the summary the sync run receives.

Restated AC (the live scope of this ticket):

• The orchestrator's per-repo tenant-sync state reports the outstanding-chunk count — derived from the totalChunks / embeddedChunks the ingest summary already carries — so count: 0 at rest is distinguishable from count: 0 with N chunks outstanding, for the lane that actually ingests tenant repos.
• It reports a staleness companion for that count: when the outstanding set last decreased, so a backlog that is converging is distinguishable from one that is stuck.
• The count is derived from the same summary the run already produces, not recomputed by a parallel implementation that can disagree with the embedder.
• It is published on the deployment-state snapshot's per-repo projection, beside the fields #16717 added — one place an operator reads the lane, not two.
• Negative control: a repo that is genuinely complete reports zero outstanding and is not indistinguishable from one whose delta has never been computed. An unknown must not render as a zero — that is the empty-is-not-success defect this ticket family exists to close.
• Scope control: nothing is added to the KB server's get_ingestion_progress or healthcheck. Its this-process-only disclosure stays accurate, and this observable does not quietly widen it.

(The KB-server framing in the two paragraphs above this section is retained as the reasoning that produced the observable; only its placement was wrong.)

What this disposition deliberately does not do

It does not close the ticket. AC-3's intent was never delivered, and on a plane whose corpus is at zero it is the observable that tells an operator whether anything is happening at all. Retiring ACs 1-2 narrows the work; it does not retire the work.

Out of Scope

• Provider co-residency, requireParallelModels, and ollama parallelism tuning — routed to dialogue, needs measurement first.
• The backoff cap and its resumption condition — sibling lane, boundedRetryGate adoption.
• KB healthcheck's inability to observe the embedding dependency — sibling lane.
• Any change to what counts as a rejected embedding.

Avoided Traps

• Raising the inline timeout. Moves the cliff, does not remove it; the deadline would have to bound the slowest legitimate provider on the slowest box, which is a number nobody can name. The measured failure was >120 s for three words — no fixed deadline survives that honestly.
• Retrying harder. The retries are plausibly feeding the starvation: a client-side timeout likely does not cancel server-side inference, so with OLLAMA_NUM_PARALLEL=1 each abandoned probe may extend the queue that starves the next. Flagged as unverified — I cannot see the provider queue from outside the deployment.
• Treating this as a deployment-config problem. It presents as one, and a faster provider would hide it. But an ingest architecture that discards completed work whenever embedding is slower than an awaited call is wrong on any box; the external deployment only made it visible.

Responsibility map (lead-role convergence artifact)

One incident, five lanes. Three filed — each has a distinct owning substrate and a named in-tree precedent. Two routed rather than filed.

Decision Record impact

none — this adopts an existing in-tree discipline into a second consumer; it does not amend ADR authority.

Related

• #16646 — health probes that spawn a process report host load; same deployment, the instrument layer rather than the ingest layer.
• #16647 — embed-failure classification; this ticket must preserve its receipt behaviour.
• #16563 — an empty KB export is degraded, not complete; same "empty is not success" family.

Sweep record

• Live latest-open sweep: latest 20 open issues at 2026-08-08T13:01:01Z — no equivalent found.
• A2A in-flight claim sweep: unavailable — the Memory Core MCP surface was wedged (the failure mode of #16677). Substituted an alternate-transport check: open PRs and the 12 most recent remote branches. No overlapping claim on this lane. A peer holding an unfiled claim should say so and this ticket stands down per first-claim-timestamp-wins.

Origin Session ID: 9ced67a1-8f21-4da2-a1bf-a2a968c47ed2

Retrieval Hint: query_raw_memories("KB ingest deferred embedding slow provider tenant repo sync backoff"); deployment evidence captured 2026-08-08T12:41-12:47Z.