An ingest run has two outcomes and needs three: a late embedding must not fail it

Context

Split from #16690 during review, at @neo-gpt's close-target RA. That ticket carries two separable halves and PR #16713 delivers exactly one of them, so closing #16690 on that head would have silently retired work nobody decided to drop.

• This leaf — the ingest run's third outcome. A late embedding must not fail the run.
• Stays on #16690 — drain-side observability: embeddings becoming queryable without a further ingest run, plus pending-depth and oldest-age reporting.

The split is also the sanctioned remedy for the tension it exposed: an agent PR requires a standalone Resolves, and the honest close target for #16713 did not exist yet.

The Problem

assertErrorFreeIngestionSummary had two outcomes where the lane needs three. Any error failed the whole run, so a single slow embedding discarded the checkpoint for every chunk that did embed, the repo took a backoff step, and the corpus never grew.

Measured on an external deployment: four tenant repos at consecutiveFailures: 13, cadence pinned to its 2 h cap, lastIngestedRev: null, count: 0. The parse and chunk work of every one of those runs was thrown away because the tail of it was late.

The Architectural Reality

• TenantRepoSyncService.mjs — assertErrorFreeIngestionSummary threw on summary.errors.length > 0; the caller's success path advances the checkpoint and resets the streak, its catch path increments it. There was no third path.
• ai/services/knowledge-base/helpers/embedFailureClassification.mjs — owns the bounded KB_* vocabulary and the credential boundary that makes those codes safe to persist.
• IngestionService.mjs — 14 distinct errors.push sites, only two of them the embed path. The summary stream is mixed, which is what makes a deferral-by-default classifier unsafe without a domain gate.
• VectorService.mjs:1103-1106 — a chunk enters chunksToProcess only when !existingIds.has(chunkId), ids content-derived, existingIds corpus-scoped. Already-embedded chunks are never re-embedded, so re-running converges rather than restarting — which is why no write-ahead store is required for the deferral half.
• #16692 / PR #16712 — the durable per-repo cadence stays the sole scheduler; a recovery generation is the only sanctioned bypass. Deferral composes with that rather than adding a second authority.

The Fix

classifyIngestionOutcome returns complete | deferred, and throws on rejected.

• Deferred is incomplete, not failed. Checkpoint holds; consecutiveFailures is neither incremented nor reset; lastRunAttemptAt advances.
• Deferral is opt-in by domain and default within it. Only codes the embed classifier can emit are eligible; within those, deferral is the default because the production specimen was the unclassified sentinel and a recognised-transient allow-list would have rejected exactly the case it was built to survive.
• A deferral arms the shared recovery episode from its bounded cause, so the existing canary owns the rearm.
• Deferrals are counted and an all-deferred sweep is not clean — otherwise a sweep deferring every repo reports 1 completed, 0 failed.

Acceptance Criteria

• A run whose errors are all deferrable embed failures completes without failing, holds its checkpoint, and does not increment consecutiveFailures.
• One rejected code, one non-embed error, or one error with no code at all still fails the run, preserving the #16647 receipt behaviour.
• A deferral arms the shared embedding-recovery episode and re-uses it rather than minting a new one per sweep.
• An all-deferred sweep reports a non-clean top-level outcome and does not advance lastSuccessAt.
• The deferred per-repo projection publishes its recovery classification.
• Specs fail against the pre-fix implementation; mutation-proved in both directions.

Out of Scope

• Async drain, pending-depth and oldest-age observability — those stay on #16690.
• Provider capacity. Nothing here makes a starved provider fast; it makes a starved provider survivable.
• Any change to the durable cadence or to #16692's recovery authority.

Avoided Traps

• A write-ahead store for parsed chunks. #16690 prescribed one; the vector store already provides the property, so it would have re-implemented incremental resume in a second place.
• A transient-allow-list discriminator. It reads as the careful choice and would not have fired on the specimen that motivated the work.
• Deferral-by-default over the whole summary stream. A parse failure would defer forever — never failing, never advancing, never surfacing a cause. Silently stuck is worse than loudly broken.
• A second cadence bypass. Deferral hands the recovery lane a retained cause instead of inventing its own resumption authority.

Related

#16690 (parent; drain-side observability remains open there), #16692 / PR #16712 (recovery authority this composes with), #16647 (embed-failure receipt), #16706 (deployment readiness — this is a MUST item), PR #16713

Live latest-open sweep: latest 20 open issues checked 2026-08-08T16:31:03Z; no equivalent found — this is a deliberate split of #16690 rather than a new discovery.

Origin Session ID: 9ced67a1-8f21-4da2-a1bf-a2a968c47ed2