A poisoned first embedding batch masquerades as provider outage

Context

A deterministic tenant-ingestion embedding failure was reproduced twice on the same corpus: once during provider contention and once in a clean window. The identical result falsifies contention as the sufficient explanation. Current dev still contains the source mechanism that can produce that steady state, even though the observed deployment is behind current dev.

This is a successor to #16843, not a reopening. #16843 restored forward progress when a later batch fails after at least one batch has landed. It explicitly preserved the zero-success rule: when batch 1 fails, treat the provider as down and abort.

The Problem

VectorService.embedChunks() uses embeddedCount === 0 as the discriminator between a poisoned batch and a dead provider. That is a continuation policy, not a diagnosis:

• a dead provider fails batch 1;
• a healthy provider given one deterministic poison in batch 1 also fails batch 1;
• both take the same abort branch;
• every later sweep selects that missing content first and repeats the same abort, so recoverable later chunks are never offered.

The next hop then erases the useful cause. VectorService carries lastError as abort.cause, while IngestionService.embedChunkGroups() classifies only error.code and reports the generic outer abort message. A diagnosed content/provider rejection therefore arrives durably as KB_VECTOR_EMBED_FAILED.

That generic code is deferrable. The repo checkpoint remains held and the same first batch is offered again on later scheduler cycles. There is no durable content-scoped disposition that lets unchanged proven poison stop consuming provider work while a changed chunk can re-enter.

Architectural Reality

• ai/services/knowledge-base/VectorService.mjs
  • input guardrails already split or refuse ordinary over-budget text before provider compute;
  • timeout/circuit-open failures already end the entire sweep and must remain on that path;
  • non-timeout retry exhaustion records lastError, but batch 1 still aborts solely because no earlier batch succeeded.
• ai/services/knowledge-base/IngestionService.mjs
  • the catch path ignores error.cause, losing the provider classification and bounded reason.
• ai/services/knowledge-base/helpers/embedFailureClassification.mjs
  • KB_VECTOR_EMBED_FAILED is deliberately retryable/deferred; it cannot stand in for a proven content poison.
• ai/daemons/orchestrator/services/TenantRepoSyncService.mjs
  • preserves checkpoint/failure state and retries later, but owns no per-content quarantine authority.
• Tenant-aware chunk IDs are content-derived and repo-scoped. They are the narrow identity for a poison disposition: changed content yields a different ID and must re-enter automatically.
• embeddingResumeStateDir is the existing declared KB plane-state directory. Any small durable disposition belongs under that authority; this ticket adds no config leaf.

Intended Solution

Replace the zero-success guess with bounded paired evidence for non-timeout failures:

1. Preserve a safe, bounded classification from the original provider error across the abort wrapper.
2. When the first batch exhausts non-timeout retries, run a bounded isolation probe that can demonstrate content dependence without walking the full corpus against a dead provider.
3. Persist only proven poison identities, scoped to tenant/repo content ID and the active embedding-generation coordinates. Unproven failures remain provider failures.
4. Skip an unchanged proven poison on later sweeps, keep it visible in the durable partial receipt, and continue recoverable chunks.
5. Revalidate automatically when content identity or embedding-generation coordinates change; retain an explicit operator replay path.
6. Keep timeout/circuit-open behavior exactly where it is. A timeout is evidence that work may still be running and must never enter isolation.

The isolation algorithm must be bounded by a dead-provider control. It may bisect a failed batch or use an equivalent paired control, but it may not turn one failed batch into a full-corpus retry walk.

Contract Ledger Matrix

Decision Record: NOT_NEEDED — this refines the existing tenant-ingestion failure/continuation contract and reuses the declared KB state authority; no new config or topology authority is introduced.

Acceptance Criteria

• A deterministic non-timeout rejection caused by one chunk in batch 1 is isolated without requiring an earlier successful batch; recoverable chunks after it are embedded in the same sweep.
• The proven poison content identity is durably recorded with bounded safe fields and appears in the partial ingestion receipt.
• A second sweep over unchanged content does not offer the proven poison to the provider again, while still reporting the unresolved hole.
• Changing the chunk content identity or embedding-generation coordinates makes it eligible again; an explicit replay path does the same.
• A dead-provider control stops within a fixed, asserted provider-call ceiling and does not walk the corpus.
• Timeout, circuit-open, abort, and cooperative-yield paths never enter isolation and retain their current dispatch ceilings.
• abort.cause classification/disposition reaches the bounded durable receipt; unknown inner causes remain unknown.
• No raw chunk text, prompt, endpoint, credential, tenant identity, or repository identity is persisted in the poison artifact or public projection.
• Mutation controls redden independently when isolation is removed, when the dead-provider ceiling is removed, when cause propagation is removed, and when the second-sweep skip is removed.

Out of Scope

• Timeout-class batch-size adaptation: #16972.
• Work that may continue after client abort: #16853.
• Stale provider-activity row liveness: #16987.
• Changing batchSize, maxRetries, timeouts, provider/model selection, or deployment topology.
• Diagnosing the exact external input before the preserved cause is available.

Avoided Traps

• “Skip batch 1 after failure.” That turns a dead provider into success-shaped data loss.
• “Retry each chunk individually.” Without a paired control and a hard ceiling, a provider outage becomes a corpus-wide load amplifier.
• “Log the raw cause.” Provider messages can contain endpoints or request material; the durable/public shape must remain bounded.
• “Remember only the chunk ID forever.” A provider/model/generation change must reopen the evidence.
• “Fix the old deployment first.” This ticket is grounded in current dev source and tests; it does not gate shipping the already-promised cumulative deployment update.

Related

• Parent: #16566
• Predecessor: #16843
• Timeout adaptation: #16972
• Abort-stranded work: #16853
• Provider activity liveness: #16987

Live latest-open sweep: checked the newest 30 open issues and a 30-message all-state A2A claim window on 2026-08-12; no exact leaf or in-flight claim owns first-batch non-timeout poison isolation plus wrapper-cause preservation.

Structure map: existing ai/services/knowledge-base service/helper placement; no new config surface. If a new state helper is required, it belongs beside the existing KB embedding resume store.

Origin Session ID: 019fe0b1-114b-7c30-aaf4-8317c1f99d4b

Retrieval Hint: query_raw_memories("first embedding batch deterministic poison zero success provider outage abort cause KB_VECTOR_EMBED_FAILED")