A lease yield on dev discards completed provider chunks, so a repeatedly-yielding sweep never advances

Context

PR #16823 merged at c40003db01 (18:02:43Z) and moved the heavy-maintenance yield consultation to the provider-chunk boundary. It shipped with a livelock, found by @neo-gpt as an observer finding while the PR was in review, landing minutes after the merge.

Duplicate sweep basis: this is a regression introduced by a PR merged minutes ago, so no pre-existing ticket can cover it. #16822 is closed by that PR and is not reopened — the merged work is correct in direction, incomplete in durability.

The Problem

At dev head, completed provider-chunk vectors live only in TextEmbeddingService's local data array. The typed yield throws them away, and VectorService.embedChunks upserts only after embedTexts fully resolves.

@neo-gpt's arithmetic, using this ticket family's own worst case — 20 min per provider chunk against a 30 min maxActiveHoldMs:

Net progress per acquisition: zero. Repeats forever.

completedChunkCount > 0 was written as a forward-progress guarantee. It proves a provider call completed, which is not a durable unit advancing — the guard guaranteed the wrong noun. This is strictly worse than the behaviour it replaced: the pre-#16823 outer-batch yield fired only where the previous batch had already been upserted, so progress was preserved by construction.

The irony is on the record: PR #16823's body argued that returning a partial array would "upsert a prefix and drop the rest" and chose to throw instead — then dropped the entire prefix. Right about the hazard, wrong about which branch carried it.

The Fix

1. The yield error carries the embeddings it obtained. createEmbeddingBatchYieldError attaches the ordered partial payload.
2. embedChunks upserts that prefix before releasing, under the matching ids, and counts it toward embedded.
3. Extract the two producers rather than replicate them. toOrderedEmbeddings serves both the resolved batch and the yield payload; buildChunkMetadata serves both the full and partial upsert. Two hand-rolled copies of either would drift, and drift here means a vector stored under the wrong id.

Acceptance Criteria

• A fixture proves an inner yield persists the completed prefix under the correct ids in order — never a suffix, never re-indexed.
• The anti-livelock control: repeated acquisitions that always yield after two chunks still advance monotonically and terminate. Strict growth asserted every sweep, with a sweep ceiling so a livelock fails red instead of hanging.
• Ordered-embedding derivation and chunk-metadata derivation each have exactly one producer, used by both the full and partial paths.
• Coverage fails against dev head and passes against the repair.
• Negative control: a sweep whose predicate never fires is unchanged; the ordinary-failure retry path is unchanged.

Out of Scope

• The checkpoint interval itself — delivered and correct in #16822. This is purely its durability half.
• #16780's reporting half (AC-3 / AC-5 / AC-7).

Avoided Traps

• Replicating the ordering instead of extracting it. The partial and full paths must map index→embedding identically; a second copy is how a vector ends up under a neighbour's id.
• Treating "a provider call completed" as progress. It is the same substitution-of-the-adjacent-thing that produced the defect.
• Reopening #16822. Its delivered scope is correct; this is a distinct, later defect and deserves its own close target.

Related

#16822 / PR #16823 (the merged predecessor that introduced this) · #16566 (the 13-hour starvation) · #16780 (parent; reporting half) · #16706 (deployment-readiness tracker)

Origin Session ID: 87f453f9-aa80-4487-9ed1-b5d91e052c43

Retrieval Hint: lease yield discards completed provider chunks livelock partial upsert monotonic advance embedChunks resume