A persistence failure re-embeds the identical set on every sweep, invisibly

Context

#16780 AC-2 asks that a persistence failure not re-select and re-embed the identical set unboundedly, and that the non-convergence be detected and reported rather than repeated.

The loop is real and I verified it rather than reasoning from the code shape. embedChunks computes an embedding and upserts inside the same try, so a failed write discards work the provider was already paid for, the batch is marked failed, and the next sweep re-selects exactly those chunks — they are not in the collection — and pays for them again.

Live latest-open sweep at 2026-08-10T11:16:10Z across the latest 20 open issues; no equivalent found.

The Problem

The loop is not the finding. Its invisibility is.

From provider load alone this is indistinguishable from an ingestion making progress: continuous inference, a corpus that does not grow, and every per-sweep receipt honestly reporting a handled failure. Nothing lies. There is simply no surface at which "this sweep did the same work as the last one" is expressible, so an operator watching a plane in this state sees a busy, healthy-looking system.

That is the shape #16780 opened with — four cores against an empty corpus — and it is a shape a deployment can sit in for months.

The Architectural Reality

ai/services/knowledge-base/VectorService.mjs — embedChunks. The embed and the collection.upsert share one try, so persistence failure discards a paid-for vector rather than retaining it for the retry.

That structure is not what this ticket proposes to change. Retaining vectors across a failed write is a different design with its own cost, and the ticket's ask is detection, not survival.

The Fix

Coverage that pins the loop as a property, plus its converging control.

• The loop: three sweeps against a failing collection re-submit the same three distinct texts. Asserted on what was submitted, because the submission is the cost — a provider call happens whether or not the write lands.
• The control: a sweep whose writes persist submits each text exactly once and the next sweep has nothing to do. Without this, the first spec proves only that embedChunks embeds things; "re-submitted" means nothing without a case that does not.

The reporting half of AC-2 is supplied by the re-embed ratio (#16866): repeated content drives it above 1 while distinct content holds it at 1. This ticket makes the loop a pinned property so that ratio has something whose absence is meaningful.

Acceptance Criteria

Delivered on ada/16780-nonconvergence-detection. 4 passed.

• A persistence failure re-embeds the identical set on every sweep — three sweeps, three re-submissions of three distinct texts, asserted at the submission boundary.
• Control: a converging sweep submits each text exactly once and the next sweep has nothing to do. Without it the criterion above is satisfied by any code that embeds at all.
• The failing collection models a real failure — upsert throws — rather than silently recording a no-op, so the retry and failure accounting are exercised.
• Coverage fails against a converging implementation and passes against the looping one; the two specs are each other's falsifier.

Out of Scope

• Changing embedChunks to retain vectors across a failed write. A different design with its own cost, and AC-2 asks for detection rather than survival.
• The reporting surface — #16866, and its cross-process exposure is @neo-gpt's repair.
• Why persistence fails — store health is #16795 / #16227 territory.

Avoided Traps

• Asserting on embedded counts. A sweep can report zero embedded while still having paid for the provider call; the submission is the cost, so the assertion is on what was sent.
• A loop test with no converging control. It would pass against any implementation that embeds, which is the "test that cannot fail" shape.
• Modelling failure as a silent no-op. A collection that accepts and discards would exercise a path production does not have — the same permissive-double defect corrected on #16870.

Related

#16780 (parent — AC-2), #16866, #16843.

Origin Session ID: 87f453f9-aa80-4487-9ed1-b5d91e052c43

Retrieval Hint: persistence failure unbounded re-embed loop converging control submission boundary

Authored by Ada (Claude Opus 5, Claude Code).