Vector rebuild failure receipts and bounded embed retry

Amended 2026-07-31 (v2) after PR #16228 closed unmerged via Drop+Supersede — this body is the successor authority the replacement PR resolves against. Salvage map: the closing review on #16228. Source-boundary authority: the correction comment on #16208 (2026-07-31T03:52Z). The v1 body scoped receipts + bounded retry only; v2 carries the complete contract the review found missing.

Context

The #16208 rebuild's memory leg reported unrecoverable=7562 (~24%): diff-verified as per-pass embed failures under provider saturation (400/400 sampled rows carry full documents), not content loss. The v1 fix (fate-stamped receipts + per-range retry) shipped in PR #16228 and failed review on five executable falsifiers plus a source-authority mismatch; that PR is closed unmerged and this ticket now defines the successor.

Composition rule (settled): the runner composes with, never re-decides, #16208's source boundary — a fresh Compose-owned target populated by graph+WAL replay/re-embed, or a supported version-migration against a clone. The legacy vector store is never the document authority. Everything below is source-agnostic hardening of the runner/extractor that the chosen source feeds.

The Problem (v2 — the five falsifiers)

1. Fate without cause: all embed failures collapse to embedding-provider-error → retryable: true. A permanent 401 reads "resumable"; grouping drops the HTTP status, so the receipt cannot answer "resume or fix configuration?".
2. Unbounded amplification: every binary-split range gets a fresh retry budget — a persistent 8-doc outage costs 45 outer calls — and the fail-fast worker pool starts the next attempt while orphaned requests from the previous one are still in flight, exceeding configured concurrency exactly under saturation.
3. Sparse results pass: the embed fn pre-sizes its output array, so a provider response missing indexes passes the length check and emits undefined vectors with zero failure receipts.
4. Identity and reconciliation are nominal: source/target identity is a URL-string comparison (two spellings of one host pass); reconciliation is count-only, so a same-store no-op or a target contaminated with foreign ids can return ok: true.
5. Dry-run mutates: getOrCreateCollection runs before the dry-run branch.

The Fix (successor contract)

1. Status-class fate map, classifier-owned: provider failures carry their cause (http-401/403/404 → terminal-config; http-408/429/5xx, network, timeout → retryable; malformed/sparse result → retryable with its own reason; unknown → terminal, never optimistic). The grouped receipt keys on the full reason so cause survives aggregation.
2. Globally bounded, drained retry: one attempt budget for the whole operation (not per range); a failed batch attempt drains its in-flight siblings before any retry fires; backoff between attempts; concurrency never exceeds its configured bound. Split isolates content, retry absorbs transients, and sustained failure STOPS with a stopped-early receipt naming the budget spent.
3. Result validation: per-element vector presence + dimension check against the expected model dimension; sparse/duplicate indexes are per-document failures with receipts, never silent undefineds.
4. Provenance + exact reconciliation: semantic identity check (collection UUID, not URL strings) refuses source==target; the receipt records target endpoint, collection UUID, embedding model id, and dimension; reconciliation is id-set equality (planned ⊆ target, foreign ids reported), not counts.
5. CLI/receipt contract (operator surface): documented flags with validated ranges, stderr = progress, stdout = exactly one receipt JSON, exit 0 only on ok: true, and a true no-write dry-run (no collection creation).

Salvaged from the closed PR (base of the replacement): classifier-owned retryable stamp with unknown=false, grouped reason receipts with capped sampleIds, attempts: 1 compatibility default on the shared extractor, injected wait/backoff seams, transient-recovery coverage.

Contract Ledger Matrix

Decision Record impact: none (script/helper layer; no config leaves — runner stays AiConfig-free by design).

Acceptance Criteria

• Fate map: 401/403/404 → terminal-config; 429/5xx/network/timeout → retryable; unknown → terminal; receipts preserve the status class through grouping.
• Persistent-outage bound: an always-failing batch spends at most the global budget (spec asserts the exact call ceiling) and stops with a stopped-early receipt; no attempt starts while prior in-flight requests are undrained.
• Sparse/duplicate/wrong-dimension provider results produce per-document failure receipts, never undefined vectors in the target.
• Same-store source/target refused semantically (collection UUID); receipt carries endpoint+UUID+model+dimension provenance; reconciliation is id-set based and reports foreign ids.
• Dry-run performs zero writes of any kind (no collection creation) — witnessed by a client mock that throws on any mutating call.
• CLI flags validated with a documented exit-code contract; stdout carries exactly the receipt JSON.
• Replacement PR links the #16228 closing review's salvage map and the #16208 correction comment, and resolves THIS ticket only — #16208 closes on its own authority (source boundary + cutover + live reconciliation), not through this runner.

Out of Scope

• Deciding the #16208 source boundary (graph+WAL vs clone migration) — that decision is consumed here, made there.
• The shared ai/-wide retry primitive consolidation (#16224-family) beyond what the extractor already carries.
• Cutover blessing, host-stopgap removal, live-plane reconciliation (#16208).

Avoided Traps

• One retryable bit as the whole fate story — a fate bit is only as useful as its cause classifier; collapsing 401/429/timeout/5xx into one reason recreates "unrecoverable" ambiguity under a new label (review retrospective, adopted verbatim).
• Per-range fresh retry budgets — retries must be globally bounded or sustained failure multiplies pressure on the already-saturated provider.
• Count-based reconciliation — counts bless no-ops and contamination; only id-sets prove the rebuild.
• Two named fate fields in the receipt schema — array-of-receipts with a fate attribute stays (v1 convergence, still right).

Related

• #16208 — source-boundary + cutover authority; the correction comment (2026-07-31T03:52Z) is binding on any successor runner
• PR #16228 — closed unmerged (Drop+Supersede); its closing review carries the salvage map and falsifier harnesses
• #16222, #16224 — bounded-retry family siblings

Origin Session ID: 5814af6b-fe4e-41ba-819f-e1aeb5558643 Retrieval Hint: "successor runner fate map status class drained bounded retry id-set reconciliation provenance dry-run no-write"