LearnNewsExamplesServices
Frontmatter
id16963
titleA failed vector write re-buys identical embeddings on every retry
stateClosed
labels
bugaiperformanceagent-os
assigneesneo-opus-ada
createdAtAug 11, 2026, 1:44 PM
updatedAtAug 11, 2026, 3:04 PM
githubUrlhttps://github.com/neomjs/neo/issues/16963
authorneo-gpt
commentsCount0
parentIssue16780
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 11, 2026, 3:04 PM

A failed vector write re-buys identical embeddings on every retry

Closed Backlog/active-chunk-15 bugaiperformanceagent-os
neo-gpt
neo-gpt commented on Aug 11, 2026, 1:44 PM

Context

#16780 requires persistence failures to stop causing unbounded repeated embedding work. #16875 and merged PR #16876 made that non-convergence observable, but they explicitly left retaining vectors across a failed write out of scope.

Current dev still places TextEmbeddingService.embedTexts() and collection.upsert() in the same retry try inside VectorService.embedChunks(). A transient vector-write failure therefore discards an already-successful embedding result and re-submits the identical texts to the provider on the next retry.

A measured two-text / three-write-attempt falsifier produces 6 provider submissions on current dev; retaining the successful result across write-only retries produces 2. The implementation prototype is on ada/16780-embed-paid-once at 9ea462ac02cc91f9e963ba79fdeb97208a0dd69b and must be rebased onto current dev before PR creation.

Memory mining surfaced the prior #16875 decision record (session 87f453f9-aa80-4487-9ed1-b5d91e052c43): detection was delivered, while survival across a failed write was deliberately deferred. A final duplicate sweep at 2026-08-11T11:44:26.702Z covered the 20 newest open GitHub issues, the 30 newest all-state A2A messages, and the local issue/discussion corpus; it found no separate live owner for this repair.

This ticket does not claim to explain or cure the permanent 400% Ollama CPU incident. It removes a separately verified redundant-work multiplier.

The Problem

The retry boundary currently conflates two different failures:

  1. the embedding provider failed, so provider work must be retried;
  2. embedding succeeded but persistence failed, so only the write must be retried.

Treating both as one attempt re-buys deterministic vectors, inflates provider activity, and can multiply expensive batch work by maxRetries while the collection makes no progress.

Architectural Reality

  • textsToEmbed and ids are already stable for the batch.
  • A successful embedding array is valid input for each retry of the same collection.upsert().
  • JavaScript assignment does not occur when the awaited provider promise rejects, so a nullable result cached with ??= preserves ordinary provider-failure retries.
  • result.embedded must still advance only after persistence succeeds.
  • The existing VectorService.leaseYield.spec.mjs negative control proves an ordinary embedding failure remains retryable.
  • This is a local retry-boundary correction: no AiConfig, public API, worker topology, schema, or service-ownership change.

The Fix

  1. Hoist a nullable embedding result outside the persistence retry loop.
  2. Resolve it with embeddings ??= await TextEmbeddingService.embedTexts(...).
  3. Reuse that exact result for write-only retries.
  4. Preserve provider retry behavior when embedding itself rejects.
  5. Add a mutation-sensitive production-path test at the provider-submission boundary.

Contract Ledger

Surface Required behavior Edge / fallback Evidence
VectorService.embedChunks() provider step A successfully computed batch is purchased once across retries of its matching write. A provider rejection leaves no cached result and remains retryable. New transient-upsert falsifier + existing ordinary-provider-failure retry control.
collection.upsert() Continues to retry according to maxRetries, using the same ids, metadata, and vectors. Exhausted write retries retain existing failure accounting. Three-attempt write fixture.
result.embedded Advances only after the matching upsert succeeds. Failed persistence never reports embedded progress. Receipt assertion after the third successful write.

Decision Record Impact

None. This narrows an existing retry boundary without changing configuration authority, public contracts, or architectural ownership.

Acceptance Criteria

  • Two texts with two transient upsert failures and success on the third attempt produce exactly 3 upsert attempts, 2 provider submissions, and embedded: 2.
  • Moving provider computation back inside the write retry loop makes the new test fail with 6 provider submissions.
  • An ordinary embedding-provider rejection is still retried; the persistence repair does not cache failure or suppress a required provider attempt.
  • Exhausted persistence retries preserve current failure accounting and do not report false embedded progress.
  • The focused VectorService suites and the Knowledge Base unit directory pass on the rebased exact head.

Post-Merge Validation

  • During a controlled transient vector-write rejection, provider activity records one logical submission per selected text while the write attempts retry independently.

Out of Scope

  • Diagnosing or curing the permanent 400% Ollama runner.
  • Explaining why the vector database rejected a write.
  • Cross-sweep non-convergence detection already delivered by #16875 / PR #16876.
  • Provider timeout or saturation retry behavior tracked by #16012.
  • Canary, admission, model-residency, or upstream Ollama cancellation policy.

Avoided Traps

  • Do not present reduced duplicate work as proof of the 400% incident's cause.
  • Do not cache a rejected provider promise or a partial/invalid result.
  • Do not remove bounded write retries.
  • Do not broaden this leaf into a pipeline or provider-policy rewrite.

Related

  • Parent: #16780
  • Prior detection-only leaf: #16875 / PR #16876
  • Distinct provider-timeout retry mechanism: #16012
  • Prototype branch: ada/16780-embed-paid-once @ 9ea462ac02

Origin Session ID: ef6172cf-3843-4358-aa46-6275e47c9b87

Retrieval Hint: VectorService persistence retry re-buys identical embeddings upsert succeeds provider submissions paid once

Measured and prototyped by Ada (Claude Opus 5, Claude Code); leaf authored by Euclid (GPT-5.6, Codex Desktop).

tobiu referenced in commit d25c397 - "fix(knowledge-base): a failed vector write no longer re-buys identical embeddings (#16963) (#16964) on Aug 11, 2026, 3:04 PM
tobiu closed this issue on Aug 11, 2026, 3:04 PM