Rows embedded before the provider-header fix keep their stale vectors, and no signal can find them

Context

Filed on @neo-gpt's Required Action 3 against PR #17426 (review): "give existing corrupted rows a concrete owner… The parent epic alone is not an executable residual." He was right — the residual on #17425 pointed at epic #17411, which had no child for this.

#17425 corrected the provider-input header: buildEmbeddingInputHeader read chunk.type, the published parsed-chunk-v1 schema requires kind and declares no type, so every chunk from a schema-conforming parser embedded behind the literal token undefined. That fix is forward-only by construction.

Live latest-open sweep: checked latest 20 open issues at 2026-08-20T17:31Z; A2A claim sweep over the current window; no equivalent found.

This body was materially corrected after filing — see The Architectural Reality. The original claimed a working global trigger; there is none.

The Problem

The affected rows are invisible to every existing signal, and the reason is structural rather than an oversight.

1. Their chunk ids did not change. createChunkHash folds only the hashInputs field list (['kind','name','content','sourcePath','parserId','parserVersion'], IngestionService.mjs:990). The provider input string is derived and is not a member, so re-ingestion computes the same id, finds the row present, and does no work.
2. The provider input is not stored. collection.upsert writes {ids, embeddings, metadatas} — there is no documents field, so the stale text is not on disk to be pattern-matched.
3. Metadata cannot date them. buildChunkMetadata copies every chunk field, so an affected row carries kind and no type — but so does a row embedded after the fix. The metadata identifies the affected population, never which members are stale.

So the symptom is: a bounded set of rows whose vectors were computed from a string beginning with a meaningless token, indistinguishable from correct rows by id, content, or metadata.

The Architectural Reality

⛔ CORRECTED 2026-08-20, hours after filing. The original version of this section was wrong, and it was the section the whole ticket rested on.

It claimed "the guaranteed trigger already exists" — that EMBEDDING_POISON_STRATEGY_FAMILY feeds createVectorGenerationIdentity, and that per generationElectionStore.mjs:7 a strategy change therefore "invalidates every existing vector". @neo-gpt falsified it on PR #17426 (closure review). Verified independently before accepting:

• strategyVersion is consumed by resolveEmbeddingPoisonGeneration (VectorService.mjs:551), whose own docblock scopes it precisely: "a provider, model, vector-schema, or input-strategy change invalidates prior poison evidence". It flows to createEmbeddingGenerationId and the poison-suppression scope. Nothing else.
• VectorService never calls createVectorGenerationIdentity. Its only occurrence in that file is inside a comment (:205) — the comment I quoted. It names that function for its id-uniqueness property, not as a call.
• Incremental selection collects existingIds and skips them (:2461-2481), so a present row is never re-embedded no matter what any generation says.

So no global re-embed trigger is wired. Bumping the strategy family would invalidate poison/suppression evidence and re-embed nothing.

How the error was made, since it is more reusable than the fact: I read a docblock's contract and a comment's mention as a wired mechanism, and never ran the one-line falsifier — "does anything in the KB path call createVectorGenerationIdentity?" — which returns a comment. Prose describing a guarantee is not evidence its producer emits it.

What actually exists

✅ RESOLVED BY THE OPERATOR 2026-08-20 — build the detector

The escalation this section used to carry is answered, and the framing it escalated was wrong in a way worth recording because the error is reusable.

I wrote three shapes "in ascending cost" and called the detector refuted. Both halves were mistakes:

1. I ranked by implementation cost while the real constraint is compute. Operator: "they spend DAYS on current embeddings. so if building a detector takes a couple of hours… so what?" A few developer hours against days of recomputation is not a trade-off, and calling the cheap-to-build option the expensive one inverted the decision.

2. I evaluated the detector against what is ALREADY stored, concluded no discriminator exists, and wrote "refuted". The question I should have asked is what is the cheapest thing to start storing. Same wrong-subject error as the one in The Architectural Reality above, one step along: there I read documentation instead of the call graph, here I read the current schema instead of the reachable one.

And the plane facts kill the global option's premise anyway. Operator: for neo's own Agent OS the multi-tenant repos present are the tiniest possible, and neo itself is not yet ingested as a tenant — it still runs the old kbSync path, which is a separate lane to change. So the "global bump rebuilds a large correct corpus" cost I used to argue against it barely exists here; the corpus it would invalidate is small.

The detector, which is cheap and was reachable all along

buildChunkMetadata (VectorService.mjs:175) copies every chunk field into row metadata. So a stored marker naming the provider-input format needs no new storage mechanism — it rides the existing copy.

Absence of the marker is the discriminator. Rows written before the marker exists carry no such field; rows written after carry the current value. That separates a stale row from a freshly-correct one with identical kind-without-type — which is the exact pair I claimed nothing on disk could distinguish.

The marker is a contract fact about the provider input, so its home is the authority that owns that string — helpers/embeddingInputFormat.mjs — not a second declaration beside it.

Retained from the refuted framing: the chunk id genuinely does not change (the input string is derived and not a hashInputs member), incremental selection genuinely does skip existing ids, and parserVersion genuinely is the only existing re-mint mechanism with nothing scheduling it. Those observations stand; only the conclusion drawn from them was wrong.

Contract Ledger Matrix

Intake amendment 2026-08-20 — absence is not queryable, so the detector scans

Added at ticket-intake on the drift probe firing. The chosen shape survives; one mechanism it would have been natural to reach for does not exist, and the repo had already learned this four times:

$ne does not rescue it either: a row missing the key is not returned by {marker: {$ne: current}}, because the clause mentions a field the row does not have. Absence is only observable by reading returned metadata.

This does not change what the detector must do — the Ledger row above already specified a behaviour rather than a query — but it fixes the cost model and names the shape:

• Detection is O(corpus) metadata reads, not a targeted query. For a ~87k-chunk corpus at the existing 2000-row page size that is ~44 pages of metadata, which is still trivial against days of embedding. The targeting benefit is in what gets re-embedded, not in what gets scanned.
• Two siblings own the shape and neither needs inventing. HealthService.#scanChromaMetadata is the scan-and-count precedent (and its #checkChromaMigrationState already reports an untagged population as observable debt — the same artifact this ticket's before/after count AC asks for). ai/scripts/migrations/backfillChromaSharedUserId.mjs is the runner precedent for tagging the rows a scan finds.

New acceptance criterion, from this amendment:

• The detector does not depend on a where filter to find absence. Selection reads returned metadata; an arm proves a marker-absent row is found. A version that filters on the marker key would silently return zero affected rows and read as "nothing to repair" — which is the failure mode of this entire lane, one layer down, and it is why this is an AC rather than a comment.

Decision Record impact

none for shape 1 (advance parserVersion). Shape 2 would need one: wiring a KB vector-generation election changes when vectors are considered invalid deployment-wide, and generationElectionStore's contract would move from describing elected generations to governing the KB embed path. That is an ADR-shaped decision, not a leaf's.

Acceptance Criteria

• The shape is chosen by the operator, on the record, before any implementation. — Resolved 2026-08-20: build the detector. See the section above, including the two errors in the framing I escalated.
• The marker's ABSENCE is the discriminator, asserted directly. A fixture with a pre-marker row and a freshly-written row carrying identical kind-without-type selects exactly the first. Without this arm the detector is indistinguishable from one that selects the whole affected population including rows it already repaired.
• The marker lives with the format it names. One authority — the module that owns the provider-input string — and no second declaration beside it. Asserted by mutation: changing the format without changing the marker must turn an arm red.
• Re-selection is idempotent. A repaired row is not selected again. A detector that re-selects its own output turns a bounded repair into a loop, which on a plane paying days of compute is worse than not repairing.
• The repair's ACs are NOT here. They moved to #17443 when the split was made — the RED-PROOF that a stale row receives a new vector, the EMBEDDING_POISON_STRATEGY_FAMILY no-op arm, before/after counting, and no-silent-partial-repair. Kept as an explicit line rather than a silent deletion, so a reader of this ticket can see the boundary rather than wonder whether the repair was forgotten.
• Neo's own corpus is accounted for explicitly. Its headers never changed; a repair that rebuilds it must say so as a cost, and one that skips it must prove the skip is safe.

Delivery status — the DETECTOR is this ticket; the REPAIR is #17443

Corrected 2026-08-21 (reviewer-named, @neo-gpt on PR #17444). An earlier revision of this section said "Not split into a child ticket, and not claimed as resolved. One PR advances this ticket; a second will close it." That was true when written and became false the moment #17443 was filed — and it left four repair ACs standing here that no detector PR could ever satisfy, so Resolves #17428 could not be truthful.

The split is real and the boundary is delivery authority: a read-only census is a leaf, while re-embedding is a plane operation that spends days of provider compute and whose scheduling is an operator decision. This ticket is now the detector alone, and its PR closes it. #17443 owns the repair. Handing over the number first is what makes the operator's scheduling decision informed rather than blind.

Closed by the detector PR:

• the marker's absence as the discriminator — a pre-marker row and a freshly-written row with identical kind-without-type classify differently, which is exactly the pair this ticket said nothing on disk could separate;
• the marker living with the format — and stronger than the AC asked: the identity is DERIVED from the format's own output over a probe set covering each of its branches, so "format changed, identity did not" is removed for every branch the probe set reaches rather than merely detected — bounded, not absolute: a branch no probe exercises can change invisibly, which the module states at its probe-set contract. Mutation-verified: reversing the header's type-first contract reddens both identity arms and no census arm;
• re-selection idempotence at the classification boundary — a row carrying the current identity is never selected, and the classifier is a pure function of the row, so there is no state that could flip it. The repair inherits this rather than re-implementing it;
• neo's own corpus, with a number instead of an adjective: 68,039 rows scanned, 68,039 pre-marker, 0 current on this deployment, measured through the shipped census;
• the scan-not-filter constraint from the intake amendment.

Still open, and owned by the repair PR:

• the RED-PROOF that a present, stale row actually receives a NEW VECTOR. This is the one that must not be satisfied by a hash or id difference, and it cannot be satisfied by a detector at all — nothing in this PR re-embeds anything;
• the arm pinning that an EMBEDDING_POISON_STRATEGY_FAMILY bump re-embeds nothing, so the corrected finding in The Architectural Reality stays behaviour rather than prose;
• no silent partial repair — a run that repairs some rows and stops must report the remainder with its count;
• the after half of the before/after count. The instrument exists and is verified; the second number requires a repair to have run.

Out of Scope

• The header fix itself — shipped in #17425 / PR #17426, forward-only by construction.
• Adding the provider input string to hashInputs. It would make future format changes self-repairing and it would also re-mint every chunk id in existence; that is a larger contract change with its own blast radius. Noted as the general fix that shape 2 approximates without the id churn.
• Storing documents in Chroma to make the text inspectable. Same class: a storage-shape change proposed to serve one migration.

Avoided Traps

• Reading "repair naturally on the next parserVersion advance" as a plan. parserVersion is a hash input, so an advance does re-mint ids and re-embed — but nothing schedules one. That sentence was in PR #17426's body and it was a hope with no trigger; the review was right to reject it.
• Assuming the stale rows are findable. They are not, by any of id, content, or metadata, and the targeted branch stands or falls on that.
• ⛔Reading a docblock's contract as a wired mechanism. This ticket shipped that error in its own most load-bearing section: generationElectionStore.mjs:7 describes what invalidates an elected generation, and VectorService.mjs:205 names createVectorGenerationIdentity — so I reported a guaranteed trigger that no code invokes. The falsifier was one grep for a call site, and it returns a comment. Prose describing a guarantee is not evidence that its producer emits it, and a function named in a comment is not a function that is called.

Related

• Parent epic: #17411. Sibling leaves: #17412 (throughput), #17413 (the lane guide).
• #17443 — the REPAIR split out of this ticket. Owns every AC about re-embedding; this ticket owns finding the rows.
• #17425 / PR #17426 — the forward-only header fix whose residual this owns.
• #17392 — parser-identity orphans; adjacent but keyed on identity change, which this population does not have.

Origin Session ID: 046f993e-13ba-47dd-827d-d786428e318b

Retrieval Hint: query_raw_memories("rows embedded before the provider header fix keep stale vectors; chunk id unchanged because embedding text is not in hashInputs; EMBEDDING_POISON_STRATEGY_FAMILY bump invalidates every vector globally versus a targeted detector that does not exist")