LearnNewsExamplesServices
Frontmatter
id14102
titleData-integrity detect: embedding-dimension consistency producer (#14026 leaf)
stateClosed
labels
enhancementaitestingarchitecture
assigneesneo-opus-ada
createdAtJun 26, 2026, 4:14 PM
updatedAtJun 26, 2026, 4:42 PM
githubUrlhttps://github.com/neomjs/neo/issues/14102
authorneo-opus-ada
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 26, 2026, 4:42 PM

Data-integrity detect: embedding-dimension consistency producer (#14026 leaf)

Closed v13.1.0/archive-v13-1-0-chunk-6 enhancementaitestingarchitecture
neo-opus-ada
neo-opus-ada commented on Jun 26, 2026, 4:14 PM

Context

A further leaf of the #14026 data-integrity detect-signal class, parallel to coverage-drift (#14075), vector-count monotonicity (#14094), and SQLite-integrity (#14096). Coordinated leaf-division — claimed non-wakeSuppressed per #14100.

The Problem

Every stored Memory Core vector must be the configured embedding dimension (e.g. 4096). A vector of the wrong dimension is corruption — it breaks similarity search and is unambiguous (unlike a count regression, there is no legitimate "wrong-dim" case). No detect signal currently flags dimension drift across the corpus.

The Fix

A pure detect-producer mirroring dataIntegrityCoverageDiagnosis.mjs / vectorCountMonotonicityDiagnosis.mjs: buildDimensionConsistencyDiagnosis({samples, observedAt, serviceId}) → emits a data-integrity / escalate recovery-diagnosis when any collection reports mismatchedVectorCount > 0; null when every sampled vector matches the expected dimension. Detect-only — samples (the per-collection dimension audit) are injected; the audit + scheduling are the consumer's concern.

Contract Ledger

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
buildDimensionConsistencyDiagnosis({samples, observedAt, serviceId}) (new pure producer) #14026 detect-signal class + ADR-0025 §2.4 (#14089) Per-collection {collection, expectedDimension, mismatchedVectorCount} samples → a recovery-diagnosis when any mismatchedVectorCount > 0; null when all match Non-finite/absent mismatchedVectorCount ignored; empty samples → null; missing serviceId / non-finite observedAt throw TypeError producer JSDoc + ticket unit spec
recovery-diagnosis payload createRecoveryDiagnosisEvent (recoveryRunStateStore.mjs) recoveryClass: 'data-integrity', details.actionClass: 'escalate', targetIdentity: {kind:'compose-service', id: serviceId}, evidenceFacts: [{type:'vector-dimension-mismatch', collection, expectedDimension, mismatchedVectorCount}], source: 'data-integrity-dimension-monitor', reasonCode: 'data-integrity-dimension-mismatch' Reuses the merged #14075 contract — no new enum values producer JSDoc mirrors #14075/#14094 payload contract
Consumer boundary configured embedding dimension (localModels.embedding) A wrong-dimension vector is unambiguous corruption (no legitimate case — unlike the monotonicity deletion caveat). Escalate-only; no auto-repair. The dimension audit + cadence are the future scheduled-consumer's concern. n/a ticket Out-of-Scope

Acceptance Criteria

  • A collection with mismatchedVectorCount > 0 → a contract-valid data-integrity/escalate diagnosis; all-matching samples → null.
  • Pure (no I/O); samples injected; diagnosisId target+instant scoped.
  • Unit coverage: mismatch → diagnosis; clean → null; non-finite/absent counts ignored; arg-rejection.

Out of Scope

  • The dimension audit + scheduled consumer (separate slice, like #14075's scheduling).
  • The other deferred #14026 heuristics (store-bloat #Grace, cross-collection, exportability-canary, WAL-age, orphan).

Related

Authored by Ada (Claude Opus 4.8, Claude Code).