LearnNewsExamplesServices
Frontmatter
id14193
titleField↔document de-dup (~910MB) — consumer-aware refactor to one canonical representation
stateOpen
labels
enhancementaiarchitecture
assigneesneo-opus-ada
createdAtJun 27, 2026, 5:12 AM
updatedAtJul 2, 2026, 7:26 PM
githubUrlhttps://github.com/neomjs/neo/issues/14193
authorneo-opus-ada
commentsCount9
parentIssue14079
subIssues
14203 composeTurnDocumentText pure helper — single-source the turn-document derivation (slice 1 of the #14193 de-dup)
14206 #14193 slice-2: single-source the MemoryService turn-document build to composeTurnDocumentText
14211 De-dup completeness: route DreamService + GoldenPathSynthesizer turn-document reads through resolveTurnDocumentForRead before slice-4 drops documents
subIssuesCompleted3
subIssuesTotal3
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]

Field↔document de-dup (~910MB) — consumer-aware refactor to one canonical representation

Open Backlog/active-chunk-2 enhancementaiarchitecture
neo-opus-ada
neo-opus-ada commented on Jun 27, 2026, 5:12 AM

Sub of #14079 (Memory Core Chroma 2.5GB bloat) — the largest slice, and the careful one.

The duplication (V-B-A confirmed, 2026-06-27)

Memory rows store the turn text TWICE: the split metadata fields (prompt/thought/response) AND chroma:document (≈ the joined split-fields). Both are CONSUMED — naive removal of either breaks retrieval/summarization/display:

  • Split fields read at MemoryService.mjs:955 (results), :1704 (summarization), :1863 (results), :1692 (presence check); documented as the full metadata payload in memoryWalStore.mjs.
  • chroma:document read via include:['documents'] at MemoryService.mjs:1922/2016, SummaryService.mjs:286/310/375, SessionService.mjs:529/725, DatabaseService.mjs:94/114.

So this is a consumer-aware refactor (pick ONE canonical representation, migrate every read path), NOT a storage-only delete — ~910MB of the bloat.

Scope (careful, sequenced — NOT a single rushed PR)

  1. Characterize: a test pinning the current dual-storage + every consumer above (the migration safety net).
  2. Choose canonical: document (Chroma-native, one include:['documents'] read) vs the split fields (already structured). Decide + record the rationale.
  3. Migrate consumers to the canonical read path (one PR per cluster, or a shared accessor).
  4. Stop writing the dup on new rows; backfill/migrate existing rows (governed, ADR-0027 envelope).
  5. Reclaim (VACUUM, after the dup column is gone — coordinate with the VACUUM slice).

Acceptance Criteria

  • One canonical representation; every read path above migrated, with the characterization test green throughout.
  • New rows store the turn text once; existing rows migrated under a governed envelope (reversible, backed up).
  • No retrieval/summarization/display regression (behavior tests over the live collections).

Refs #14079 (parent). Coordinate with @neo-opus-grace (the MemoryService/kbSync owner). Sibling slices: #14192 (FTS index), the SQLite VACUUM, the abandoned-shadow promotion.


🤖 Authored by Ada (@neo-opus-ada · Claude Opus 4.8, Claude Code). The "both copies are consumed" V-B-A corrects the parent ticket's "redundant → removable" framing.

tobiu referenced in commit ef32808 - "feat(ai): composeTurnDocumentText helper — single-source the canonical turn-document derivation (#14193) (#14202) on Jun 27, 2026, 11:08 AM
tobiu referenced in commit 1f27a2d - "refactor(ai): single-source the MemoryService turn-document build via composeTurnDocumentText (#14206) (#14207) on Jun 27, 2026, 12:28 PM
tobiu referenced in commit 3400050 - "feat(ai): reconstruct turn-document on read from split metadata (#14209) (#14210) on Jun 27, 2026, 5:07 PM
tobiu referenced in commit 744448e - "fix(ai): route DreamService REM turn-document read through resolveTurnDocumentForRead (#14211) (#14213) on Jun 27, 2026, 6:07 PM
tobiu referenced in commit 82255a3 - "fix(ai): reconstruct dropped turn-documents in the embed-repair re-embed — recover de-duped turns (#14218) (#14219) on Jun 27, 2026, 6:07 PM