Problem
The .neo-ai-data cross-clone SSOT topology symlinks each clone's substrate subdirs → the canonical (github) clone via symlinkDataDir in ai/scripts/migrations/bootstrapWorktree.mjs. The linked set is a static whitelist — DATA_SUBDIRS_TO_LINK = [sqlite, chroma, wake-daemon, backups, datasets, neo-sqlite].
memory-wal was never added to that whitelist. Result: every non-canonical clone (claude, opus-vega, codex, antigravity, clio, fable) wrote its add_memory WAL to its OWN un-drained .neo-ai-data/memory-wal, invisible to the single github drainer → ~2,500 memory records across 6 clones un-embedded for ~8 days (semantic recall frozen since ~06-10). Not a code regression (the github daemon is healthy + draining its own dir) — the whitelist's structural failure mode: a new substrate subdir is silently un-linked until someone remembers to add it.
The Fix (operator directive: "blacklist, not whitelist")
Invert to a blacklist: symlinkDataDir enumerates the canonical .neo-ai-data children and symlinks every one EXCEPT a small blacklist — initially just concepts (the only git-tracked item; .gitignore:103 !.neo-ai-data/concepts/ IS the boundary SSOT). New subdirs (memory-wal + anything future) link automatically → drift-proof.
Contract / API change
- Replace
export const DATA_SUBDIRS_TO_LINK (allowlist) with export const DATA_SUBDIRS_BLACKLIST = ['concepts'].
symlinkDataDir({mainCheckout, projectRoot, blacklist = DATA_SUBDIRS_BLACKLIST, force, log}): enumerate <canonical>/.neo-ai-data children (dirs and files), skip blacklist, symlink the rest (dir→'dir', file→'file'). Graceful when canonical .neo-ai-data is absent (fresh repo).- Preserve ALL existing semantics: parent
.neo-ai-data stays a real dir (never symlinked → protects tracked concepts/), idempotent already-linked skip, force-guard (refuse to clobber a non-symlink without force), skip-no-source, mainCheckout no-op, per-item result buckets.
Affects ("ensure new repo setups use it")
- Git worktrees:
hydrateCurrentWorktree / --link-data (same code path).
- New-peer onboarding: independent-clone setup via
--canonical-root (same code path) → a fresh clone gets memory-wal + all substrate linked by construction.
Test plan
Update bootstrapWorktree.spec.mjs #10432 symlinkDataDir: seed fixture subdirs in fake-main, call without an allowlist, assert all-seeded-except-blacklist linked; KEEP the load-bearing NEVER touches concepts/ safety test; ADD a regression guard that a previously-non-whitelisted subdir (memory-wal) IS now linked (the guard for THIS bug). UNIT_TEST_MODE=true unit suite green.
Sequencing (cross-lane — critical)
The memory-wal symlink flip must NOT be applied on live clones until the per-append WAL write-lock (Part 2, sibling leaf, @neo-opus-vega) is live on every clone's MC — else an un-locked clone writes the shared wal-DATE.jsonl concurrently and corrupts (one-writer-per-segment invariant; getWalSegmentKey is UTC-date-only, not clone-unique). This ticket is the code; the operational flip + the ~2,500-record one-shot recovery are coordinated separately (ephemeral, NOT committed — no repo bloat).
Out of scope
- Per-append WAL write-lock (@neo-opus-vega).
- One-shot recovery merge of the orphaned backlog (ephemeral op, with @tobiu).
- WAL-backlog liveness watchdog (@neo-opus-vega).
Refs #13495, #12864.
Problem
The
.neo-ai-datacross-clone SSOT topology symlinks each clone's substrate subdirs → the canonical (github) clone viasymlinkDataDirinai/scripts/migrations/bootstrapWorktree.mjs. The linked set is a static whitelist —DATA_SUBDIRS_TO_LINK = [sqlite, chroma, wake-daemon, backups, datasets, neo-sqlite].memory-walwas never added to that whitelist. Result: every non-canonical clone (claude, opus-vega, codex, antigravity, clio, fable) wrote itsadd_memoryWAL to its OWN un-drained.neo-ai-data/memory-wal, invisible to the single github drainer → ~2,500 memory records across 6 clones un-embedded for ~8 days (semantic recall frozen since ~06-10). Not a code regression (the github daemon is healthy + draining its own dir) — the whitelist's structural failure mode: a new substrate subdir is silently un-linked until someone remembers to add it.The Fix (operator directive: "blacklist, not whitelist")
Invert to a blacklist:
symlinkDataDirenumerates the canonical.neo-ai-datachildren and symlinks every one EXCEPT a small blacklist — initially justconcepts(the only git-tracked item;.gitignore:103!.neo-ai-data/concepts/IS the boundary SSOT). New subdirs (memory-wal+ anything future) link automatically → drift-proof.Contract / API change
export const DATA_SUBDIRS_TO_LINK(allowlist) withexport const DATA_SUBDIRS_BLACKLIST = ['concepts'].symlinkDataDir({mainCheckout, projectRoot, blacklist = DATA_SUBDIRS_BLACKLIST, force, log}): enumerate<canonical>/.neo-ai-datachildren (dirs and files), skip blacklist, symlink the rest (dir→'dir', file→'file'). Graceful when canonical.neo-ai-datais absent (fresh repo)..neo-ai-datastays a real dir (never symlinked → protects trackedconcepts/), idempotent already-linked skip, force-guard (refuse to clobber a non-symlink withoutforce), skip-no-source, mainCheckout no-op, per-item result buckets.Affects ("ensure new repo setups use it")
hydrateCurrentWorktree/--link-data(same code path).--canonical-root(same code path) → a fresh clone getsmemory-wal+ all substrate linked by construction.Test plan
Update
bootstrapWorktree.spec.mjs#10432 symlinkDataDir: seed fixture subdirs in fake-main, call without an allowlist, assert all-seeded-except-blacklist linked; KEEP the load-bearingNEVER touches concepts/safety test; ADD a regression guard that a previously-non-whitelisted subdir (memory-wal) IS now linked (the guard for THIS bug).UNIT_TEST_MODE=trueunit suite green.Sequencing (cross-lane — critical)
The
memory-walsymlink flip must NOT be applied on live clones until the per-append WAL write-lock (Part 2, sibling leaf, @neo-opus-vega) is live on every clone's MC — else an un-locked clone writes the sharedwal-DATE.jsonlconcurrently and corrupts (one-writer-per-segment invariant;getWalSegmentKeyis UTC-date-only, not clone-unique). This ticket is the code; the operational flip + the ~2,500-record one-shot recovery are coordinated separately (ephemeral, NOT committed — no repo bloat).Out of scope
Refs #13495, #12864.