LearnNewsExamplesServices
Frontmatter
id10224
titleUnify .neo-ai-data across worktrees via bootstrap symlink
stateClosed
labels
enhancementaiarchitecturecore
assigneesneo-opus-ada
createdAtApr 23, 2026, 11:34 AM
updatedAtJun 7, 2026, 7:20 PM
githubUrlhttps://github.com/neomjs/neo/issues/10224
authorneo-opus-ada
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtApr 23, 2026, 11:43 AM

Unify .neo-ai-data across worktrees via bootstrap symlink

Closed v13.0.0/archive-v13-0-0-chunk-5 enhancementaiarchitecturecore
neo-opus-ada
neo-opus-ada commented on Apr 23, 2026, 11:34 AM

Context

The Claude Code worktree convention (.claude/worktrees/<name>/) mechanically isolates filesystem state per session — a genuine safety net for concurrent sessions, test artifacts, and mid-edit crashes, and the structural enforcement of pull-request-workflow §2.1. But the isolation extends beyond source code: each worktree ALSO gets its own .neo-ai-data/ directory, which contains the Memory Core's shared SQLite graph (memory-core-graph.sqlite), Chroma vectors, concept extraction byproducts, and JSONL backups.

This shatters ADR 0001's foundational assumption (§1): "Every MCP server process has its own singleton; processes share only the SQLite-WAL backing file at .neo-ai-data/sqlite/memory-core-graph.sqlite." Worktrees silently create N parallel substrates instead of one.

Empirical evidence from session 2581f466-d3ac-4a4a-a50e-5184b03ccca1 (2026-04-23, post-#10190-merge restart):

  • Main checkout .neo-ai-data/sqlite/memory-core-graph.sqlite: 48.9 MB (populated, AgentIdentity nodes seeded, active)
  • Worktree .neo-ai-data/sqlite/memory-core-graph.sqlite: 36 KB (schema-only, no AgentIdentity nodes, isolated)
  • Consequence: bindAgentIdentity('neo-opus-ada') returns null in the worktree → mailbox unbound → A2A blocked → the exact #10184 symptom, but with a different root cause than cache coherence.

#10176 §"Multi-harness dev" already codified the tactical convention: "the .neo-ai-data folder must be unified across checkouts (symlink is the current tactical convention)." This ticket automates that convention so every worktree bootstrap unifies the data plane by default.

The Problem

Three-layer problem analysis:

  1. Data plane split: every worktree boot creates/uses a fresh, empty .neo-ai-data/. AgentIdentity seed data, mailbox messages, concept ontology, and session memories all land in whichever substrate the MCP server happened to spawn under.
  2. ADR assumption violated: ADR 0001's cache-coherence fix (#10190) assumes one SQLite file shared across N processes. Worktrees create N SQLite files with one process each — the fix can't bridge physically separate files.
  3. Manual workaround doesn't scale: #10176 documented the symlink pattern, but requires each agent to execute rm -rf .neo-ai-data && ln -s ../../../.neo-ai-data .neo-ai-data per worktree. Humans forget; agents don't find it in the boot sequence unless hitting the symptom first.

Historical context from session 7ee23599-3f94-4f75-b54c-97ba92c9aaef (2026-04-22): Gemini initially proposed disabling worktrees entirely after hitting this pain; Opus pushed back with the symlink-resolves-it framing; Gemini accepted and withdrew the disable proposal. This ticket formalizes that architectural conclusion.

The Architectural Reality

Key files and service boundaries:

  • ai/scripts/bootstrapWorktree.mjs — existing module that copies gitignored config.mjs files from main into worktrees. Natural home for the new behavior; already well-tested.
  • ai/mcp/server/memory-core/config.mjs — resolves .neo-ai-data/ via path.resolve(neoRootDir, ...) where neoRootDir comes from __dirname of the config file itself. Since bootstrap copies config.mjs INTO the worktree, the worktree's config resolves .neo-ai-data/ to the worktree root. Symlinking the directory is transparent to path.resolve and better-sqlite3.
  • .neo-ai-data/ contents are pure data: SQLite DB files, Chroma vectors, JSONL exports, concept CSVs. Zero ESM imports — which is the load-bearing distinction from the existing "do not symlink" warning in bootstrapWorktree.mjs JSDoc (that warning is about src/core/Base.mjs and config.mjs, where canonical-path resolution matters for Node's namespace discipline).

The Fix

Extend ai/scripts/bootstrapWorktree.mjs with a symlinkDataDir helper that creates a worktree→main symlink for .neo-ai-data/:

export async function symlinkDataDir({mainCheckout, projectRoot, dir = '.neo-ai-data', force = false}) {
    if (path.resolve(projectRoot) === path.resolve(mainCheckout)) return 'main-checkout';
    const src = path.join(mainCheckout, dir);
    const dst = path.join(projectRoot, dir);

    const lstat = await fs.lstat(dst).catch(() => null);
    if (lstat?.isSymbolicLink()) return 'already-linked';
    if (lstat?.isDirectory()) {
        if (!force) throw new Error(`Refusing to replace non-symlink ${dst}; pass force=true (CLI --force) to opt in`);
        await fs.rm(dst, {recursive: true, force: true});
    }
    await fs.symlink(src, dst, 'dir');
    return 'linked';
}

CLI additions:

  • --link-data flag (opt-in initially) — runs symlinkDataDir after config copy.
  • --force flag — allows replacement of an existing worktree-local .neo-ai-data/ directory (data-loss guard opt-in).

JSDoc updates at the top of bootstrapWorktree.mjs:

  • Carve out the data-vs-code distinction: existing "Do NOT use symlinks" warning applies only to ESM-imported source code (src/core/Base.mjs, config.mjs). Data directories (.neo-ai-data/) have no canonical-path semantics to worry about.
  • Document the symmetry with #10176's tactical convention.

Test coverage additions to test/playwright/unit/ai/scripts/bootstrapWorktree.spec.mjs (if spec exists; else create):

  • Symlink creation when dst missing
  • 'already-linked' short-circuit when dst is already a symlink (idempotent)
  • Refusal when dst is a non-empty directory without force
  • 'main-checkout' no-op when run from main checkout
  • force: true clobbers existing non-symlink dir and creates the link

Acceptance Criteria

  • symlinkDataDir({mainCheckout, projectRoot, dir?, force?}) function exported from bootstrapWorktree.mjs
  • CLI --link-data flag wires symlink creation into the bootstrap pipeline
  • CLI --force flag enables overwrite of an existing non-symlink dir (gated behind explicit flag)
  • Refuses to clobber non-symlink directories without --force (data-loss guard)
  • Idempotent — re-running the script with --link-data on an already-linked worktree is a no-op
  • JSDoc at file head updated to carve out the data/code symlink distinction
  • Playwright unit tests cover: fresh-link, already-linked, non-symlink-dir-guard, main-checkout-noop, force-clobber
  • AGENTS_STARTUP.md §5 Worktree Bootstrap section updated to mention the --link-data flag
  • Dogfooded: the authoring session's worktree was unified via the new script; healthcheck returns populated mailboxPreview; list_messages returns without "no agent identity context bound" error (post-merge validation)

Out of Scope

  • Changing the default to enable symlinking automatically (requires one session's worth of empirical validation first; ship opt-in, flip to default in a follow-up).
  • Handling Chroma's dataDir (different mount point, not in .neo-ai-data/) — out of scope; Chroma is already shared via its external server process.
  • Windows symlink support — if the swarm ever includes Windows dev boxes, handle via follow-up; macOS + Linux are the current deployment surface.
  • Migration of worktree-local data written before symlinking — if a worktree had accumulated unique data, that's a one-off merge problem not worth automating.

Avoided Traps

  • "Just disable worktrees." Rejected (session 7ee23599, 2026-04-22 discussion). Worktrees provide mechanical main/dev branch-exclusivity (pull-request-workflow §2.1), filesystem isolation for crashes/test artifacts, and parallel-session safety. The data-isolation cost is a bug, not a feature — fix the bug, keep the benefits.
  • "Symlink src/ too." Rejected — the existing bootstrapWorktree.mjs JSDoc correctly warns against this. Node's ESM resolver walks to the canonical path of symlinked modules, causing Namespace collision in unitTestMode when the same namespace gets registered from two different worktrees' canonical paths. Applies to all ESM-imported code, NOT to data directories.
  • "Move .neo-ai-data/ out of the project root." Rejected — the config layer resolves it via path.resolve(neoRootDir, '.neo-ai-data/'), which is load-bearing for test harnesses and existing deployments. Relocating the directory is a 10× larger change for zero additional benefit.

Related

  • Predecessor (tactical convention): #10176 (Mailbox identity observability + Claude Desktop onboarding docs)
  • Adjacent substrate: #10190 (cache coherence — PR #10221, merged; ADR 0001 at learn/agentos/decisions/0001-cross-process-cache-coherence.md)
  • Parent Epic: #10186 (MCP concurrency audit + single-writer enforcement) — this ticket removes the worktree-induced concurrency-split that made the audit harder
  • Files: ai/scripts/bootstrapWorktree.mjs, test/playwright/unit/ai/scripts/bootstrapWorktree.spec.mjs (if exists; else create), AGENTS_STARTUP.md §5

Origin Session ID: 2581f466-d3ac-4a4a-a50e-5184b03ccca1

Handoff Retrieval Hints

  • query_raw_memories(query="worktree .neo-ai-data symlink bootstrap unify")
  • query_raw_memories(query="cross-harness MCP cache coherence worktree isolation")
  • query_summaries(query="A2A mailbox bootstrap worktree data plane")
  • Commit-range anchor: PR #10221 (substrate fix merged) + PR #10220 (broadcast-receipt merged) → this ticket (follow-up architectural unification)
tobiu referenced in commit ee24826 - "feat(bootstrap): granular per-subdir symlinking in --link-data (#10432) (#10433) on Apr 27, 2026, 6:15 PM