Backup: atomic timestamped bundle across all persistent subsystems

Context

Flagged during #10000 pr-review (PR #10128) by @tobiu: the existing buildScripts/ai/exportDatabase.mjs has structural gaps surfaced while running a safety backup before the tenant-isolation polish tests. Initial ticket framing was narrow ("add KB to the MC-only exporter"). A codebase exploration requested by @tobiu afterward surfaced buildScripts/ai/defragChromaDB.mjs, which already creates physical-copy backups of both KB and MC as a side-effect of its Nuke-and-Pave strategy (introduced via #8490 on 2026-01-10). That redefines the actual gap, and the ticket scope widens accordingly.

Architectural driver (added 2026-04-20, follow-up session): The root problem is discoverability, not just coverage. When the backup mechanism is hidden as a defrag side-effect, a future agent (or the repo owner himself, as literally happened in the originating session when he pointed at a .neo-ai-data/backups/chroma/ path that didn't exist in the fresh worktree) cannot find it without archaeology. Backup must be a first-class, obviously-named, top-level script — visible to ls buildScripts/ai/ and grep backup. This reframes Phase 3 below: defrag and backup are peer scripts with orthogonal responsibilities, not a delegation chain.

The Problem

Problem A — fragmented backup layout

Today's backup state spans two roots with mixed formats:

• .neo-ai-data/backups/ — flat JSONL files at root:
  • memory-backup-<timestamp>.jsonl
  • summaries-backup-<timestamp>.jsonl
  • graph-backup-<timestamp>.jsonl
• dist/chromadb-backups/<target>/backup-<timestamp>/ — physical directory copies (KB + MC), only created as a defrag side-effect
• No coverage at all for two subsystems that do contain important state:
  • Concepts (.neo-ai-data/concepts/nodes.jsonl, edges.jsonl — the Concept Ontology from #10030)
  • Trajectories (.neo-ai-data/datasets/rlaif/trajectories.jsonl — RLAIF training data)

One "full-system snapshot" requires correlating a JSONL at one root with a physical copy at another root — and you have to actually run defrag to get the physical copies. Inconsistent timestamps across the two roots mean a restore operation may stitch data from different points in time.

Problem B — SDK bypass + no KB JSONL export

buildScripts/ai/exportDatabase.mjs imports LifecycleService and DatabaseService directly from ai/mcp/server/... rather than routing through ai/services.mjs — bypassing the Zod runtime validation applied by makeSafe(). And the KB has no manageDatabaseBackup equivalent surface at all, so there's no JSONL export option for it.

Problem C — defrag and backup are tangled (discoverability failure)

defragChromaDB.mjs does three independent things in one script:

1. Backup (physical copy, as safety step)
2. Defragment (nuke-and-pave ETL)
3. Cleanup (vacuum + orphan folder removal)

The backup step is implicit and coupled to the defrag action, and the resulting physical-copy artifacts live in dist/chromadb-backups/ — a path that is not discoverable when an operator asks "where does the backup script live?". Operators who want "just back up, no defrag" have no obvious affordance.

The Architectural Reality

Subsystems requiring backup coverage

Existing scripts

• buildScripts/ai/exportDatabase.mjs — MC-only, direct imports, flat output
• buildScripts/ai/defragChromaDB.mjs — defrag + side-effect physical backup, dist/chromadb-backups/ output
• buildScripts/ai/importBackupToSQLite.mjs — MC restore, hardcoded-path legacy

The Fix

Three coordinated changes.

Phase 1 — Unified timestamped bundle structure

Replace the current fragmented layout with a single atomic-snapshot convention:

.neo-ai-data/backups/
└── backup-<ISO-timestamp>/
    ├── kb/                 # KB ChromaDB JSONL export
    ├── mc/                 # MC memories + summaries JSONL export
    ├── graph/              # MC SQLite graph JSONL export
    ├── concepts/           # nodes.jsonl + edges.jsonl file copies
    └── trajectories/       # trajectories.jsonl file copy

One timestamp = full system snapshot. Restore semantics are obvious (pick a backup folder → all subsystems restore to the same point in time). Retention policy is a single directory-level sweep instead of 5 separate file patterns.

Phase 2 — Backup script as canonical, discoverable entrypoint

Rename buildScripts/ai/exportDatabase.mjs → buildScripts/ai/backup.mjs (preserve history via git mv) and widen its scope:

• Imports exclusively from ai/services.mjs (Zod-validated SDK boundary)
• Orchestrates KB + MC + graph + concepts + trajectories into a single backup-<timestamp>/ bundle
• Exposes npm run ai:backup in package.json
• Non-destructive: captures current DB state, whatever shape it is in. Does not trigger defrag or any compaction step.

Prerequisite: add manageDatabaseBackup({action: 'export'}) to KB_DatabaseService + register in KB's openapi.yaml so ai/services.mjs applies the makeSafe Zod wrapper.

Export methods for concepts + trajectories can be new helpers in the services layer (or pure file-copy operations in the script — whichever fits the substrate boundary cleanest).

Phase 3 — Defrag and backup as peers (not delegation)

Earlier drafts of this ticket proposed that defragChromaDB.mjs delegate its pre-nuke snapshot to the canonical backup script. That is now explicitly rejected. Delegation would:

1. Re-create the exact discoverability failure this ticket addresses (backup becomes a defrag side-effect again, just at a different layer).
2. Couple the fast physical-copy safety snapshot — which preserves exact HNSW index state — to the slower JSONL-export pipeline.
3. Introduce a latent circular-dependency risk if backup.mjs ever gains pre-backup compaction logic.

Instead, backup.mjs and defragChromaDB.mjs stay as peer scripts with orthogonal responsibilities:

Operators who want a compacted backup for release artifacts compose at the shell layer: npm run ai:defrag && npm run ai:backup. Two independent commands; no recursion, no coupling. If that chain is common enough to warrant sugar, a thin npm run ai:release-snapshot composition wrapper can be added later — but that is shell-level orchestration, not architectural coupling.

The physical-copy path at dist/chromadb-backups/ stays as defrag-exclusive pre-nuke snapshot. HNSW index state preservation has value beyond JSONL re-ingest (instant restore if defrag nuke-and-pave fails mid-flight) and keeps the fast physical-copy path cheap.

Acceptance Criteria

• KB_DatabaseService exposes manageDatabaseBackup({action: 'export'}); registered in KB's openapi.yaml.
• buildScripts/ai/exportDatabase.mjs renamed to backup.mjs via git mv (history preserved); produces a backup-<ISO-timestamp>/ bundle containing kb/, mc/, graph/, concepts/, trajectories/ subfolders.
• Script imports exclusively from ai/services.mjs — no direct ai/mcp/server/... imports.
• Script is non-destructive: captures current DB state without triggering defrag or any compaction step.
• npm run ai:backup invokes the script.
• defragChromaDB.mjs retains its private pre-nuke physical-copy snapshot helper; it does not call backup.mjs. Backup and defrag are peer scripts with orthogonal responsibilities.
• Playwright unit coverage for KB_DatabaseService.manageDatabaseBackup (happy path + empty collection).
• Integration assertion (Playwright or documented manual steps) that the script produces all 5 subfolders with non-empty content when the source subsystems have data.
• Retention policy documented inline + at least a TODO for implementation (delete backup folders older than N days, keep last K — same semantics as defragChromaDB.cleanOldBackups but applied to the unified tree).
• Legacy migration: existing flat-file backups at .neo-ai-data/backups/ and physical copies at dist/chromadb-backups/ are either migrated into the new structure or explicitly documented as "archive, preserve" in a README.

Out of Scope

• Restore tooling — separate ticket. This ticket establishes the atomic-bundle layout; restore is a downstream consumer.
• Automatic / scheduled backup triggers beyond defrag's existing pre-nuke call — e.g. pre-release snapshots, cron-driven snapshots. Establish the script first; automation layer is a separate concern.
• Cross-machine backup sync / cloud upload — local-disk layout only.
• Trajectory / concept backup format choices — trivial file-copy is sufficient; no need to convert to a different format.
• GitHub Workflow / Neural Link backup coverage — those MCP servers are stateless protocol adapters without persistent data stores.
• ai:release-snapshot composition wrapper — optional shell-level sugar for the defrag && backup chain; defer until that 2-step compose pattern proves common in practice.

Related

• Surfaced during #10000 pr-review (PR #10128) safety-backup step on 2026-04-20
• Consumes ai/services.mjs SDK pattern from #9922 Two-Pillar Hybrid RAG
• Defrag backup history: #8490 (2026-01-10) introduced dist/chromadb-backups/ physical copies
• Complements #10001 / #10007 / #10000 multi-tenant deployment target — operators will run the backup script as part of their deployment rollback safety net
• Concept subsystem backup dependency: #10030 (Concept Ontology) defines the JSONL files to be backed up
• Trajectory subsystem backup dependency: RLAIF dataset path from aiConfig.datasets.rlaif.trajectories

Origin Session ID: 1c001810-be28-4554-bb56-c98f9b91bbfb Architectural revision session: 5a521819-dc75-4549-888e-fcea818d0401 (peer-architecture lockdown — Phase 3 rewritten to reject defrag→backup delegation on discoverability + coupling grounds)