LearnNewsExamplesServices

Restoration Runbook

This document provides consistent recovery procedures for all persistent subsystems within the Neo.mjs AI substrate, leveraging the atomic bundles generated by the Daily Snapshot Pipeline (npm run ai:backup).

Bundle Layout Overview

Backups are stored in .neo-ai-data/backups/backup-<timestamp>/ and contain the following atomic directories:

  • kb/: Knowledge Base ChromaDB as JSONL
  • mc/: Memory Core memories and summaries as JSONL
  • graph/: Memory Core SQLite graph as JSONL
  • concepts/: Concept Ontology JSONL
  • trajectories/: RLAIF training trajectories JSONL

Prerequisites

Before initiating any restoration, ensure that all AI MCP servers and daemon processes are stopped (terminate any running npm run ai:server processes).

Restoration Procedures

1. Knowledge Base (KB)

The Knowledge Base is the neo-knowledge-base collection inside the one flat unified ChromaDB store (ADR 0017) — a cache, not a store. Recover it by deterministic rebuild from source, at collection scope. Never delete the store folder: chroma/unified also holds the irreplaceable Memory Core collections.

Procedure:

  1. Re-synchronize the KB from the source files (deterministic rebuild — clears and repopulates only the neo-knowledge-base collection):
       npm run ai:sync-kb
    (Note: Direct JSONL import of the kb/ bundle is deferred to #10871. Do not rm -rf the chroma/unified folder — it is shared with Memory Core; KB recovery is collection-scoped, handled by the rebuild above.)

2. Memory Core (MC) - Memories & Summaries

Memory Core memories and session summaries live as the neo-agent-memory and neo-agent-sessions collections inside the same flat unified ChromaDB store (ADR 0017). Unlike the KB, MC is the irreplaceable store — recover it from the backup bundle, at collection scope via the SDK. The pre-unification chroma/memory-core/ folder is retired.

Procedure:

  1. Re-import the MC JSONL from the backup bundle via the SDK (mode: 'replace' clears and repopulates the MC collections at collection scope — no folder deletion):
       node -e "import('./ai/services.mjs').then(s => s.default.memory.manageDatabaseBackup({action: 'import', file: '.neo-ai-data/backups/backup-<timestamp>/mc/memory-backup-<timestamp>.jsonl', mode: 'replace'}))"
    (Note: A dedicated restore.mjs CLI is deferred to #10871. Do not rm -rf the chroma/unified folder — it is shared with the Knowledge Base; MC restore is collection-scoped via the SDK above.)

3. Memory Core - Native Edge Graph

The Memory Core Edge Graph is persisted in SQLite.

Procedure:

  1. Move the corrupted SQLite database aside:
       mv .neo-ai-data/sqlite/memory-core-graph.sqlite .neo-ai-data/sqlite/memory-core-graph.sqlite.bak
  2. Re-import the Graph JSONL from the backup bundle via the SDK:
       node -e "import('./ai/services.mjs').then(s => s.default.memory.manageDatabaseBackup({action: 'import', file: '.neo-ai-data/backups/backup-<timestamp>/graph/graph-backup-<timestamp>.jsonl', mode: 'replace'}))"

4. Concept Ontology

The Concept Ontology consists of nodes and edges defined in JSONL.

Procedure:

  1. Clear the active concepts directory:
       rm -rf .neo-ai-data/concepts/*
  2. Copy the concepts from the backup bundle:
       cp -r .neo-ai-data/backups/backup-<timestamp>/concepts/* .neo-ai-data/concepts/

5. RLAIF Trajectories

The RLAIF trajectories capture interaction feedback and metadata for offline RL alignment.

Procedure:

  1. Replace the active trajectories file:
       cp .neo-ai-data/backups/backup-<timestamp>/trajectories/trajectories.jsonl .neo-ai-data/datasets/rlaif/trajectories.jsonl

Verification

After restoration, restart the subsystem and verify health.

npm run ai:server

Validate recovery by ensuring healthcheck.backup.lastSuccessful surfaces your desired timestamp and no subsystem connection errors are reported.

Important: Never perform destructive restores on shared multi-tenant databases without confirming coordinate isolation. Only restore into intended environments.