[Epic] Two-Pillar Hybrid RAG: ChromaDB Re-integration & Topological Re-ranking

Architectural Bottleneck

Currently, Swarm Episodic Memory (memory-core) utilizes sqlite-vec to manage embeddings. Extensive analysis verified that sqlite-vec executes brute-force exact nearest neighbor (KNN) via full table scans $O(N)$ and does not construct a $O(\log N)$ Hierarchical Navigable Small World (HNSW) indexing graph. Furthermore, attempting to bridge this gap by mathematically layering an explicit HNSW topology inside the V8 engine over standard ai/graph SQLite edges fails severely on execution latency and GC constraints when operating on dense 4096-Dimensional embedding geometries.

The Objective

We must decisively separate the Hybrid RAG components to utilize the exact engines explicitly built for their mathematical constraints:

1. Pillar A (Semantic Search): Revert the raw embedding sink to ChromaDB. Chroma natively uses hnswlib internally via C++/Rust to handle billion-scale MMAP logic and geometric spatial bounds efficiently, resolving the Swarm memory context limits flawlessly.
2. Pillar B (Topological Proximity): Retain SQLite categorically for explicit, relational Native Edge Graph paths.

The Strategy

1. Restore Vector Engine: Prioritize and restore ChromaManager.mjs as the default high-scale embedding driver within the memory-core server configuration, phasing out SQLiteVectorManager.mjs.
2. Implement Dual-Phase Re-Ranking Middleware: Abstract a routing filter inside StorageRouter.mjs to execute true hybrid RAG:
   • Pass 1: Execute a similarity heuristic against ChromaDB to extract the Top K semantically relevant Node IDs.
   • Pass 2: Filter those retrieved IDs explicitly through the SQLite Native Edge Graph to weight them via Topological proximity (e.g., connected actively to Frontier, shared sequence topologies).

Rationale & Avoided Pitfalls

• Avoided V8 HNSW Implementations: Building a multi-layer Skip List index inside interpreted JavaScript over SQLite triggers introduces colossal V8 Garbage Collection pauses and fundamentally negates low-level SIMD scaling.
• Genuine Hybrid Execution: Previously, SessionService isolated SQLite Vector tables from SQLite Graph tables without continuous systemic crossover rendering. This Two-Pillar re-ranker enforces intersection at the absolute base of the architecture.