What is the Neural Link?

The Neural Link is a bi-directional bridge that connects AI agents directly to the Neo.mjs runtime. It lets agents inspect the Scene Graph, component state, event listeners, computed styles, and DOM rectangles, and mutate the running application in real time.

Why is Neo.mjs called an Application Engine instead of a framework?

Neo.mjs maintains persistent application objects in a worker-backed Scene Graph instead of compiling application state away into ephemeral DOM nodes. That architecture enables multi-window orchestration, runtime permutation, and deep AI introspection.

What is Context Engineering?

Context Engineering shapes the information and tool environment around AI agents. Neo.mjs implements it through Knowledge Base, Memory Core, GitHub Workflow, and Neural Link MCP servers for frontier harnesses, plus a File System MCP server for internal Neo.ai.Agent local loops.

What is the Neo.mjs Agent OS?

The Neo.mjs Agent OS is the repository Brain: source code and services for Memory Core, Knowledge Base, Active Hybrid GraphRAG, DreamService, Golden Path synthesis, A2A coordination, and Neural Link tooling.

Frontmatter

id	13467
title	Investigate malformed Chroma FTS5 index
state	Closed
labels	bugaiarchitecture
assignees	neo-gpt
createdAt	Jun 18, 2026, 1:12 PM
updatedAt	Jun 19, 2026, 6:40 AM
githubUrl	https://github.com/neomjs/neo/issues/13467
author	neo-gpt
commentsCount	2
parentIssue	null
subIssues	[]
subIssuesCompleted	0
subIssuesTotal	0
blockedBy	[]
blocking	[]
closedAt	Jun 19, 2026, 6:40 AM

Investigate malformed Chroma FTS5 index

Closed v13.1.0/archive-v13-1-0-chunk-3 bugaiarchitecture

neo-gpt commented on Jun 18, 2026, 1:12 PM

Context

During the 2026-06-18 local Neo Chroma corpus investigation, the KB corpus itself proved present and queryable by count: live Chroma SQLite and the newest backup both showed 30466 neo-knowledge-base rows. However, a read-only SQLite integrity probe against the unified Chroma store returned:

malformed inverted index for FTS5 table main.embedding_fulltext_search

The exact probe was:

sqlite3 .neo-ai-data/chroma/unified/chroma.sqlite3 "pragma quick_check;"

That does not prove KB/MC collection loss. It does prove an internal Chroma SQLite FTS index is in an unhealthy state and should not be ignored.

Live latest-open sweep: checked the latest 20 open issues/PRs via gh api --method GET repos/neomjs/neo/issues -f state=open -f per_page=20 on 2026-06-18; no equivalent ticket found.

A2A in-flight claim sweep: checked latest 30 messages with list_messages({status:'all', limit:30}); no overlapping [lane-claim] / [lane-intent] found.

Exact local sweep: searched resources/content/issues, resources/content/discussions, ai, learn, and .agents for FTS5, embedding_fulltext_search, quick_check, integrity_check, and malformed inverted index; only unrelated mailbox-FTS discussion surfaced.

Release classification: post-release / operational-hardening follow-up; boardless unless the operator promotes it to release-blocking.

The Problem

Neo now runs KB and MC over a unified Chroma persist directory. Chroma owns the SQLite schema, including embedding_fulltext_search, while Neo uses Chroma APIs for collection count, get, query, and backup/defrag flows.

A malformed internal FTS5 index may be harmless for vector/count paths, or it may indicate a broader store-integrity risk that only shows up under specific Chroma API operations. We need to determine which is true before anyone runs a destructive repair on the live store.

The Architectural Reality

Relevant paths and surfaces:

.neo-ai-data/chroma/unified/chroma.sqlite3 is the unified Chroma SQLite store for KB and MC in the local topology.
ai/scripts/maintenance/defragChromaDB.mjs performs extract/delete/recreate/reload and then runs SQLite VACUUM on the Chroma database.
ai/scripts/maintenance/backup.mjs exports portable JSONL backups through service boundaries.
ai/services/knowledge-base/* and ai/services/memory-core/* should treat Chroma as the API boundary, not hand-edit Chroma internals casually.
Chroma's embedding_fulltext_search table is an internal Chroma SQLite FTS5 table. Repair strategy must account for Chroma's schema/version expectations.

The Fix

This ticket is investigation-first, repair-second:

Reproduce the malformed FTS5 state on a copied snapshot of the Chroma DB, not by mutating the live store.
Identify which Chroma operations, if any, fail because of the malformed FTS5 index.
Determine whether the safe repair path is Chroma-native rebuild/export-import, SQLite REINDEX, full collection defrag, or something else.
Document the exact live-store safety procedure before any repair is run against production/local-live data.
Add a narrow diagnostic script or checklist only if it reduces future repair risk without making healthcheck heavier.

Contract Ledger Matrix

Target Surface	Source of Authority	Proposed Behavior	Fallback	Docs	Evidence
Chroma integrity probe	SQLite `pragma quick_check` / `pragma integrity_check` on a copied Chroma DB	Detect and report malformed internal indexes without mutating live data	If probe cannot run, report the command/environment failure separately	Maintenance script JSDoc or operator runbook if a script is added	Reproducible read-only probe output
Chroma repair procedure	Chroma API + SQLite behavior on copied snapshot	Establish a safe repair/rebuild sequence before touching live data	Restore from known-good JSONL/physical backup if repair is unsafe	Maintenance docs or ticket comment	Snapshot repair validation
Neo health surfaces	KB/MC healthchecks	Do not add heavyweight SQLite integrity checks to normal healthcheck hot paths	Provide on-demand script/checklist instead	Out-of-band maintenance docs	Healthcheck remains bounded

Decision Record impact

none. This ticket investigates operational integrity of the Chroma backing store; it does not change an ADR.

Acceptance Criteria

Reproduce malformed inverted index for FTS5 table main.embedding_fulltext_search on a copied snapshot or otherwise document why it no longer reproduces.
Identify whether KB/MC Chroma operations used by Neo (count, get, query, backup export, defrag extract) are affected by the malformed FTS5 index.
Determine a safe repair/rebuild path and validate it on a copy before any live mutation.
Document the required pre-repair backup/stop-daemon sequence.
If a diagnostic script is added, it must be on-demand only and must not make KB/MC healthchecks heavier.
Leave the live store untouched unless an operator explicitly authorizes the repair step.

Out of Scope

Fixing stale long-lived MCP collection handles after Chroma recycle. That is tracked in #13466.
Redesigning Chroma defrag's delete/recreate window.
Running live repair without explicit operator approval.
Treating pragma quick_check failure as proof of KB/MC corpus loss without API-level evidence.

Avoided Traps

Do not run REINDEX, VACUUM, manual table edits, or collection rebuilds directly on the live Chroma DB as the first move.
Do not put this probe into the normal healthcheck path; it is an integrity diagnostic, not a cheap liveness check.
Do not conflate Chroma-internal FTS corruption with vector collection row loss.

Related: #13466

Origin Session ID

Origin Session ID: 4ce60429-2986-4543-be2d-741957c75b6c

Handoff Retrieval Hints

query_raw_memories: malformed Chroma FTS5 index embedding_fulltext_search quick_check
query_raw_memories: Chroma restart stale collection handle affects Memory Core and Knowledge Base MCP healthcheck counts
Local evidence anchor: sqlite3 .neo-ai-data/chroma/unified/chroma.sqlite3 "pragma quick_check;" returned malformed inverted index for FTS5 table main.embedding_fulltext_search during the 2026-06-18 corpus investigation.