ChromaDB Concept Embedding & Hybrid Search

Objective

Embed concept nodes into ChromaDB as type: 'concept' documents, enabling hybrid semantic search across concepts, guides, and source code.

Context

Phase 2 sub-issue of the Concept Ontology epic (#10030). Depends on #10032 (ConceptService with loaded JSONL data). Requires a working ChromaDB instance.

Rationale

Concept nodes have short, focused descriptions (1-3 sentences) — the opposite of the Macro KB semantic dilution problem. Embedding them enables:

1. Concept discovery: queryDocuments({query: "how does reactivity work?", type: "concept"}) → ranked concept nodes
2. Automated wiring: queryDocuments({query: "Neo.core.EffectManager", type: "concept"}) → discover which concept a class implements
3. Edge validation: Compare concept description vectors against linked guide chunk vectors → verify EXPLAINED_BY edges are semantically accurate

Deliverables

1. Concept Embedding Sync

Extend the KB DatabaseService sync pipeline (or create a standalone script) to:

• Read .neo-ai-data/concepts/nodes.jsonl
• For each concept node, upsert into ChromaDB with metadata:

    { "type": "concept", "name": "Two-Tier Reactivity", "tier": 1, "source": ".neo-ai-data/concepts/nodes.jsonl" }

• Document content = "${name}: ${description}"
• Use the same embedding model as all other KB content (NEO_GLOBAL_EMBEDDING)

2. QueryService Scoring Update

Add conceptMatch weight to queryScoreWeights in KB config:

conceptMatch: 15  // Concept results get a scoring boost when type='concept' is queried

3. Cross-Type Semantic Validation

Utility method (in ConceptService or as a script) that:

• For each CONCEPT with EXPLAINED_BY edges, query ChromaDB for the concept description against type: 'guide'
• If the linked guide doesn't appear in the top-N results → flag as potentially misattributed edge
• Report for human review

Three Representations, Three Strengths

Architectural Note

ConceptService lives in the SDK (ai/services.mjs). Uses KB_ChromaManager directly through SDK imports — no new MCP tools needed.

Acceptance Criteria

• Concepts embedded in ChromaDB with correct metadata
• queryDocuments({type: 'concept'}) returns concept results
• Cross-type validation identifies misattributed edges
• Sync is idempotent (re-running doesn't duplicate)

Origin Session ID: 496c0120-29aa-4b02-a804-d5a32ff6b5eb