The Concept Ontology
The Concept Ontology is a graph that each Agent OS plane keeps as plane data. It provides the semantic stratum between source code and learning content. It is the foundation for the Dream Pipeline's deterministic documentation gap detection.
The Problem It Solves
The GapInferenceEngine (Phase 4 of the Dream Pipeline) needs to detect which parts of Neo.mjs lack adequate documentation. The original approach used regex-based token matching against file paths:
// OLD: Fragile regex token matching
const hasGuide = guideFilePaths.some(p => nodeTokens.some(term => regex.test(p)));This fails structurally — "Reactivity.md" never token-matches "Neo.button.Base".
The Concept Ontology solves this by introducing CONCEPT nodes as first-class entities
that bridge the gap between implementation files and learning guides:
// NEW: Graph traversal
const explanations = conceptService.getEdges(conceptId, 'EXPLAINED_BY');
const hasGuide = explanations.length > 0;A concept has a GUIDE_GAP if it has zero EXPLAINED_BY edges. This is deterministic,
semantically correct, and requires no embedding comparison.
What Is a Concept?
A concept is an abstract architectural idea that:
- Has a name and a hierarchical position in the knowledge tree
- Can be implemented by one or more source files
- Can be explained by one or more learning guides
- Has a tier reflecting its importance to the platform's identity
- Is connected to other concepts via typed relationships
Concepts vs. Classes
| Class | Concept | |
|---|---|---|
| Identity | Neo.core.Base |
"Instance Lifecycle" |
| Nature | Implementation artifact | Architectural idea |
| Guides map to | ❌ Not directly (many-to-many) | ✅ Directly (1-to-many) |
| Source files map to | ✅ 1-to-1 | ✅ 1-to-many |
Every class doesn't deserve a guide, but every concept deserves at least one. The concept layer is the intermediary that makes gap detection semantically meaningful.
The Teaching Test
A concept is included in the ontology only if it passes all three criteria:
- A developer needs to understand it to use Neo.mjs productively
- It cannot be learned by simply reading one API doc page
- It answers "how" or "why" questions, not just "what" questions
| ✅ Passes | ❌ Fails |
|---|---|
| "Two-Tier Reactivity" (architectural model) | Neo.util.Array (utility, API-doc-sufficient) |
| "Off-Main-Thread Execution" (mental model shift) | "Portal App About Us View" (app-specific) |
| "Config Descriptors & Merge Strategies" (complex system) | afterSetWidth (lifecycle hook instance) |
Storage Format
Each plane keeps its concept graph as JSONL files under its data root. In a container that is
/app/.neo-ai-data/concepts/, on the orchestrator's root volume, and every backup bundle carries
it as the concepts substrate:
<dataRoot>/concepts/ ├── nodes.jsonl # One concept node per line └── edges.jsonl # One relationship edge per line
Concepts are derived knowledge. A deployment may seed them, and discovery extends them from conversations, memories and the graph. A concept can span repositories, and no repository holds the store. ADR 0023 §2.5 and ADR 0024 §2.6 record that decision.
The Knowledge Base still reads the engine's Markdown copies of the explanations for now. ADR 0024 §2.6 records the move to projecting the store instead.
Why JSONL, Not JSON or SQLite
- Append-friendly: Each line is an independent record, so discovery adds a candidate by appending a line.
- Diffable: Two backup bundles compare line by line.
- Streaming: Can be processed line-by-line without loading the entire graph into memory.
- Source-owned: The store is the declaration.
ConceptIngestorprojects it into the Native Edge Graph (SQLite). Source and projection are separate representations, not two independently maintained ontologies.
Source Membership vs. Runtime Salience
The JSONL store owns the membership contract: which concept rows and typed
relationships are declared. SQLite owns the live edge instances that graph traversal consumes.
ConceptIngestor.syncConceptsToGraph() reconciles those layers on every run:
- A matching live tuple keeps its edge ID and its current, possibly decayed,
weight. - A declared tuple that decay/pruning or another mutation removed is re-derived.
- A row removed from JSONL removes only the live tuple carrying
projectionSource: "concept-ontology-jsonl"; same-type edges from other producers survive. - Node payload hashes optimize CONCEPT-node upserts only. They never suppress relationship reconciliation.
Every projected ontology edge carries four independent axes — authority, fidelity,
extractionProvenance, and lifecycle — rather than one composite confidence score. The
current curated values are repo-trusted authority, curated/non-degraded fidelity, JSONL
extraction provenance, and promoted lifecycle state.
This does not make concept edge types globally protected. GraphService may still decay
their salience and prune weak live instances. Reconciliation restores still-declared membership
without reinforcing surviving edges or resetting their weight.
Node Schema
Each line in nodes.jsonl is a JSON object:
| Field | Type | Required | Description |
|---|---|---|---|
id |
string | ✅ | Kebab-case unique identifier (e.g., "multi-threading") |
name |
string | ✅ | Human-readable display name |
tier |
number | ✅ | Importance tier (see Tiering System below) |
description |
string | ✅ | One-paragraph explanation of the concept |
uniqueToNeo |
boolean | ✅ | true if architecturally unique to Neo.mjs |
tags |
string[] | ✅ | Categorization tags for search and filtering |
aliases |
string[] | ❌ | Alternative terms that refer to the exact same concept (O(1) lookup) |
validated |
boolean | ❌ | false for an unreviewed discovery candidate; missing is treated as validated for legacy curated rows |
ontologyLayer |
string | ❌ | code (default) or process-mx; process/MX candidates do not claim source-code coverage |
codeGapEligible |
boolean | ❌ | Whether GUIDE / EXAMPLE / ORPHAN code-gap signals may be emitted; defaults to true |
source / reasoning |
string | ❌ | Discovery provenance and Teaching-Test rationale for mined candidates |
verifiedAt |
string | null | ❌ | ISO date string for the last source-grounded verification, or null / missing when never explicitly verified |
extraction_metadata |
object | null | ❌ | Present only on LLM-mined candidate rows (written by ConceptDiscoveryService): the extraction pass's objective self-report {missing_fields, ambiguous_references, confidence_score}, denormalized onto each candidate it produced. JSONL-only — not projected to graph node properties or the ConceptIngestor payload hash; legacy rows without it load unchanged. |
{"id":"off-main-thread","name":"Off-Main-Thread Execution","tier":1,"description":"Application business logic runs inside a dedicated App Worker.","uniqueToNeo":true,"tags":["architecture","workers"],"aliases":["off the main thread","OMT"],"verifiedAt":null}
{"id":"mined-example","name":"Mined Example","tier":3,"description":"An LLM-mined candidate awaiting curator review.","uniqueToNeo":false,"tags":["mined-candidate","process-mx"],"aliases":[],"validated":false,"ontologyLayer":"process-mx","codeGapEligible":false,"source":"message-concept-harvest","reasoning":"Passes the Teaching Test as a reusable process concept.","verifiedAt":null,"extraction_metadata":{"missing_fields":[],"ambiguous_references":["'the module' — three modules exist"],"confidence_score":0.7}}[!IMPORTANT] Aliases are strict synonyms within Neo.mjs. A term qualifies as an alias only if it refers to the exact same architectural concept. Cross-framework terms (e.g., "ViewModel" for State Provider, "JSX" for JSON VDOM) are not aliases — they belong in
ANALOGOUS_TOedges.
[!IMPORTANT] Freshness metadata is non-destructive.
verifiedAtexists to build a review queue: concepts withnull, missing, invalid, or older-than-90-day values emit aCONCEPT_REVERIFY_DUEhandoff signal. This must not fade graph nodes, weaken edges, reduce concept weight, or auto-retire concepts. A stale verification date means "check this against current repo reality"; it is not evidence that the concept lost value. Seeded rows start with explicitverifiedAt: nullso the first source-grounding pass can be queried directly from the store.
Curated Tags, Historical Extraction, and Scheduled Discovery
Concept knowledge enters the graph through distinct trust paths. Do not collapse the broad historical search population into the small, deliberate ontology.
| Path | Current trigger | Runtime / review signal |
|---|---|---|
| Manual message tag | addMessage({taggedConcepts: [...]}) with explicit IDs |
Synchronous TAGGED_CONCEPT edge at weight 1.0 |
| Seeded ontology & curated edge import | A deployment may seed its store, and maintainers import curated edge additions or decision-backed removals via ConceptService.importEdges() (ai/scripts/maintenance/importConceptEdges.mjs); ConceptIngestor reads <dataRoot>/concepts/*.jsonl during REM |
Source-owned projected edges plus validated, tier, and projection axes |
| Scheduled candidate discovery | The orchestrator runs message-concept-harvest; ConceptDiscoveryService reads a bounded batch of unharvested MESSAGE nodes, frequency-filters them, and sends one bounded Teaching-Test prompt. Its separate runDiscoveryCycle() mines epics and a capped recent-PR set on explicit invocation. |
Candidate rows append to nodes.jsonl as validated: false, tier 3; they remain silent until curator promotion and enter SQLite only on a later ConceptIngestor sync |
| Historical inline extraction | Retired runtime path; SemanticGraphExtractor.extractMessageConcepts() remains only as a direct/test helper and has no production caller |
Existing auto_extracted: true nodes and TAGGED_CONCEPT edges at weight 0.8 can remain as legacy provenance |
Current Message-Discovery Path
MailboxService.addMessagepersists the MESSAGE node and routing edges, then synchronously projects only explicittaggedConceptsat weight1.0. It performs no model call.- The orchestrator schedules
message-concept-harvestas an exclusive heavy-maintenance task. ConceptDiscoveryService.runMessageConceptHarvest()loads at most the configured batch, builds a cheap subject/tag frequency report, keeps only the configured top recurring terms, and makes one bounded Teaching-Test request.- Accepted proposals append to JSONL with
validated: false; only successfully processed messages receiveconceptHarvestedmarkers.ConceptIngestorprojects the candidates on a later run, whileGapInferenceEnginesuppresses unvalidated proposals.
Read-Time Consumer Pattern
- Use
validated === falseas the current explicit exclusion for unreviewed discovery candidates. Legacy rows without the field are treated as validated for compatibility. - Use the ontology edge's four axes and
projectionSourceto reason about curated membership; use its liveweightonly for salience. - Treat
auto_extracted: trueandTAGGED_CONCEPTweight0.8as historical provenance for the retained inline-extraction population, not as evidence that MailboxService still performs per-message inference.
Edge Schema
Each line in edges.jsonl is a JSON object:
| Field | Type | Required | Description |
|---|---|---|---|
source |
string | ✅ | Source node ID (concept or file reference) |
target |
string | ✅ | Target node ID (concept, file reference, or ext: external ID) |
type |
string | ✅ | Relationship type (see Edge Types below) |
note |
string | ❌ | Architectural distinction note (used with ANALOGOUS_TO) |
On projection, every valid row also receives runtime-only metadata:
| Property | Meaning |
|---|---|
projectionSource |
concept-ontology-jsonl, the ownership marker for tuple reconciliation |
axes.authority |
Repo-trusted curated authority |
axes.fidelity |
Curated source tier, not degraded |
axes.extractionProvenance |
Curated JSONL origin |
axes.lifecycle |
Promoted lifecycle state |
Edge Types
| Type | Direction | Meaning |
|---|---|---|
PARENT_CONCEPT |
parent → child | Hierarchical grouping |
IMPLEMENTED_BY |
concept → file | Source file that implements the concept |
EXPLAINED_BY |
concept → file | Guide/doc that explains the concept |
EXEMPLIFIED_BY |
concept → file | Example that demonstrates the concept |
REQUIRES |
concept → concept | Prerequisite (must understand A before B) |
ANALOGOUS_TO |
concept → ext:id | Cross-framework analogue (not equivalence) |
File Reference Format
File targets use the file: prefix with a repository-relative path:
{"source":"push-reactivity","target":"file:src/Neo.mjs","type":"IMPLEMENTED_BY"}
{"source":"push-reactivity","target":"file:learn/guides/coreengine/ConfigSystem.md","type":"EXPLAINED_BY"}file: is author-facing syntax only. During projection, FileSystemIngestor validates that
the path stays inside the repository, exists, is a regular non-ignored file, and resolves it to
the canonical runtime node ID file-<path>. Invalid rows do not become coverage evidence; the
source CONCEPT retains an exact-row projection-integrity finding for gap inference.
External Reference Format
External (cross-framework) targets use the ext: prefix to prevent collision with
internal concept IDs:
{"source":"state-provider","target":"ext:react-context","type":"ANALOGOUS_TO","note":"Both provide hierarchical state, but Neo.mjs providers use bind:{} on reactive configs — no subtree re-rendering."}[!WARNING]
ANALOGOUS_TOexpresses architectural similarity, not equivalence. Thenotefield must explain how the Neo.mjs concept differs from its cross-framework analogue. Never use this edge to suggest that concepts are interchangeable.
Tiering System
| Tier | Weight | Description | Gap Severity |
|---|---|---|---|
| 0 | — | System anchor (Neo.mjs itself) | N/A |
| 1 | ≥ 0.9 | Platform identity concepts | CRITICAL if undocumented |
| 2 | 0.5–0.8 | Major subsystem concepts | HIGH if undocumented |
| 3 | 0.1–0.4 | Implementation-level concepts | MEDIUM if undocumented |
The Concept Hierarchy (Abbreviated)
✅ = has at least one EXPLAINED_BY edge. Missing ✅ = GUIDE_GAP candidate.
How a Concept Enters the Store
- Discovery appends candidates with
validated: false. They stay out of gap inference and concept search until they are promoted. - A seed, which is optional, writes rows into a new plane's store, for example from a guide index. A plane may also start empty.
- Curated edge import (
ConceptService.importEdges()andai/scripts/maintenance/importConceptEdges.mjs) is the sanctioned entry and removal path for curatededges.jsonlrows on a plane's store:- Input rows are
{source, target, type, note?}, or{source, target, type, op: "remove", decision}wheredecisionnames an issue, pull request, or ADR. --tenant <id>is required and recorded in the report. Today only the shared core corpus tenant (kbConfig.defaultTenantId) is admitted; other tenants are refused until tenant-keyed concept rows and per-tenant file roots land (#473).- Each row is classified as
added,equal,differs,removed,absent, orrefused(sharing target validation withConceptIngestor.normalizeOutboundEdges()and refusing unvalidatedvalidated: falsecandidate endpoints).differsrows are never overwritten silently; changing anoterequires a decision-backedop: "remove"followed by the new row. - The CLI reads JSONL from
--file <path>orstdin, supports--dry-run, accepts no store path argument, and replacesedges.jsonlatomically only whenaddedorremovedrows apply.
- Input rows are
- Promotion has no surface yet. Evaluating and validating candidates is an open design question (neomjs/neo#19096 OQ9). Until it is answered, a seedless plane has no validated concepts, and therefore no Knowledge Base concepts. That is by design.
Every row follows the node and edge schemas above and passes the Teaching Test, however it
enters: its PARENT_CONCEPT edges place it in the hierarchy, and its EXPLAINED_BY and
IMPLEMENTED_BY edges link guides and source files.
JSONL Format Rules
- One JSON object per line — no multi-line JSON
- No trailing commas — strict JSON per line
- Append-only nodes, atomic whole-file updates for edges — Discovery appends candidate lines to
nodes.jsonl;ConceptService.importEdges()preserves untouchededges.jsonllines verbatim, applies decision-backed removals and additions, and swapsedges.jsonlatomically whileConceptIngestorremains strictly read-only on the store