LearnNewsExamplesServices
Frontmatter
title>-
authortobiu
stateMerged
createdAtApr 19, 2026, 12:20 PM
updatedAtApr 19, 2026, 12:26 PM
closedAtApr 19, 2026, 12:26 PM
mergedAtApr 19, 2026, 12:26 PM
branchesdev ← agent/10035-gapinferenceengine-concept-graph-refactor
urlhttps://github.com/neomjs/neo/pull/10084
Merged
tobiu
tobiu commented on Apr 19, 2026, 12:20 PM

Summary

Replaces the pre-#10035 regex + LLM-Boolean-verification GUIDE_GAP path in GapInferenceEngine with deterministic outbound-edge traversal over curator-maintained EXPLAINED_BY / EXEMPLIFIED_BY edges in the Native Edge Graph. Introduces ConceptIngestor (new daemon service) to bridge the version-controlled JSONL concept ontology into the runtime graph as DreamService Phase 0. Extends GoldenPathSynthesizer with a new [EXAMPLE_GAP] display section. TEST_GAP regex logic is preserved unchanged — out of scope for this refactor.

The Problem

The pre-refactor GapInferenceEngine.executeCapabilityGapInference regex-matched tokenized node names against learn/guides/* paths, then ran a per-match gemma4-31b Boolean verification to reject false positives. N structural nodes × M matched guides inferences per REM cycle — the slowest step in the pipeline and architecturally fragile (LLM semantic judgment against 3000-char truncated guide content). With the concept ontology shipped in #10031/#10032/#10049 and covered by tests in #10033, the authoritative source of "does concept X have a guide?" is a curator-asserted EXPLAINED_BY edge, not a regex fuzzy-match.

Architectural Reality (verified against precedent, not docs)

Read and verified before code:

  • ai/graph/Database.mjs — getAdjacentNodes(nodeId, direction, type) exists; getNodesByLabel does NOT. Current filter pattern: db.nodes.items.filter(n => n.label === 'CONCEPT'). Precedent confirmed in GapInferenceEngine's pre-refactor code and GoldenPathSynthesizer.
  • ai/graph/storage/SQLite.mjs — enforces FOREIGN KEY from edges.target → nodes.id. Concept edges pointing at file:... / ext:... identifiers must have stub nodes seeded before insertion, or the transaction aborts. Missed this on first pass — test surfaced it, ConceptIngestor now handles via ensureEdgeTargetsExist.
  • ai/daemons/services/IssueIngestor.mjs — template pattern for sibling ingestors (GraphService.upsertNode({id, type, name, properties}); DTO type maps to NodeModel label).
  • ai/daemons/services/GoldenPathSynthesizer.mjs — downstream consumer parses capabilityGap as JSON-array string with tag prefixes ([TEST_GAP], [GUIDE_GAP], else-fallback). New [EXAMPLE_GAP] tag required explicit display branch.
  • Local model capability (via WebSearch, April 2026): Gemma 4 31B is frontier-class (85.2% MMLU Pro, 89.2% AIME 2026, 256K context, distilled from Gemini 3). "SLM weakness" was NOT the correct framing for dropping the LLM verification. The correct framing: curator-maintained graph edges are the authoritative source of truth; LLM verification was rubber-stamping when the data model changed from regex-matched to concept-curated.

Gold Standards Leveraged / Traps Avoided

  • Gold Standard (IssueIngestor pattern): sibling-service shape with singleton Base subclass, single public sync*ToGraph() entry point, GraphService.upsertNode for persistence
  • Gold Standard (edges-direct traversal): pattern borrowed from GoldenPathSynthesizer (.edges.getByIndex('source', id).filter(e => e.type === ...)) instead of getAdjacentNodes (which would return empty for unmaterialized file: / ext: targets)
  • Gold Standard (hash-based differential sync): sha256 of stable-stringified payload — simple, auditable, deterministic. No clever algorithms. I/O reduction matters for the REM pipeline (gemma4-31b is throughput-bound, not capability-bound)
  • Trap avoided (ticket's pseudo-code): graphDb.getNodesByLabel('CONCEPT') doesn't exist. Used the current filter-on-label pattern
  • Trap avoided (Ghost node hallucination in traversal): concept edges point at file: / ext: string targets; getAdjacentNodes resolves to existing nodes only and would mis-report every concept as an uncovered gap. Switched to edges-direct
  • Trap avoided (LLM verification retention): the per-match gemma4-31b inference added no signal once concept edges became the source of truth. Dropped. Reframed from "SLM can't verify" to "curated graph is truth; verification is rubber-stamp"

Implementation Footprint (5 commits, +397 / -104)

Commit Change
184f3e623 feat(ai): ConceptIngestor New ai/daemons/services/ConceptIngestor.mjs (~210 lines) — singleton reading ConceptService, upserting CONCEPT nodes + typed edges with sha256 differential sync. DreamService wires it as Phase 0 before FileSystemIngestor.
6d14e4449 docs(ai): reframe rationale Corrected JSDoc framing from "SLM weakness" to "I/O throughput" after empirical reading of Gemma 4 31B benchmarks
22d3d0dd7 refactor(ai): GapInferenceEngine Split into two passes: TEST_GAP (unchanged regex on session-artifact structural nodes) and GUIDE_GAP/EXAMPLE_GAP (new concept-graph edge traversal over ingested CONCEPT nodes). Removed OpenAiCompatible import, file reads, LLM verification prompt. GUIDE_GAP_WEIGHT_THRESHOLD = 0.8 constant with documented derivation from the ConceptService weight formula
3411bb2f7 feat(ai): sandman_handoff EXAMPLE_GAP GoldenPathSynthesizer parses [EXAMPLE_GAP] into its own array and renders the ### 💡 Example Disconnects section with the same 5-item limit pattern used by TEST_GAP and GUIDE_GAP
91722d561 test(ai): rewrite test #194 Replaces the LLM-verification test with three new tests: GUIDE_GAP via graph traversal, EXAMPLE_GAP detection, low-weight threshold skip. Seeds SQLite FK-safe edge-target stub nodes. Also adds ensureEdgeTargetsExist to ConceptIngestor so production FK constraint holds.

Acceptance Criteria (from #10035)

  • CONCEPT nodes visible in SQLite graph after DreamService run (via ConceptIngestor Phase 0)
  • All concept edges correctly wired (PARENT_CONCEPT, IMPLEMENTED_BY, EXPLAINED_BY, EXEMPLIFIED_BY, ANALOGOUS_TO) with FK-safe stub-node seeding
  • GapInferenceEngine uses graph traversal, not regex (for the guide/example gap path; TEST_GAP preserves regex by design)
  • Gap report priorities align with concept weights (GUIDE_GAP_WEIGHT_THRESHOLD = 0.8, derivation documented)
  • Existing DreamService tests still pass (5 green; the 6th was the LLM-verification test replaced with 3 new tests; one pre-existing test-isolation bug incidentally fixed by removing the LLM mock setup)

Test Plan

npm run test-unit -- test/playwright/unit/ai/services/ConceptService.spec.mjs test/playwright/unit/ai/daemons/

Expected: 30 passed (23 ConceptService + 7 daemon).

Behavioral test-of-success (observable across REM cycles after merge):

  • sandman_handoff.md emits 🗺️ Guide Disconnects only for concepts with weight >= 0.8 AND no EXPLAINED_BY edge
  • sandman_handoff.md emits 💡 Example Disconnects for documented concepts lacking EXEMPLIFIED_BY
  • No per-match LLM inference spike in the gap inference phase (cycle-time drop measurable via existing DreamService phase-time logging)

Out of Scope / Follow-Up

  • ORPHAN_CONCEPT signal — the ticket mentioned this as a possible gap type. Implemented as logger.warn during ingestion instead of a capabilityGap entry, since orphans are ontology data-quality issues rather than coverage gaps. Can be promoted to a distinct channel if human tuning shows value
  • guideGapWeightThreshold as config — currently a named constant in GapInferenceEngine with documented derivation. Promote to aiConfig.data.guideGapWeightThreshold if tuning becomes a real need
  • ConceptIngestor.spec.mjs (dedicated) — dimension-specific service tests (idempotency, hash-based differential sync, orphan detection, edge-type normalization) deserve their own spec file. Currently covered indirectly via DreamService integration tests. Follow-up worth filing if coverage gaps show up

Related

  • Parent: #10030
  • Unblocks: #10036 (Memory Core Concept Discovery — consumes concept graph), downstream of this refactor
  • Related siblings filed this session: #10080 (relevance-bounded queries), #10081 (agent FAQ telemetry), #10082 (tech-debt-radar unbounded-list detection)
  • Discipline applied: feedback_challenge_numerical_thresholds_in_acs.md (weight threshold derivation), feedback_verify_written_claims_against_precedent.md (five instance callouts this session)

Origin Session ID

62d6f155-e57f-4279-9b59-36c9e4ecbc5e

Resolves #10035

tobiu
tobiu commented on Apr 19, 2026, 12:25 PM

Input from Claude Opus 4.7 (Claude Code):

❋ # PR Review Summary

Status: Approved (pending human merge)

Self-Review Opening: Self-review of #10035. This refactor chose (a) the IssueIngestor pattern over inline modification of DreamService, (b) edges-direct traversal over getAdjacentNodes, (c) stub-node seeding at ingest over schema relaxation, and (d) dropping the LLM verification step entirely over retaining it as a fallback. Rationale for each is documented in the PR body and code; deductions below focus on the parts that didn't land as cleanly as they could have.

📊 Evaluation Metrics

  • [ARCH_ALIGNMENT]: 92 — Pattern-matches the existing ingestor family (IssueIngestor, FileSystemIngestor). Edges-direct traversal mirrors the proven pattern in GoldenPathSynthesizer. Stub-node seeding preserves graph FK integrity without schema changes. Deduction: the concept-graph pass (inferConceptGraphGaps) is ontology-scoped — same output every invocation within a single REM cycle — but fires once per session inside DreamService's per-session loop. Correctness is unaffected (gap writes are idempotent) but it's wasted work at N×M scale. Architecturally cleaner: hoist the concept pass to cycle-scope, call it once before the per-session loop. Not blocking; follow-up polish.
  • [CONTENT_COMPLETENESS]: 87 — Anchor & Echo JSDoc throughout; constants carry derivation rationale; PR body has five-row empirical verification table and explicit out-of-scope flags. Deductions: (a) no dedicated ConceptIngestor.spec.mjs — behavior tested indirectly via DreamService integration. I flagged this in the PR body, but it's still a real coverage gap that deserves its own ticket, not a hand-wave. (b) DreamService calls await ConceptIngestor.syncConceptsToGraph() and discards the returned stats — the return shape exists for observability and isn't surfaced. Simple fix: const stats = await ...; logger.info('[DreamService] Concept ingestion: ...', stats);. (c) Orphan warnings emit per-concept via logger.warn; at 17 orphans they fire 17 times per cycle. Summary warn (count + IDs) would be quieter without losing signal.
  • [EXECUTION_QUALITY]: 88 — 5 commits, 30 tests pass, no regressions, clean branch lineage. Deductions: (a) SQLite FK constraint on edges.target → nodes.id missed on first pass. Fixed by test surfacing + ensureEdgeTargetsExist, but should have been caught by reading ai/graph/storage/SQLite.mjs proactively. Sixth in-session instance of accept-without-verify class. (b) Per-session concept-graph pass inefficiency noted in ARCH_ALIGNMENT.
  • [PRODUCTIVITY]: 95 — All 5 acceptance criteria met. Four explicit out-of-scope follow-ups flagged (dedicated spec, config-lift of threshold, ORPHAN_CONCEPT channel, stats logging), none smuggled in.
  • [IMPACT]: 85 — Load-bearing. Replaces N×M gemma4-31b inferences per REM cycle with deterministic edge checks. Unblocks #10036 downstream. Shifts the gap-detection authority from regex heuristics to curator-maintained graph edges. Not UI-visible on its own; it's substrate.
  • [COMPLEXITY]: 65 — 5 files, ~500 line delta, multi-service coordination (DreamService orchestration, new ingestor, refactored gap inference, downstream display, test rewrite). Not a single-file change.
  • [EFFORT_PROFILE]: Architectural Pillar — fundamental change to how the REM pipeline detects guide gaps. Net-new service, behavioral change for downstream consumers, new signal type (EXAMPLE_GAP), preserved backward compatibility with the TEST_GAP channel.

🕸️ Context & Graph Linking

  • Target Epic / Issue ID: Resolves #10035
  • Related Graph Nodes:
    • Parent: #10030 (Concept Ontology epic)
    • Unblocks: #10036 (Memory Core Concept Discovery) — consumes the ingested concept graph
    • Sibling tickets filed this session: #10080 (bounded queries), #10081 (FAQ telemetry), #10082 (tech-debt-radar unbounded-list detection), #10079 (Hypothesis gate broadening), #10083 (AGENTS.md §9 framing audit)
    • Prior art consumed: session aae16459 (ConceptService shape), session 897b45ae (DreamService decomposition)

🧠 Graph Ingestion Notes

  • [TOOLING_GAP]: SQLite FK constraints aren't discoverable from ai/graph/Database.mjs's exposed API surface — db.addEdge(edge) accepts any object and defers validation to the storage layer. Future implementers will hit the same FK-FAILED error I did unless they proactively read ai/graph/storage/SQLite.mjs. A JSDoc @throws SqliteError note on db.addEdge with the specific FK constraint spelled out would close this gap. Worth a follow-up polish ticket.

  • [KB_GAP]: The concept-graph ingestion architecture (Phase-0 ingestor, stub-node seeding, edges-direct gap traversal) isn't yet indexed in the knowledge base. ask_knowledge_base("how do concept edges land in the SQLite graph?") will return stale answers until a KB sync runs post-merge. Same failure mode as #10075's self-review flagged — post-merge KB sync is a recurring implicit requirement. Candidate for automation.

  • [RETROSPECTIVE]: This PR landed cleanly despite six in-session instances of the accept-without-verify failure class (see #10079's observation table — 100-line threshold, ask vs query_documents, universal dispatch vs per-server recorder, Architectural-Pillar = fresh session, 25-turn guardrail misread, SQLite FK constraint). Every miss was caught — most by tobi's challenges, one (FK) by tests. The compensating mechanism worked, but the failure rate reveals that checkpoint-with-human remains the enforcement of the Hypothesis gate until #10079's skill-level strengthening ships. The durable-memory chain (feedback_challenge_numerical_thresholds_in_acs.md, feedback_verify_written_claims_against_precedent.md) and the skill-strengthening ticket (#10079) together form the canonical remediation for this class — future sessions will either validate the fix or surface it's still insufficient. Observable falsifiable test.

📋 Required Actions

All items below are polish / follow-up, not merge blockers. The PR's core scope (ticket ACs) is complete and green.

  • (Optional, follow-up ticket) Hoist inferConceptGraphGaps call from per-session to cycle-scope in DreamService. Correctness is preserved today; efficiency only matters at larger ontology scale (post-#10050/#10036/#10037).
  • (Optional, follow-up ticket) Create test/playwright/unit/ai/daemons/services/ConceptIngestor.spec.mjs for dimension-specific service coverage: hash-based differential sync skip paths, orphan detection, edge-type normalization, stub-node seeding by namespace, idempotency. The DreamService integration tests cover behavior indirectly but don't lock in the service's contract at unit level.
  • (Optional, tiny polish) Surface syncConceptsToGraph's returned stats via logger.info in DreamService. Currently discarded.
  • (Optional, tiny polish) Consolidate per-orphan logger.warn into a single summary warning with count + IDs to reduce log noise at scale.
  • (Post-merge, recurring) Trigger manage_knowledge_base sync so agents querying the concept-graph ingestion architecture get current synthesized answers.

Approved for human merge. Zero behavior regressions, 30 tests green, rollback is a single-PR revert. Honest deductions above — the refactor landed well but isn't above critique, and I'd rather flag the per-session inefficiency + missing spec coverage explicitly than pretend they don't exist.