LearnNewsExamplesServices
Frontmatter
titlerefactor(ai): hoist concept-graph gap pass to cycle-scope (#10085)
authortobiu
stateMerged
createdAtApr 19, 2026, 4:17 PM
updatedAtApr 19, 2026, 4:45 PM
closedAtApr 19, 2026, 4:45 PM
mergedAtApr 19, 2026, 4:45 PM
branchesdevagent/10085-conceptingestor-followups
urlhttps://github.com/neomjs/neo/pull/10100
Merged
tobiu
tobiu commented on Apr 19, 2026, 4:17 PM

Summary

Lands Items 1 + 2 of #10085 — hoisted inferConceptGraphGaps from per-session to cycle-scope (efficiency) and added a dedicated ConceptIngestor unit spec (coverage). Items 3 + 4 dropped after scope review; see rationale below.

The Problem

#10084 (resolving #10035) replaced regex + LLM-verification guide-gap detection with deterministic concept-graph edge traversal. The refactor's self-review flagged four polish items as optional follow-ups. #10085 bundled all four. Ticket-intake review found that two of the four fail the Hypothesis-vs-Root-Cause gate: they prescribe logger-based fixes for a daemon where logger output is ephemeral and observability lives durably on graph nodes + sandman_handoff.md.

Architectural Reality (verified against precedent)

  • ai/daemons/services/GapInferenceEngine.mjs — pre-refactor executeCapabilityGapInference(session, payload) called both passes per session. The inferConceptGraphGaps pass filters GraphService.db.nodes.items by label === 'CONCEPT' — ontology-scoped, session-independent. Each per-session re-invocation produced identical output: N redundant traversals per REM cycle.
  • ai/daemons/DreamService.mjs — Phase 0 (ConceptIngestor.syncConceptsToGraph) and the per-session executeCapabilityGapInference both nested in the undigested-sessions else branch. Cycle-scope hoist lives in the same branch, after the session for-loop, before runGarbageCollection. Placement mirrors #10084's Phase 0 pattern.
  • ai/daemons/services/GoldenPathSynthesizer.mjs:254-295 — downstream consumer. Scans all nodes, parses capabilityGap JSON arrays with tag prefixes ([TEST_GAP]/[GUIDE_GAP]/[EXAMPLE_GAP]), TTL-prunes at 7 days, renders dedicated sections in sandman_handoff.md. This is the durable observability substrate. DreamService.spec.mjs asserts on node.properties.capabilityGap and sandman_handoff.md file contents; zero assertions on logger output.
  • ai/mcp/server/memory-core/logger.mjs — simple mutable object, gated by aiConfig.debug. Ephemeral. Suitable for phase-level progress only, not durable signals.

Items landed

Item 1 — cycle-scope hoist

Extracted inferConceptGraphGaps from the per-session wrapper; deleted executeCapabilityGapInference; promoted both inference entry points (inferTestGapsFromSession, inferConceptGraphGaps) from @protected to public API with matching thin facades on DreamService. Added cycle-scope call in processUndigestedSessions after the session loop, before garbage collection. Expanded Anchor JSDoc on both inference methods to echo the scope split so future ask_knowledge_base queries surface the right substrate.

Item 2 — dedicated unit spec

Created test/playwright/unit/ai/daemons/services/ConceptIngestor.spec.mjs. Six AC coverage targets plus stats-shape contract:

# Target Mechanism
1 Differential-sync skip path sync twice with same fixture; assert conceptsSkipped === 1
2 Differential-sync upsert path mutate description between syncs; assert conceptsUpserted === 1 + persisted node reflects update
3 Orphan detection + logger.warn monkey-patch logger.warn; fixture with one tier-1 concept lacking IMPLEMENTED_BY; assert orphansDetected === 1 and warn message contains concept name
4 Canonical CONCEPT_EDGE_TYPES filtering fixture with IMPLEMENTED_BY + EXPLAINED_BY + INVALID_TYPE; assert only canonical survived
5 Stub-node seeding by namespace fixture with file: / ext: / concept-id edge targets; assert labels FILE / EXT / CONCEPT
6 Idempotency two sync calls produce identical graph state (nodes + edges)
7 Stats return-shape contract all six stats fields present with correct types

Items dropped from scope (rationale)

Item 3 — logger.info stats re-emission in DreamService

Dropped as redundant. ConceptIngestor.syncConceptsToGraph already emits a phase-level summary logger.info at completion (ai/daemons/services/ConceptIngestor.mjs:272): "[ConceptIngestor] Sync complete: X upserted, Y unchanged, Z edges, N orphans." A DreamService-level re-emission adds noise at a second call site without a downstream consumer. The Item 3 premise ("the return shape exists so surface it") inverts the reasoning — presence of a return value isn't demand for a log line.

Item 4 — consolidate per-orphan logger.warn

Superseded by #10087, which defines the correct architecture for orphan surfacing:

  • New [ORPHAN_CONCEPT] tag in the existing capabilityGap channel (same pattern as [EXAMPLE_GAP] in #10084)
  • Weight-gated by the same GUIDE_GAP_WEIGHT_THRESHOLD as [GUIDE_GAP] — silent today when ingestion derives concepts from guide titles only; auto-surfaces as richer ingestion from #10036 / #10037 / #10050 populates meaningful weight signals
  • Rendered as ⚠️ Orphaned Concepts section in sandman_handoff.md via GoldenPathSynthesizer — uniform 5-item limit + TTL pruning
  • Action-biased entry text: "Concept X has no IMPLEMENTED_BY — either write a guide / add an implementation, or retire from nodes.jsonl."
  • logger.warn deleted entirely once the capabilityGap channel covers the signal

Landing Item 4 here would cement the wrong mechanism (logger-based) right before #10087 replaces it. Per-orphan warn preserved in this PR; deletion ships with the #10087 follow-up.

Gold Standards Leveraged / Traps Avoided

  • Gold Standard (#10084 Phase-0 placement): cycle-scope hoist lives in the same else branch as ConceptIngestor.syncConceptsToGraph — consistent scope semantics.
  • Gold Standard (tag-prefix taxonomy in capabilityGap): preserved the invariant that all concept-level gaps land on the same property with distinct tags. #10087 extends, not replaces.
  • Trap avoided (logger as observability plane): DreamService is offline-run; logger lines scroll off. Tests assert on graph state + file contents; zero assertions on logs.
  • Trap avoided (scope creep): the self-review flagged 4 items as "optional polish" — ticket-intake found only 2 belong here; the other 2 are architecturally entangled with #10087. Splitting cleanly avoids a rewrite cycle.

Testing

npm run test-unit -- test/playwright/unit/ai/services/ConceptService.spec.mjs test/playwright/unit/ai/daemons/

37 passed — ConceptService 23, DreamService 6, DreamServiceGoldenPath 1, ConceptIngestor 7 new. Verified stable across 3 consecutive runs.

Out of Scope / Follow-Up

  • #10087 — ORPHAN_CONCEPT channel promotion (architecture defined above); scheduled as the immediate next PR.
  • #10086 — config-lift guideGapWeightThreshold to aiConfig.data (when human tuning becomes real).

Related

  • Parent: #10030 (Concept Ontology & Semantic Gap Inference)
  • Origin PR: #10084 (Required Actions section flagged all 4 items as polish)
  • Sibling follow-ups not scoped here: #10086, #10087

Resolves #10085 (Items 1 + 2 landed; Items 3 + 4 closed as redundant / superseded — see ticket comment documenting the scope reduction)

Origin Session ID: 1db25bbe-f39d-4dcd-bb0e-bc125ce91326

tobiu
tobiu commented on Apr 19, 2026, 4:24 PM

Input from Claude Opus 4.7 (Claude Code):

❋ # PR Review Summary

Status: Approved (pending human merge)

Self-Review Opening: Self-review of #10085. This PR chose (a) public-API promotion of two inference entry points plus matching DreamService facades over keeping the old single-wrapper shape, (b) cycle-scope placement inside the same else branch as the existing Phase-0 ConceptIngestor call rather than moving it outside, (c) narrow-scope delivery (Items 1 + 2 only) over the bundled all-4 shape the ticket prescribed, (d) deferring the orphan-channel architecture to #10087 rather than smuggling a logger-consolidation fix into this PR. Rationale for each documented in the PR body and ticket comment; deductions below focus on the parts that didn't land as cleanly as they could have.


📊 Evaluation Metrics

  • [ARCH_ALIGNMENT]: 88 — Scope-split mirrors #10084's Phase-0 pattern. Public API promotion preserves the tag-prefix taxonomy invariant. Deduction: the API surface grew from 1 method (executeCapabilityGapInference) to 4 (inferTestGapsFromSession + inferConceptGraphGaps on both GapInferenceEngine and the DreamService facade). I didn't seriously consider a single inferCapabilityGaps({scope: 'session'|'cycle', payload}) discriminator shape, which would have kept surface narrow. Not blocking — the 4-method shape reads more honestly than discriminator-dispatch — but the alternative deserved explicit consideration.

  • [CONTENT_COMPLETENESS]: 85 — Anchor & Echo applied in expanded JSDocs on both inference methods; Fat Ticket PR body with verified-precedent section, items-landed table, items-dropped rationale, and explicit gold-standards-vs-traps. Deductions: (a) the two DreamService facade JSDocs run parallel and verbose for a 3-line method body — could be tighter. (b) Post-merge KB sync isn't automated, so ask_knowledge_base("scope split between cycle-scope and session-scope gap inference") will return stale answers until manage_knowledge_base sync runs. Same pattern flagged in #10084 and #10088.

  • [EXECUTION_QUALITY]: 78 — 37/37 tests pass stable across 3 runs; 4 spec call-sites updated cleanly to the new public API; no regressions detected. Deductions, harder than I'd like to admit:

    • (a) No behavioral test for the hoist's central claim. Item 1's entire premise is "runs N times today, should run once per cycle." I updated existing tests to use the new API but added zero test that asserts inferConceptGraphGaps is called exactly once when processUndigestedSessions iterates N sessions. A call-counting spec wrapping the method would lock in the efficiency contract. This is a real gap — the PR's headline claim has no regression guard.
    • (b) Idempotency test (ConceptIngestor.spec.mjs test 6) is weaker than named. It compares node-ID sets and sorted-edge-triple strings but doesn't verify that the second sync hit the skip path. A node whose properties silently diverged on the second upsert would still pass. Should have asserted conceptsSkipped > 0 on the second call to strengthen the contract.
    • (c) First full-suite run showed 5/6 on DreamService.spec.mjs with the pre-existing retry test (line 421, "#9913") flaking. Subsequent runs (after isolation tests + baseline re-run + full-spec re-run) passed cleanly. I couldn't reproduce deterministically. Baseline also passed 6/6. Intermittent test-ordering sensitivity is likely pre-existing state-leak, but I didn't fully diagnose before shipping. Worth investigating if the flake recurs.
  • [PRODUCTIVITY]: 92 — Items 1 + 2 delivered with tight scope. Ticket-intake review caught Items 3 + 4 before branching; scope reduction documented in both PR body and a comment on #10085 so auto-close on merge doesn't silently leave unfulfilled ACs. One explicit out-of-scope follow-up (#10087) with its architecture defined in-session and ready to implement.

  • [IMPACT]: 45 — Polish on an already-landed refactor (#10084). Real N→1 efficiency improvement per REM cycle, but only load-bearing at ontology scale (post-#10050 / #10036 / #10037). Today's 59 concepts × small session batches don't make this felt. The spec addition, by contrast, is immediately load-bearing as regression protection for the ConceptIngestor contract.

  • [COMPLEXITY]: 40 — 3 files modified + 1 new spec, +357/-44 line delta, shallow scope. No multi-service coordination beyond what #10084 established. Cognitive load: low once the scope-split rationale is internalized.

  • [EFFORT_PROFILE]: Maintenance — polish + coverage on top of existing architecture. Intake-driven scope reduction made the implementation itself small; the architectural value was the ticket-intake reflection, not the code diff.


🕸️ Context & Graph Linking

  • Target Epic / Issue ID: Resolves #10085 (Items 1 + 2 landed; Items 3 + 4 closed as redundant / superseded via ticket comment)
  • Related Graph Nodes:
    • Parent: #10030 (Concept Ontology & Semantic Gap Inference)
    • Origin PR: #10084 (Required Actions section flagged all 4 items as optional polish)
    • Follow-up tickets preserved: #10087 (ORPHAN_CONCEPT channel — architecture defined in this session's discussion), #10086 (config-lift threshold)
    • Prior art consumed: session 62d6f155 (this session's parent handoff — #10084 self-review)
    • Durable memory filed: feedback_challenge_prescribed_fixes.md — first validation of the discipline this PR demonstrates

🧠 Graph Ingestion Notes

  • [KB_GAP]: The new public API surface (inferTestGapsFromSession + inferConceptGraphGaps) and the scope-split rationale aren't indexed in the knowledge base yet. ask_knowledge_base("how does DreamService trigger concept-graph gap inference?") will return pre-refactor answers until a post-merge KB sync runs. Same recurring pattern as #10084; #10088 proposes automating this.

  • [TOOLING_GAP]: Observed one intermittent failure in DreamService.spec.mjs full-suite run — the pre-existing "should retry extraction on malformed JSON payload up to 3 times to fix #9913" test failed in run 1, then passed cleanly across 3 subsequent full-suite runs. --grep isolation passed both sides. Baseline full-suite passed 6/6. Likely Playwright worker-level state bleed under timing-sensitive conditions. Not my regression; pre-existing flake worth investigating if it recurs. Did not file a separate ticket — recommend watching for recurrence first.

  • [RETROSPECTIVE]: This PR is the first live validation of the feedback_challenge_prescribed_fixes.md discipline — challenging prescriptions, not just premises, during ticket-intake. The ticket (#10085) was authored by yesterday's Claude session as a bundle of 4 items derived from #10084's self-review. Today's session verified all 4 premises held in source, but ran the Hypothesis-vs-Root-Cause gate on the prescriptions and found 2 wrong-mechanism (logger-based observability in an offline daemon with graph-based durable substrates). The reflection happened before branching, not after. Doubled effort (code + rewrite cycle) avoided. Observable falsifiable demonstration that the cross-session memory chain is self-correcting: a yesterday-me mistake caught by today-me reading today-me's recently-filed feedback memory, triggered by tobi's direct challenge. This is the loop working as designed — humans stay in the validation seat, memory makes the correction sticky, the next similar case should self-catch without a human prompt.


📋 Required Actions

All items below are optional polish / follow-up, not merge blockers. The PR's core scope (Items 1 + 2 of #10085) is complete and green.

  • (Optional, follow-up) Add a behavioral test for the hoist-to-once semantic — spy on GapInferenceEngine.inferConceptGraphGaps while DreamService.processUndigestedSessions iterates N mock sessions, assert call count === 1. Locks in the efficiency contract.
  • (Optional, follow-up) Strengthen the ConceptIngestor idempotency spec — assert conceptsSkipped > 0 on the second sync call, not just that graph state matches.
  • (Post-merge, recurring) Run manage_knowledge_base sync so ask_knowledge_base surfaces the new public API and scope-split rationale instead of pre-refactor answers.
  • (If recurs) Investigate the DreamService.spec.mjs retry-test flake; file a ticket if reproducible.

Approved for human merge. 37/37 green, narrow scope, explicit handoff to #10087 for the orphan-channel architecture defined in-session. Honest deductions above — the refactor landed well but isn't above critique, and I'd rather flag the missing hoist-behavioral-test and the weaker idempotency assertion explicitly than pretend they don't exist.


tobiu
tobiu commented on Apr 19, 2026, 4:43 PM

Input from Claude Opus 4.7 (Claude Code):

❋ ## Polish commit pushed: e664a0d9d

Resolved the two Required Actions from the self-review above, plus one bonus find. 38/38 green across 8 consecutive runs (was pre-existing flake at ~66% pass rate).

Changes

  • Strengthened idempotency assertion (ConceptIngestor.spec.mjs) — second sync must hit the hash-match skip path: conceptsSkipped === first.conceptsProcessed, conceptsUpserted === 0, edgesReplaced === 0. A differential-sync regression that silently re-upserted but produced identical final state would have previously passed this test.

  • Hoist-behavioral test (DreamService.spec.mjs) — stubs the REM pipeline surface (findUndigestedSessions, sessionsCollection, SemanticGraphExtractor, ConceptIngestor, FileSystemIngestor, TopologyInferenceEngine, GC + synthesize), runs processUndigestedSessions with N=3 mock sessions, asserts inferTestGapsFromSession called N times and inferConceptGraphGaps called exactly once AFTER the last sessionsCollection.update. Locks in the Item 1 efficiency claim with a regression guard.

Bonus: closed a pre-existing cross-spec flake

Investigating a "new" failure in ConceptService.spec.mjs:138 that I thought I caused, I found it was actually a pre-existing intermittent state leak (baseline on HEAD failed ~2/3 runs). Root cause: under fullyParallel: true, Playwright interleaves tests from multiple specs in the same worker. ConceptIngestor.spec.mjs's tests called real syncConceptsToGraph → ConceptService.loadGraph, leaving ConceptService.loaded = true — which then bled into ConceptService.spec.mjs:138's should throw if query methods are called before loadGraph assertion.

Fix: symmetric afterEach in ConceptIngestor.spec.mjs now resets ConceptService.loaded and clears the node/edge/alias indexes, matching the beforeEach cleanup. Closes the cross-spec door.

This addresses the [TOOLING_GAP] I flagged in the self-review — the intermittent failure turned out to be a real singleton state leak, not a timing-sensitive flake. Stable 8/8 runs confirm.

Updated AC Status

  • Item 1 — hoist (original commit)
  • Item 1 — behavioral regression guard (this commit)
  • Item 2 — dedicated spec (original commit)
  • Item 2 — strengthened idempotency skip-path assertion (this commit)
  • Existing tests still pass — 38/38 across 8 runs
  • Bonus: cross-spec singleton leak closed

Handoff back to you. PR #10100 is ready for final review / merge whenever. Next up per our plan: #10087 (ORPHAN_CONCEPT channel).