LearnNewsExamplesServices
Frontmatter
id10085
title[enhancement] ConceptIngestor follow-ups post-#10035 — hoist concept-graph pass to cycle-scope + dedicated spec + observability polish
stateClosed
labels
enhancementaiarchitecture
assigneestobiu
createdAtApr 19, 2026, 12:31 PM
updatedAtApr 19, 2026, 6:19 PM
githubUrlhttps://github.com/neomjs/neo/issues/10085
authortobiu
commentsCount1
parentIssue10030
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtApr 19, 2026, 4:45 PM

[enhancement] ConceptIngestor follow-ups post-#10035 — hoist concept-graph pass to cycle-scope + dedicated spec + observability polish

Closed v13.0.0/archive-v13-0-0-chunk-4 enhancementaiarchitecture
tobiu
tobiu commented on Apr 19, 2026, 12:31 PM

Context

Follow-up items surfaced during the self-review of PR #10084 (resolves #10035). The refactor landed cleanly (all 30 tests green), but four polish items were deferred as explicit out-of-scope flags rather than smuggled into the main PR. This ticket bundles them because all four touch the same two files (ai/daemons/DreamService.mjs + ai/daemons/services/ConceptIngestor.mjs) and sequencing them together avoids conflicting patches.

The Four Items

1. Hoist inferConceptGraphGaps call from per-session to cycle-scope (efficiency)

Problem: GapInferenceEngine.executeCapabilityGapInference(session, payload) invokes both inferTestGapsFromSession (session-scoped, correct) AND inferConceptGraphGaps (ontology-scoped, wasteful in a loop) once per session. DreamService's REM loop iterates N sessions per cycle, so the concept-graph pass runs N times with the same output. Correctness is unaffected (writes are idempotent), but it's wasted work that grows with ontology + session count.

Fix: Extract inferConceptGraphGaps to a standalone method callable on GapInferenceEngine; invoke it once from DreamService.processUndigestedSessions after the per-session loop, before runGarbageCollection.

2. Create test/playwright/unit/ai/daemons/services/ConceptIngestor.spec.mjs (coverage)

Problem: Currently tested only indirectly via DreamService integration tests. The service's contract at the unit level isn't locked in.

Coverage targets:

  • Hash-based differential sync — skip path (unchanged payload) + upsert path (changed payload)
  • Orphan detection (no IMPLEMENTED_BY → logger.warn)
  • Edge-type normalization via CONCEPT_EDGE_TYPES constants
  • Stub-node seeding by namespace (file: → FILE, ext: → EXT, fallback → CONCEPT)
  • Idempotency (two consecutive sync calls produce identical graph state)
  • Stats return shape ({conceptsProcessed, conceptsUpserted, conceptsSkipped, edgesReplaced, orphansDetected, errors})

3. Log syncConceptsToGraph returned stats in DreamService (observability)

Problem: await ConceptIngestor.syncConceptsToGraph() returns a stats object that DreamService currently discards. The return shape exists for observability; dropping it on the floor means cycle-time / differential-sync benefits aren't visible in REM cycle logs.

Fix: Capture the returned stats; emit via logger.info('[DreamService] Concept ingestion: ...', stats) before the per-session loop begins.

4. Consolidate per-orphan logger.warn into summary (log hygiene)

Problem: ConceptIngestor.syncConceptsToGraph emits one logger.warn per orphan concept. At 17 orphans today × N REM cycles per day = log spam. Grows linearly with ontology size.

Fix: Accumulate orphan IDs/names into an array during the sync loop; emit a single summary warn at completion with count + top N names (e.g., first 10 by weight).

Acceptance Criteria

  • inferConceptGraphGaps hoisted from per-session to cycle-scope in DreamService
  • test/playwright/unit/ai/daemons/services/ConceptIngestor.spec.mjs created with the six coverage targets above
  • Stats logging surfaces in DreamService REM cycle output
  • Orphan warnings consolidated into summary form
  • Existing PR #10084 tests still pass (regression guard)

Origin Session ID

62d6f155-e57f-4279-9b59-36c9e4ecbc5e — surfaced in PR #10084 self-review comment

Related

  • Parent epic: #10030 (Concept Ontology & Semantic Gap Inference)
  • Origin PR: #10084 (merged) — Required Actions section of the self-review
  • Sibling follow-ups filed alongside: #[config-lift threshold], #[ORPHAN_CONCEPT channel]
tobiu added the enhancement label on Apr 19, 2026, 12:31 PM
tobiu added the ai label on Apr 19, 2026, 12:31 PM
tobiu added the architecture label on Apr 19, 2026, 12:31 PM
tobiu added parent issue #10030 on Apr 19, 2026, 12:32 PM
tobiu cross-referenced by #10030 on Apr 19, 2026, 12:33 PM
tobiu assigned to @tobiu on Apr 19, 2026, 4:03 PM
tobiu referenced in commit 6f0263f - "refactor(ai): hoist concept-graph gap pass to cycle-scope (#10085) on Apr 19, 2026, 4:16 PM
tobiu cross-referenced by PR #10100 on Apr 19, 2026, 4:17 PM
tobiu
tobiu Apr 19, 2026, 4:19 PM

Input from Claude Opus 4.7 (Claude Code):

❋ ## Scope Reduction — PR #10100 closes Items 1 + 2 only

Ticket-intake review (per .agent/skills/ticket-intake/references/ticket-intake-workflow.md §1.4, Hypothesis-vs-Root-Cause Validation) found that Items 3 and 4 fail the prescription-level gate: both prescribe logger-based fixes for a daemon where logger output is ephemeral and durable observability lives on graph-node capabilityGap + sandman_handoff.md rendering.

Items landed in #10100

  • Item 1 — Hoisted inferConceptGraphGaps from per-session to cycle-scope in DreamService.processUndigestedSessions. Dropped the executeCapabilityGapInference wrapper; promoted both inference entry points to public API with matching thin facades on DreamService.
  • Item 2 — New test/playwright/unit/ai/daemons/services/ConceptIngestor.spec.mjs with 6 AC coverage targets plus stats-shape contract (7 tests total). Full daemon + ConceptService suite: 37/37 green across 3 runs.

Items dropped (rationale)

  • Item 3logger.info stats re-emission in DreamService Redundant. ConceptIngestor.syncConceptsToGraph already emits a phase-level summary logger.info at completion (ai/daemons/services/ConceptIngestor.mjs:272). A DreamService-level re-emission adds noise without a consumer. The Item 3 premise ("the return shape exists so surface it") inverts the reasoning — presence of a return value isn't demand for a log line.

  • Item 4 — Consolidate per-orphan logger.warn Superseded by #10087. The correct architecture for orphan surfacing is:

    • New [ORPHAN_CONCEPT] tag in the existing capabilityGap channel (same pattern as [EXAMPLE_GAP] in #10084)
    • Weight-gated by the same GUIDE_GAP_WEIGHT_THRESHOLD as [GUIDE_GAP] — silent today when ingestion derives concepts from guide titles only; auto-surfaces as richer ingestion from #10036 / #10037 / #10050 populates meaningful weight signals
    • Rendered as ⚠️ Orphaned Concepts section in sandman_handoff.md via GoldenPathSynthesizer — uniform 5-item limit + TTL pruning
    • logger.warn deleted entirely once the capabilityGap channel covers the signal

    Landing Item 4 here would cement the wrong mechanism (logger-based) right before #10087 replaces it. Per-orphan warn preserved in #10100; deletion ships with #10087.

AC status after scope reduction

  • inferConceptGraphGaps hoisted from per-session to cycle-scope in DreamService
  • test/playwright/unit/ai/daemons/services/ConceptIngestor.spec.mjs created with the six coverage targets
  • Stats logging surfaces in DreamService REM cycle output — dropped (redundant with ConceptIngestor.mjs:272)
  • Orphan warnings consolidated into summary form — superseded by #10087 (architectural channel promotion)
  • Existing PR #10084 tests still pass (regression guard) — 37/37 green

On merge, closing the ticket is correct: Items 3 and 4 are addressed by reframing, not by code here. The full rationale is captured in the PR body and this comment so humans and future agents landing on #10085 see why 2 of 4 items were dropped without silent abandonment.

Origin Session ID: 1db25bbe-f39d-4dcd-bb0e-bc125ce91326

tobiu referenced in commit e664a0d - "test(ai): strengthen ConceptIngestor spec + hoist-behavioral test (#10085) on Apr 19, 2026, 4:42 PM
tobiu referenced in commit 45da7fc - "refactor(ai): hoist concept-graph gap pass to cycle-scope (#10085) (#10100) on Apr 19, 2026, 4:45 PM
tobiu closed this issue on Apr 19, 2026, 4:45 PM
tobiu cross-referenced by PR #10101 on Apr 19, 2026, 4:59 PM
tobiu cross-referenced by PR #10102 on Apr 19, 2026, 5:27 PM
tobiu cross-referenced by #10104 on Apr 19, 2026, 5:39 PM
tobiu cross-referenced by PR #10105 on Apr 19, 2026, 6:05 PM
tobiu cross-referenced by PR #10116 on Apr 20, 2026, 1:40 AM