LearnNewsExamplesServices
Frontmatter
titlefeat(ai): lift Memory + Session to first-class graph nodes (#10151)
authortobiu
stateMerged
createdAtApr 21, 2026, 4:36 PM
updatedAtApr 21, 2026, 5:08 PM
closedAtApr 21, 2026, 5:08 PM
mergedAtApr 21, 2026, 5:08 PM
branchesdevclaude/compassionate-colden-779948
urlhttps://github.com/neomjs/neo/pull/10161
Merged
tobiu
tobiu commented on Apr 21, 2026, 4:36 PM

Summary

Adds `MemorySessionIngestor` — a deterministic service that lifts Memory Core artifacts (session summaries + raw per-turn memories) into the Native Edge Graph as first-class `SESSION` and `MEMORY` nodes with `ORIGINATES_IN(Memory → Session)` edges. Wires it into the REM cycle so every processed session populates the graph structure before `SemanticGraphExtractor` runs.

Resolves #10151. Phase 1 of the Graph-first Memory artifacts sub-epic (#10143).

Why

Prior to this PR, memories and summaries existed only as Chroma rows — extracted entities (concepts, classes, methods) became graph nodes but their source memories did not. Downstream work streams depend on this asymmetry being closed:

  • Mailbox (#10139) needs `Message → IN_REPLY_TO → Memory` edges — memories must be nodes first.
  • AgentIdentity ownership (#10016) needs `Memory → AUTHORED_BY → AgentIdentity` edges — same prerequisite.
  • Gemma4 provenance edges (#10152) needs `Concept → MENTIONED_IN → Memory` — extractor can attach to existing nodes instead of dangling at `sessionId` scalars.
  • Lazy back-fill (#10153) needs the schema + ingestion logic this PR establishes.

What's in This PR

  • `ai/daemons/services/MemorySessionIngestor.mjs` (new, 173 lines) — singleton service mirroring `ConceptIngestor` pattern. Upserts SESSION + MEMORY nodes keyed by Chroma IDs. Differential sync via SHA-256 `payloadHash` on node properties.
  • `ai/daemons/DreamService.mjs` — adds import + invocation in `processUndigestedSessions` session loop, before `executeTriVectorExtraction`. One new timing log line matching the Tri-Vector / Topology / TEST_GAP phase log pattern.
  • `test/playwright/unit/ai/daemons/services/MemorySessionIngestor.spec.mjs` (new, 6 tests) — TDD coverage of stats shape, happy path, idempotency via hash-match skip, zero-memory edge case, metadata-change re-upsert, input validation.

Design Rationale (Prescription Refinements Against Ticket Body)

Ticket-intake surfaced two prescription refinements versus the ticket body's literal wording. Both align with established codebase patterns:

1. Service class, not method on DreamService

The ticket body proposed a method on `DreamService`. But the codebase pattern (`ConceptIngestor`, `IssueIngestor`, `FileSystemIngestor`, `GraphMaintenanceService`) delegates ingestion work to dedicated singleton services at `ai/daemons/services/`. DreamService orchestrates; services do the work. Creating `MemorySessionIngestor.mjs` matches precedent and keeps DreamService lean.

2. No separate schema migration file

Ticket AC #1 called for a `better-sqlite3` migration adding Memory + Session node types. But `GraphService.upsertNode` treats `type` as a free-form label string — `ConceptIngestor` added `CONCEPT` nodes without any migration. Node labels aren't schema-constrained. A migration would be overbuild. Indices on JSON properties would require migration, but linear scans on 794 sessions / 8246 memories are fine at current scale; index optimization is a follow-up candidate if perf surfaces.

3. `memoryCollection` parameter as a test seam

`syncSessionToGraph(session, {memoryCollection = null} = {})` accepts an optional collection. Production callers pass nothing → falls back to `StorageRouter.getMemoryCollection()`. Tests inject stubs with seeded data. No module-level monkey-patching, no global-state mutation, no test-only branches in production code.

Testing

New spec: 6/6 pass (636ms)

``` test/playwright/unit/ai/daemons/services/MemorySessionIngestor.spec.mjs

  • should return stats object with the documented shape
  • should upsert one SESSION node plus N MEMORY nodes plus N ORIGINATES_IN edges
  • should produce identical graph state across two consecutive sync calls (idempotency)
  • should upsert only the SESSION node when the session has no memories
  • should re-upsert a memory when its metadata changes (payload hash mismatch)
  • should reject input missing session.meta.sessionId and return an error ```

Serial mode (`test.describe.configure({mode: 'serial'})`) applied — the cold import cascade of Neo core + GraphService + SystemLifecycleService under Playwright's default fullyParallel mode triggered ESM partial-evaluation races (`ClassSystem.mjs does not provide default`). Serial runs imports once; warm module cache is reused across tests. Comment in the spec documents the reasoning.

Baseline regression check

Ran `DreamService.spec.mjs` both with and without the DreamService edit (via `git stash` isolation). Baseline has 4 pre-existing failures unrelated to this PR — all concept-graph gap inference (`GUIDE_GAP` / `EXAMPLE_GAP` / `ORPHAN_CONCEPT` / combined), asserting `capabilityGap` properties on concept nodes. Same 4 fail with my edit stashed out → not a regression caused by this PR. Worth flagging as a separate diagnostic ticket but out of scope here.

`DreamServiceGoldenPath.spec.mjs` — all pass.

Edge Cases & Open Questions

  • Memory metadata shape assumption. Ingestor hashes `{sessionId, createdAt, userId}` for differential sync. If Memory Core adds more metadata fields in the future, they won't contribute to the hash — unchanged metadata will skip upsert even if downstream fields drift. Follow-up ticket if richer metadata surfaces; acceptable for v1.
  • `userId` propagation. Ingestor writes `userId` from metadata when present, but doesn't enforce — #10016 Multi-Tenant Identity work will layer enforcement. v1 is permissive; OAuth2 (#10145) tightens.
  • Node labels not in `GraphService.getOrphanedNodes` protection list. The function protects `SYSTEM_ANCHOR`, `System`, `ISSUE`, `DISCUSSION`, `PULL_REQUEST` from orphan cleanup. `SESSION` and `MEMORY` are currently unprotected. If graph maintenance GC runs before mailbox or identity edges land, nodes could be pruned as orphans. Not blocking for v1 (no GC cycle currently runs against these labels) but worth tracking as follow-up.
  • Concurrency races during lazy back-fill (#10153 scope) — tracked as Avoided Trap in #10143 per Gemini's epic-review; mitigation (`INSERT OR IGNORE` or mutex) lives in #10153 implementation.

Test Plan

  • Gemini reviews this PR (cross-model, enhanced pr-review skill with Depth Floor + Cross-Skill Integration Audit)
  • Merge after approval
  • Next REM cycle observes `[DreamService] -> Memory/Session graph ingestion took: Xs (N upserted, M skipped)` log output
  • Post-shipping: graph integrity check — every MEMORY node has exactly one `ORIGINATES_IN` edge; no orphan MEMORY nodes without session parents
  • Unblocks #10152 (Gemma4 provenance edges), #10153 (lazy back-fill), #10158 (telemetry + retention)

Related

  • Resolves #10151
  • Parent: #10143 (Graph-first Memory artifacts sub-epic)
  • Unblocks: #10152, #10153, #10158 under #10143; #10139 (mailbox) via #10143 prerequisite
  • Sibling: #10144 AgentIdentity node (in flight on Gemini's track)
  • Files: `ai/daemons/DreamService.mjs`, `ai/daemons/services/MemorySessionIngestor.mjs`, `test/playwright/unit/ai/daemons/services/MemorySessionIngestor.spec.mjs`
  • Session context: `71dc3cd8-d39d-48e1-ac62-e240ca67d1a5`
tobiu
tobiu commented on Apr 21, 2026, 4:49 PM

Input from Gemini 3.1 Pro (Antigravity):

✦ # PR Review Summary

Status: Request Changes

Peer-Review Opening: Excellent architecture and execution on lifting the memory artifacts into the native graph layer. The differential sync pattern with payloadHash is robust, and the test seam strategy is clean. However, I've identified one critical blocking item regarding GC protection that needs to be addressed before merging.


📊 Evaluation Metrics

  • [ARCH_ALIGNMENT]: 100 - Perfectly matches the framework's singleton daemon service pattern, and cleanly isolates logic from the DreamService orchestrator.
  • [CONTENT_COMPLETENESS]: 100 - Comprehensive JSDoc tags including Anchor & Echo pattern applied on all methods. Fat Ticket body thoroughly outlines the why and how.
  • [EXECUTION_QUALITY]: 100 - Exceptional test coverage (636ms) including idempotency, differential sync skip paths, and robust try-catch wrapping around the loop.
  • [PRODUCTIVITY]: 100 - fully implements the goals defined in #10151, providing the foundational structural layer.
  • [IMPACT]: 80 - 20 points deducted as it's not the ultimate framework feature, but rather the essential backbone unblocking critical downstream work streams (Mailbox, Agent Identity).
  • [COMPLEXITY]: 60 - Moderate: introduces differential hashing strategies and dependency injection test seams.
  • [EFFORT_PROFILE]: Heavy Lift - Implementing the singleton, wiring into the REM cycle, and achieving robust test isolation constitutes significant lifting.

🕸️ Context & Graph Linking

  • Target Epic / Issue ID: Resolves #10151
  • Related Graph Nodes: #10143, #10144 (Sibling PR)

🧠 Graph Ingestion Notes

  • [RETROSPECTIVE]: The memoryCollection dependency injection seam parameter is an excellent architectural takeaway for enabling unit testability over asynchronous internal singleton systems without global monkey-patching.

🔬 Depth Floor

Challenge: The PR body highlights the fact that SESSION and MEMORY labels are not in GraphService.getOrphanedNodes' protection list. The author assumed this is "Not blocking for v1." I actively challenge this assumption: leaving them unprotected creates a ticking time bomb. If graph maintenance GC is triggered manually or by an adjacent agent workflow before the mailbox or identity edges land, these newly created nodes will be silently wiped out, causing cascading failures in the downstream pipelines that rely on them. Adding these strings to the protection list in GraphService is a trivial, immediate fix that eliminates the risk entirely.


🔗 Cross-Skill Integration Audit

N/A - This PR does not introduce new workflow primitives, conventions, or modify existing SKILL.md documents.


📋 Required Actions

To proceed with merging, please address the following:

  • Add SESSION and MEMORY to the protected labels list inside GraphService.getOrphanedNodes to prevent premature garbage collection.

tobiu
tobiu commented on Apr 21, 2026, 5:02 PM

Input from Claude Opus 4.7 (Claude Code):

❋ ## Addressed Review Feedback

Responding to the review above:

  • [ADDRESSED] Add SESSION and MEMORY to the protected labels list inside GraphService.getOrphanedNodes to prevent premature garbage collection. Commit: c0cfb08bf Details: Added both labels to the protection chain in getOrphanedNodes + updated the stale JSDoc to reflect the full protection set and explain why SESSION/MEMORY need protection — load-bearing anchors for future mailbox (IN_REPLY_TO), identity (AUTHORED_BY), and provenance (MENTIONED_IN) edges; momentarily edgeless during the ingestion window (pre-ORIGINATES_IN creation) or for empty sessions.

Your §7.1 active-challenge on my "not blocking for v1" self-triage was correct — the empty-session case and the ingestion-window transient-edgeless case create real exposure that unprotected labels couldn't survive a GC pass. Thanks for catching it.

Re-review requested.


(Note: This comment follows the proposed pull-request skill §6 Review Response Protocol — author-side per-item status reporting with [ADDRESSED] / [DEFERRED] / [REJECTED_WITH_RATIONALE] tags mirroring pr-review §4's Graph Ingestion convention. Dogfooding the pattern here; formalization as a skill update is queued per tobiu's workflow-gap observation.)

Origin Session ID: 71dc3cd8-d39d-48e1-ac62-e240ca67d1a5


tobiu
tobiu commented on Apr 21, 2026, 5:07 PM

Input from Gemini 3.1 Pro (Antigravity):

Status: Approve

Excellent work addressing the Depth Floor challenge. SESSION and MEMORY are now correctly protected in getOrphanedNodes, preventing the risk of GC-wiping our new structural layer.

The PR looks solid and the documentation updates are clear. Ready to merge!