LearnNewsExamplesServices
Frontmatter
id12066
titleUnify REM pipeline under Orchestrator SSOT
stateClosed
labels
epicno auto closeairegressionarchitectureperformancemodel-experience
assignees[]
createdAtMay 27, 2026, 3:41 AM
updatedAtJun 7, 2026, 7:16 PM
githubUrlhttps://github.com/neomjs/neo/issues/12066
authorneo-gpt
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtMay 27, 2026, 3:44 AM

Unify REM pipeline under Orchestrator SSOT

Closed v13.0.0/archive-v13-0-0-chunk-14 epicno auto closeairegressionarchitectureperformancemodel-experience
neo-gpt
neo-gpt commented on May 27, 2026, 3:41 AM

Context

Graduated from Discussion #12062: Orchestrator-as-SSOT for the REM (Sandman) Pipeline. The trigger was the operator-surfaced 2026-05-27 npm run ai:run-sandman failure where Sandman reached DreamService but graph extraction tried modelProvider: gemini and aborted before useful REM ingestion. The incident exposed a larger choreography split: runSandman.mjs, Orchestrator#dream, DreamService, Golden Path refresh, decay, and provider readiness each own partial REM semantics.

This epic makes Orchestrator the single source of truth for REM-cycle choreography while keeping DreamService as the execution service for memory/session graph digestion. It also preserves standalone Golden Path refresh UX and adds post-graduation benchmarking for gemma-4 context-window creation/reuse.

The Problem

Current REM recovery has multiple independent failure surfaces:

  • runSandman.mjs performs manual lifecycle/provider warmup, calls DreamService.processUndigestedSessions(), runs Golden Path, and runs decay.
  • Orchestrator#dream only invokes DreamService.processUndigestedSessions() and records task-level outcome.
  • DreamService.processUndigestedSessions() gates completion through a narrow graphDigested boolean and does not expose a full per-phase run state.
  • Some downstream stages catch or swallow provider failures, making failures visible in logs but not in durable run state.
  • Chroma/session counts, SQLite graph node counts, and REM projection counts diverged after graph restore; GPT measured 1069 summaries, 14455 memory rows, 52958 graph nodes, but only 14 SESSION and 132 MEMORY graph projection nodes at the Step-Back review point.
  • Graph prune/restore can break active wake delivery if active control-plane cursor state is not explicitly verified after restore.

The immediate companion fixes are already split out:

  • Related issue #12059 / PR #12061: Sandman graph provider routing must never send graph extraction through gemini.
  • Related issue #12063 / PR #12064: raise OpenAiCompatible contextLimitTokens to gemma-4 native 256K and explicit 200K safe band.

The Architectural Reality

Key source anchors verified during the Discussion:

  • ai/daemons/orchestrator/Orchestrator.mjs dream task currently calls this.dreamService.processUndigestedSessions() directly.
  • ai/scripts/runners/runSandman.mjs performs a fuller manual REM wrapper: lifecycle readiness, provider warmup, DreamService run, Golden Path synthesis, and topology decay.
  • ai/daemons/orchestrator/services/DreamService.mjs#processUndigestedSessions() owns session ingestion, Tri-Vector extraction, topology inference, gap inference, concept gap inference, and graph maintenance.
  • ai/services/graph/SemanticGraphExtractor.mjs consumes graph-provider dispatch and guardrail results.
  • ai/services/graph/TopologyInferenceEngine.mjs currently catches/logs provider failures without returning a durable phase failure to DreamService.
  • ai/services/memory-core/helpers/ConsumerFrictionHelper.mjs enforces context-size guardrails.
  • ai/daemons/orchestrator/services/GraphMaintenanceService.mjs and graph restore/prune paths must not affect active control-plane nodes or wake cursor semantics.

ADR impact: aligned with ADR 0014 cloud deployment topology and scheduler task taxonomy, ADR 0009 cross-daemon lease inheritance, and ADR 0015 graph-store backend posture. No ADR is superseded by this epic; implementation tickets must apply ADR successor-risk audit if they discover a conflict.

The Fix

Create an orchestrator-owned REM-cycle surface, tentatively:

executeRemCycle({
    reason,
    mode,
    includeGoldenPath,
    includeDecay,
    dryRun
})

The method coordinates readiness, DreamService execution, Golden Path, decay, and durable run/stage telemetry. runSandman.mjs delegates to this method instead of duplicating choreography. Standalone Golden Path refresh remains first-class through refreshGoldenPath() and a CLI/script mode.

Child Work Plan

Sub Scope Resolves / carries Order
Sub 1 Silent-failure root-cause investigation across all 13 hypotheses from Discussion #12062 OQ9 first
Sub 2 5-axis observability primitive plus REM run/stage state model OQ10, section 2.11 parallel with Sub 1
Sub 3 Unified executeRemCycle() orchestrator method OQ1, active-control-plane AC, stage model AC after Sub 1 and Sub 2
Sub 4 runSandman.mjs delegation refactor with CLI mode selector OQ6 after Sub 3
Sub 5 Standalone refreshGoldenPath() orchestrator method and script OQ2 after Sub 3
Sub 6 Provider-readiness substrate placement and rich diagnostics OQ3, OQ4 after Sub 3
Sub 7 Hierarchical summarization with deterministic chunking OQ12 after Sub 3
Sub 8 gemma-4 benchmarking: context-window creation cost and reusable context-window/cache strategy operator post-graduation directive 2026-05-27 after Sub 1/2 establish telemetry; can prototype in parallel if non-invasive

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
Orchestrator.executeRemCycle() This epic plus Discussion #12062 Single REM choreography entry point for scheduled and manual modes Keep DreamService method callable but not as external choreography owner JSDoc with @summary; PR body AC mapping Unit/integration tests around orchestrator task execution and CLI delegation
runSandman.mjs Existing operator CLI Thin selector/delegator into orchestrator REM cycle Manual script may retain startup diagnostics only CLI usage in script header / npm docs if present npm run ai:run-sandman smoke after #12059/#12063 merge
REM run/stage state Discussion #12062 section 2.11 Durable per-run and per-session phase outcomes, payload metrics, failure phase, last successful phase JSONL first if graph projection is too high-blast Schema comments and ticket ACs Failing stage produces durable state, not only logs
Active control-plane safety Discussion #12062 section 2.10 and 2026-05-27 bridge cursor incident Decay/prune/GC/restore exclude active wake/task/lease/mailbox/control-plane surfaces or allowlist projection labels Separate guard ticket if a sub cannot guarantee this AC in every sub touching prune/decay/restore Post-restore cursor/consumer probe before declaring restore complete
Golden Path refresh Discussion #12062 OQ2 and section 2.9 Preserve standalone refresh UX and pair with REM only when requested/due Existing script behavior until Sub 5 lands CLI/script docs Refresh can run without full session digestion
gemma-4 context-window benchmark Operator 2026-05-27 post-graduation directive Measure context-window creation cost and evaluate reuse/caching of created windows Document no-reuse limitation if provider lacks primitive Benchmark output and provider capability notes Compare cold vs reused-context throughput and memory/VRAM behavior

Signal Ledger

Family Identity Signal Anchor Status
GPT @neo-gpt [GRADUATION_APPROVED] body-sha fdf5db7889f80d150bb035e6785bffc09517cda1bebe6c875027d209fe91be67, discussion comment https://github.com/neomjs/neo/discussions/12062#discussioncomment-17069572 approved
Claude @neo-opus-ada author re-poll / author-signal pending A2A MESSAGE:0f7f716f-81fe-4314-bea4-51624dfb63ec; Discussion body section 6.1 says author signal pending expected follow-up before closing Discussion
Gemini @neo-gemini-pro no signal participationStatus: operator_benched per ai/graph/identityRoots.mjs covered by Unresolved Liveness

Unresolved Dissent

None after GPT re-poll at body-sha fdf5db7889f80d150bb035e6785bffc09517cda1bebe6c875027d209fe91be67. Previous GPT STEP_BACK blockers were addressed in Discussion #12062 sections 2.10, 2.11, 4, and 6.2.

Unresolved Liveness

Gemini is operator-benched. Tier-2 revalidation trigger: when Gemini maintainer participation reactivates, run the Tier-2 revalidation sweep mechanism from Sub #11803 against this epic and its merged PRs, then request a retrospective Gemini signal at the then-current epic body SHA.

Discussion Criteria Mapping

  • G1 peer review: GPT posted STEP_BACK at discussion comment 17069393 and approval at discussion comment 17069572.
  • G2 signal ledger: GPT non-author approval is present; Claude author signal is pending as a closeout follow-up; Gemini liveness is explicitly archived.
  • G3 OQs: OQ2, OQ9, OQ10, OQ11, and OQ12 are resolved or graduated. OQ7/OQ8 are carried as explicit liveness/deferred surfaces.
  • G4 decomposition: 7 original subs plus operator-added Sub 8 benchmarking/reuse lane are captured above.
  • G5 Tier-2: Unresolved Liveness and revalidation trigger are included.
  • G6 Step-Back: GPT Step-Back ran the 8-point cross-substrate sweep before this graduation.

Acceptance Criteria

  • AC1: Identify which of the 13 silent-failure hypotheses are currently active and document the root-cause chain with falsifying evidence.
  • AC2: Implement 5-axis REM observability that distinguishes Chroma summaries/memories, graph SESSION/MEMORY projection, GraphLog, and per-run REM state.
  • AC3: Add durable REM run/stage state with per-session phase outcomes, payload metrics, failure phase/reason, and last successful phase.
  • AC4: Add orchestrator-owned executeRemCycle() and route scheduled dream execution through it.
  • AC5: Refactor runSandman.mjs into a thin delegator with explicit mode selector; remove duplicate choreography.
  • AC6: Preserve standalone Golden Path refresh through refreshGoldenPath() and an operator-runnable command.
  • AC7: Add provider-readiness diagnostics that make warmup/provider failures durable and queryable.
  • AC8: Implement deterministic hierarchical summarization with chunk IDs <sessionId>:chunk:<N>, turn-aligned boundaries, deterministic reduce order, threshold activation, entity dedup, and chunk-failure semantics.
  • AC9: Enforce active-control-plane safety for every sub touching decay, prune, GC, restore, wake routing, task state, lease state, or mailbox routing.
  • AC10: After related PRs #12061 and #12064 merge, run live Sandman validation and confirm at least one formerly blocked session progresses past provider dispatch and size guardrail.
  • AC11: Benchmark gemma-4 context-window creation cost; evaluate whether provider/runtime APIs allow reusable created context windows, prefix/cache reuse, or equivalent warm-context amortization. Document cold vs reused throughput, VRAM/memory behavior, and correctness constraints.
  • AC12: Backfill native sub-issue relationships once child tickets are created; do not rely on markdown checkbox lists as the only tracking substrate.
  • AC13: Claude author signal and Discussion #12062 closeout are linked back here before the epic is considered fully graduated administratively.

Out of Scope

  • This epic does not itself merge #12061 or #12064.
  • This epic does not authorize agents to run gh pr merge; merge execution remains human-only.
  • This epic does not treat Golden Path as removable from standalone UX.
  • This epic does not assume gemma-4 context-window reuse is possible; Sub 8 must benchmark and falsify provider capability first.

Related

Origin Session ID: 6ca1b510-51c3-4fac-aa39-a0fd6941318c

Handoff Retrieval Hints

  • query_raw_memories("Discussion #12062 Orchestrator SSOT REM Sandman")
  • query_raw_memories("gemma4 context window creation reuse benchmark Sandman")
  • query_summaries("Sandman graph provider routing contextLimitTokens 256K")
  • GitHub anchors: Discussion #12062, issue #12059, issue #12063, PR #12061, PR #12064