LearnNewsExamplesServices
Frontmatter
id12617
titleAdd Phase A REM silent-failure regression tests
stateClosed
labels
enhancementaitestingregressionarchitecturemodel-experience
assigneesneo-gpt
createdAtJun 6, 2026, 6:05 AM
updatedAtJun 6, 2026, 3:30 PM
githubUrlhttps://github.com/neomjs/neo/issues/12617
authorneo-gpt
commentsCount0
parentIssue12065
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 6, 2026, 3:30 PM

Add Phase A REM silent-failure regression tests

Closed v13.0.0/archive-v13-0-0-chunk-16 enhancementaitestingregressionarchitecturemodel-experience
neo-gpt
neo-gpt commented on Jun 6, 2026, 6:05 AM

Context

This ticket creates the explicit phase split that the live #12075 thread now requires.

Live freshness sweep: checked the latest 20 open issues on 2026-06-06 via gh issue list --state open --limit 20 --json number,title,author,labels,url. The only direct duplicate surfaced was #12075 itself; no existing Phase A current-dev test ticket was found.

Additional duplicate evidence:

  • Exact GitHub search for "Phase A" "silent failure" OR "aborted-size-cap" OR "REM silent-failure regression tests" returned #12075 only.
  • Local resource/source sweep across resources/content/issues, resources/content/discussions, learn/agentos, test/playwright/unit/ai, ai/services, and ai/daemons found the parent epic, source discussion, runbook, and related closed subs, but no equivalent Phase A implementation ticket.
  • Knowledge Base semantic search was attempted, but the KB server is currently unhealthy (collections.knowledgeBase: null), so this ticket relies on live GitHub + repo-local evidence instead of stale KB state.

The Problem

#12075 asks for full regression coverage for all thirteen REM/Sandman silent-failure hypotheses. That ticket is no longer blocked by Sub 1/Sub 2: the runbook and REM state model exist. It is still not valid as a full closeout as written because hypothesis 9 crosses the unresolved chunking lane.

Current live evidence:

  • The #12075 thread now contains a 2026-06-06 GPT intake classification: full 13-hypothesis closeout is not claimable until #12073/#12423/#12439 resolves, but a Phase A batch can cover current-dev silent-failure surfacing.
  • PR #12423 is still open with CHANGES_REQUESTED; chunk activation is not merged into origin/dev.
  • ai/services/graph/SemanticGraphExtractor.mjs:148-182 still sends the whole session through one guardrailed call and returns null when the guardrail emits no result. That is current-dev aborted-size-cap / guardrail telemetry behavior, not chunk activation.

Without a fresh split, agents will either keep pausing on stale full-scope wording or overclaim a PR as resolving #12075 while silently skipping the chunking-dependent part.

The Architectural Reality

The current-dev substrate already has enough surface to test a broad, non-micro Phase A batch:

  • learn/agentos/incidents/sandman-silent-failure-forensics.md:63-79 is the Sub 1 thirteen-hypothesis source map.
  • learn/agentos/rem-state-model.md:82-118 defines the durable REM run-state JSONL shape: outcome, reasonCode, failurePhase, failureReason, lastSuccessfulPhase, perPhaseStates, and perSessionStates.
  • ai/daemons/orchestrator/services/DreamService.mjs:510-714 writes typed REM outcomes and phase/session state for provider, dry-run, concurrent guard, session-query, decay, and extraction failure paths.
  • ai/services/memory-core/helpers/RemRunStateStore.mjs:77-123 derives lastSuccessfulPhase, cycleScopePhases, overflow signal, and session/phase state payloads.
  • Existing sibling specs already establish the canonical test home: test/playwright/unit/ai/daemons/orchestrator/services/DreamService.executeRemCycle.spec.mjs, HeavyMaintenanceLeaseService.spec.mjs, and MaintenanceBackpressureService.spec.mjs.

Pre-Flight (structural fast-path): authoring any new .spec.mjs for this work under test/playwright/unit/ai/daemons/orchestrator/services/ matches sibling patterns in that directory; sibling-file-lift applies; no novel directory choice.

The Fix

Add a current-dev Phase A regression test batch under the existing unit-test substrate. The implementation may either extend existing sibling specs or add one focused sibling spec if that keeps the hypothesis mapping easier to audit.

Required scope:

  • Cover hypotheses 1-8 and 10-13 from the Sub 1 runbook using already-merged current-dev behavior.
  • Cover hypothesis 9 only as current aborted-size-cap / guardrail telemetry behavior. Do not assert chunk activation, chunk recomposition, or semantic-fidelity preservation in this phase.
  • Add/reuse deterministic failure-injection helpers only where they make the batch clearer. Avoid a helper abstraction if the result is less readable than focused per-test stubs.
  • Each test or grouped describe block must document the hypothesis number(s) it protects and cite Discussion #12062 §2.4 / the Sub 1 runbook.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
REM cycle outcome assertions DreamService.executeRemCycle() + REM state model Tests assert typed completed / skipped / failed outcomes and diagnostic fields for current-dev failure paths If a path is not yet implemented, test the current explicit skip/failure telemetry rather than inventing future behavior learn/agentos/rem-state-model.md npm run test-unit -- <specific spec>
Phase/session state assertions RemRunStateStore + DreamService.processUndigestedSessions() Tests verify reasonCode, failurePhase, lastSuccessfulPhase, and perSessionStates where applicable For non-state-backed paths, assert the absence is intentionally out of Phase A learn/agentos/incidents/sandman-silent-failure-forensics.md Unit-test assertions on JSONL / returned envelopes
Hypothesis 9 boundary Current SemanticGraphExtractor guardrail path Tests assert current oversized-session abort / null-result telemetry only Chunk activation coverage waits for #12073/#12423/#12439 disposition #12075 live intake thread PR body says Related: #12075, not Resolves
Unit-test CI lane unit-test skill and learn/guides/testing/UnitTesting.md Run with npm run test-unit -- <specific spec> Do not use raw npx playwright test/playwright/setup.mjs Command output in PR body

Decision Record Impact

aligned-with current REM state-model architecture. No ADR change is required because this ticket adds tests around already-accepted service/run-state contracts; it does not introduce a new architectural primitive.

Acceptance Criteria

  • AC1: Adds Phase A regression coverage for hypotheses 1-8 and 10-13 from learn/agentos/incidents/sandman-silent-failure-forensics.md.
  • AC2: Adds hypothesis 9 coverage only for current-dev aborted-size-cap / guardrail telemetry behavior; no test or PR text claims chunk activation or semantic-fidelity completion.
  • AC3: Each test or grouped describe block cites its hypothesis number(s) and Discussion #12062 §2.4 / the Sub 1 runbook.
  • AC4: Tests live under test/playwright/unit/ai/ using existing sibling patterns; no legacy test/playwright/mcp/ placement.
  • AC5: Verification runs through npm run test-unit -- <specific spec>; raw npx playwright is not used.
  • AC6: PR body uses Related: #12075, not Resolves #12075, and explicitly leaves #12075 open for Phase B/full closeout.
  • AC7: The batch is broad enough to justify full CI/review cost; do not split into a one-test or cosmetic micro PR.

Out of Scope

  • Implementing chunked Tri-Vector extraction.
  • Asserting 300K-token payload chunk activation.
  • Closing #12075.
  • Mutating live REM/Memory Core graph state via npm run ai:run-sandman.
  • Reworking #12423/#12439 semantic-fidelity decisions.

Avoided Traps

  • Full-scope overclaim: A PR before chunking lands must not claim all thirteen hypotheses are fully closed.
  • Micro PR: This should be a meaningful Phase A batch, not an eight-line cleanup.
  • Stub-only proof: Prefer real service contracts and deterministic stubs over mocks that bypass the consumed behavior.
  • Wrong test runner: Use the repo unit-test script, not raw Playwright invocation.

Related

Parent Epic: #12065

Related: #12075 #12073 #12423 #12439 #12077 #12087 #12088

Origin Session ID: dbb1a88c-987f-4519-9645-8f13e9d71000

Handoff Retrieval Hints:

  • query_summaries(query='Sub 9 REM silent-failure Phase A #12075 aborted-size-cap')
  • query_raw_memories(query='Phase split #12075 current-dev silent-failure tests hypothesis 9 chunking')
  • Source anchors: learn/agentos/incidents/sandman-silent-failure-forensics.md, learn/agentos/rem-state-model.md, DreamService.executeRemCycle.spec.mjs
tobiu referenced in commit 96f23bf - "test(ai): add REM Phase A regression anchors (#12617) (#12620)" on Jun 6, 2026, 3:30 PM
tobiu closed this issue on Jun 6, 2026, 3:30 PM