LearnNewsExamplesServices
Frontmatter
id12075
titleSub 9: Regression test coverage for 13 silent-failure modes
stateClosed
labels
enhancementaitestingarchitecturemodel-experience
assigneesneo-opus-grace
createdAtMay 27, 2026, 3:45 AM
updatedAtJun 7, 2026, 6:27 PM
githubUrlhttps://github.com/neomjs/neo/issues/12075
authorneo-opus-ada
commentsCount11
parentIssue12065
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 7, 2026, 6:27 PM

Sub 9: Regression test coverage for 13 silent-failure modes

Closed v13.0.0/archive-v13-0-0-chunk-14 enhancementaitestingarchitecturemodel-experience
neo-opus-ada
neo-opus-ada commented on May 27, 2026, 3:45 AM

Parent Epic

#12065 — Sub 9 of 9. NEW per operator-direct addition 2026-05-27 ~01:36Z post-STEP_BACK. Depends on Sub 1 (needs hypothesis inventory + mitigation specs as input).

Premise

Operator: "and we need more unit tests. if our corrupted state did not get catched in tests, we are missing something."

The 13 silent-failure hypotheses in Discussion #12062 §2.4 + §2.4.1 represent regression surfaces that current test coverage missed (proven empirically: operator saw fans-glow disappear for a week+ before noticing; no test would have flagged the cap-vs-payload mismatch silently). Each hypothesis needs a regression test that would fail if the failure mode re-emerged in production.

Prescription

Author targeted test for each of the 13 hypotheses (per Discussion #12062 §2.4 enumeration). Tests live in test/playwright/unit/ai/daemons/ or equivalent canonical location per unit-test.md. Each test:

  • Sets up a substrate condition that would trigger the failure mode
  • Asserts the system either (a) prevents the failure OR (b) surfaces it via the REM run state model (Sub 2)
  • Documents the hypothesis it targets in test docblock

Test inventory:

Hypothesis Test focus
1 — isProcessing stuck flag Spawn REM, kill mid-flight, verify next REM either resets flag OR records guard-skip state
2 — Provider probe gate skipped for gemini/ollama Inject modelProvider: 'ollama' + offline provider; assert REM marks failed-due-to-provider with rich diagnostic
3 — Cadence silent-loop Inject markStarted + markFailed without doing work; assert next cadence detects pattern + escalates
4 — MaintenanceBackpressureService indefinite-deferral Hold another heavy task indefinitely; assert dream task records 'deferred' state model entry + escalates after N cycles
5 — Lease held by orphan Write orphan lease to disk; assert orchestrator either reclaims or marks blocked
6 — DreamService.initAsync hung on dependency Mock StorageRouter.ready() to never resolve; assert initAsync timeout + REM marked failed
7 — autoDream config flip Verify periodic path doesn't depend on autoDream flag
8 — Pre-PR #11966 hardwire — N/A (already fixed) Smoke test confirms providerDispatch handles all 3 providers
9 — invokeWithGuardrail size-cap (PRIMARY) Inject 300K-token session payload + verify either (a) chunking activates per Sub 7 OR (b) REM run records aborted-size-cap state
10 — TopologyInferenceEngine returns void Mock Topology to throw; assert REM run records topology: 'failed' (NOT silent void)
11 — 3-retry JSON parse exhaustion Mock LLM to return malformed JSON × 3; assert REM run records failed + payload preserved for analysis
12 — Provenance edge culling Inject Tri-Vector output with all-orphan provenance edges; assert lazy-edge queue receives them (not silent drop)
13 — Lazy edge queue overflow Verify LazyEdgeDrainer runs periodically OR alarm fires if JSONL exceeds N MB

Acceptance Criteria

  • AC1: 13 targeted tests authored, one per hypothesis (some may share fixtures)
  • AC2: Each test has docblock citing the hypothesis number + Discussion #12062 §2.4 reference
  • AC3: Tests run under npm run test-unit per unit-test.md (NOT npx playwright — UNIT_TEST_MODE injection required per feedback_npx_bypass_test_isolation)
  • AC4: Test placement: test/playwright/unit/ai/daemons/ or equivalent canonical location
  • AC5: All 13 tests pass on current dev (proves regressions documented in Discussion would now be caught)
  • AC6: Failure injection patterns documented as reusable test helpers (e.g., test/playwright/helpers/sandman-failure-injection.mjs)
  • AC7: CI integration — tests run in unit-test CI lane; regression fails CI

Avoided Traps

  • ❌ Stub-only tests masking real-substrate failure modes per feedback_stub_tests_miss_adapter_drift — at least one test per hypothesis MUST exercise real substrate (DreamService + Orchestrator + ChromaManager) not just mocks
  • ❌ Test placement in legacy test/playwright/mcp/ per feedback_mcp_test_location — use canonical test/playwright/unit/ai/
  • ❌ Use npx playwright per feedback_npx_bypass_test_isolation — UNIT_TEST_MODE injection required
  • ❌ Land before Sub 1 (no hypothesis V-B-A → tests would target hypothetical not empirically-validated failure modes)

Related

  • Epic #12065
  • Operator directive 2026-05-27 ~01:36Z: "we need more unit tests. if our corrupted state did not get catched in tests, we are missing something."
  • Discussion #12062 §2.4 (13-hypothesis enumeration; required input for test design)
  • Sub 1 (hypothesis V-B-A produces the empirically-validated failure-mode inventory)
  • Sub 2 (REM run state model — tests assert state-model recording, NOT silent failures)
  • feedback_stub_tests_miss_adapter_drift + feedback_npx_bypass_test_isolation + feedback_mcp_test_location (memory anchors for test discipline traps)
tobiu referenced in commit 17c8ec6 - "feat(orchestrator): unified executeRemCycle() with typed cycle outcome (#12069) (#12096) on May 27, 2026, 6:10 PM
tobiu referenced in commit 85860ab - "test(ai): real-service-integration guards for REM silent-failure modes H9+H11 (#12075) (#12680)" on Jun 7, 2026, 6:27 PM
tobiu closed this issue on Jun 7, 2026, 6:27 PM