LearnNewsExamplesServices
Frontmatter
id12007
titleDreamService.spec.mjs — 2 tests failing on dev (synthesizeGoldenPath + malformed-JSON-retry)
stateClosed
labels
bugaitesting
assigneesneo-gpt
createdAtMay 26, 2026, 2:29 AM
updatedAtMay 26, 2026, 9:19 AM
githubUrlhttps://github.com/neomjs/neo/issues/12007
authorneo-opus-ada
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtMay 26, 2026, 9:19 AM

DreamService.spec.mjs — 2 tests failing on dev (synthesizeGoldenPath + malformed-JSON-retry)

neo-opus-ada
neo-opus-ada commented on May 26, 2026, 2:29 AM

Two specific tests in test/playwright/unit/ai/daemons/orchestrator/services/DreamService.spec.mjs are persistently failing on origin/dev. The failures predate Epic #11993 and PR #12005 (the substrate-cleanup PR that surfaced this consolidation triage) and have been flagged as "unrelated pre-existing failures" in PR evidence sections multiple times now (#11999, #12005, possibly earlier). Without a tracking ticket, the failures act as anchored noise in the test-suite floor, degrading future-regression signal quality.

Empirical anchor

Run on origin/dev HEAD 6c1e3c86c (post-#12004 merge):

npm run test-unit -- test/playwright/unit/ai/daemons/orchestrator/services/DreamService.spec.mjs

13/15 PASS, 2 FAIL. Failing tests + exact failure assertions:

  1. synthesizeGoldenPath should mathematically select and inject Golden Path while rejecting BLOCKS (DreamService.spec.mjs:817):

       Expected substring: "Math synthesis works natively."
    Received string:    "# Autonomous Handoff (Dream Pipeline & Golden Path)..."

    The assertion expects the synthesized Golden Path output to contain the literal "Math synthesis works natively." — but the actual output is an "Autonomous Handoff" template body. Strong hypothesis: the test was authored against a pre-refactor synthesis output shape; downstream the synthesis logic was changed to return a handoff template instead, and the test fixture didn't follow. Could also be a stale mock LLM response that no longer matches.

  2. should retry extraction on malformed JSON payload up to 3 times to fix #9913 (DreamService.spec.mjs:967):

       Expected: 3
    Received: 0

    Expected 3 retries (per the original #9913 fix), got 0. The retry logic isn't firing in this test path. Could be (a) the retry threshold semantic changed in DreamService, (b) the mock setup that triggers the retry condition broke, (c) the assertion target moved (e.g., the variable being checked no longer reflects retry count).

Both failures are deterministic across runs (not flaky); ran 3× in the diagnostic session 2026-05-26, same output each time.

The Fix

Investigation-shaped (the fix prescription depends on bisect findings):

  1. git bisect each failing test against dev history to identify the commit that broke each:

    • For test 1: when did synthesizeGoldenPath start returning a handoff template instead of the math-synthesis output? Did the test author update the assertion at that commit and miss the spec, OR did the substrate change without spec update?
    • For test 2: when did the retry count drop to 0? Was the retry threshold reset to 0/disabled, OR did the mock-injection mechanism change?
  2. Based on bisect findings, classify each failure:

    • Substrate-drift fix — test fixture stale; update the assertion to match new substrate behavior (and document why in the test comment).
    • Logic-bug fix — substrate broke its own contract; fix the substrate code + keep the test as the regression anchor.
    • Mock-staleness fix — test mock no longer reflects current dependency shape; refresh the mock to match current production behavior.
  3. Decision recorded in PR body — even if both turn out to be "substrate-drift, test fixtures need updating," that decision is worth documenting because it identifies the substrate change that lacked spec update at the time.

Acceptance Criteria

  • AC1 — DreamService.spec.mjs line 817 (synthesizeGoldenPath) passes deterministically.
  • AC2 — DreamService.spec.mjs line 967 (malformed JSON payload retry) passes deterministically.
  • AC3 — PR body documents git bisect findings: commit that broke each test + classification (substrate-drift / logic-bug / mock-staleness) + rationale.
  • AC4 — Full npm run test-unit -- test/playwright/unit/ai/daemons/orchestrator/services/DreamService.spec.mjs reports 15/15 PASS.
  • AC5 — Broader orchestrator tree (test/playwright/unit/ai/daemons/orchestrator/) reports 326/326 PASS (full-green floor restored).

Out of Scope

  • Re-authoring the DreamService production code wholesale. Only the minimum necessary to make the specs pass per ground truth.
  • Changing the DreamService API or test surface — narrow per-test fixes only.
  • Adding new test coverage beyond fixing the 2 broken tests.
  • Audit of OTHER specs that might also have anchored noise — that's a separate sweep ticket.

Avoided Traps

  • Skip-marking the failing tests as a "fix" — anchored skips are worse than anchored failures (skip-marking removes the regression signal entirely; failures at least surface the breakage in test output).
  • Removing the failing tests as "stale" without bisect investigation — if the substrate broke its own contract, the test is correct and should stay.
  • Bundling the fix with another lane's PR — these failures are independent of any active substrate-evolution work; should ship as its own focused fix.

Related

  • Discovered during: PR #11999 (Epic #11993 Sub-iii) + PR #12005 (daemon.mjs router cleanup) test-evidence sections that both flagged "324/326 PASS — 2 unrelated pre-existing DreamService failures"
  • Original #9913 referenced in the malformed-JSON-retry test name — that's the fix-anchor for the retry-count=3 expectation; bisect should reveal whether the contract from #9913 was unintentionally walked back
  • Sibling DreamService tickets (not direct dependencies, just nearby substrate context): #9890 (DreamService 4th REM Vector executeNLActionDigest), #9891 (Strategic Constraint Nodes), #9843 (Quantitative Reward Signal for Golden Path Edge Reinforcement)

Handoff Retrieval Hint: "DreamService spec 2 failing tests synthesizeGoldenPath malformed JSON retry git bisect substrate-drift noise floor".

tobiu referenced in commit 220c550 - "test(ai): restore DreamService provider isolation (#12007) (#12010) on May 26, 2026, 9:19 AM
tobiu closed this issue on May 26, 2026, 9:19 AM