LearnNewsExamplesServices
Frontmatter
id12067
titleSub 1: Silent-failure root-cause investigation across 13 hypotheses
stateClosed
labels
enhancementaiarchitecturemodel-experience
assigneesneo-gpt
createdAtMay 27, 2026, 3:41 AM
updatedAtMay 27, 2026, 2:15 PM
githubUrlhttps://github.com/neomjs/neo/issues/12067
authorneo-opus-ada
commentsCount0
parentIssue12065
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtMay 27, 2026, 2:15 PM

Sub 1: Silent-failure root-cause investigation across 13 hypotheses

Closed v13.0.0/archive-v13-0-0-chunk-14 enhancementaiarchitecturemodel-experience
neo-opus-ada
neo-opus-ada commented on May 27, 2026, 3:41 AM

Parent Epic

#12065 — Sub 1 of 9. MUST land FIRST per GPT migration blast-radius sweep ordering.

Premise

REM pipeline silently produces no fans-glow despite the orchestrator's dream task code calling processUndigestedSessions(). Discussion #12062 §2.4 + §2.4.1 enumerate 13 distinct silent-failure hypotheses. Until we identify which hypothesis is actively firing in production, the unified orchestrator-as-SSOT shape (Sub 3) could replicate the failure at a different layer.

Prescription

V-B-A each of the 13 hypotheses against production state. For each, document: (a) is it actively firing? (b) what evidence proves/disproves it? (c) what mitigation is needed?

13 hypotheses (full descriptions in Discussion #12062 §2.4):

  1. DreamService.isProcessing stuck flag
  2. Provider probe gate skipped for gemini/ollama
  3. Cadence silent-loop (markStarted + markFailed without work)
  4. MaintenanceBackpressureService indefinite-deferral
  5. Lease held by orphan
  6. DreamService.initAsync hung on dependency
  7. autoDream config flip
  8. Pre-PR #11966 OpenAiCompatible-hardwire (last-2-days only)
  9. invokeWithGuardrail size-cap pre-check (PRIMARY — addressed by hot-fix #12063/#12064)
  10. TopologyInferenceEngine.extractTopology returns void (most insidious)
  11. 3-retry JSON parse exhaustion
  12. Provenance edge culling
  13. Lazy edge queue overflow

Acceptance Criteria

  • AC1: V-B-A report per hypothesis (active/inactive/inconclusive + evidence + mitigation strategy)
  • AC2: Forensic protocol documented for each hypothesis (e.g., "how to detect isProcessing-stuck: check .neo-ai-data/wake-daemon/* for orphan lease + DreamService log debug-line freq")
  • AC3: Inventory document at learn/agentos/sandman-silent-failure-forensics.md (or similar canonical location) serves as runbook for future regressions
  • AC4: Hypothesis-mitigation specs handed off to Sub 3 (unified REM method) as preconditions for implementation

Avoided Traps

  • ❌ Fix-the-symptom-without-V-B-A — must actually verify each hypothesis empirically; don't guess
  • ❌ Skip hypothesis 9 since #12063 handles it — STILL document the forensic protocol so future cap regressions are diagnosable
  • ❌ Defer mitigation specs to Sub 3 implementation — Sub 3 needs the specs INPUT; Sub 1 produces them OUTPUT

Related

  • Epic #12065
  • Discussion #12062 §2.4 + §2.4.1 (hypothesis enumeration)
  • Memory anchor f04dd0ba (the deferral pattern this sub structurally prevents)
tobiu referenced in commit 117d693 - "docs(agentos): add sandman silent-failure forensics runbook (#12067) (#12077) on May 27, 2026, 2:15 PM
tobiu closed this issue on May 27, 2026, 2:15 PM