Bound REM parser calls with live chunk diagnostics

Context

Live nightshift evidence on 2026-06-25: LM Studio is processing a REM graph extraction with google/gemma-4-26b-a4b and the model accounting climbed past 198k tokens while still streaming. My earlier estimate that the run would stay near the combined chunk estimate was wrong; the live counter already falsified it.

This is not the closed #13981 hypothesis. #13981 focused on LMS structured-output generation caps. This ticket is broader and stricter: REM parsing itself is broken and needs an end-to-end bounded parser contract with exact-session/chunk forensics and resumable failure semantics.

Live latest-open sweep: checked latest 25 open issues on 2026-06-25; no open replacement for the closed #13981 scope was found. A2A in-flight sweep: checked latest 30 all-state messages; only the closed #13981 / #13982 lane matched, so this ticket supersedes the rejected shape.

What happened

• REM reached it at 2026-06-24T23:27:55.880Z.
• Raw payload: 500,586 chars, 504,892 bytes, /3 estimate 168,298 tokens, 192 raw turns.
• Tri-Vector chunking activated: 2 chunks, estimated 167,916 tokens.
• Chunk 1 prompt estimate: 99,179 tokens. It completed at 2026-06-24T23:30:15.840Z.
• There is no following Graph entities committed, no Tri-Vector Synthesis took, and no topology line.
• Chunk 2 prompt estimate: 70,549 tokens. That is almost certainly the active generation now.
• My previous estimate was wrong: LM Studio passed 198k tokens, then 204k tokens, and was still not done.
• Compact deterministic fallback is rejected; this ticket must make REM graph parsing bounded and scalable, not bypass it.

The Problem

REM parsing can still dispatch a model call that grows far beyond the intended usable context/safety envelope. The system has invested in turn-boundary chunking and guardrails, but the live behavior proves that at least one REM parsing path still lets a single provider call keep expanding until the chat model is unusable for the rest of the team.

This breaks backlog processing and makes cloud deployment economics impossible: a team-size backlog cannot rely on one-hour parser calls, especially on slower cloud hardware.

The Architectural Reality

• REM session digestion feeds the Native Edge Graph; if it stalls, Golden Path freshness rots and scheduled Golden Path work may never get a free chat model slot.
• DreamService delegates graph extraction work into SemanticGraphExtractor and related graph services.
• Prior context shows sessionChunker / turn-boundary chunking exists, but live LMS accounting proves the current parser contract is still not sufficient.
• The parser needs observable per-session/per-chunk state before and after provider calls. Without that, operators only see a model counter climbing and cannot tell which session/chunk is responsible.
• This is LMS/OpenAI-compatible REM parsing, not Ollama and not embedding truncation.

The Fix

• Add exact REM parser run-state for active session id, chunk index, estimated prompt tokens, provider, model, start time, and parser phase before each model call.
• Enforce an end-to-end per-call budget for REM parsing: prompt estimate, expected output budget, provider options, and fail-loud classification when provider accounting exceeds the budget.
• Make oversized or runaway chunks terminal-for-current-cadence and resumable; do not retry the same pathological call indefinitely.
• Add tests using this marathon-session shape that prove the parser splits/limits/fails before a single call can monopolize the chat model.

Contract Ledger

Acceptance Criteria

• The exact session/chunk evidence above is covered by a test or deterministic harness reproduction.
• A REM parser call cannot exceed the configured per-call prompt + output budget without fail-loud termination.
• Active REM diagnostics identify the exact session id, chunk index, phase, provider, model, estimate, and start timestamp while the call is in flight.
• Over-budget/runaway parser chunks are terminal for the current cadence and do not get immediately re-served.
• The solution is not a wall-clock timeout as the primary parser correctness mechanism.
• The solution does not switch graph extraction to mini-turn summaries.

Out of Scope

• Handoff PR snapshot freshness.
• Golden Path Strategic Interpretation / handoff summary contract.
• Model-choice optimization or parallelism tuning.
• Cloud deployment documentation.

Related

• Closed #13981 / PR #13982 — rejected narrower output-cap shape.
• #12073 / PR #13976 — Tri-Vector map/reduce context.
• #13835 / #13977 — terminal digest-state handling.
• #13750 — Golden Path freeze from undigested sessions.

Release classification: boardless P0 Agent OS reliability hardening; live REM backlog drain is currently unsafe. Retrieval Hint: REM parsing broken 198k tokens LM Studio gemma4 session 2d993feb chunk 2 parser diagnostics