Investigate REM 400k token counter overflow

Context

During adversarial self-review of PR #13986 / issue #13984, the operator reported that the chat model stayed locked overnight while LM Studio showed 400k+ generated tokens. The identified source session was not large enough to explain that counter by input size alone, and #13986 now explicitly treats that symptom as generated-output runaway rather than as proof that input chunking alone explains the failure.

This follow-up exists because the 400k+ counter still exceeds the expected shape even after bounding a single REM Tri-Vector provider call. The remaining concern is cross-call or cross-session contamination: provider context reuse, stale KV/cache state, repair-loop message growth, or multiple full sessions being streamed into one effective request.

Release classification: post-#13986 hardening; boardless unless post-merge validation reproduces the multi-session/context-contamination symptom as release-blocking.

Live latest-open sweep: checked the latest 20 open issues at 2026-06-25T08:05:46Z. #13984 is related and covers bounding a single REM parser call, but no equivalent ticket covered 400k+ counter overflow exceeding the source session size or possible multi-session/context contamination.

A2A in-flight sweep: checked recent all-status A2A traffic at 2026-06-25T08:05:46Z. Recent claims/updates were my own #13984/#13986 lifecycle messages plus unrelated #6777/#13936 traffic; no competing claim for this follow-up scope was present.

KB/local prior-art sweep: ask_knowledge_base("Existing issue or documentation about REM 400k token LM Studio context reuse multiple sessions SemanticGraphExtractor DreamService", type="ticket") surfaced related prior art (#12074 context reuse benchmark, #13918 parser repair overflow, #12073 chunking, #13984 current cap work), but no explicit 400k overflow / multi-session contamination ticket. Local search across resources/content/issues, resources/content/discussions, and learn/agentos found REM/context prior art but no exact equivalent.

The Problem

The observed counter violates the simple single-session theory:

• The source session was reported around marathon scale, not 400k+ tokens.
• A correct chunker can still dispatch multiple chunks, but each chunk should be bounded and independently diagnosed.
• PR #13986 caps provider completion for a single Tri-Vector call, records active call diagnostics, and marks provider-size parser failures terminal for cadence.
• If a post-#13986 run still shows counters materially above promptPlusOutputTokens for one active REM call, the bug is no longer "chunk too large". It is likely context state, request assembly, streaming lifecycle, repair-loop contamination, or multiple sessions sharing one provider-side context/accounting window.

The Architectural Reality

SemanticGraphExtractor.createTriVectorChunks() plans prompt chunks from session.turnDocuments or session.document; after #13986 it clamps the effective prompt budget by graphChunkLimitTokens, safeProcessingLimitTokens, and contextLimitTokens - graphOutputLimitTokens before subtracting envelope tokens (ai/services/graph/SemanticGraphExtractor.mjs:293).

SemanticGraphExtractor.extractTriVectorPayload() records promptTokensEstimate, outputLimitTokens, promptPlusOutputTokens, provider, model, chunk index/count, and aborts before dispatch when prompt plus output reserve exceeds the context cap (ai/services/graph/SemanticGraphExtractor.mjs:514). It passes maxCompletionTokens to the provider (ai/services/graph/SemanticGraphExtractor.mjs:560).

OpenAiCompatible.preparePayload() maps maxCompletionTokens to max_tokens, preserving the caller's configured structured-output cap for LM Studio/OpenAI-compatible requests (ai/provider/OpenAiCompatible.mjs:97).

The repair loop still appends the assistant's failed output plus a repair prompt when schema extraction fails, and only guards the next repair payload against the safe band (ai/services/graph/SemanticGraphExtractor.mjs:690). That is a plausible contamination axis if provider accounting includes prior context, partial streams, or retained assistant output unexpectedly.

learn/agentos/measurements/gemma4-rem-benchmark.md:12 documents that Tri-Vector calls were historically understood as fresh gemma4 contexts per invocation, while lines 22-27 preserve the residual question of backend-specific keep_alive / context reuse behavior.

The Fix

Instrument and prove the provider/request lifecycle around REM Tri-Vector calls:

1. Add a deterministic probe or unit/integration harness that dispatches two REM graph calls with distinct session ids and verifies the second request body does not contain the first session's full payload.
2. Capture request-side evidence for provider calls: session id, asset ref, chunk index/count, prompt estimate, output cap, request message count, repair attempt, and bounded content fingerprints for each message slot.
3. Characterize LM Studio/OpenAI-compatible context reuse behavior for REM: prove whether keep_alive, provider request ids, streaming state, or server-side KV reuse can make the visible token counter accumulate across calls while Neo's request body remains bounded.
4. If Neo is contaminating the request body, fix the assembly path and add regression coverage. If the provider counter is cumulative by design, document that interpretation and adjust diagnostics so operators can distinguish provider-accounting accumulation from a real prompt leak.
5. Add a post-#13986 validation script or documented operator probe that compares active REM diagnostics (promptPlusOutputTokens, outputLimitTokens, chunk metadata) against provider-visible token counters for the same call.

Contract Ledger Matrix

Decision Record impact

Aligned with ADR 0019: any config or provider-control surface must remain rooted in AiConfig leaves with concrete use-site reads. No ADR amendment expected unless the investigation changes the provider lifecycle contract.

Acceptance Criteria

• A probe or test proves two sequential REM Tri-Vector calls do not include the prior session's full payload in the second request body.
• The probe records whether LM Studio/OpenAI-compatible visible token counters are per-call, cumulative across retained context, or evidence of contamination.
• REM diagnostics include enough bounded fingerprints to correlate request-body contents with the provider-visible counter without logging full session payloads.
• If repair-loop amplification is implicated, the repair path is bounded by both prompt-plus-output context cap and max retained assistant-output length before appending repair context.
• If provider context reuse is implicated, the fix either disables/isolates reuse for REM graph calls or documents why the counter is expected cumulative accounting and not a Neo leak.
• Post-merge validation explicitly reruns against the session family that triggered the 400k+ report and records the observed counter shape.

Out of Scope

• Replacing Tri-Vector extraction with a deterministic fallback.
• Adding a public MCP tool for this bug.
• Changing generic provider behavior for non-REM consumers without evidence that they share the same contamination path.
• Treating 50k as a completion budget; large REM context belongs to input chunking, not structured JSON output.

Avoided Traps

• Do not assume a bigger output cap is the fix. REM output is a compact structured graph artifact, not a 50k-token summary.
• Do not assume chunking proves the 400k symptom. A counter larger than the source session means the investigation must include provider/request lifecycle state.
• Do not hide new config behind dynamic AiConfig helper indirection; keep ADR-19 concrete leaf reads.

Related

• #13984
• PR #13986
• #13918
• #12073
• #12074

Origin Session ID: 019ef378-527d-7393-bc74-ec3a1d3f2ddf

Handoff Retrieval Hint: REM 400k token counter SemanticGraphExtractor LM Studio context reuse multiple sessions PR #13986 graphChunkLimitTokens