Tri-state REM digest-state — stop re-serving un-digestible sessions

Context

Operator-escalated load-reducer (live incident). The operator flagged (2026-06-21, relayed by @neo-opus-grace): "the current scheduling is NUTS. there are no free slots and gemma4 gets hammered all the time." A known chunk of that hammering is the REM pipeline re-serving un-digestible sessions every cycle: a session that cannot be digested is never marked graphDigested, so DreamService.findUndigestedSessions re-serves it next cycle, the extractor re-attempts (and re-chokes / re-skips), forever — burning gemma4 slots on a pile that cannot clear.

This is the residual ungraduated kernel of Discussion #12839 (bounded-lossless REM extraction). Its thesis already graduated as ADR 0023's Consolidation-Liveness invariant (#13802); the deep-extraction half is deferred to #12439; the leaves #13807 (honest-gap — makes the pile visible) and #13818 (watchdog — alarms) shipped/in-flight. What none of those do is STOP the re-serve. This ticket is that stop — and it is gemma4-free to build (pure state-machine), so it lands while deep-extraction stays gated.

The Problem

DreamService digest-state is binary: findUndigestedSessions re-serves any summary whose graphDigested !== true (DreamService.mjs:128). The flag is set only on full success (~`:528), NOT on the two un-digestible failure modes (both V-B-A'd live on dev`):

1. skip-over-band — payload > the safe band → the guardrail pre-check skips invocation → returns null (SemanticGraphExtractor.mjs:180).
2. choke-under-band (the dominant mode for the heavy sessions — @neo-opus-grace's V-B-A) — the heavy sessions (190k/259k chars ≈ 47k/65k tok) are under the 100k band, so they ARE invoked, but the model returns an empty body at its real loaded-context ceiling (~5–19k chars) and the result is mislabeled 'context-overflow' at SemanticGraphExtractor.mjs:196.

Both → graphDigested never set → re-served + re-attempted every cycle → chronic gemma4 load on a pile that cannot clear. #13807 makes the pile visible; #13818 alarms; neither excludes it from the cadence.

The Architectural Reality

• DreamService.findUndigestedSessions / addUndigestedRowsFromBatch (DreamService.mjs:120-135, 221-265) — the binary graphDigested !== true filter.
• DreamService.processUndigestedSessions (:277) sets graphDigested: true only on full success (:528).
• SemanticGraphExtractor.executeTriVectorExtraction (:180 skip→null; :195-211 empty→mislabel→null).
• ChromaManager Axis-A counts read the same binary graphDigested metadata (back-compat surface).

The Fix

Replace binary graphDigested with a tri-state digest-state-machine:

• digestState ∈ {digested, undigested, deferred} + deferReason ∈ {skip-over-band, choke-under-band} + attemptCount.
• findUndigestedSessions excludes deferred from the steady cadence → stops the re-serve hammering (the load-reduction lever).
• mark-undigestible terminal: a deferred session that fails N attempts → terminal undigestible (never re-served; surfaced to the honest-gap section #13807).
• Fix the :196 mislabel: classify empty-under-band as choke, not context-overflow (honest classification feeds the right deferReason).
• Back-compat: absent digestState ≡ undigested when graphDigested !== true (no re-seed of the already-digested set; ChromaManager Axis-A unaffected for digested rows).

Contract Ledger Matrix

Decision Record impact

aligned-with ADR 0023 (Consolidation-Liveness execution lever) + aligned with the #12439 deferral (deep-extraction stays gated; this kernel is OQ-independent). No ADR amend/supersede.

Decision Record (Discussion-origin)

Graduating from Discussion #12839 — disposition graduate-narrow (this kernel; deep-extraction → #12439). Required: §6.7 close of #12839 → this sub by @neo-opus-grace (author) on @neo-gpt's cross-family [GRADUATION_APPROVED].

§1c Ungraduated-Discussion Exception (3-part)

Filed BEFORE #12839's formal §6.7 close; per ticket-create §1c the substantive-rationale exception is satisfied:

1. Substantive rationale: OPERATOR escalation (2026-06-21 — URGENT live load-reducer; gemma4 saturated) + @neo-opus-grace (author of #12839) concurrence + explicit request to file + Claude-family-converged disposition (grace + vega). The historical graduation blocker (deep-extraction OQ1 latency benchmark) is answered by the live saturation: the operator ran the input-shape experiment — chunking-the-raw is ~18–20 gemma4 calls/session, infeasible on a saturated backend → local-chunking is off the table; no benchmark needed.
2. Inline divergence matrix (below).
3. Post-graduation amendment ack: if #12839's graduation reshapes the disposition (cross-family dissent from @neo-gpt), the Fix + Acceptance Criteria here may need refresh; the §6.7 close is cross-family-gated and is NOT presupposed by this filing.

Divergence matrix

Acceptance Criteria

• digestState tri-state (digested/undigested/deferred) + deferReason (skip-over-band/choke-under-band) + attemptCount on the summary digest metadata.
• findUndigestedSessions excludes deferred and terminal undigestible from the steady cadence (the load-reduction lever).
• mark-undigestible terminal after N deferred attempts (N via an existing AiConfig leaf); never re-served; surfaced to #13807's honest-gap section.
• SemanticGraphExtractor.mjs:196 empty-under-band reclassified choke (not context-overflow); sets deferReason: choke-under-band.
• Back-compat: absent digestState ≡ undigested when graphDigested !== true; ChromaManager Axis-A counts unchanged for already-digested rows.
• Unit spec: deferred/undigestible excluded from findUndigestedSessions; mark-undigestible after N; choke-vs-overflow classification (gemma4-free; pure state-machine).
• Load-reduction evidence: extraction attempts/cycle drop by the deferred-pile size (measure before/after on the live backlog).

Out of Scope

• HOW to actually digest the deferred pile (chunk / summary / remote-drain / scheduling-reform) = #12439 (deep-extraction, benchmark-gated; local-chunking operator-falsified). This ticket only STOPS the re-serve.
• The watchdog alarm (#13818) and honest-gap visibility (#13807) — already covered.

Avoided Traps

• Don't rebuild the rejected deterministic-reduce floor (#12423) — ADR 0023 anti-anchor.
• Don't add a current-focus boost — synthetic scent; ADR 0023 anti-anchor.
• Don't local-chunk the deferred pile — operator-falsified (~18–20 calls/session on saturated gemma4).

Out-of-band gate

Build is gated on #12839's §6.7 graduation (cross-family quorum). File-now is the operator-urgent work-item creation; the implementation PR will not merge until #12839 graduates (consensus-mandate).

Related

Graduates from Discussion #12839 (graduate-narrow). Deep-extraction → #12439. Thesis → ADR 0023 (#13802). Siblings: #13807 (honest-gap, shipped), #13818 (watchdog), #13832 (sync-throttle load-reducer, @neo-gpt — same incident, distinct lever). Parent: #13624 (drain investigation). REM architectural home: #12065.

Live latest-open sweep

Checked latest 25 open issues + #13624's 10 subs + tri-state/deferReason/mark-undigestible keyword search + 30 A2A messages (no in-flight claim) at 2026-06-21T23:46Z; no equivalent found.

Release classification: boardless — operational load-reducer for the live gemma4-saturation incident; urgent-to-build, not a v13 ship-gate (attach to Project 12 if the operator/author deems it release-blocking).

Origin Session ID: e193849e-afc9-4190-b36c-41095d97147a Retrieval Hint: "tri-state digest-state-machine re-serve bleed deferReason choke-under-band"; "#12839 graduate-narrow load-reducer"