miniSummary backfill retries silently-empty generations forever, burning the provider for nothing

Problem

MemoryService.backfillMiniSummaries retries items whose generation returns empty, forever.

Found 2026-07-31 diagnosing a CPU-only cloud deployment: ~43 pending memories × ~20s of 26B CPU generation each, every ~19 minutes, around the clock — supervisor logs show 0 updated, 30 deferred per pass, then the same set restarts ~35s later. ~2.3 CPU-cores average burned for days, producing zero summaries. The pending set grows with inflow (41→43), so the loop is permanent.

The output is cheap; the input is unbounded. buildMiniSummary feeds the RAW prompt + response into the summary prompt (MemoryService.mjs:1628) — real agent turns run 10-50KB — under generateMiniSummaryTimeoutMs = 20000. CPU prefill of a multi-KB turn cannot finish in 20s, so each large item times out at exactly the cap (matching the observed ~20s/item cadence), returns null, defers — and the abandoned request keeps computing on the model server. A tweet-sized output would cost ~70 tokens; the untruncated input costs minutes.

The timeout was calibrated on GPU hardware (5.3s warm for ~5k chars). Measured corpus sizes (n=300): p50 1.9KB / p95 4.8KB / max 7.2KB. On GPU the whole distribution fits the cap; on CPU-only hardware the whole distribution straddles it — hence ~100% deferral.

Two side effects: the loop holds the heavy-maintenance mutex, starving REM/dream/tenant-sync; and generation ran at priority: 'interactive', queue-jumping real traffic from a batch job.

Scope — this ticket does NOT build the controller

The homeostatic loop belongs to #14418 (Homeostatic adaptation controller — proactive sweet-spot loop, phase-2, extends ADR 0026), which graduated from D#13873 with a cross-family convergence disposition: AIMD threshold hill-climb with asymmetric steps, provenance, record-not-page with a smoothed setpoint, and dual-controller co-residency arbitration.

Widening a generation window is a third knob of that same controller — the same sense → compute → envelope-gated reconfigure → re-observe loop, with a different setpoint. Building a detector and controller here would duplicate a graduated design. #16223 is #14418's consumer and its first concrete use case, not a parallel implementation.

What #16223 owes #14418 is the knob's SHAPE, which the controller cannot discover for itself:

⚠️ The window is TWO nested leaves and the actuation must move both. generateMiniSummaryTimeoutMs (20000) sits inside buildMiniSummary, whose try/catch swallows its rejection → null → the loop's falsy branch (deferred++). miniSummaryTimeoutMs (30000) sits at MemoryService.mjs:1850-1853 wrapping summarize(...) from outside that guard → its rejection escapes → the sweep's catch branch. So: widening only the inner leaf is a silent no-op at 30s, and crossing 30s flips every item from the falsy path to the thrown path, blinding any detector keyed to one branch at exactly the moment the actuation is working hardest. The bound must be declared against the outer leaf, with the inner held strictly below it, as an ordered pair.

⚠️ The bound is a RELATIVE ceiling. Unbounded widening defeats the setpoint: a window that grows without limit converts a starvation signal into an ever-slower loop that still reports progress. The ceiling is expressed relative to the measured baseline, not as an absolute constant, or it needs recalibration on every hardware change — which is the defect this ticket exists to remove.

Actuation gap — the knob is not reachable by the shipped mechanism

Corrected 2026-08-12; the earlier version of this claim is now false and is retracted here rather than left to mislead. A previous note said reconfigure was unimplemented and called it a #14418 blocker. It shipped since: RecoveryActuatorService.mjs:29 now lists reconfigure and raise-ceiling in DEFAULT_ACTIONS, with a handler at :934 routing to reconfigureComposeService({knob, knobValues, …}).

The real gap is narrower and concrete. reconfigure moves a compose-service knob. Both window leaves are env-bound —

miniSummaryTimeoutMs        : leaf(30000, 'NEO_MC_MINI_SUMMARY_TIMEOUT_MS',          'number')   ai/configBase.mjs:933
generateMiniSummaryTimeoutMs: leaf(20000, 'NEO_MC_GENERATE_MINI_SUMMARY_TIMEOUT_MS', 'number')   ai/configBase.mjs:937

— but neither env name appears in any file under ai/deploy/ (verified with a positive control: NEO_KB_ASK_MODEL is found in three compose files by the same search, so the absence is real). The knob has a binding and no surface. Exposing the pair as compose knobs is a prerequisite for #14418 actuating this window at all, and it is small, concrete work.

Restore obligation — currently unimplementable as specified

@neo-opus-grace, verified at ff96657e38:

1. The risk is live, not hypothetical. #16314 merged 08-02 and the revision handed to the CPU-only deployment on 08-07 contains it, on hardware where the deferral rate measured ~100%. That deployment held ~46 memories with ~43 pending. The set this ticket wants to restore may already be most of that Memory Core.
2. The only unarchive that exists cannot serve as the restore. unarchiveMemoriesByAgentIdentity (MemoryService.mjs:1304) is the sole writer clearing archivedAt/archivedReason, and it is reason-blind (selection at :1317 tests archivedAt only, so it cannot restore generation-timeout rows without also restoring deliberate no-content and operator tombstones), identity-scoped rather than row-scoped, and cannot clear the tally — recordMiniSummaryAttempt has no reset path by design (:1866-1870).
3. Ordering is load-bearing. Restore-then-widen re-archives everything on the first pass; widen-then-restore is the only sequence that terminates. Neither is expressible as an operation today — unarchiveMemoriesByAgentIdentity is not surfaced as an MCP tool.

The private-summary tier

Including thought in the summary input shipped in #16313 and was withdrawn there on privacy grounds. thought is private: queryRecentTurns forces projection: 'public' for peer reads, but the resulting miniSummary is returned ungated by both public shapes — so feeding thought to the summarizer laundered private reasoning into a public field as derived text. Derived text cannot launder the boundary.

The enabling gap is one argument at one dispatch site (MemoryService.mjs:1478-1480): effectiveProjection is already computed and in scope, the full path receives it and gates thought on it (:1551), the summary path never receives it. There is no second generation pass and no extra cost — an earlier cost claim was retracted by its author.

Acceptance Criteria

• Expose the window pair as compose knobs (NEO_MC_MINI_SUMMARY_TIMEOUT_MS, NEO_MC_GENERATE_MINI_SUMMARY_TIMEOUT_MS), so #14418's shipped reconfigure can reach them. Without this the controller has nothing to actuate.
  • ⚠️ RETRACTED 2026-08-24 — this AC was FALSE, and implementing it would have disabled the actuation it claimed to enable. Measured, not argued:
    • The actuation channel is a config OVERLAY, not the environment. recoveryOverrideStore.mjs writes "a validated knob transaction to the durable config overlay the target reads at boot", keyed by leaf path. The running mc-server's PID-1 cmdline shows it live: --config /app/.neo-ai-data/deployment-state/recovery-actuator-overrides.json.
    • An env pin REFUSES the actuation. writeKnobOverride rejects a transaction when any of the knob's leaves is env-pinned (recoveryOverrideStore.mjs:131-132): "the env layer outranks this overlay, so the write would be discarded" — because "the config layer re-asserts the env layer after merging an overlay" (:99-100). Setting either variable to a value in compose would permanently disable reconfigure for this knob.
    • A leaf's env field is the CONFLICT KEY, not a delivery address. That is the inversion this AC rested on.
    • The knob is already declared and bounded. RECOVERY_KNOBS['minisummary-generation-window'] carries both leaves with role: 'inner'/'outer', the inner-strictly-below-outer invariant, and outer-leaves-room-for-a-draining-sweep — a ceiling expressed relative to memoryService.miniSummaryBackfillMaxRunMs that refuses on an unresolvable budget. Spec-covered in recoveryKnobRegistry.spec.mjs, recoveryOverrideStore.spec.mjs, RecoveryActuatorService.spec.mjs.
    • So AC-2 below ("declare the knob shape to the controller as a bounded, ordered pair") is also already satisfied, more thoroughly than it was specified.
    • Full derivation in the closing comment of #17731, the leaf I filed for this AC and then closed as not planned.
• Declare the knob shape to the controller as a bounded, ordered pair: the inner leaf stays strictly below the outer, the bound is expressed against the outer, and the ceiling is relative to measured baseline rather than an absolute constant.
• The detector counts a timeout on BOTH branches (falsy return and thrown). Falsifier: drive a stub past 30s with only the inner leaf widening — items must NOT silently move to the sweep's catch while the pass reports the same starved shape.
• A reason-scoped restore for archivedReason === 'generation-timeout' that resets miniSummaryAttempts in the same transaction, so the two cannot drift. Falsifier: a restored row keeping its spent tally has zero budget and re-archives on its FIRST failure, turning "N failures at the maximum window" into "one".
• The restore does not disturb deliberately-archived rows — positive control: a no-content or operator-archived row is still archived afterwards.
• Widen-before-restore ordering asserted directly, since the reverse silently no-ops.
• Timed-out requests are canceled provider-side (no orphaned compute), witnessed by a stub asserting cancellation.
• The summary input includes thought when present, with no content-lossy truncation, and a thought-derived summary is unreachable from a default/public peer read. Falsifier (already written, reusable): a same-tenant peer reads the target row, the row IS returned (positive control), and no canary planted in thought appears in the summary or full shape.
• A row failing N consecutive times is archived with a machine-readable reason and tallied. (#16313 — at the FIXED window.)
• Batch generation no longer runs at interactive priority. (#16313)
• ADR 0019 §3 self-audit for any new leaves recorded in the PR.

Contract Ledger

Out of scope

• The homeostatic controller itself — #14418 owns the detector, controller, actuation envelope and heal-ledger events, including the convergence and reversibility criteria.
• The healthcheck-canary and tenant-sync siblings — #16222 owns the family framing.
• Summarization quality.

Related

• #14418 — the controller this ticket consumes; #16223 is its first concrete use case
• #16222 — failure-handling family (canary pole)
• #16313 — the bounded half, shipped: attempt budget, reason ledger, batch priority

depends-on ADR 0025 (detector) · depends-on ADR 0026 (config-tier actuation) · aligned-with ADR 0019 (window bounds as declarative leaves)

Origin Session ID: 449fb365-2bce-4451-bf99-35b0cffd015b

Retrieval Hint: miniSummary backfill deferred forever empty generation attempt budget archive reason restore reset tally widened window compose knob homeostatic controller consumer