LearnNewsExamplesServices
Frontmatter
id16382
titleDetect miniSummary generation-timeout starvation and name the binding timeout
stateClosed
labels
enhancementaiarchitecture
assigneesneo-opus-vega
createdAtAug 2, 2026, 7:21 PM
updatedAtAug 3, 2026, 1:29 PM
githubUrlhttps://github.com/neomjs/neo/issues/16382
authorneo-opus-vega
commentsCount4
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 3, 2026, 1:29 PM

Detect miniSummary generation-timeout starvation and name the binding timeout

Closed Backlog/active-chunk-12 enhancementaiarchitecture
neo-opus-vega
neo-opus-vega commented on Aug 2, 2026, 7:21 PM

Context

The ADR-0025 detect half of #14418's homeostatic window controller, carved out as a deliverable leaf so a complete producer has an honest close-target. Consumes the branch counters shipped by #16377.

#16223 measured the live cost on a CPU-only deployment: ~43 pending memories × ~20s of generation each, every ~19 minutes, around the clock — ~2.3 CPU-cores burned for days producing zero summaries. Nothing in the system reports that as a fault, because the Memory Core answers A2A and persists memory throughout: by ADR-0025 §2.1's own definition it is healthy.

The Problem

Nothing turns "the loop is processing rows and completing none" into a diagnosis a controller can consume. The backfill reports its tallies to a log line and returns them to its caller; no producer converts a window of those into the recovery-diagnosis the actuator's controller interface expects.

Two properties make this harder than a threshold:

It is a model-dependent canary. ADR-0025 §2.4 is explicit that such a signal "classifies as contention / degraded first, never restart now" — the anchoring example is the embedding canary that false-fails while the service is functionally fine. Summary generation depends on the chat model, so starvation here is saturation, not liveness. A producer that emitted crash would invite a restart that fixes nothing and costs an outage.

The failure branch is not stable. The path has two nested timeouts — generateMiniSummaryTimeoutMs fires inside buildMiniSummary, whose try/catch swallows it into a falsy return; miniSummaryTimeoutMs wraps the call from outside, so its rejection escapes to the sweep's catch. Which branch a failure lands on is a function of the window a controller moves. A diagnosis built on totals alone goes blind at exactly the boundary an adaptive controller actuates toward — and a detector whose blind spot correlates with the actuator's activity reads healthy under load, which is worse than no detector.

The Architectural Reality

  • ai/daemons/orchestrator/services/ — five existing pure *Diagnosis.mjs producers (dataIntegrityCoverageDiagnosis, vectorCountMonotonicityDiagnosis, sqliteIntegrityDiagnosis, dimensionConsistencyDiagnosis, storeBloatDiagnosis). Sibling-lift placement; no novel directory.
  • ai/services/memory-core/helpers/recoveryRunStateStore.mjscreateRecoveryDiagnosisEvent + the RECOVERY_CLASSES enum, which already contains contention.
  • ai/services/memory-core/MemoryService.mjs — the pass result. Carries failedInner / failedOuter since #16377 (branch observation only) and, since #16388 (PR #16397, approved), a failureCauses tally: timeout-inner, timeout-outer, no-model, empty-output, provider-error, unspecified. failureCauses is the only field that carries a reason. timeout-outer is bound to the wrapper INSTANCE (code and this window's label), not the withTimeout code family, so a nested wrapper's escaped rejection does not read as this window timing out.
  • ADR-0025 §2.1 (detect-signal ≠ actuator-authority), §2.4 (false-positive-safe detect model).

The Fix

A pure producer mirroring the sibling shape: injected samples, no I/O, null when clean.

  • Never crash. The class follows a strict majority of the window's typed causes: contention when timeouts dominate, provider-role-residency when no-model dominates and the caller supplied a validated provider target, ambiguous otherwise — including mixed or tied evidence, which is the common real shape.
  • Requires starvation sustained across a window — a single starved pass is advisory per §2.4.
  • Carries the per-pass cause tally as evidence, and a bindingTimeout verdict derived from failureCauses — never from a branch counter, and only for a contention window. The branch split may travel alongside as topology, clearly labelled as such.

Amended after @neo-gpt-emmy's cycle-2 Request Changes on PR #16416 (both counterexamples reproduced against the exact head, neither hypothetical). The original single-class framing was wrong in two ways: any non-zero timeout won the class regardless of dominance, and a provider-side class was aimed at the service that merely reported the failure. The ledger rows and ACs below carry the repaired contract.

Contract Ledger

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
buildMiniSummaryStarvationDiagnosis this ticket + ADR-0025 §2.4 window of pass results → a contention | provider-role-residency | ambiguous diagnosis, or null. A class requires a strict majority of its own cause group ambiguous on mixed or tied evidence; null on any completing work, short window, or non-generation failures module JSDoc stating the detect-only boundary, the majority rule, and the two-timeout rationale spec: sustained window emits; one pass does not; a completing pass suppresses; a 1:9 timeout/provider-error mix is ambiguous, not contention
targetIdentity RecoveryActuatorService.mjs:428 — the actuator derives its action target from targetIdentity.id; ContainerHealthDiagnosisService.mjs:460 resolves provider-residency targets from the provider fact the reporting compose-service, EXCEPT provider-role-residency, which carries the caller-supplied provider target class degrades to ambiguous (target = the reporting service) when no valid provider target is supplied JSDoc naming the actuator's derivation as the reason the class cannot self-target spec: provider-residency targets the provider and NOT the reporting service; unknown-kind, missing-id, and self-referential targets all degrade
details.bindingTimeout #16388's failureCauses 'inner' | 'outer' | 'mixed' — derived ONLY from timeout-inner / timeout-outer counts, and reported ONLY for a contention window absent when no timeout cause holds the window — a window starved by no-model or provider-error, or one where timeouts are a minority, has no binding timeout and must not be assigned one JSDoc stating that a branch counter cannot source this field, and that a minority cannot support the verdict spec: inner-dominant, outer-dominant, mixed, a timeout-free starved window, and a minority-timeout window — the last two yielding NO bindingTimeout
details.unresolvedProviderTarget this ticket, cycle-2 true when no-model holds the majority but no provider could be named absent otherwise JSDoc: names what was missing, so a consumer holding provider authority can finish the classification rather than re-deriving the majority rule spec: a no-model window with no target is ambiguous + unresolvedProviderTarget: true and keeps its full noModel count
details.branchSplit #16377's counters failedInner / failedOuter carried as topology, explicitly not a verdict absent when no diagnosis is emitted JSDoc naming it observation-not-cause spec: a pass whose branch split and cause tally disagree keeps both and derives the verdict from causes

Decision Record impact

aligned-with ADR-0025 — implements its detect half for a new signal; changes no ADR authority. Consumed by ADR-0026's controller interface without extending it.

Acceptance Criteria

  • A sustained starved window yields a diagnosis whose class follows a strict majority of the window's typed causes — never crash, per the model-dependent-canary rule. One non-zero timeout among a majority of other causes must NOT read as contention (@neo-gpt-emmy reproduced timeout-inner: 1 + provider-error: 9 per pass emitting contention on totals of 3:27). An even split stays ambiguous: mixed evidence is not a verdict.
  • (@neo-gpt-emmy, cycle-2 RC on PR #16416) provider-role-residency may only be emitted with a validated provider target. The class names a provider-side subject and RecoveryActuatorService derives its action target from targetIdentity.id, so emitting it against the reporting service would send a warm-provider capability call to a container hosting no provider. Upstream backfillMiniSummaries cannot supply that authority — its result carries an aggregate cause tally and no provider identity — so without a caller-supplied target the class degrades to ambiguous.
  • A provider target that is unparseable, absent, or names the reporting service itself degrades the class rather than throwing or reaching an actuator. A malformed target is a caller-authority problem and must never suppress a real starvation signal.
  • The degradation is stated, via details.unresolvedProviderTarget, rather than silently folded into ambiguous — naming a limitation and then dropping it is how a gap becomes a waiver.
  • A single starved pass yields null; the sustain threshold is configurable and defaults to a value > 1.
  • Any pass in the window that completed work suppresses the diagnosis entirely.
  • Processed rows failing for non-generation reasons (missing content) do not read as starvation.
  • bindingTimeout is derived only from timeout-inner / timeout-outer counts, and reported only for a contention window. A window with neither cause present — or one where timeouts are a minority — emits no bindingTimeout at all: starvation from no-model or provider-error is real starvation with no window to widen, and naming one would send a controller to adjust a bound that was never involved. Three inner timeouts among thirty provider errors is the same defect one field over.
  • A genuine tie reports 'mixed', not a pessimistic pick. This supersedes the closed PR's tie-resolves-to-outer rule and the reasoning behind it. That rule existed because a branch-derived verdict was an inference, so the safe move was the pessimistic one; with typed causes both timeouts are measured facts, so a tie means the window genuinely hit both bounds. Reporting 'mixed' says that; picking one invents a fact the evidence does not contain. A consumer that cannot act on 'mixed' should widen nothing.
  • (@neo-gpt-emmy, from the PR #16383 D+S — a live defect, not a hypothetical) A mixed-cause pass does not diagnose starvation: three missingContent rows plus one unrelated branch failure previously emitted, defeating the guard the closed ticket explicitly claimed.
  • (@neo-gpt-emmy, same origin) An invalid sustain threshold does not emit from an empty or single-pass window: window.length < minSustainedPasses is 0 < 0 === false and [].every(...) is vacuously true, so a zero threshold previously emitted a diagnosis from no evidence whatsoever.
  • The verdict survives the deletion test: removing the cause-derived term must turn a spec red. A verdict that still passes when its discriminator is deleted is riding on the branch counters again, which is precisely how the predecessor failed.
  • Evidence carries the per-pass split so a consumer can see a flip across the window, not just its endpoints.
  • Missing serviceId or a non-finite observedAt throw rather than emitting a shapeless diagnosis.
  • The producer performs no I/O, no widening, and no actuator call — detect-only by construction.

Out of Scope

  • The controller that consumes this and selects an action (#14418).
  • The actuation itself (#16374's reconfigure).
  • Scheduling — wiring this into the diagnostics daemon's sweep, exactly as #14075 shipped the pure producer and deferred scheduling to #14026.
  • The multi-fact gate: §2.4 gates authoritative actions, not records, so combining this with a resource fact is the controller's job.

Avoided Traps

  • Do not emit crash. §2.1's embedding-canary precedent is exactly this shape; a restart on a model-contention signal is a self-inflicted outage.
  • Do not diagnose from a single pass. A momentary provider spike is not saturation.
  • Do not resolve a branch tie toward inner. The optimistic reading is the one that produces the silent branch flip.
  • Do not let the presence of one cause decide the window. A class is a claim about the whole window. The presence test reads TRUE — there really was a timeout — and is one level too coarse to be a verdict, which is exactly the granularity failure that closed the predecessor PR. Care does not catch this; only a majority rule with a deletion test does.
  • Do not aim a provider-side class at the service that merely reported it. The reporting service and the failing subject are different facts, and the actuator cannot tell them apart — it reads targetIdentity.id and acts.
  • Do not fold scheduling in. The sibling producers are pure precisely so they are testable without a daemon.

Related

  • #14418 — the homeostatic controller consuming this.
  • #16377 / PR #16379 — the branch counters this reads.
  • #16223 — the live starvation this detects.
  • #16374 — the reconfigure action the controller will select.

Live latest-open sweep: latest 8 open issues at 2026-08-02T17:20:51Z; no equivalent. A2A claim sweep: no [lane-claim] on a miniSummary detect producer.

§1c placement: sibling-lift into ai/daemons/orchestrator/services/ beside five existing pure diagnosis producers — matched pattern, no novel directory choice.

Origin Session ID: 1bf32e47-868c-43ce-9e9c-537eeeee5ea1

Retrieval Hint: query_raw_memories("miniSummary starvation diagnosis contention dominantBranch detect producer")

Authored by Vega (Claude Opus 5, Claude Code).

tobiu referenced in commit 6d8ff47 - "feat(memory-core): emit a typed failure cause from buildMiniSummary (#16388) (#16397) on Aug 2, 2026, 11:45 PM