LearnNewsExamplesServices
Frontmatter
id16443
titleRetry-growth bounds are knowable only by reading each caller
stateClosed
labels
enhancementaiarchitecture
assigneesneo-opus-grace
createdAtAug 3, 2026, 4:24 PM
updatedAtAug 4, 2026, 9:59 AM
githubUrlhttps://github.com/neomjs/neo/issues/16443
authorneo-opus-grace
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 4, 2026, 9:59 AM

Retry-growth bounds are knowable only by reading each caller

Closed Backlog/active-chunk-12 enhancementaiarchitecture
neo-opus-grace
neo-opus-grace commented on Aug 3, 2026, 4:24 PM

Context

Filed 2026-08-03 after a cloud deployment sat unable to recover from a fixed credential: its tenant-repo lane was backoff-suppressed at 2^consecutiveFailures × 30min with no cap, so the next retry was ~128h away and the suppression survived restart. #16224 fixed that lane (2h backoffCapMs + starved status), and the operator asked for a standing rule — "backoff times must NOT grow indefinitely."

The standing rule turned out to already hold, and finding that out was the expensive part. I ran the census twice and got it wrong both times before @neo-gpt-emmy corrected the population. That cost is the ticket.

The Problem

Every retry-growth site in the tree is bounded — but boundedness is only knowable by reading each site's caller, and the bound lives in four structurally different places. There is no declaration, no convention, and no way to answer "is this one bounded?" without a manual trace.

Verified census (2026-08-03, dev), genuine retry-growth sites only:

site bound carrier
ai/daemons/embed/drainCycle.mjs:81 max-delay (Math.min(…, MAX_RECORD_COOLDOWN_MS))
ai/daemons/orchestrator/scheduling/tenantRepoSync.mjs:123 max-delay (backoffCapMs leaf)
ai/daemons/orchestrator/services/RecoveryActuatorService.mjs:1022 max-delay (Math.min(maxBackoffMs, …))
src/data/connection/WebSocket.mjs:23 max-delay (Math.min(…, 30000))
ai/agent/Loop.mjs:421 attempt budget (retryCount < this.maxRetries)
ai/daemons/message/drainCycle.mjs:21 attempt budget (caller's attempt <= maxRetries loop)
ai/services/knowledge-base/VectorService.mjs:684 attempt budget (retries < maxRetries)
ai/services/memory-core/WebhookDeliveryService.mjs:194 attempt budget (attempt <= maxRetries)
ai/services/memory-core/helpers/freezeReprobeDecision.mjs:70 terminal state (attempts >= maxUnfreezeAttemptscontained)

Three properties make this un-reviewable by inspection:

  1. The bound is usually not at the growth site. Four of nine bound in the caller. message/drainCycle.mjs:21 returns a raw backoffBaseMs * 2 ** attempt and looks identical to an unbounded defect; its caller's loop counter is the bound. I flagged it as a defect on first read and was wrong.
  2. freezeReprobeDecision returns a genuinely growing delayminReprobeIntervalMs * Math.pow(backoffMultiplier, attempts) — and is bounded by a terminal state that stops the loop, not by any cap on the value. No delay-shaped check can see that.
  3. The syntax is not one pattern. Math.pow(2, n), 2 ** n, backoff *= 2, and Math.pow(configurableMultiplier, n) all occur. My first census matched only literal base-2 exponentiation and reported five sites; the real count is nine, and I asserted "all bounded, nothing to fix" from the incomplete set.

And the naive check is worse than nothing. A Math.pow( sweep returns a large false-positive family — easing curves in apps/portal/canvas/HomeCanvas.mjs, apps/portal/canvas/ServicesCanvas.mjs, src/canvas/Header.mjs, src/canvas/Sparkline.mjs, and a power-law random offset in apps/devindex/services/Spider.mjs. A gate that fails on match would train contributors to add suppressions, which is how the invariant actually dies.

The Architectural Reality

  • The growth sites span three substrates — orchestrator scheduling lanes, Memory Core / Knowledge Base services, and Body-side src/data/connection/WebSocket.mjs. No single owner.
  • Bound carriers differ by state lifetime: in-cycle loop counters (message/drainCycle), process-local (agent/Loop), and persisted/cross-cycle (tenantRepoSync's consecutiveFailures in the orchestrator state volume). Only the persisted class can strand a deployment across restarts, which is why #16224 needed a delay cap where an attempt budget would have sufficed elsewhere.
  • Sibling precedent for the mechanism already exists: ai/scripts/lint/config-leaf-parity.json is a JSON census consumed by ai/scripts/lint/lint-config-template-ssot.mjs, which fails CI on drift. This ticket reuses that shape rather than inventing one.
  • #16307's body records why a process-local flight gate is the wrong mechanism for persisted tenant cadence state — so a universal boundedRetryGate mandate is explicitly not the prescription.

The Fix

A candidate discovery + explicit bound classification check, in the existing lint substrate:

  1. Discovery — match retry-growth syntax broadly: Math.pow(<any base>, <var>), <any> ** <var>, and multiplicative mutation (x *= n) on delay/backoff-named bindings.
  2. Classification registry — a JSON manifest beside config-leaf-parity.json. Each production candidate declares:
    • lifetime: in-cycle | process-local | persisted
    • bound carrier: max-delay | max-attempts | max-window | terminal-state
    • witness: a direct spec or exported-policy reference proving the bound
  3. Reporting discipline — a new match reports unclassified, never unbounded. The check asserts the registry is complete, not that any site is defective.
  4. Drift gate — CI fails when the discovered candidate set and the registry disagree in either direction (new unclassified candidate, or a registry entry whose site is gone).

Non-retry matches (easing, animation, random distributions) are classified once as not-a-retry with the same witness requirement, so the false-positive family is recorded rather than silently filtered by a regex nobody can audit.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback / Error Semantics Docs Evidence
retry-bound registry JSON (ai/scripts/lint/) this ticket; shape mirrors config-leaf-parity.json Each candidate declares lifetime, bound carrier, and witness A candidate absent from the registry is unclassified and fails the gate; it is never reported as unbounded registry header comment spec: a synthetic new growth site fails the gate as unclassified
discovery check (ai/scripts/lint/) this ticket; sibling lint-config-template-ssot.mjs Discovers candidates and diffs against the registry Fails CI on set drift in either direction; a discovery-engine error fails loudly rather than reporting an empty candidate set check JSDoc spec: removing a registry entry whose site still exists fails; deleting a site whose entry remains fails
not-a-retry classification this ticket Records the easing/animation/random false-positive family explicitly Same witness requirement as a real bound — this is not a suppression allowlist registry header spec: an easing-curve match is classified, not silently dropped

Decision Record impact

none — this adds a lint-substrate census over existing behavior. It does not change any retry policy, and it explicitly does not mandate a shared retry primitive (see Avoided Traps).

Acceptance Criteria

  • The discovery pass finds every genuine retry-growth site from a single run — no hand-maintained site list in the checker. (Filed as "all nine sites"; corrected to the discovered set, which is 16. The census table above is left as filed — its drift from the shipped registry is the argument for the tool. See the AC-1 correction comment.)
  • A newly introduced growth site with no registry entry fails CI, and the failure text says unclassified, never unbounded.
  • Each registry entry names a lifetime, a bound carrier, and a witness that resolves to a real spec or exported policy; a witness that does not resolve fails the gate.
  • message/drainCycle.mjs:21 classifies as in-cycle / max-attempts with its caller's loop as the witness — the case that defeats any "returned vs consumed" heuristic.
  • freezeReprobeDecision.mjs:70 classifies as terminal-state, proving the schema admits a bound that is not a cap on the delay value.
  • Deleting a classified site fails the gate as registry drift.
  • The easing/animation matches are classified not-a-retry with witnesses, not excluded by a path or filename filter.

Out of Scope

  • Changing any retry policy. Every site in the census is bounded today; this ticket adds no cap and alters no cadence.
  • A universal shared retry primitive. #16307 records why a process-local flight gate is wrong for persisted cadence state; mandating one mechanism across three lifetimes would re-import that error.
  • The cause-proven access-recovery admission successor — a separate shape (#16224 family) proposed by @neo-gpt-emmy: admit one immediate sync attempt when a persisted remediable access cause meets a currently-READY probe, without pre-clearing evidence. Distinct concern, distinct ticket.
  • Static proof of boundedness. The check discovers and diffs; the bound is declared by a human/agent with a witness. Deciding boundedness automatically is not attempted.

Avoided Traps

  • A fail-on-match lint. The false-positive family (easing curves, power-law random) is larger than the true-positive set. A gate that flags matches as violations trains contributors to suppress it, and the invariant dies quietly.
  • "Returned vs consumed" as the heuristic. It is unsound: message/drainCycle.mjs:21 returns a raw exponential and is correctly bounded by its caller. This was my first proposed rule and it would have produced a false positive on its first run.
  • Trusting a single regex to define the population. Two censuses in this ticket's own history were wrong — five sites, then nine — because the pattern, not the problem, set the scope. Discovery must match multiple syntaxes and report what it cannot classify.
  • Treating the registry as a suppression allowlist. Every entry must name the actual bound and a resolving witness; "known exception" without a witness is the failure mode this replaces.

Related

  • #16224 — bounded the tenant-repo-sync backoff; the specimen that prompted the standing rule.
  • #16307 — the implementing PR; its body records the process-local-gate rejection this ticket honors.
  • #16227 — bounded embed retry for one lane; adjacent site-level work, distinct from this census layer.
  • ai/scripts/lint/config-leaf-parity.json / lint-config-template-ssot.mjs — the census + drift-gate precedent reused here.

Concept reviewed pre-filing by @neo-gpt-emmy, who corrected the census population and reframed the check from semantic proof to classification. The nine-site table above is my re-verification of that correction, not a restatement of it.

Live latest-open sweep: checked latest 20 open issues at 2026-08-03T14:22Z; A2A claim sweep over 30 most-recent messages; no equivalent found.

Origin Session ID: 9f05cd72-5457-4ec2-926c-ef1406041f19

Retrieval Hint: query_raw_memories("retry growth bound classification registry unclassified never unbounded backoff census")