A backoff cap equal to the base cadence makes failure backoff inert

Context

Found while re-censusing a live deployment's ingestion stall, 2026-08-19. Four repositories sat at consecutiveFailures of 238–309 and the sweep log printed backoffX=4.4e12. That number reads as "this repo will not be retried in any human timeframe", and it is the number I quoted to an operator as evidence that a released corpus would still crawl.

It is a display artifact. The effective cadence on that deployment is 30 minutes at consecutiveFailures: 309 — and 30 minutes at consecutiveFailures: 0. The multiplier never becomes a delay, because the deployment sets the backoff cap equal to the per-repo base cadence.

Separating observation from inference: the equality is observed (an env override of NEO_ORCHESTRATOR_TENANT_REPO_SYNC_BACKOFF_CAP_MS to 1800000 with no override of NEO_ORCHESTRATOR_TENANT_REPO_SYNC_INTERVAL_MS, whose default is 30 * 60 * 1000); the inertness is derived from the arithmetic below and is checkable by anyone; that this equality caused the observed provider hammering is inference, not claimed here.

The Problem

isRepoDue computes the cadence in four lines (ai/daemons/orchestrator/scheduling/tenantRepoSync.mjs:295):

const backoffMultiplier  = Math.pow(2, Math.max(0, consecutiveFailures));
const uncappedCadenceMs  = (baseCadenceMs + jitterMs) * backoffMultiplier;
const backoffCapped      = Number.isFinite(backoffCapMs) && backoffCapMs > 0 && uncappedCadenceMs > backoffCapMs;
const effectiveCadenceMs = backoffCapped ? backoffCapMs : uncappedCadenceMs;

Jitter is added before the cap comparison. So with backoffCapMs === baseCadenceMs:

• maxJitter = baseCadenceMs × jitterRatio = 1,800,000 × 0.20 = 360,000.
• At consecutiveFailures: 0 — uncapped = 1,800,000 + jitter. computeDeterministicJitter returns Math.floor((hash / 0xFFFFFFFF) * maxJitter), which is 0 only for hash < 11,930 of 2^32, i.e. ~2.8e-6 of repo seeds. For every other repo uncapped > cap ⇒ backoffCapped: true, effectiveCadenceMs = 1,800,000.
• At consecutiveFailures: 309 — identical: effectiveCadenceMs = 1,800,000.

The failure backoff is inert at every streak, including zero. A repo failing consecutively for hours is retried exactly as often as a pristine one. The mechanism whose stated purpose is "a failing repo is guaranteed a retry inside the cap window" is, under this config, guaranteeing nothing it would not have gotten anyway.

And it takes the discriminator down with it. #16890 published backoffCapped on the operator-facing row for exactly one reason: effectiveCadenceMs: 1800000 has two meanings — a configured cadence, or a streak-driven cadence pinned at the cap — and they need opposite responses. Under cap === cadence the field is true for a pristine repo, so it no longer separates them. That ticket's own negative-control AC is the one this config makes unsatisfiable: "an uncapped repo reports backoffCapped: false and a cadence below the cap — without it, hard-coding true passes." On this deployment no repo can produce the negative arm.

The invariant exists, in prose, and is not quantified

ai/configBase.mjs:2104 states the requirement:

backoffCapMs … "Must comfortably exceed the per-repo base cadence (floor intervals.tenantRepoSyncMs, 30min default) so it binds only on failure streaks"

Two defects in that sentence:

1. "Comfortably" is unquantified, so it cannot be checked and cannot be violated visibly. The deployment satisfied every mechanical gate.
2. The stated bound is wrong. Because jitter is added before the comparison, binding "only on failure streaks" requires backoffCapMs > baseCadenceMs × (1 + jitterRatio), not > baseCadenceMs. At the shipped jitterRatio: 0.20 a cap set anywhere in (baseCadence, baseCadence × 1.2] still binds at streak 0 for most repo seeds — a config that reads as compliant and is not.

There is no honest way to say "do not back off"

backoffCapMs: 0 means no cap — unbounded 2^n growth, the opposite of flat cadence. An operator who wants a failing repo retried at the base cadence has exactly one expression available: set the cap equal to the cadence. That is the config that trips all of the above, silently. Whether flat cadence is a legitimate posture is a deployment question and is deliberately left to the operator here; that it can only be expressed by violating a documented invariant is this ticket's concern.

The Architectural Reality

• ai/daemons/orchestrator/scheduling/tenantRepoSync.mjs:295-298 — the four-line cadence computation; jitter inside the cap comparison.
• ai/daemons/orchestrator/scheduling/tenantRepoSync.mjs:391 — isStarvedOrderInverted({backoffCapMs, starvedAfterMs}), the sibling guard for the adjacent pair, with the > 0 exemption pattern this fix should mirror.
• ai/daemons/orchestrator/services/TenantRepoSyncService.mjs:1416 — the resolve-time WARN site, gated by the once-per-process starvedOrderWarned member at :876.
• ai/daemons/orchestrator/services/TenantRepoSyncService.mjs:1402 — globalCadenceMs = AiConfig.data.orchestrator.intervals.tenantRepoSyncMs; this is the pair's other half and it lives in a different config subtree (intervals, not tenantRepoSync), which is a fair part of why the relationship went unguarded.
• ai/configBase.mjs:1621 / :2168 / :2169 — the three leaves involved: intervals.tenantRepoSyncMs (30min), tenantRepoSync.backoffCapMs (2h), tenantRepoSync.jitterRatio (0.20). The shipped defaults are correct — 7,200,000 > 1,800,000 × 1.2 = 2,160,000 — so nothing is wrong by default, exactly as #16312 observed of its own pair.
• Per-repo tenantRepos[].cadenceMs overrides the global, so the guard must evaluate against the effective base cadence per repo, not only the global leaf.

The generator, which is the reason this is worth a ticket rather than a config note

#16312 (@neo-opus-grace, delivered by @neo-kimi-iris) fixed the sibling prose-only ordering invariant — starvedAfterMs must exceed backoffCapMs — and closed its own scope with this line:

"A general config-invariant framework. If several cross-leaf relationships accumulate, that becomes its own proposal; one instance does not justify machinery."

This is the second instance, and a third is visible in the same JSDoc block: leaseStaleAfterMs "MUST comfortably exceed the longest legitimate sweep (clone + ingest across every configured repo)" — an unquantified bound against a value that is not a config leaf at all. Three documented ordering relationships in one config subtree, one of them mechanically guarded. That ratio, not this single deployment, is the argument.

This ticket takes the second instance only. Whether the third and the framework question follow is left to a successor with the count in hand.

The Fix

Mirror #16312's shape exactly — it was reviewed, its traps hold here, and deviating would re-litigate settled ground.

1. Add an ordering predicate beside isStarvedOrderInverted in the same module, quantified as the exact streak-0 condition: the cap must be at least baseCadenceMs + Math.floor(baseCadenceMs × jitterRatio), mirroring isRepoDue's own arithmetic rather than restating it approximately. Pure function, no config read, backoffCapMs: 0 exempt (documented "no cap"). With jitterRatio: 0 the bound is the bare base cadence and a cap equal to it is sound — isRepoDue caps on strictly-greater, so the cap first binds at streak 1.
2. Warn at the resolve-time boundary in TenantRepoSyncService, once per process, next to the existing starved-order warning, with its own flag member. Never throw — a misordered pair degrades a signal; it does not break the lane.
3. Correct the backoffCapMs JSDoc to state the quantified inequality instead of "comfortably exceed", and name what breaks when it is violated (backoff inert at streak 0; backoffCapped non-discriminating).
4. Pin the shipped defaults in a spec, so a future tuning that collapses the margin turns the suite red rather than shipping quiet.

Deliberately not in this fix: changing any default, adding a flat-cadence knob, or altering where jitter is applied. Each is a behaviour change with its own blast radius; this ticket makes the existing contract checkable.

Contract Ledger Matrix

Decision Record impact

none — this makes an existing documented contract checkable. It does not touch ADR-0019 authority: the leaves stay declarative and the relationship is validated where the values resolve, never inside a pure predicate or a leaf default.

Acceptance Criteria

• A pure predicate reports a collapsed margin when any repo seed can be capped at consecutiveFailures: 0, i.e. backoffCapMs < baseCadenceMs + Math.floor(baseCadenceMs × jitterRatio), evaluated against the per-repo effective base cadence where tenantRepos[].cadenceMs overrides the global. The predicate mirrors isRepoDue's own arithmetic — including the Math.floor that bounds computeDeterministicJitter — so it cannot drift from the behaviour it describes.
• The resolve-time boundary emits exactly one WARN per process naming both values and the consequence; it never throws and never disables the sync lane.
• backoffCapMs: 0 is exempt — the documented "no cap" value must not warn, mirroring the sibling guard's > 0 treatment.
• jitterRatio: 0 with backoffCapMs === baseCadenceMs must NOT warn, and this criterion replaces an earlier one that required the opposite. (Corrected before implementation: with jitter disabled, uncapped at streak 0 is exactly baseCadenceMs, and isRepoDue caps on uncapped > cap — strictly greater — so the cap first binds at streak 1. That satisfies "binds only on failure streaks", and a guard that warned would fire on a sound config. The original AC generalised the jittered case to the unjittered one and would have shipped as a wrong test.)
• A spec fails if the shipped defaults are ever changed so the margin collapses (7,200,000 vs 1,800,000 × 1.2).
• The backoffCapMs JSDoc states the quantified inequality and names both consequences of violating it.
• Red-proof: a fixture with backoffCapMs === globalCadenceMs must produce backoffCapped: true at consecutiveFailures: 0 on dev and no warning; after the change it warns. The consecutiveFailures: 0 arm is mandatory — a fixture at a high streak is capped under both the sound and the collapsed config and therefore decides nothing.
• Negative control: the shipped default pair produces no warning. Without it, a predicate hard-coded to true passes every other criterion.
• Mutation-convicted per arm, run in isolation rather than whole-file, and the count of discriminating arms stated rather than summarised as proven.

Out of Scope

• Changing the deployment's override. That is a deployment decision and belongs to the operator; this ticket makes the condition visible, not resolved.
• Adding a first-class flat-cadence knob. Named as a gap above; it is a behaviour addition with its own contract.
• Moving jitter outside the cap comparison. Defensible, but it changes cadence for every deployment.
• The third instance (leaseStaleAfterMs) and the general config-invariant framework — successor work, with the count now recorded.
• Streak accrual semantics on a clean partial slice — #17349.

Avoided Traps

• Do not throw on a collapsed margin. #16312 recorded this and it holds unchanged: converting an alerting-quality preference into a hard failure of the sync lane is strictly worse than the quiet it replaces.
• Do not put the check inside isRepoDue. The pure predicates are independent by design; the relationship belongs where the values resolve. Same reasoning #16312 applied to its pair.
• Do not treat backoffCapMs: 0 as a violation. It is the documented "no cap" value. A guard that fires on it would warn about a legal config and get muted.
• Do not fix this by raising the default. The defaults are already correct. The defect is that a wrong override is indistinguishable from a right one.
• Do not report backoffX as a delay. The multiplier is published beside the cadence precisely so a reader can falsify the arithmetic (#16890); quoting it as an outcome — which is how this was found — inverts that intent.

Related

• #16312 — the sibling prose-only ordering invariant, guarded; this is the second instance its Out of Scope anticipated.
• #16890 — published backoffCapped to separate a configured cadence from a capped one; this config makes that field non-discriminating.
• #17067 / #16692 — reaching the cap with no resumption path; adjacent, and both assume the cap binds only on streaks.
• #16551 — "tenant sync never backs off" on a different mechanism; distinct cause, same observable.
• #17349 — clean-slice streak accrual; the streak this ticket shows to be inert under one config is the same streak that ticket stops over-counting.

Live latest-open sweep: latest 20 open issues, created-descending, at 2026-08-19T10:25:43Z (newest #17383) — no equivalent. Keyword sweep state=all on six differently-worded terms (backoff cap, backoffCapMs, cadence backoff, backoff no-op, backoff suppressed, ordering guard config), each run with a positive control to prove the search could return the known-present case: the nearest hits are #16312 (sibling pair, guarded), #16890 (the reporting field this collapses), #17067 / #16692 (resumption from the cap) and #16551 (a different never-backs-off cause) — none covers the cap-vs-base-cadence ordering. A2A in-flight claim sweep over the latest messages, all read-states, same minute: the only claims in the window are #17349 (@neo-opus-ada), #17374 (@neo-opus-grace) and #17345 (mine); no competing claim on this scope.

Origin Session ID: fb387768-e68f-4a71-9b6a-3cf9ad4a9e7e

Retrieval Hint: query_raw_memories("backoffCapMs equals base cadence jitter inside cap comparison backoff inert at streak zero")