Tenant-repo sync backs off to its cap with no resumption condition

Context

Measured on an external deployment, 2026-08-08. All four configured tenant repos, identical state:

status               backoff-suppressed
consecutiveFailures  13
backoffMultiplier    8192
effectiveCadenceMs   7200000        <- the configured cap
nextDueAt            +1h from observation
lastIngestedRev      null
accessReadiness      ready 4/4

The underlying cause was a slow embedding provider (sibling lanes cover the ingest and health sides). This ticket is about what happens after that cause is fixed: nothing, for up to two hours per repo, because the only event that can lower the backoff is a successful run, and runs are what the backoff suppresses.

The Problem

// ai/daemons/orchestrator/scheduling/tenantRepoSync.mjs:123
const backoffMultiplier = Math.pow(2, Math.max(0, consecutiveFailures));

Doubling per failure, capped. There is no path from "the dependency recovered" to "try now" — the schedule is a pure function of past failures and wall-clock. A deployment that fixes its provider at minute one discovers the fix at minute one-hundred-and-twenty, four times over, with nothing in any surface explaining the wait.

The failure is worse than slow recovery: it is indistinguishable from the original fault. An operator who repairs the provider and sees the corpus still at zero has no signal separating "still broken" from "repaired, waiting out a backoff earned while it was broken". Both look like count: 0.

Thirteen consecutive failures also means the backoff was earned against a condition that was never going to clear on its own, which is precisely the case exponential backoff is not built for. Backoff protects a struggling dependency from a retry storm. It should not outlive the struggle.

The Architectural Reality

This is not a new notion. ai/services/shared/boundedRetryGate.mjs already names this exact consumer, in its module doc:

*"The family primitive for bounded-retry-with-reason: three independently-discovered failures (an uncached canary retrying at probe frequency, a sync lane in infinite backoff, a backfill in infinite retry) are one missing notion — retry with a bounded attempt budget, a recorded stop reason, and a guaranteed resumption condition. This module is that notion, consumer-agnostic: the embedding write canary (HealthService) is the first adopter; the other two poles adopt in their own lanes."*

The sync lane in infinite backoff is this lane. The primitive was written with it in view, the canary adopted it, and this pole did not. The three properties it supplies are the three the tenant-sync scheduler lacks: a bounded attempt budget, a retained stopReason, and a guaranteed resumption condition.

The deployment-state snapshot already carries accessReadiness per repo, refreshed independently of the sync schedule — ready 4/4 throughout the outage. So the surface where a readiness signal would live already exists and is already being maintained; it simply has no authority over the schedule.

The Fix — corrected 2026-08-08 after an intake challenge from @neo-gpt

The original prescription said "adopt boundedRetryGate as tenantRepoSync's second pole", and a literal reading of that regresses the thing this ticket is defending. Verified both halves:

• boundedRetryGate.mjs exports exactly one symbol, createBoundedRetryGate. There is no hydrate, restore, persist, or serialize seam — its generation and failureStreak are process-local.
• isRepoDue({repo, persistedRepoState, …}) reads consecutiveFailures and lastRunAttemptAt from durable per-repo state that survives orchestrator restarts.

So swapping the durable scheduler for the gate would either erase restart truth or create two competing backoff authorities. On the deployment that motivated this, the orchestrator has already restarted once today — an in-memory streak would have silently zeroed. (I diagnosed exactly this shape in ConfiguredTaskDefinitionsService's closure-local consecutiveStuckFailures earlier the same day, then prescribed it here. Recognising a defect pattern is not the same act as designing against it.)

A second gap the challenge surfaced, which the original ticket simply assumed away: there is no embedding oracle in the orchestrator process. #16704 classifies Docker probe execution only. #16702's embedding-probe producer is started by the Knowledge Base Server lifecycle, while the orchestrator resolves KB_IngestionService in-process and never starts that producer. So "a recovered dependency re-arms the lane" named a signal that does not exist where the scheduler runs.

Corrected shape:

• The durable per-repo cap / due computation stays the sole sync scheduler. boundedRetryGate is not the scheduler and must not become it.
• After a retained embedding-class source failure, arm one process-owned embedding recovery canary in the orchestrator, using the shared attempt boundary plus boundedRetryGate for probe cadence and backoff only — the same cadence-not-verdict split #16691 settled.
• Persist a recovery episode / generation receipt. A healthy canary generation grants exactly one due-bypass, and only for repos whose retained cause belongs to that episode. A restart can neither synthesise a recovery nor zero the streak.
• The snapshot distinguishes four states, not two: still-failing, recovery-probe-backoff, recovery-observed/retry-pending, and ordinary repo backoff.
• The suppressed state still carries a retained stop reason, so backoff-suppressed says why rather than only how long.

Acceptance Criteria

• A repo in backoff-suppressed whose dependency becomes available again re-arms without waiting out the full effectiveCadenceMs.
• The suppressed state exposes a retained stop reason naming the failure class that earned it.
• The deployment-state snapshot distinguishes suppressed because still failing from suppressed on a backoff earned by an already-resolved condition.
• Backoff still lengthens under sustained failure — a recovery signal that is itself unreliable must not defeat the protection.
• A test drives a repo to a capped multiplier, signals recovery, and asserts the next run is scheduled ahead of the un-recovered schedule — failing against the current Math.pow(2, consecutiveFailures) implementation.
• Restart-safety witness. A process restart between the earned backoff and the next sweep neither zeroes consecutiveFailures nor synthesises a recovery. The test restarts across the boundary and asserts the durable streak and the retained cause both survive — the failure mode a literal gate adoption would have introduced.
• The recovery episode receipt is durable, and a due-bypass is granted only to repos whose retained cause belongs to that episode — a healthy canary does not blanket-release repos that failed for an unrelated reason.

Out of Scope

• Making ingestion survive a slow embedding provider — sibling lane, and the reason these repos failed at all.
• KB healthcheck's embedding probe — sibling lane.
• The two remaining boundedRetryGate poles beyond this one (the backfill in infinite retry). Adopting this pole does not oblige the other.
• Changing the configured cadence or cap values.

Avoided Traps

• Lowering the cap. Treats the symptom and weakens the protection that is otherwise working correctly. The cap is not the defect; the absence of a resumption condition is.
• Resetting the counter on any readiness signal. accessReadiness was ready 4/4 throughout the outage — it measures repo access, not the embedding dependency that was actually failing. A resumption trigger wired to the wrong readiness surface would have re-armed all four repos every sweep for thirteen failures, converting a backoff into a retry storm against a struggling provider. The resumption condition must be tied to the dependency that earned the backoff.
• Hand-rolling the resumption. The primitive exists, names this consumer, and has a spec encoding its concurrency floor. A second implementation of bounded-retry-with-reason is how the notion got independently discovered three times in the first place.
• Literal gate adoption — rejected on a source falsifier. Making boundedRetryGate the sync scheduler regresses durable, restart-surviving per-repo backoff into process-local state, or stands up a second competing backoff authority. The primitive is right for probe cadence; it was never right for the durable schedule. See the corrected Fix above.
• Wiring resumption to accessReadiness. It measures git access and read ready 4/4 throughout the embedding outage — it would have re-armed all four repos on every sweep for thirteen consecutive failures, converting a backoff into a retry storm against an already-struggling provider.

Responsibility map (lead-role convergence artifact)

One incident, five lanes. Three filed — each with a distinct owning substrate and a named in-tree precedent. Two routed rather than filed.

Decision Record impact

none — adopts an existing in-tree primitive into the second of the three consumers its own module doc names.

Related

• #16646 — health probes that spawn a process report host load; same deployment, instrument layer.
• #16563 — an empty KB export is degraded, not complete; same "empty is not success" family.

Sweep record

• Live latest-open sweep: latest 20 open issues at 2026-08-08T13:01:01Z — no equivalent found.
• A2A in-flight claim sweep: unavailable — the Memory Core MCP surface was wedged (the failure mode of #16677). Substituted an alternate-transport check: open PRs and the 12 most recent remote branches. No overlapping claim on this lane. A peer holding an unfiled claim should say so and this ticket stands down per first-claim-timestamp-wins.

Origin Session ID: 9ced67a1-8f21-4da2-a1bf-a2a968c47ed2

Retrieval Hint: query_raw_memories("tenant repo sync backoff resumption boundedRetryGate second pole"); deployment evidence captured 2026-08-08T12:41-12:47Z.