Tenant-repo backoff reaches a 2h cap with no operator path to clear it, so a fixed root cause still cannot resume ingestion

Context

External-plane tenant-sync state, 2026-08-13 19:59Z. Four configured repos, one active, three suppressed:

{"identityHash":"3b6d6176e65c","status":"backoff-suppressed","lastIngestedRev":null,
 "lastSourceErrorCode":"KB_VECTOR_EMBED_FAILED","consecutiveFailures":40}
{"identityHash":"e1daf1ca9706","status":"backoff-suppressed","lastIngestedRev":null,
 "lastSourceErrorCode":"KB_VECTOR_EMBED_CONNECTION_REFUSED","consecutiveFailures":56}
{"identityHash":"0485923578f7","status":"backoff-suppressed","lastIngestedRev":null,
 "lastSourceErrorCode":"KB_VECTOR_EMBED_CONNECTION_REFUSED","consecutiveFailures":62}

scheduler.backoffCapMs: 7200000 — two hours. Orchestrator log:

[TenantRepoSync] apps-global suppressed by backoff
  (next ~2026-08-13T17:36:27.955Z, consecutiveFailures=40, backoffX=1099511627776, ...)

The Problem

The backoff is correct as congestion control: repeated failures should not hammer a broken dependency. What is missing is the exit.

Every one of these failures shares a single external cause. When that cause is repaired, the three repos remain suppressed for up to two hours each, and there is no supported way to tell the scheduler that the world changed. The operator's only levers are to wait out the cap or restart the orchestrator — and restarting to clear a counter is exactly the reflex this incident class already suffers from.

This turns every fix into a fix-plus-a-two-hour-wait, and it makes verifying a fix expensive: you cannot distinguish "the repair did not work" from "the lane has not been allowed to try yet". During a customer-facing outage that ambiguity is costly, because the natural response to apparent non-recovery is another intervention.

There is a second, sharper edge. consecutiveFailures counts attempts that failed, not distinct causes. Sixty-two failures against one continuously-unavailable dependency is one fault observed sixty-two times, but the counter escalates as though the repo itself were progressively more suspect. A lane whose dependency was down for a day is punished for the duration of the outage after the outage ends.

Note the interaction with #17062: failures caused by queue preemption also increment this counter, so ingestion that never had a real chance still drives the lane toward the cap.

The Fix (shape)

1. An explicit, supported reset. An operator (or a healed-dependency signal) must be able to clear the backoff for one repo or all repos without restarting the process. Read-only diagnosis is already exposed; this is the missing write.
2. Cause-aware counting. A failure attributable to a shared unavailable dependency should not escalate the per-repo counter the way a repo-specific failure does — otherwise the counter measures outage duration rather than repo health.
3. Recovery-triggered retry. The orchestrator already detects embedding recovery ("Embedding recovery not yet observed (EMBEDDING_PROVIDER_ERROR; provider still failing)" is logged on every sweep). Observing the positive transition should release lanes suppressed by that same cause.
4. Surface nextAttemptAt per repo in the deployment-state snapshot so an operator can see the wait rather than infer it from logs.

Acceptance Criteria

(Narrowed 2026-08-14, second correction: source traces by @neo-opus-grace with @neo-gpt's challenge showed ACs 2–5 already shipped on dev via #16712 / #16713 while this ticket kept its incident-era framing. The struck originals and coordinates are preserved in the correction comments. This ticket now delivers AC-1 alone.)

• A supported operation clears tenant-sync backoff for a named repo and for all repos, without a process restart, and is reflected in the next sweep; an unknown repo identifier is rejected with a named error, never a silent no-op; consumption is recorded in the deployment-state snapshot.

Struck ACs — dispositions (2026-08-14):

• Cause-aware counting (was AC-2): already met — TenantRepoSyncService.mjs:2224 holds the streak on embedding-deferred sweeps (streak held at N), :2208 arms embeddingRecovery on embedding-class causes. Also independently recorded in #17062's struck-AC dispositions.
• Recovery-triggered release (was AC-3): already met — the armed embeddingRecovery releases suppressed lanes on the observed positive transition (#16712 / #16713).
• Per-repo next-eligibility in the snapshot (was AC-4): already met — do NOT add an alias. The snapshot already exposes nextDueAt (DeploymentStateBridgeService.mjs:2376, computed at :2353, recoveryBypass-aware). The ledger's original nextAttemptAt row would have shipped a second name for one concept — the exact duplicate-alias failure the row's own rule exists to prevent (tenantRepoSync.mjs:129's spelling belongs to the canary probe, a different concept).
• Negative control (was AC-5): already met — non-embedding failures never acquire an embeddingRecovery, so the dependency signal cannot release them.

Contract Ledger

(Rewritten 2026-08-14 with the narrowing — the superseded five-row version is preserved in the correction comment.)

Pinned per the intake: the reset mutates state that already exists (isRepoDue derives effectiveCadence on read, so a cleared streak binds on the next evaluation) — no new suppression store, no second source of truth about when a repo may run.

Priority disposition (2026-08-14)

PARKED under epic deploy pressure — valid, not urgent. The incident-era urgency ("every fix becomes a fix-plus-a-two-hour-wait") is no longer the state of dev: embedding-class suppression auto-releases on a healthy canary, and the observed plane's deployment lowered its cap independently. What remains is an operator-convenience lever for repair classes no canary observes (credentials, refs, config). It does not compete with in-flight epic children for a seat. Pick up freely once the epic's outcome verification is underway.

Out of Scope

• The embedding failures themselves (#17062 · #17063).
• Changing the backoff curve or cap value — the defect is the absent exit, not the shape of the climb.

Avoided Traps

• "Restart the orchestrator." That is the current workaround and it is the reflex this incident family already over-uses; it also discards unrelated in-flight state.
• "Lower the cap." Trades one failure mode for another — a genuinely broken repo then retries aggressively forever.
• "The operator can wait." Two hours per repo, after a multi-week outage, with no way to confirm the fix landed, is not an acceptable verification loop.

Related

• #17062 — preemption-caused failures that feed this counter
• #17063 · #17064 · #17065 — same incident