Tenant-repo backoff reaches a 2h cap with no operator path to clear it, so a fixed root cause still cannot resume ingestion

Context

External-plane tenant-sync state, 2026-08-13 19:59Z. Four configured repos, one active, three suppressed:

{"identityHash":"3b6d6176e65c","status":"backoff-suppressed","lastIngestedRev":null,
 "lastSourceErrorCode":"KB_VECTOR_EMBED_FAILED","consecutiveFailures":40}
{"identityHash":"e1daf1ca9706","status":"backoff-suppressed","lastIngestedRev":null,
 "lastSourceErrorCode":"KB_VECTOR_EMBED_CONNECTION_REFUSED","consecutiveFailures":56}
{"identityHash":"0485923578f7","status":"backoff-suppressed","lastIngestedRev":null,
 "lastSourceErrorCode":"KB_VECTOR_EMBED_CONNECTION_REFUSED","consecutiveFailures":62}

scheduler.backoffCapMs: 7200000 — two hours. Orchestrator log:

[TenantRepoSync] apps-global suppressed by backoff
  (next ~2026-08-13T17:36:27.955Z, consecutiveFailures=40, backoffX=1099511627776, ...)

The Problem

The backoff is correct as congestion control: repeated failures should not hammer a broken dependency. What is missing is the exit.

Every one of these failures shares a single external cause. When that cause is repaired, the three repos remain suppressed for up to two hours each, and there is no supported way to tell the scheduler that the world changed. The operator's only levers are to wait out the cap or restart the orchestrator — and restarting to clear a counter is exactly the reflex this incident class already suffers from.

This turns every fix into a fix-plus-a-two-hour-wait, and it makes verifying a fix expensive: you cannot distinguish "the repair did not work" from "the lane has not been allowed to try yet". During a customer-facing outage that ambiguity is costly, because the natural response to apparent non-recovery is another intervention.

There is a second, sharper edge. consecutiveFailures counts attempts that failed, not distinct causes. Sixty-two failures against one continuously-unavailable dependency is one fault observed sixty-two times, but the counter escalates as though the repo itself were progressively more suspect. A lane whose dependency was down for a day is punished for the duration of the outage after the outage ends.

Note the interaction with #17062: failures caused by queue preemption also increment this counter, so ingestion that never had a real chance still drives the lane toward the cap.

The Fix (shape)

1. An explicit, supported reset. An operator (or a healed-dependency signal) must be able to clear the backoff for one repo or all repos without restarting the process. Read-only diagnosis is already exposed; this is the missing write.
2. Cause-aware counting. A failure attributable to a shared unavailable dependency should not escalate the per-repo counter the way a repo-specific failure does — otherwise the counter measures outage duration rather than repo health.
3. Recovery-triggered retry. The orchestrator already detects embedding recovery ("Embedding recovery not yet observed (EMBEDDING_PROVIDER_ERROR; provider still failing)" is logged on every sweep). Observing the positive transition should release lanes suppressed by that same cause.
4. Surface nextAttemptAt per repo in the deployment-state snapshot so an operator can see the wait rather than infer it from logs.

Acceptance Criteria

• A supported operation clears tenant-sync backoff for a named repo and for all repos, without a process restart, and is reflected in the next sweep.
• Failures classified as shared-dependency-unavailable do not escalate consecutiveFailures on the same schedule as repo-specific failures.
• When embedding recovery is observed, lanes suppressed solely by embedding failure become eligible on the next sweep rather than waiting out the cap.
• nextAttemptAt (or equivalent) appears per repo in the snapshot.
• Negative control: a repo failing for its own reasons (bad credentials, missing ref) still backs off normally and is not released by a dependency-recovery signal.

Out of Scope

• The embedding failures themselves (#17062 · #17063).
• Changing the backoff curve or cap value — the defect is the absent exit, not the shape of the climb.

Avoided Traps

• "Restart the orchestrator." That is the current workaround and it is the reflex this incident family already over-uses; it also discards unrelated in-flight state.
• "Lower the cap." Trades one failure mode for another — a genuinely broken repo then retries aggressively forever.
• "The operator can wait." Two hours per repo, after a multi-week outage, with no way to confirm the fix landed, is not an acceptable verification loop.

Related

• #17062 — preemption-caused failures that feed this counter
• #17063 · #17064 · #17065 — same incident