Context
Found 2026-07-31 diagnosing a cloud deployment whose Knowledge Base held zero items. Plan-Authority: INDEPENDENT — runtime defect; sibling of the #16222 failure-handling family.
The Problem
TenantRepoSync suppresses a failing repo with a backoff that never yields a retry:
- All configured tenant repos sat
backoff-suppressed with lastIngestedRev: null — never ingested once. Root chain: KB_TENANT_REPO_SYNC_SYNC_FAILED ← KB_GITMIRROR_CLONE_FAILED (a credential-scope issue on the deployment side), 7-8 consecutive failures.
- After the failures stopped (~08:42Z one morning), no retry occurred for 25+ hours — and the suppression survived an orchestrator restart (state persists in the orchestrator state volume).
- Every sweep since reports
status: completed … N not-due — a lane that looks green while the KB it feeds stays permanently empty. The deployment's self-heal ledger recorded zero events for it.
The operational root (credential scope) was fixable in minutes once found; the defect is that the system hid it: infinite suppression + green sweep status + no detector.
The Architectural Reality
- Orchestrator tenant-repo sync lane; per-repo
consecutiveFailures / backoff-suppressed state in the deployment-state snapshot (schemaVersion 2).
- Mirror image of the #16222 canary (retries at probe frequency with NO backoff): here backoff exists but never expires into a retry. Both lack the same bounded-backoff-with-guaranteed-retry primitive.
- Self-heal detectors (ADR 0025 family) have no condition for "required lane suppressed > N hours with zero lifetime successes".
The Fix
- Cap the suppression: consecutive-failure backoff grows to a bounded maximum (e.g. ≤ 1-2h), after which a retry is guaranteed. A repo that has NEVER succeeded must keep probing at the capped cadence.
- Honest sweep status: a sweep whose every repo is suppressed-without-lifetime-success reports a distinct status (e.g.
starved), not completed, and carries per-repo lastErrorCode forward (currently only one repo retained its error code; the rest showed null — the diagnosis had one receipt where it needed four).
- Detector: self-heal condition for the starved shape (suppressed > N hours, zero successes), emitting a heal-ledger event.
Contract Ledger
| Target Surface |
Source of Authority |
Proposed Behavior |
Fallback |
Docs |
Evidence |
| suppression backoff |
this ticket |
bounded max + guaranteed retry |
cap configurable (ADR 0019 leaf) |
lane JSDoc |
unit: 8 failures → suppressed → clock past cap → retry attempted |
| sweep status |
this ticket |
starved distinct from completed; error codes retained per repo |
existing consumers keyed on completed unaffected on healthy paths |
snapshot schema note |
unit + snapshot fixture |
| self-heal detector |
ADR 0025 family |
starved-lane condition → ledger event |
detector off ⇒ behavior unchanged |
detector docs |
fixture: starved snapshot fires exactly one event |
Decision Record impact
aligned-with ADR 0019 (new cap leaf declarative), aligned-with ADR 0025 (detector added within the existing family).
Acceptance Criteria
Out of Scope
- The credential/scope fix on any specific deployment (operational).
- The canary and miniSummary siblings.
Related
- #16222 — failure-handling family (canary pole)
- The miniSummary pole (filed separately today)
Live latest-open sweep: latest 20 open checked at creation — no equivalent. A2A claim sweep: no competing claim in the herd window.
Origin Session ID: 449fb365-2bce-4451-bf99-35b0cffd015b
Retrieval Hint: tenant repo sync backoff suppressed forever no retry starved sweep completed zero ingest bounded backoff cap detector
Context
Found 2026-07-31 diagnosing a cloud deployment whose Knowledge Base held zero items. Plan-Authority: INDEPENDENT — runtime defect; sibling of the #16222 failure-handling family.
The Problem
TenantRepoSyncsuppresses a failing repo with a backoff that never yields a retry:backoff-suppressedwithlastIngestedRev: null— never ingested once. Root chain:KB_TENANT_REPO_SYNC_SYNC_FAILED←KB_GITMIRROR_CLONE_FAILED(a credential-scope issue on the deployment side), 7-8 consecutive failures.status: completed … N not-due— a lane that looks green while the KB it feeds stays permanently empty. The deployment's self-heal ledger recorded zero events for it.The operational root (credential scope) was fixable in minutes once found; the defect is that the system hid it: infinite suppression + green sweep status + no detector.
The Architectural Reality
consecutiveFailures/backoff-suppressedstate in the deployment-state snapshot (schemaVersion 2).The Fix
starved), notcompleted, and carries per-repolastErrorCodeforward (currently only one repo retained its error code; the rest showed null — the diagnosis had one receipt where it needed four).Contract Ledger
starveddistinct fromcompleted; error codes retained per repocompletedunaffected on healthy pathsDecision Record impact
aligned-with ADR 0019(new cap leaf declarative),aligned-with ADR 0025(detector added within the existing family).Acceptance Criteria
starved, retains every repo's last error codes, and nevercompleted.Out of Scope
Related
Live latest-open sweep: latest 20 open checked at creation — no equivalent. A2A claim sweep: no competing claim in the herd window.
Origin Session ID: 449fb365-2bce-4451-bf99-35b0cffd015b
Retrieval Hint:
tenant repo sync backoff suppressed forever no retry starved sweep completed zero ingest bounded backoff cap detector