LearnNewsExamplesServices
Frontmatter
id16224
titleTenantRepoSync suppression never retries: green sweeps over a permanently starved Knowledge Base
stateClosed
labels
bugai
assigneesneo-kimi-iris
createdAtJul 31, 2026, 12:11 PM
updatedAtAug 2, 2026, 1:18 AM
githubUrlhttps://github.com/neomjs/neo/issues/16224
authorneo-opus-vega
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[x] 16243 Joined boundedRetryGate waiters share nested delivery values
blocking[]
closedAtAug 2, 2026, 1:18 AM

TenantRepoSync suppression never retries: green sweeps over a permanently starved Knowledge Base

neo-opus-vega
neo-opus-vega commented on Jul 31, 2026, 12:11 PM

Context

Found 2026-07-31 diagnosing a cloud deployment whose Knowledge Base held zero items. Plan-Authority: INDEPENDENT — runtime defect; sibling of the #16222 failure-handling family.

The Problem

TenantRepoSync suppresses a failing repo with a backoff that never yields a retry:

  • All configured tenant repos sat backoff-suppressed with lastIngestedRev: null — never ingested once. Root chain: KB_TENANT_REPO_SYNC_SYNC_FAILEDKB_GITMIRROR_CLONE_FAILED (a credential-scope issue on the deployment side), 7-8 consecutive failures.
  • After the failures stopped (~08:42Z one morning), no retry occurred for 25+ hours — and the suppression survived an orchestrator restart (state persists in the orchestrator state volume).
  • Every sweep since reports status: completed … N not-due — a lane that looks green while the KB it feeds stays permanently empty. The deployment's self-heal ledger recorded zero events for it.

The operational root (credential scope) was fixable in minutes once found; the defect is that the system hid it: infinite suppression + green sweep status + no detector.

The Architectural Reality

  • Orchestrator tenant-repo sync lane; per-repo consecutiveFailures / backoff-suppressed state in the deployment-state snapshot (schemaVersion 2).
  • Mirror image of the #16222 canary (retries at probe frequency with NO backoff): here backoff exists but never expires into a retry. Both lack the same bounded-backoff-with-guaranteed-retry primitive.
  • Self-heal detectors (ADR 0025 family) have no condition for "required lane suppressed > N hours with zero lifetime successes".

The Fix

  1. Cap the suppression: consecutive-failure backoff grows to a bounded maximum (e.g. ≤ 1-2h), after which a retry is guaranteed. A repo that has NEVER succeeded must keep probing at the capped cadence.
  2. Honest sweep status: a sweep whose every repo is suppressed-without-lifetime-success reports a distinct status (e.g. starved), not completed, and carries per-repo lastErrorCode forward (currently only one repo retained its error code; the rest showed null — the diagnosis had one receipt where it needed four).
  3. Detector: self-heal condition for the starved shape (suppressed > N hours, zero successes), emitting a heal-ledger event.

Contract Ledger

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
suppression backoff this ticket bounded max + guaranteed retry cap configurable (ADR 0019 leaf) lane JSDoc unit: 8 failures → suppressed → clock past cap → retry attempted
sweep status this ticket starved distinct from completed; error codes retained per repo existing consumers keyed on completed unaffected on healthy paths snapshot schema note unit + snapshot fixture
self-heal detector ADR 0025 family starved-lane condition → ledger event detector off ⇒ behavior unchanged detector docs fixture: starved snapshot fires exactly one event

Decision Record impact

aligned-with ADR 0019 (new cap leaf declarative), aligned-with ADR 0025 (detector added within the existing family).

Acceptance Criteria

  • A repo with consecutive failures retries after the capped backoff, verified by a unit clock test; suppression cannot exceed the cap even across restarts.
  • A sweep over only-suppressed, never-succeeded repos reports starved, retains every repo's last error codes, and never completed.
  • The self-heal detector fires exactly one ledger event for the starved shape and none for healthy backoff.
  • ADR 0019 §3 self-audit for the new leaf recorded in the PR.

Out of Scope

  • The credential/scope fix on any specific deployment (operational).
  • The canary and miniSummary siblings.

Related

  • #16222 — failure-handling family (canary pole)
  • The miniSummary pole (filed separately today)

Live latest-open sweep: latest 20 open checked at creation — no equivalent. A2A claim sweep: no competing claim in the herd window.

Origin Session ID: 449fb365-2bce-4451-bf99-35b0cffd015b

Retrieval Hint: tenant repo sync backoff suppressed forever no retry starved sweep completed zero ingest bounded backoff cap detector

tobiu referenced in commit d8d8e66 - "feat(ai): bound the tenant-repo-sync backoff and report starved lanes (#16224) (#16307) on Aug 2, 2026, 1:18 AM
tobiu closed this issue on Aug 2, 2026, 1:18 AM