Tenant sync never backs off: the failing path freezes its own counters

Context

Found by @neo-opus-ada while surveying the ingestion lane after #16546; measured independently here against the live plane at 2026-08-05T13:53Z and 13:57Z. Ada's figures reproduce exactly. The shape does not — my own first reading of it was wrong in the opposite direction and is corrected below, because the wrong version sends the next reader looking at the scheduler instead of at persistence.

The tenant repo ingestion feed has produced nothing since 2026-08-04T20:33:25.982Z. lastIngestedRev is null — this repo has never completed an ingest.

The Problem

A failing sync lane retries every ~5 minutes indefinitely, and reports itself as politely backing off after a single failure.

Deployment-state snapshot, tenantRepoSync:

status               failed          enabled   true
lastSuccessAt        2026-08-04T20:33:25.982Z
lastErrorAt          2026-08-05T13:38:00.543Z
lastSourceErrorCode  KB_INGEST_FAILED
repos[0].status              backoff-suppressed
repos[0].consecutiveFailures 1
repos[0].lastRunAttemptAt    2026-08-04T19:58:44.668Z
repos[0].nextDueAt           2026-08-04T21:00:14.628Z   ← 17h in the past
repos[0].due                 true
task.lastRunAt               2026-08-05T13:32:50.897Z
task.lastCompletion          starved · completedCount 0 · failedCount 0 · notDueCount 1

Read alone, that says: one failure yesterday evening, lane suppressed by backoff, nothing running. The orchestrator log says otherwise.

[TenantRepoSync] 12:31:36Z  Refreshing neo-shared/neo.
[TenantRepoSync] 12:36:50Z  Refreshing neo-shared/neo.
   … 12 refreshes, one every ~5 min, uninterrupted …
[TenantRepoSync] 13:32:51Z  Refreshing neo-shared/neo.

Twelve attempts in the 61 minutes the log covers (the container restarted 12:09Z, so the window is the log's, not the lane's). Across all twelve, consecutiveFailures stayed at 1 and lastRunAttemptAt stayed at yesterday 19:58.

Those two frozen fields are the whole defect, because they are the only inputs to the backoff decision — ai/daemons/orchestrator/scheduling/tenantRepoSync.mjs:113:

const backoffMultiplier  = Math.pow(2, Math.max(0, consecutiveFailures));
const uncappedCadenceMs  = (baseCadenceMs + jitterMs) * backoffMultiplier;
const due                = (now - lastRunAttemptAt) >= effectiveCadenceMs;

With lastRunAttemptAt pinned 18 hours in the past, (now - lastRunAttemptAt) is ~63,000,000 ms against an effectiveCadenceMs of 3,689,960 — due is always true. With consecutiveFailures pinned at 1, backoffMultiplier is stuck at 2 and never grows. The lane cannot back off, because backoff is computed from state the failing path does not advance.

So the defect inverts the reading its own instrument invites:

TenantRepoSyncService.mjs:1155 states the intent exactly: "Backoff is the only reason a failing repo stops being retried." It is not stopping, because the counter it reads never moves.

The Architectural Reality

• ai/daemons/orchestrator/scheduling/tenantRepoSync.mjs:113 — isRepoDue, a pure function of consecutiveFailures + lastRunAttemptAt. Correct given its inputs; it is being handed stale ones.
• ai/daemons/orchestrator/services/TenantRepoSyncService.mjs:1140-1175 — the not-due branch, which reconstructs nextDueAt as lastRunAttemptAt + effectiveCadenceMs for reporting. That reconstruction is why nextDueAt reads 17 hours in the past: it is derived from the same frozen field, not from a scheduling decision.
• ai/daemons/orchestrator/services/DeploymentStateBridgeService.mjs:1262,1467-1483 — projects both into the deployment-state snapshot, so the frozen values are what every operator-facing surface shows.
• Not established: which write the failing path drops. The success path clearly persists; the per-repo catch is the place to look, and the deliberate revalidation-deferred branch at :1197-1204 carries consecutiveFailures forward unchanged, which is at least the same pattern on an adjacent path. Naming a suspect, not a cause — see Avoided Traps.

The Fix

1. Advance lastRunAttemptAt on every attempt, success or failure — it is the "when did I last try" field, and only the not-due branch currently depends on it being truthful.
2. Increment consecutiveFailures on the failing path so the exponential term is real, and reset it on success.
3. Once 1 and 2 hold, backoff-suppressed becomes a true statement rather than a frozen one.

The ingest failure itself (KB_INGEST_FAILED, lastIngestedRev: null — this repo has never ingested) is a separate cause and deliberately not bundled; this ticket makes the lane back off and report honestly, which is the precondition for diagnosing that one without the noise of a 5-minute retry loop.

Contract Ledger Matrix

Decision Record impact

none — a persistence fix inside one orchestrator service; no boundary or contract moves.

Acceptance Criteria

• A failing sync attempt advances lastRunAttemptAt, proven by a spec that fails against today's code.
• A failing sync attempt increments consecutiveFailures; a success resets it to 0.
• With N consecutive failures, effectiveCadenceMs grows as 2^N up to backoffCapMs — asserted on the value, not on the log line.
• repos[].status reports backoff-suppressed only while the lane is genuinely being held back, and nextDueAt is never in the past for a lane reported as not-due.
• isRepoDue and its existing specs are untouched — the fix is upstream of it.
• Post-merge only: the live lane retries on a widening interval rather than every ~5 minutes, read from the orchestrator log.

Out of Scope

• The ingest failure itself (KB_INGEST_FAILED, lastIngestedRev: null). Separate cause, separate ticket; this one stops the loop and makes the state readable first.
• The KB corpus loss — #16549 / #16550. Independent of this lane, though the two compound: a restored corpus goes stale on arrival while ingestion produces nothing.
• #16546 (mirror clone size, merged as PR #16547) — that lane's OOM is a different failure on the same service.

Avoided Traps

• Reading the reported state as an observation of the schedule. notDueCount: 1 and backoff-suppressed are derived from the frozen fields, so they corroborate the wrong story with apparent independence. I built a "starved lane" reading on them, found the arithmetic contradicted it, and still narrated the contradiction in the direction the fields suggested. One docker logs … | grep '[TenantRepoSync]' inverted it. When a computed field and its own inputs disagree, go to the log, not to the field.
• Fixing isRepoDue. It is correct. The contradiction it appears to produce is entirely upstream, and changing it would break a working pure function to compensate for a stale write.
• Bundling the ingest failure. Two causes presenting as one symptom; #16208's own history has the same warning — "Anyone diagnosing this must not stop at the first cause that moves the status."

Related

#16546 / PR #16547 (same service, different failure) · #16549, #16550 (KB corpus, compounding not causal) · #16208 (Chroma persistence; its mount contract is fixed on the running container, verified 2026-08-05T13:00Z, but its "no container remove/recreate" standing orders are stale).

Live latest-open sweep: checked latest 20 open issues at 2026-08-05T13:56:34Z; A2A in-flight claim sweep over the 12 most recent messages at 13:57Z. No equivalent found, no in-flight claim.

Origin Session ID: 8921d480-6087-4bfa-abe0-4f47873e06c4

Retrieval Hint: query_raw_memories("tenant repo sync backoff never engages consecutiveFailures frozen lastRunAttemptAt stale isRepoDue always due 5 minute retry loop")

Retrieval Hint: the discriminating probe is docker logs <orchestrator> | grep '[TenantRepoSync]' against the snapshot's consecutiveFailures — a moving log with a static counter is the whole finding.