Backup has been failing on a cloud plane with retries exhausted and no escalation, while off-host durability reads unmet

Context

[L2-deferred — operator handoff needed] The 15-second inner retry horizon is hermetically verified but still lacks an external-plane Chroma restart witness. #16706 owns that post-merge validation; this ticket must not be read as proving wall-clock restart coverage.

External-plane maintenance section, 2026-08-13 19:59Z:

"lastBackup": {"backup": {"durationMs": 5028, "error":
  "Error: DATABASE_EXPORT_ERROR: Knowledge base collection resolve failed after 5/5 attempts and 5000ms retry delay:
   Failed to connect to chromadb. Make sure your server is running and try again. ..."}}

"retry": {"retriesRemaining": 0, "phase": "exhausted",
          "windowEndsAtMs": 1786124554430, "streakStartedAtMs": 1786120954430}

"durability": {"cloudDeployment": true, "offHostBackupRequired": true,
               "offHostSyncConfigured": false, "posture": "unmet",
               "reason": "This cloud deployment requires an off-host copy of the backup bundle,
                          but no off-host sync command is configured.
                          The bundle and the data it protects share one failure domain."}

The retry window closed at epoch 1786124554430 — 2026-08-09. The backup lane has been in exhausted with zero retries remaining for roughly four days, on a plane that is simultaneously failing to ingest.

Note the irony in the error itself: Chroma was reachable and healthy at snapshot time (chroma container healthy, uptime 9h, KB connected to it). The backup failed against a transient connect problem, exhausted its window, and then simply stopped.

The Problem

Three compounding gaps.

1. The inner dependency-resolve retry cannot survive a dependency restart (corrected 2026-08-14 — see comment; the original framing conflated two retry layers). The 5/5 attempts and 5000ms retry delay in the error string is the Chroma client's inner collection-resolve retry (chromaClientPrimitives), not the lane's budget. The lane's own budget — backupRetryDelayMs = 15 min against backupRetryWindowMs = 1 h (ai/configBase.mjs, ~4 attempts) — did retry across its window, and the ledger's streakStartedAtMs → windowEndsAtMs span is exactly that hour. But each outer attempt fails within ~5 s because the inner resolve gives up long before a restarting dependency returns, so the outer window burns four fast failures and exhausts terminally. A dependency restarting — which on this plane happens routinely — outlasts the inner budget every time.

2. exhausted is terminal and silent. Once the window closes, the lane stops and stays stopped. Nothing re-arms it, and nothing raises the fact that a data-safety task has not succeeded in days. It is visible only to someone who reads this section of the snapshot deliberately.

3. Durability posture unmet is reported and not enforced. The system correctly states that a cloud deployment requires an off-host copy and that none is configured — "the bundle and the data it protects share one failure domain" — and then proceeds indefinitely. A correct statement nobody is obliged to act on is not a control.

Together: a plane that has had no successful backup for days, no off-host copy, and no signal proportional to that risk. The ingestion incident consumed all attention because it was loud; this was silent, and it is the one with irreversible consequences.

Architectural Reality

• Backup is priority-0 in the scheduler (#16561) — the lane's importance is already established; what is missing is that failing it terminally produces no proportional signal.
• The health surface reports status for services and lanes, but a terminal backup exhaustion does not degrade any surface an operator or agent routinely reads.
• offHostSyncConfigured: false is a deployment-configuration fact the runtime can detect at boot, not only at backup time.

The Fix (shape)

1. Size the inner dependency-resolve retry against realistic dependency-restart duration, not milliseconds; a dependency that restarts must not exhaust an outer attempt in seconds.
2. exhausted must re-arm on a bounded schedule rather than terminate — a data-safety lane should never enter a state it cannot leave without intervention.
3. Terminal backup failure, and durability.posture: "unmet", must degrade a surface that is actually read (healthcheck details at minimum), so the condition cannot persist unnoticed for days.
4. Escalate age explicitly: "last successful backup was N days ago" belongs in the health output, because the absence of a recent success is the fact that matters — not the presence of a recent failure.

Acceptance Criteria

• Backup retry survives a dependency restart of realistic duration (fixture restarts the store mid-backup).
• A lane in exhausted re-arms on a bounded schedule without operator action.
• Terminal backup failure degrades the health surface with a distinguishing reason.
• Time since last successful backup is reported, and crossing a threshold degrades health independently of whether a failure occurred recently.
• durability.posture: "unmet" on a cloud deployment surfaces at boot, not only inside a maintenance section.

Out of Scope

• Configuring off-host sync for any particular deployment (that is deployment configuration).
• The ingestion incident (#17062 · #17063 · #17064 · #17065 · #17067).

Avoided Traps

• "It is already reported." It is reported where nobody looks. Four days of no backups on a plane under active incident is the evidence that reporting was insufficient.
• "Chroma was down, so the failure is correct." The failure is correct; giving up permanently after five seconds of inner retry per attempt and one hour overall is not.
• "Off-host sync is the operator's job." Agreed — and the system stating a requirement it never enforces or escalates leaves the risk unowned.

Related

• #16561 — backup is priority-0 and was starved; same lane, different failure mode
• #17062 · #17063 · #17064 · #17065 · #17067 — same plane, same window