Backup has been failing on a cloud plane with retries exhausted and no escalation, while off-host durability reads unmet

Context

External-plane maintenance section, 2026-08-13 19:59Z:

"lastBackup": {"backup": {"durationMs": 5028, "error":
  "Error: DATABASE_EXPORT_ERROR: Knowledge base collection resolve failed after 5/5 attempts and 5000ms retry delay:
   Failed to connect to chromadb. Make sure your server is running and try again. ..."}}

"retry": {"retriesRemaining": 0, "phase": "exhausted",
          "windowEndsAtMs": 1786124554430, "streakStartedAtMs": 1786120954430}

"durability": {"cloudDeployment": true, "offHostBackupRequired": true,
               "offHostSyncConfigured": false, "posture": "unmet",
               "reason": "This cloud deployment requires an off-host copy of the backup bundle,
                          but no off-host sync command is configured.
                          The bundle and the data it protects share one failure domain."}

The retry window closed at epoch 1786124554430 — 2026-08-09. The backup lane has been in exhausted with zero retries remaining for roughly four days, on a plane that is simultaneously failing to ingest.

Note the irony in the error itself: Chroma was reachable and healthy at snapshot time (chroma container healthy, uptime 9h, KB connected to it). The backup failed against a transient connect problem, burned all five retries inside five seconds (durationMs: 5028, 5/5 attempts, 5000ms retry delay), exhausted its window, and then simply stopped.

The Problem

Three compounding gaps.

1. The retry budget is spent in five seconds. Five attempts at a 5,000 ms total makes the retry policy useless against anything except an instantaneous blip. A dependency restarting — which on this plane happens routinely — outlasts the entire budget.

2. exhausted is terminal and silent. Once the window closes, the lane stops and stays stopped. Nothing re-arms it, and nothing raises the fact that a data-safety task has not succeeded in days. It is visible only to someone who reads this section of the snapshot deliberately.

3. Durability posture unmet is reported and not enforced. The system correctly states that a cloud deployment requires an off-host copy and that none is configured — "the bundle and the data it protects share one failure domain" — and then proceeds indefinitely. A correct statement nobody is obliged to act on is not a control.

Together: a plane that has had no successful backup for days, no off-host copy, and no signal proportional to that risk. The ingestion incident consumed all attention because it was loud; this was silent, and it is the one with irreversible consequences.

Architectural Reality

• Backup is priority-0 in the scheduler (#16561) — the lane's importance is already established; what is missing is that failing it terminally produces no proportional signal.
• The health surface reports status for services and lanes, but a terminal backup exhaustion does not degrade any surface an operator or agent routinely reads.
• offHostSyncConfigured: false is a deployment-configuration fact the runtime can detect at boot, not only at backup time.

The Fix (shape)

1. Size the retry budget against realistic dependency-restart duration, not milliseconds; a dependency that restarts must not exhaust the window.
2. exhausted must re-arm on a bounded schedule rather than terminate — a data-safety lane should never enter a state it cannot leave without intervention.
3. Terminal backup failure, and durability.posture: "unmet", must degrade a surface that is actually read (healthcheck details at minimum), so the condition cannot persist unnoticed for days.
4. Escalate age explicitly: "last successful backup was N days ago" belongs in the health output, because the absence of a recent success is the fact that matters — not the presence of a recent failure.

Acceptance Criteria

• Backup retry survives a dependency restart of realistic duration (fixture restarts the store mid-backup).
• A lane in exhausted re-arms on a bounded schedule without operator action.
• Terminal backup failure degrades the health surface with a distinguishing reason.
• Time since last successful backup is reported, and crossing a threshold degrades health independently of whether a failure occurred recently.
• durability.posture: "unmet" on a cloud deployment surfaces at boot, not only inside a maintenance section.

Out of Scope

• Configuring off-host sync for any particular deployment (that is deployment configuration).
• The ingestion incident (#17062 · #17063 · #17064 · #17065 · #17067).

Avoided Traps

• "It is already reported." It is reported where nobody looks. Four days of no backups on a plane under active incident is the evidence that reporting was insufficient.
• "Chroma was down, so the failure is correct." The failure is correct; giving up permanently after five seconds of it is not.
• "Off-host sync is the operator's job." Agreed — and the system stating a requirement it never enforces or escalates leaves the risk unowned.

Related

• #16561 — backup is priority-0 and was starved; same lane, different failure mode
• #17062 · #17063 · #17064 · #17065 · #17067 — same plane, same window