LearnNewsExamplesServices
Frontmatter
id14030
titleBackup reliability: alert on failure, verify restorability, beat retention
stateClosed
labels
enhancementaiarchitecture
assigneesneo-opus-vega
createdAtJun 26, 2026, 1:20 AM
updatedAtJun 27, 2026, 8:21 PM
githubUrlhttps://github.com/neomjs/neo/issues/14030
authorneo-opus-vega
commentsCount6
parentIssue14039
subIssues
14048 Backup integrity: flag empty-parity (gutted-store) exports instead of a silent pass
14053 Backup restorability pre-flight check (verifyLatestBackupRestorable) — #14030 AC2 check-half
14055 #14030 AC1 producer-core: supervised-task failure/overdue diagnosis helpers
14058 Escalate backup task failure diagnoses
14059 Detection-latency-under-retention SLA check (evaluateDetectionRetentionSla) — #14030 AC3 verdict-half
14082 Backup verification crashes on >512MB JSONL files (ERR_STRING_TOO_LONG)
14086 Backup verification skips KB parity: KB export count is string-only
subIssuesCompleted7
subIssuesTotal7
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[x] 14046 v13.1 release-gate: end-to-end corruption-injection → detect → diagnose → recover proof
closedAtJun 27, 2026, 8:21 PM
milestonev13.1

Backup reliability: alert on failure, verify restorability, beat retention

Closed v13.1.0/archive-v13-1-0-chunk-6 enhancementaiarchitecture
neo-opus-vega
neo-opus-vega commented on Jun 26, 2026, 1:20 AM

Context

In the #13999 incident the canonical backup started failing on 2026-06-20 (the first fail-loud partial export after the corruption) — but that failure sat unalerted for ~5 days until it was noticed on 06-25. The backup job already fail-louds (#13584); the gap is that its failure raised no alarm. Three related backup/recovery-substrate trust gaps surface together.

The Problem

The backups are the organism's recovery source of last resort, yet: (1) a failing backup job alerts no one — a multi-day detection lag while the situation worsens; (2) backups are assumed restorable but never verified — a corrupt/incomplete backup means no recovery; (3) if corruption isn't detected within the backup-retention window, the last good backup ages out and recovery becomes impossible.

The Fix

  1. Alert on backup-job failure — the daily backup already fail-louds; wire its failure (and a missed/overdue run) to the escalation path. Cheapest, highest-ROI detect signal (turns a 5-day lag into immediate).
  2. Verify restorability — a periodic check that the latest backup is actually restorable (e.g. a sampled logical import into a throwaday store), not just present.
  3. Detection-latency-under-retention SLA — make the data-integrity detect cadence (#14026) provably faster than backup retention, so a last-good backup never ages out before corruption is caught. Document/enforce the relationship.

Acceptance Criteria

  • A failed or overdue canonical backup raises an alert/escalation (not just a log line).
  • A periodic restorability check validates the latest backup is importable, flagging if not.
  • The detect-cadence (#14026) vs backup-retention relationship is documented and guarded (detection latency < retention).

Out of Scope

  • The data-recovery actuator itself (backup-merge / from-scratch — Ideation-bound).
  • The live coverage detect-signal (#14026) and the historical timeline (#14024) — this is about the backup substrate's own trustworthiness + alerting.

Related

  • #13999 — the incident where the backup-failure alert was missing (5-day lag).
  • #14026 — detect-signal (the retention-SLA ties to its cadence).
  • #14024 — backup-timeline diagnostic (consumes the same backups).
  • #13584 — fail-loud partial export (the signal that exists but isn't alerted).

Origin Session ID: 58acd7a6-778e-4fe6-b85a-4569802ac57b

Handoff Retrieval Hints: query_raw_memories("backup failure alert restorability verification detection latency retention SLA"); anchors: ai/scripts/maintenance/backup.mjs, .neo-ai-data/backups/.

Authored-by: @neo-opus-vega (Vega, Claude Opus 4.8)