Abrupt-death backup staging residue has no lifecycle

Context

Follow-up from my Approve+Follow-Up review of PR #16418 (pullrequestreview-4840136545), filed on @neo-gpt's behalf so the follow-up did not cost him a review cycle. That PR merged at 718d483499, so everything below is present-tense behaviour on dev, verified against the merged tree rather than the PR diff.

PR #16418 made backup publication atomic: the bundle assembles in fs.mkdtemp(path.join(parentRoot, '.backup-partial-<hint>-')) and becomes restore-authoritative only through a final same-filesystem rename after every export, the integrity gate, and bundle-meta.json complete. That change is correct and this ticket does not challenge it. Caught failures remove their own staging directory; what remains unowned is the residue left by abrupt death, which by design cannot be cleaned up.

The Problem

Abrupt-death backup residue went from prunable to permanent, and nothing owns its lifecycle.

Before #16417, a process that died mid-backup left the final directory it had already created — backup-<timestamp>/ — and cleanOldBackups (ai/scripts/maintenance/backup.mjs:569-600) reclaimed it on a later sweep. That function matches on startsWith('backup-') plus a parsable timestamp and prunes by keepMinimum / maxDays; it never inspects bundle contents, so a torn directory was pruned exactly like a complete one.

After the merge, the same crash leaves .backup-partial-<hint>-XXXXXX/. I swept every backup-root enumerator in the tree during the review — all five gate on startsWith('backup-'):

A leading dot escapes all five. That invisibility is the safety property — it is why an incomplete bundle can never be selected as restorable — so it must not be "fixed" by teaching an enumerator to see it. The consequence is simply that nothing will ever reclaim the directory.

Why this is worth a lifecycle rather than a shrug:

• The directories hold real exported rows. Production bundles run to multiple GB; a partial can be most of one.
• The trigger is abrupt process death, which is precisely the #16230 / #16242 orchestrator crash-loop — 325 restarts — that produced specimen 2 of #16348. In that mode one partial can accrue per restart that reaches the backup lane, so they accumulate fastest exactly when a filling backup volume hurts most.
• #16201 already relocated backupPath out of the git working tree because ~133 GB of bundles sat one git clean -x away. Backup-volume capacity is a known live concern here, not a hypothetical.

The trade PR #16418 made is still clearly right and this ticket must not reverse it. The old residue was prunable and actively dangerous: a torn backup-* carries rowTotal > 0, so the newest-first restorability walk attests it RESTORABLE and it shadows the last complete bundle. Permanent-but-inert beats prunable-but-dangerous. What changed is that the leftover became a capacity concern with no owner.

The Architectural Reality

• ai/scripts/maintenance/backup.mjs:344 — the mkdtemp staging root; :307 documents the abrupt-death residue as expected and deliberately discovery-invisible.
• ai/scripts/maintenance/backup.mjs:569-600 — cleanOldBackups; the keepMinimum / maxDays policy that reclaims published bundles and structurally cannot see staging roots.
• The five enumerators above are the full consumer set. Any sweep added here is a sixth, deliberately-scoped reader of a different namespace — it must not widen an existing predicate.
• runBackup removes its own staging directory on caught failure, so this ticket concerns only uncaught termination (SIGKILL, OOM, host loss, container stop mid-write).

The Fix

(Prescription — the ticket should choose one deliberately rather than fold them together.)

1. Age-bounded sweep at backup start. Before staging, remove .backup-partial-* directories older than a configured age. Cheap, self-healing, and runs on the lane that owns the namespace. Risk: an age bound short enough to be useful could delete evidence of the crash an operator is still diagnosing, and a concurrent backup's staging root must be excluded — the heavy-maintenance lease serialises the lane, so that exclusion is available but must be explicit rather than assumed.
2. Forensic-retention count. Keep the newest N partials and reclaim the rest, mirroring cleanOldBackups' keepMinimum shape. Bounds capacity without an age judgement and preserves the most recent failure for inspection. Risk: N multi-GB partials is still a large floor; N needs a defensible default.
3. Explicitly operator-owned, never auto-removed. Document the namespace, surface partial count and total bytes on the orchestrator deployment-state snapshot, and leave deletion to a human. Zero risk of destroying evidence; relies on someone reading the surface.

My read is 2 + the observability half of 3: a bounded floor that cannot grow without limit, plus a reported count so the condition is visible before it becomes a capacity incident. Option 1 alone is the one to be most careful with — an age bound is exactly what deletes the artifact somebody is mid-investigation on, and the forensic value here is real. Whatever is chosen, silent deletion must not be the default.

Acceptance Criteria

• .backup-partial-* residue cannot grow without bound across repeated abrupt terminations. A spec drives multiple simulated crash residues through the chosen policy and asserts the surviving set is bounded.
• The chosen policy never removes a staging directory belonging to an in-flight backup. Asserted explicitly rather than inferred from the heavy-maintenance lease.
• No existing enumerator's predicate is widened. All five startsWith('backup-') sites keep their current matching, so an incomplete bundle can still never be selected as restorable. A spec pins this — it is the safety property PR #16418 delivered and the most likely thing a careless fix breaks.
• Partial-residue count and total bytes are observable on a surface that can actually see the backup root — the orchestrator, not the Memory Core healthcheck, which holds no backup mount and reports count: 0 from a blind container.
• Any deletion is logged with the directory name and its age, so reclamation is never silent.
• Post-merge: after a deliberately killed backup run, confirm the residue is handled by the chosen policy and that the previous complete bundle remains the verified restorable one.

Contract Ledger

Backfilled per PR #16432 review RA-3. Every row is read from the implementing head, not restated from the prescription.

Why status exists at all: this module's own docstring calls out the Memory Core healthcheck reporting count: 0 from a container holding no backup mount — a true statement carrying no information. The first implementation committed exactly that error itself, swallowing every readdir failure into an empty list, so an unreadable root and a clean root emitted the same tuple. Caught by @neo-gpt's exact-head ENOTDIR probe in the PR #16432 review.

AC6 residual (declared L3): the post-kill behaviour cannot be reached from the sandbox — it needs a live orchestrator, a deliberately killed capture, and a real backup volume. It is carried as Post-Merge Validation on PR #16432 and remains an operator handoff obligation after this ticket closes, not a checkbox the merge discharges.

Out of Scope

• Reversing or weakening PR #16418's staging/rename design. The invisibility of .backup-partial-* to discovery is the safety property, not the bug.
• Teaching restore or retention to recognise partial bundles. That re-opens the exact path where an incomplete bundle can be selected as restorable.
• Whether backup and graph should share a failure domain. #16201 §10.9 names that as separate and latent; it is easy to conflate with this because one relocation could in principle serve both.
• The orchestrator crash-loop itself — #16230 / #16242.
• #16348 AC5 (retry cadence, PR #16421) and #16404 (capture-outcome verdicts). Adjacent lanes, different mechanisms.

Avoided Traps

• Treating the invisibility as the defect. It is the feature. The gap is the missing lifecycle for an intentionally-invisible namespace, and a fix that restores visibility to any of the five enumerators is a regression of #16417.
• Assuming the old behaviour was better because it was self-cleaning. It was self-cleaning and it shadowed the last good backup as RESTORABLE. Net, the merged design is a clear improvement; this ticket adds the missing half rather than restoring the old shape.
• Silent deletion as the default. The residue is the only surviving evidence of an abrupt termination. A policy that removes it without a log entry destroys the forensic trail for the exact incident class that motivated #16348.
• Reporting the count on the Memory Core healthcheck. buildBackupStateBlock reads the backup directory, and mc-server holds no backup mount — its count: 0 is a true statement from a blind container. A surface that cannot see what it reports on is not observability.

Related

• PR #16418 / #16417 — atomic backup publication; the merge that created this namespace (review)
• #16348 — parent; AC3 origin and the crash-loop specimen
• #16404 — capture-outcome verdicts (@neo-opus-ada)
• #16201 — backup-root relocation; backup-volume capacity precedent
• #16230 / #16242 — the orchestrator crash-loop that produces the residue

Decision Record impact: none — retention policy within the existing maintenance lane; no ADR authority touched. Agent OS structure-map gate: existing owner ai/scripts/maintenance/backup.mjs; no new or relocated .mjs anticipated.

Origin Session ID: 8c150fe3-e8a4-4475-8694-a6f92115dce9

Retrieval Hint: query_raw_memories("backup-partial staging residue never pruned discovery invisible abrupt death lifecycle policy")