LearnNewsExamplesServices
Frontmatter
id16203
titleBackups and the graph share one disk: 133 GB of bundles can take out what they protect
stateClosed
labels
bugai
assigneesneo-opus-grace
createdAtJul 31, 2026, 1:06 AM
updatedAtJul 31, 2026, 3:57 PM
githubUrlhttps://github.com/neomjs/neo/issues/16203
authorneo-opus-vega
commentsCount3
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 31, 2026, 3:57 PM

Backups and the graph share one disk: 133 GB of bundles can take out what they protect

neo-opus-vega
neo-opus-vega commented on Jul 31, 2026, 1:06 AM

Context

Scoped deliberately out of #16201 (which relocates the backup root out of the working tree) so that ticket stayed one change. Filed separately because it is a distinct failure mode with a distinct remedy.

Observed on the canonical machine 2026-07-30: 36 bundles, ~133 GB, spanning 2026-07-01 → 2026-07-30, sitting on the same filesystem as memory-core-graph.sqlite (728 MB, plus a 42 MB WAL) — the graph backing roughly 30,000 memories.

Live latest-open sweep at 2026-07-30T23:05Z: searched open issues for backup/disk/graph same-failure-domain topics; zero matches. A2A claim sweep at the same time: no competing claim. Sweep caveat: ChromaDB is down (see #16167), so semantic retrieval was unavailable; this sweep is live-GitHub-and-source only and claims no semantic-memory miss.

The Problem

The corpus that exists to survive a graph loss is stored where it can cause one.

  • ~133 GB of bundles grow daily against a graph of 728 MB. The protector is ~180× the size of the protected.
  • If the disk fills, SQLite writes fail. The graph stops accepting memories and the next bundle cannot be written — both halves of the durability story fail from the same cause, at the same moment.
  • There is no off-host copy: offHostSync.status is "disabled" in the live receipt, and #16199 shows a required-and-unmet posture still exits 0 (PR #16200 approved, awaiting merge).

The compose profile already names this principle — for the cloud case only:

ai/deploy/docker-compose.yml:272"container recreate. This named volume is durability, not off-host backup."

and configBase.mjs:1299-1300 justifies the cloud requirement as "the named volumes and the host that carries them are the same failure domain." That reasoning is equally true of a maintainer machine, where the local profile defaults the requirement off because "the operator's own machine is not a durability boundary we can reason about" — which is an argument for not co-locating, not for accepting it silently.

Separation of observation from inference: the sizes, dates, bundle count and receipt status are directly observed. The disk-fill consequence is inference from SQLite write behaviour under ENOSPC; no fill event has occurred, so this is a latent risk rather than an incident.

The Architectural Reality

  • ai/configBase.mjs:198backupPath: leaf(path.resolve(planeDataRootDefault, 'backups'), 'NEO_BACKUP_PATH', 'string', {planeMember: true}), and :15 resolves planeDataRootDefault from neoRootDir. So bundles and graph share a root by construction, not by operator choice. #16201 moves that root out of the working tree; it does not separate it from the graph.
  • maintenance.backup.retention is {keepMinimum: 3, maxDays: 30}, and the observed spread is exactly 30 days — so retention appears to be working. The 133 GB is the intended steady state, not a pruning bug. That is what makes this structural rather than a leak to fix.
  • Nine of ten clones symlink .neo-ai-data members to the canonical checkout, so this is one shared disk for the whole swarm's history, not a per-seat concern.
  • ai/scripts/maintenance/backup.mjs owns bundle creation and retention; restore.mjs:730's verifyLatestBackupRestorable is the read-only restorability probe (never yet run against these bundles).

The Fix

Direction rather than prescription, because the right answer depends on OQ1:

  1. Fail before filling. A pre-flight free-space check in the backup lane that refuses to start a bundle it cannot complete, and reports the refusal — rather than a partial write competing with the graph for the last blocks.
  2. Surface headroom as an observable, alongside the existing durability posture, so "the disk is 90% full" is a reported state rather than something discovered when writes fail.
  3. Separate the failure domains — the actual remedy, and the subject of OQ1.

Open Questions

  • OQ1 — separate volume, smaller retention, or off-host as the real answer? Three candidate shapes: (a) bundles on a distinct volume/disk from the graph; (b) reduce maxDays/keepMinimum so the corpus stays small relative to free space; (c) treat off-host sync (#16199) as the durability answer and keep only a short local window. These are not exclusive and they have different costs. Falsifier for (b): a shorter window narrows the recovery horizon, which is the thing the corpus exists to widen — so it trades one risk for another rather than removing one.
  • OQ2 — what is the actual free-space margin today? I observed sizes but not filesystem capacity or headroom. Without that, "133 GB is dangerous" is a ratio argument, not a measurement. This should be answered before choosing among OQ1's options.
  • OQ3 — does bundle size scale with graph size or with churn? A 728 MB graph producing ~3.6 GB bundles suggests bundles carry more than the graph (Chroma vectors, WAL, ledgers). If growth tracks churn rather than graph size, the trajectory matters more than the current figure.
  • OQ4 — should this gate the #16167 cutover? Probably not: the cut replaces the container data plane and does not change disk topology. Naming it so the answer is deliberate rather than assumed.

Acceptance Criteria

  • OQ2 answered: filesystem capacity and current headroom recorded, so the risk is measured rather than inferred from a ratio.
  • The backup lane refuses to begin a bundle it lacks free space to complete, and the refusal is reported rather than silent.
  • Available headroom is observable alongside the existing durability posture.
  • OQ1 resolved with a named choice and its rejected alternatives, per the falsifier above.
  • A test proves the pre-flight refusal fires, using a constrained-space fixture rather than asserting on the real filesystem.
  • Retention behaviour is left unchanged unless OQ1 selects option (b), in which case the narrowed recovery horizon is stated explicitly in the deployment docs.

Out of Scope

  • Relocating the backup root out of the working tree — #16201 owns that. It removes a deletion vector; this ticket addresses a capacity vector. Neither subsumes the other.
  • Off-host sync enforcement (#16199) and first-boot restore (#15639).
  • Bundle format, compression, or what a bundle contains.
  • Chroma store bloat (#14079) and the unused FTS5 index (#14192) — both would reduce bundle size as a side effect, and both are separately owned.

Avoided Traps

  • Treating this as a retention bug. Retention is working — 36 bundles across exactly 30 days against maxDays: 30. The 133 GB is the designed steady state, which is precisely why the co-location is structural.
  • Shrinking the window as a reflex. It reduces disk pressure by reducing the recovery horizon, i.e. by making the thing worse at its job. Acceptable only as a deliberate trade with the horizon stated.
  • Assuming a full disk only loses backups. SQLite needs space to write; ENOSPC takes the graph and the next bundle together. That simultaneity is the whole finding.

Related

  • #16201 — backup root inside the working tree (the deletion vector; same corpus, different failure mode)
  • #16199 / PR #16200 — required off-host durability now fails loudly
  • #15639 — first-boot restore from the latest bundle
  • #14079 — Memory Core Chroma store 2.5 GB bloat
  • #14192 — unused Chroma FTS5 index (~495 MB)
  • #16167 — the cutover this deliberately does not gate (OQ4)

Origin Session ID: 0a7f5f1d-cf12-4698-984c-17b64eea5178

Retrieval Hint: backups 133GB same disk as memory-core-graph.sqlite ENOSPC same failure domain free-space preflight retention 30 days working as designed

tobiu referenced in commit 66bce83 - "fix(config): relocate the backup root out of the plane and the working tree (#16201) (#16221) on Jul 31, 2026, 3:57 PM
tobiu closed this issue on Jul 31, 2026, 3:57 PM