A destructive docker compose down -v during an update removed the named volumes of a cloud deployment. The Memory Core corpus — roughly 30 entries — was destroyed, along with the self-heal and recovery ledgers. Merge-on-update is the stated intent; it could not run, because its left-hand side no longer existed. (Impact was contained: the corpus was smoke-test data, so nothing of value was lost. That bounds the damage; it does not bound the defect, which would have taken a populated deployment the same way.)
Attribution corrected 2026-07-28 — one sourcing error in the original body, and one over-correction of my own. See the pre-implementation V-B-A and its follow-up. The data loss is not in question and never was — I observed it directly on the live deployment. What I got wrong was attributing the destructive instruction to our runbook when it came from a tenant-authored checklist. My first correction then overshot in the opposite direction by labelling the gap "falsified", which reduced a real data-loss event to a citation error. It is not. The gap is real; only the document I blamed was wrong, and that makes the problem worse — see below.
The Problem
The load-bearing gap: post-mortem capability is co-located with what it would explain, and excluded from the backup bundle.
The self-heal and recovery ledgers report status: "available" with total: 0 / entries: [] after an update, because both live in a volume the update removes:
In the cloud profile that directory is the orchestrator-state named volume (ai/deploy/docker-compose.yml:259, declared :392), which docker compose down -v removes. And the bundle does not cover it — PipelineWiring.md:124-126 already admits "The current logical backup bundle does not include orchestrator-state". Both halves fail, so the daemons that exist to explain an incident are destroyed by the same operation that caused it: the one class of event they most need to describe is the one they can never describe.
The corpus itself was destroyed by the same command.shared-sqlite-data (docker-compose.yml:89) is a named volume, and -v takes it. Merge-on-update cannot merge when its left-hand side is already deleted.
The runbook contradicted the merge semantics, the runbook won, and the runbook was not ours. A checklist instructed the destructive step and acknowledged the deletion as approved. Our own runbook forbids -v (PipelineWiring.md:115) and our reference script never runs it (deploy-pipeline.sh:14-16).
That is the aggravating fact, not the mitigating one. A documentation guarantee only binds readers of that document. The instruction that actually executed lives in a file we do not own, cannot edit, and will not see change. So "our runbook already says never -v" is worth nothing here — it is precisely the reason the fix must be a refusal in code, on the path we ship, rather than any wording change. Every deployment that adapts our reference script inherits a refusal; no deployment inherits our prose.
A guarantee that lives only in documentation is not a guarantee.
One clause of the original body was imprecise and is not being fixed as written. It said "the only copies of a bundle live in the same Docker named volumes the update removes." The data did live in a destroyed named volume — that part is exact. Bundles specifically do not: docker-compose.yml:268 bind-mounts ./.neo-ai-data/backups to the host on the cloud profile, so -v spares them. The real bundle-loss mode is the documented rule-3 footgun (PipelineWiring.md:119) — the bind-mount is relative to the compose project directory, so a run from an ephemeral checkout orphans every prior bundle. That still matters, because AC4's precondition depends on locating one.
None of these are regressions — no survivability guarantee for an update ever existed.
The Architectural Reality
An update is a transition between two live states, not a teardown followed by a fresh install. Today it is modelled as the latter, so preservation is left to operator discipline — and discipline is exactly what a 2am redeploy does not have.
offHostSync cannot be part of the answer by being switched on. Enablement is not a boolean: validateOffHostSyncConfig returns enabled: false for an empty command (ai/scripts/maintenance/offHostSync.mjs), so enabled iffcommand names an executable — and we cannot know whether a deployment syncs via aws, rclone, rsync, or a bespoke script. There is also no leaf to bind: maintenance is one object leaf (ai/configBase.mjs:1424) whose offHostSync keys are plain nested values, which is why "maintenance" is a single entry in ai/scripts/lint/config-leaf-parity.json:74.
So the defaultable property is posture honesty, not enablement. A cloud deployment reporting offHostSync.status: 'disabled' is not in a benign state — it has an unmet durability posture, today indistinguishable from a deliberate opt-out. Same defect class as #16056: a suppressed state reporting itself as benign.
The Fix (one PR)
Project an explicit unmet-durability posture for a cloud deployment with no off-host command, through the existing deployment-state bridge, readable by the diagnostics tools.
Ship a preflight refusal that proves a restorable pre-transition bundle exists before anything touches containers, wired as a mandatory step in the reference deploy script.
Bring the heal/recovery ledgers into the bundle so they survive a volume replacement.
Where the refusal can and cannot live
We own no destructive path.DEPLOYMENT_RUNTIME_LIFECYCLE_OPERATIONS is frozen to ['restart'] (DeploymentRuntimeAccessService.mjs:15), and docker compose down appears in no script in this repo — the reference script only runs up -d --build plus a healthcheck wait. The destructive act is an operator command or a third-party checklist, and no guard we write can intercept it.
So the refusal goes where we do have leverage: the reference script is the CI-neutral substrate teams adapt (PipelineWiring.md:154). A preflight that exits non-zero there is inherited by every team that adapts it, and it also correctly guards up -d --build, which recreates containers. The posture projection covers the rest: a deployment already in the bad state becomes visible through the diagnostics tools without host access.
Acceptance Criteria
deploymentMode: 'cloud' with no configured off-host command projects an explicit unmet-durability posture via collectMaintenanceSnapshot (DeploymentStateBridgeService.mjs:249), distinguishing not configured from configured and failing from deliberately opted out. Uses the existing resolveCloudOnlyEnabled mode-gate (Orchestrator.mjs:118), not a raw deploymentMode read. No invented default command, no new leaf inside the object-leaf subtree.
A preflight gate refuses (exits non-zero) unless a verified pre-transition bundle exists, and is wired as a mandatory step in ai/examples/cloud-deployment/deploy-pipeline.sh before any container-affecting command. Refusal, not a warning — a guarantee that can be skipped by a tired operator at 2am is not a guarantee. Scoped honestly: this cannot intercept a hand-typed docker compose down -v; it makes the path we ship refuse.
The bundle is verified non-empty and restorable by the gate — an empty bundle must not satisfy the precondition, which is precisely the observed failure (the one bundle in the ledger completed 25 minutes after the new stack came up, capturing an already-empty plane).
The refusal distinguishes no bundle exists from the bundle location moved — rule 3's relative bind-mount makes the second reachable, and reporting "no bundle" for a bundle sitting in a prior checkout directory is a wrong-subject answer.
Self-heal and recovery ledgers survive a named-volume replacement, asserted by a test that replaces the volume and reads the ledger.
PipelineWiring.md:124-126 no longer documents the orchestrator-state exclusion as a standing caveat — the wording follows the fix rather than warning about a closed hole.
Post-merge: one rehearsal on a disposable deployment — populate, update, confirm the corpus survived and the ledgers explain the transition.
Out of Scope
Data-plane placement — #15800 (@neo-opus-grace) owns the bind-mount vs named-volume election. AC5 is satisfied by bundle inclusion, deliberately not by relocating the ledgers, so this ticket does not pre-decide her call.
Restore UX / partial-restore selection — this ticket guarantees a bundle exists and survives, not a new restore surface.
The merge algorithm itself, which is fine; only its precondition was missing.
Defaulting an off-host sync command, which is not knowable for a given deployment.
Related
#15800 (data-plane placement election) — adjacent, deliberately not pre-decided
#15641 (offHostSync contract owner) — establishes the object-leaf shape
Context
A destructive
docker compose down -vduring an update removed the named volumes of a cloud deployment. The Memory Core corpus — roughly 30 entries — was destroyed, along with the self-heal and recovery ledgers. Merge-on-update is the stated intent; it could not run, because its left-hand side no longer existed. (Impact was contained: the corpus was smoke-test data, so nothing of value was lost. That bounds the damage; it does not bound the defect, which would have taken a populated deployment the same way.)The Problem
The load-bearing gap: post-mortem capability is co-located with what it would explain, and excluded from the backup bundle.
The self-heal and recovery ledgers report
status: "available"withtotal: 0/entries: []after an update, because both live in a volume the update removes:healAttemptsPath→<planeDataRoot>/orchestrator-daemon/heal-attempts.json(ai/configBase.mjs:1232,planeMember: true)recoveryRunStateDir→<planeDataRoot>/orchestrator-daemon/recovery-runs(ai/configBase.mjs:1233,planeMember: true)In the cloud profile that directory is the
orchestrator-statenamed volume (ai/deploy/docker-compose.yml:259, declared:392), whichdocker compose down -vremoves. And the bundle does not cover it —PipelineWiring.md:124-126already admits "The current logical backup bundle does not includeorchestrator-state". Both halves fail, so the daemons that exist to explain an incident are destroyed by the same operation that caused it: the one class of event they most need to describe is the one they can never describe.The corpus itself was destroyed by the same command.
shared-sqlite-data(docker-compose.yml:89) is a named volume, and-vtakes it. Merge-on-update cannot merge when its left-hand side is already deleted.The runbook contradicted the merge semantics, the runbook won, and the runbook was not ours. A checklist instructed the destructive step and acknowledged the deletion as approved. Our own runbook forbids
-v(PipelineWiring.md:115) and our reference script never runs it (deploy-pipeline.sh:14-16).That is the aggravating fact, not the mitigating one. A documentation guarantee only binds readers of that document. The instruction that actually executed lives in a file we do not own, cannot edit, and will not see change. So "our runbook already says never
-v" is worth nothing here — it is precisely the reason the fix must be a refusal in code, on the path we ship, rather than any wording change. Every deployment that adapts our reference script inherits a refusal; no deployment inherits our prose.A guarantee that lives only in documentation is not a guarantee.
One clause of the original body was imprecise and is not being fixed as written. It said "the only copies of a bundle live in the same Docker named volumes the update removes." The data did live in a destroyed named volume — that part is exact. Bundles specifically do not:
docker-compose.yml:268bind-mounts./.neo-ai-data/backupsto the host on the cloud profile, so-vspares them. The real bundle-loss mode is the documented rule-3 footgun (PipelineWiring.md:119) — the bind-mount is relative to the compose project directory, so a run from an ephemeral checkout orphans every prior bundle. That still matters, because AC4's precondition depends on locating one.None of these are regressions — no survivability guarantee for an update ever existed.
The Architectural Reality
An update is a transition between two live states, not a teardown followed by a fresh install. Today it is modelled as the latter, so preservation is left to operator discipline — and discipline is exactly what a 2am redeploy does not have.
offHostSynccannot be part of the answer by being switched on. Enablement is not a boolean:validateOffHostSyncConfigreturnsenabled: falsefor an emptycommand(ai/scripts/maintenance/offHostSync.mjs), so enabled iffcommandnames an executable — and we cannot know whether a deployment syncs viaaws,rclone,rsync, or a bespoke script. There is also no leaf to bind:maintenanceis one object leaf (ai/configBase.mjs:1424) whoseoffHostSynckeys are plain nested values, which is why"maintenance"is a single entry inai/scripts/lint/config-leaf-parity.json:74.So the defaultable property is posture honesty, not enablement. A cloud deployment reporting
offHostSync.status: 'disabled'is not in a benign state — it has an unmet durability posture, today indistinguishable from a deliberate opt-out. Same defect class as #16056: a suppressed state reporting itself as benign.The Fix (one PR)
Where the refusal can and cannot live
We own no destructive path.
DEPLOYMENT_RUNTIME_LIFECYCLE_OPERATIONSis frozen to['restart'](DeploymentRuntimeAccessService.mjs:15), anddocker compose downappears in no script in this repo — the reference script only runsup -d --buildplus a healthcheck wait. The destructive act is an operator command or a third-party checklist, and no guard we write can intercept it.So the refusal goes where we do have leverage: the reference script is the CI-neutral substrate teams adapt (
PipelineWiring.md:154). A preflight that exits non-zero there is inherited by every team that adapts it, and it also correctly guardsup -d --build, which recreates containers. The posture projection covers the rest: a deployment already in the bad state becomes visible through the diagnostics tools without host access.Acceptance Criteria
deploymentMode: 'cloud'with no configured off-host command projects an explicit unmet-durability posture viacollectMaintenanceSnapshot(DeploymentStateBridgeService.mjs:249), distinguishing not configured from configured and failing from deliberately opted out. Uses the existingresolveCloudOnlyEnabledmode-gate (Orchestrator.mjs:118), not a rawdeploymentModeread. No invented default command, no new leaf inside the object-leaf subtree.ai/examples/cloud-deployment/deploy-pipeline.shbefore any container-affecting command. Refusal, not a warning — a guarantee that can be skipped by a tired operator at 2am is not a guarantee. Scoped honestly: this cannot intercept a hand-typeddocker compose down -v; it makes the path we ship refuse.PipelineWiring.md:124-126no longer documents theorchestrator-stateexclusion as a standing caveat — the wording follows the fix rather than warning about a closed hole.Out of Scope
Related
offHostSynccontract owner) — establishes the object-leaf shape