LearnNewsExamplesServices
Frontmatter
id16199
titleFail required off-host backup runs after the local receipt
stateClosed
labels
bugaitesting
assigneesneo-gpt-emmy
createdAtJul 31, 2026, 12:17 AM
updatedAtJul 31, 2026, 12:54 AM
githubUrlhttps://github.com/neomjs/neo/issues/16199
authorneo-gpt-emmy
commentsCount0
parentIssue16167
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 31, 2026, 12:54 AM

Fail required off-host backup runs after the local receipt

Closed Backlog/active-chunk-11 bugaitesting
neo-gpt-emmy
neo-gpt-emmy commented on Jul 31, 2026, 12:17 AM

Context

The canonical local Agent OS cutover in #16167 requires a restorable copy that survives losing the machine before the Docker-owned data plane replaces the current host graph.

A live intake probe on 2026-07-30 establishes the current deployment state without exposing config values or credentials:

{
  "deploymentMode": "cloud",
  "offHostBackupRequired": true,
  "offHostSyncConfigured": false,
  "offHostSyncConfigValid": true
}

The latest durable receipt independently records:

{
  "backupStatus": "success",
  "backupErrorPresent": false,
  "offHostSyncStatus": "disabled",
  "offHostExitCode": null
}

That combination is not merely a diagnostic posture. The operation itself exits successfully.

Live source confirms why:

  • resolveDurabilityPosture() applies orchestrator.cloudOnly.offHostBackupRequired and correctly projects this cloud state as unmet.
  • runBackupWithOffHostSync() does not read the requirement. After a local bundle succeeds, disabled, validation-failed, failed, and timeout are receipt statuses plus warnings; the wrapper returns the local result.
  • The CLI footer treats that return as success and calls process.exit(0).
  • Existing unit coverage deliberately proves the older #15641 contract: a malformed sync subtree leaves backup.status: 'success' and offHostSync.status: 'validation-failed'.

Those receipt facts are correct and must remain correct. The defect is the terminal contract after them.

The historical sweep found no open equivalent. Closed #15641 / PR #15716 intentionally made sync failure non-fatal so an optional hook could not falsify a completed local bundle. Closed #16055 / PR #16077 later introduced the separate deployment requirement and made an unmet requirement observable, but did not connect it to the backup command's exit status. This successor joins those two already-correct halves.

Memory Core and Knowledge Base semantic retrieval were unavailable during intake because ChromaDB was down, so this ticket does not claim a semantic-memory miss; its prior-art authority is the live GitHub/source/receipt sweep above.

The Problem

offHostBackupRequired currently affects only a diagnostic projection. It does not constrain the operation whose durability it declares.

A required deployment can therefore:

  1. create another local bundle on the same host;
  2. skip or fail its off-host copy;
  3. persist an honest disabled / validation-failed / failed / timeout receipt;
  4. print a warning; and
  5. exit 0.

The orchestrator sees a successful backup task even though the deployment's declared durability requirement was not met. Before a one-machine data-plane replacement, this is fail-open on the only copy that survives host loss.

The Architectural Reality

  • Local bundle truth and operation admission are different axes. A completed bundle remains backup.status: 'success'; off-host outcome remains offHostSync.status.
  • #15641 intentionally defined off-host success narrowly as “the configured direct child exited 0 for the named bundle.” This ticket must not invent remote-object attestation.
  • resolveCloudOnlyDefault() in deploymentDurabilityPosture.mjs is already the canonical tri-state resolver for offHostBackupRequired; the backup path must reuse it rather than re-read env or duplicate deploymentMode === 'cloud'.
  • The lease-owning wrapper owns sync, receipt persistence, and the CLI terminal. Exported runBackup() remains the pure local-bundle primitive.
  • A required failure must be raised only after the receipt attempt, so observability is retained before the supervisor receives a non-zero terminal.

The Fix

  1. Resolve whether off-host backup is required from the existing AiConfig leaves through the existing canonical resolver.
  2. Preserve the local bundle and write the truthful receipt exactly as today.
  3. After the receipt attempt, reject the lease-owned operation when off-host backup is required and the observed sync status is anything other than success.
  4. Keep optional deployments' current behavior: a disabled or failed optional sync warns and returns the completed local bundle.
  5. Surface only the stable sync status in the terminal error. Do not interpolate config, argv, stderr, paths, or credential-bearing values.

Contract Ledger

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
Off-host requirement resolution AiConfig.orchestrator.cloudOnly.offHostBackupRequired + resolveCloudOnlyDefault() Reuse the cloud/local tri-state rule at the backup terminal Missing config/programming defect fails loud through AiConfig; no env re-read ADR 0019 + #16055 Required/optional matrix
Local bundle receipt #15641 receipt schema Completed local bundle remains backup.status: success regardless of sync outcome Receipt write failure retains its existing warning; it cannot turn sync failure into success Existing JSDoc Receipt asserted before terminal rejection
Off-host receipt offHostSyncStore closed status set Preserve exact disabled, validation-failed, failed, timeout, or success outcome No invented remote-attestation state #15641 Status matrix
Lease-owned operation terminal runBackupWithOffHostSync() + CLI footer Required + non-success sync rejects after receipt; required + success resolves; optional behavior unchanged Stable status-only error, then existing CLI exit-1 path This ticket Wrapper rejection + CLI/supervisor witness
Exported local primitive runBackup() owner boundary No off-host side effect or requirement read Unchanged Existing JSDoc Source/behavioral boundary test

Acceptance Criteria

  • Requirement resolution reuses the existing canonical tri-state resolver; no raw env read, duplicate deployment-mode rule, runtime AiConfig mutation, or new config leaf is introduced.
  • A completed local bundle still records backup.status: 'success' even when required off-host sync is disabled or fails.
  • The truthful off-host receipt is attempted before the required-durability error is raised.
  • When off-host backup is required, each non-success outcome—disabled, validation-failed, failed, and timeout—rejects the lease-owned operation and makes the CLI exit non-zero.
  • When off-host backup is required and the configured direct child exits 0, the operation resolves normally.
  • When off-host backup is not required, disabled and failed sync outcomes preserve the existing warning + successful local-bundle terminal.
  • The terminal error exposes only a stable status/reason code; config values, argv, stderr, filesystem paths, and credentials do not cross the boundary.
  • The orchestrator's existing backup child supervision observes the non-zero exit without a new scheduling or process-management mechanism.
  • Exported runBackup() remains a pure local bundle/retention primitive with no configured sync or requirement side effects.
  • Focused tests mechanize the full required/optional × sync-status matrix and preserve the existing receipt schema.

Out of Scope

  • Selecting or defaulting an off-host sync command.
  • Implementing S3, rsync, rclone, restic, encryption, or remote-object verification.
  • Changing the meaning of backup.status or offHostSync.status: 'success'.
  • Changing bundle contents, retention, restore behavior, deployment scheduling, or the cutover procedure.
  • Treating a host bind or Docker named volume as off-host durability.

Avoided Traps

  • Marking the local bundle failed: that would falsify an artifact that was successfully created.
  • Logging louder while still exiting zero: the supervisor consumes the terminal, not operator attention.
  • Making every sync failure fatal: optional deployments deliberately retain the #15641 behavior.
  • Duplicating the cloud-default rule: posture and operation could then disagree on whether durability is required.
  • Throwing before the receipt: loses the exact evidence needed to diagnose the refusal.
  • Claiming remote attestation: exit 0 proves only the configured direct child completed successfully for the named bundle.

Decision Record impact

Aligned with ADR 0019: consumers read resolved AiConfig leaves at the use site and reuse the existing pure resolver. No amendment.

Aligned with #15641 and #16055: this connects optional-hook outcome truth to the later deployment-requirement gate without changing either predecessor's owned semantics.

Related

Parent: #16167

Related: #15641, PR #15716, #16055, PR #16077, #16197

Origin Session ID: b1ebc46a-5a83-496c-aa8b-385af785e9cb

Retrieval Hint: required off-host backup local bundle success sync disabled CLI exit zero fail after receipt

tobiu referenced in commit 54d7cc8 - "fix(backup): fail unmet required off-host sync (#16199) (#16200)" on Jul 31, 2026, 12:54 AM
tobiu closed this issue on Jul 31, 2026, 12:54 AM