Backup lane: off-host bundle sync hook with receipts

Context

The cloud backup lane writes daily atomic bundles to a host bind-mount. That survives rebuilds, --force-recreate, and down — everything except host loss. Redeploy recovery is solved; disaster recovery is not: if the host dies, the backups die with it.

Amended 2026-07-22 per Emmy's intake cycles 1-5 (issuecomment-5044662992, -5044813796, -5045032310, -5045126841, -5045289785). Cycle 5 (this fold): the receipt gains exact non-sensitive bundle provenance (bundleName + bundleCompletedAt, null on pre-bundle failure; never an absolute host path); offHostSync.status: 'success' is honestly defined as the configured command exited 0 for that named bundle (the generic hook cannot attest remote state); and the configured side effect gets a declared owner — the lease-owning CLI wrapper, NOT exported runBackup() — with the AiConfig overlay loaded before the first backupPath read. Cycle 4 (this fold): her descendant probe proved the lease bound ≠ sync-side-effect bound (direct child SIGKILLed at 202ms, grandchild alive at 603ms) — the completion contract is now truthfully direct-child-scoped; and runBackup/cleanOldBackups still use the DEFAULT_BACKUP_ROOT module constant (backup.mjs:162,215 — verified), so the ticket now also repairs the bundle writer + retention to resolve from AiConfig.backupPath before the receipt may derive from it. Cycle 1: config SSOT is AiConfig.maintenance.backup; launch is executable+argv; backup task doesn't opt into structured capture. Cycle 2: my "existing receipt path" was invented — the receipt transport is a durable atomic store (below). Cycle 3 (this fold): execFile timeout is not a hard completion bound (her probe: SIGTERM-ignoring child survived 6× timeout, completing only on SIGKILL — an awaited uncooperative sync can hold the exclusive-heavy backup lease indefinitely); path/config validation must be ticket-owned; the receipt schema lacked the not-run state and the store/snapshot/consumer contracts; the snapshot's schema owner and its four agent-facing tool echoes were unnamed.

The Problem

The bundle's persistence boundary is the host filesystem. Operators who want off-host durability hand-roll a sync outside the platform — invisible to the backup lane's cadence, retention, and receipts. Nobody can answer "is our latest bundle safely off the host?" from inside the system.

The Architectural Reality (intake-verified)

• Bundle writer: orchestrator backup task (taskDefinitions.mjs:373-378, command: nodeBin, args: [backup.mjs], no captureStdoutJson) → ai/scripts/maintenance/backup.mjs.
• Config: AiConfig.maintenance.backup is ONE object leaf (ai/configBase.mjs:1264); new keys are plain nested keys, so this ticket owns their validation (ADR-0019 successor-risk: owner aligned, validation must be explicit). Path SSOT: AiConfig.backupPath (configBase.mjs:62, env-bound NEO_BACKUP_PATH, leaf default == the current hardcoded root) — and the ticket repairs the current divergence: runBackup's default bundle root and cleanOldBackups still read the DEFAULT_BACKUP_ROOT module constant; they move to the AiConfig.backupPath read so bundle, retention, receipt, and the existing Health consumer share one root under the env override.
• Owner boundary (cycle-5): two materially different call surfaces exist — exported runBackup({bundleRoot, ...}) (used directly by unit + deployed integration tests) and the lease-owning CLI footer (withHeavyMaintenanceLease(() => runBackup()), orchestrator-injected lease token). The configured off-host sync + deployment-global receipt belong to the CLI wrapper, not the exported primitive: direct module callers (including disposable-bundleRoot tests) must never fire a configured off-host command or overwrite the deployment-global receipt outside the lease boundary.
• AiConfig load-order (cycle-5): current source calls loadTopLevelAiConfig() at line 212, AFTER the bundle root resolves at line 162 — merely swapping DEFAULT_BACKUP_ROOT → AiConfig.backupPath at 162 would still read the pre-overlay value. Invariant: the top-level overlay loads BEFORE bundle, retention, receipt, and snapshot-root resolution (witness pinned).
• Snapshot pipeline: DeploymentStateBridgeService.collectSnapshot → writes the snapshot → ai/services/memory-core/helpers/deploymentStateBridgeStore.mjs owns snapshot schema/section inventory + compatibility → agent-consumed via four documented surfaces: KB get_deployment_state_snapshot + inspect_deployment, MC get_deployment_state_snapshot + inspect_deployment. Snapshot byte cap: 256 KiB.
• execFile({timeout}) sends SIGTERM only; an uncooperative child completes only on SIGKILL. Awaiting the sync inside backup.mjs extends the exclusive-heavy backup lease for the sync's full lifetime — a deliberate behavior change, bounded below.

The Fix — post-bundle sync step owned by the lease-owning CLI wrapper (supervisor + exported primitive untouched)

Exported runBackup() remains the local bundle/retention primitive — unchanged module API, no off-host side effects, no deployment-global receipt writes. The CLI wrapper (withHeavyMaintenanceLease path) owns: the configured sync step, the deployment-global receipt (success around runBackup(), or a truthful not-run-backup-failed receipt when runBackup() throws), and the AiConfig overlay load before any path resolution.

1. Config keys (five nested keys inside the existing maintenance leaf; validation is this ticket's contract):
   • offHostSync.command (string, default '' = disabled) — executable name/path. Enabled iff non-empty string.
   • offHostSync.argv (string[], default []) — whole-token placeholders {bundleDir}/{bundleName} only; any token containing (not equaling) a placeholder, or any other {...} token, is a validation failure.
   • offHostSync.envAllowlist (string[], default []) — names matching /^[A-Z_][A-Z0-9_]*$/, non-empty; only set, non-empty values are forwarded.
   • offHostSync.timeoutMs (integer, default 600000, bounds 1000..1800000) + offHostSync.killGraceMs (integer, default 5000, bounds 0..60000).
   • Any type/bounds violation → validation-failed receipt, no sync attempt, loud log. No leaf() nodes are added; no manual env checks anywhere (ADR-0019).
2. Launch: execFile(command, argv, {shell: false, env: minimalBase + allowlisted, timeout: timeoutMs, killSignal: 'SIGTERM'}). Credential boundary: base env = PATH, HOME, USER, TMPDIR + allowlisted names; no implicit process.env inheritance.
3. Completion contract — direct-child-scoped, honest about the rest (cycle-4): SIGTERM at timeoutMs; if the direct child is still alive after killGraceMs, SIGKILL follows. The TERM→KILL escalation is bounded at timeoutMs + killGraceMs at the direct-child boundary (her probe confirmed the callback completes); lease release occurs after the bounded-size receipt fsync/rename and process exit — local filesystem completion is not a hard real-time guarantee, so no exact wall-clock lease-release bound is claimed. The sync side effect is NOT claimed bounded: the receipt is written at direct-child completion and carries completionScope: 'direct-child' + descendants: 'unknown' + terminatedVia: 'exit' | 'sigterm' | 'sigkill', exitCode (nullable), signal (nullable), durationMs (measured). Overlap posture (stated, not hidden): a surviving descendant may continue after a timeout receipt — acceptable at the daily cadence (~24h to the next run), and operators choose tools with well-behaved children (rsync/awscli/restic). Rejected: process-group tree-kill (detached + kill(-pgid)) — the platform matrix cost (POSIX vs Windows job control) outweighs a daily-cadence sync; a truthful field beats an unprovable guarantee.
4. Receipt store (durable, atomic): path.join(AiConfig.backupPath, 'last-backup-receipt.json') (after the root repair in §Architectural Reality, this IS the bundle tree's root under every override). Envelope (exact primitive types):

{ schemaVersion: 1,                                    // integer, required
  finishedAt: string | null,                           // ISO-8601
  bundleName: string | null,                           // e.g. 'backup-2026-07-22T11-00-00-000Z'; null on pre-bundle failure — never an absolute host path
  bundleCompletedAt: string | null,                    // runBackup()'s own completedAt; null on pre-bundle failure
  backup: { status: 'success' | 'failed',
            durationMs: number,
            error: string | null },                    // redacted + ≤4 KiB
  offHostSync: { status: 'disabled' | 'not-run-backup-failed' | 'success' | 'failed' | 'timeout' | 'validation-failed',
                 durationMs: number | null,
                 exitCode: integer | null,
                 signal: string | null,
                 terminatedVia: 'exit' | 'sigterm' | 'sigkill' | null,
                 completionScope: 'direct-child',
                 descendants: 'unknown',
                 stderrTail: string } }                // redacted + ≤4 KiB

• offHostSync.status ∈ the union above — not-run-backup-failed covers the promised receipt-on-backup-failure case truthfully. success means exactly: the configured command exited 0 for the named bundle (bundleName); the generic hook does not attest remote object state.
• The projection's unreadable outcome is one machine-consumable shape: {status: 'unreadable', kind: 'corrupt' | 'oversize' | 'unsupported-version', finishedAt: string | null} (finishedAt when parseable, else null).
• error/stderrTail: redact before bounding — allowlisted env values replaced with *** (longest-value-first, overlap-safe), then bounded to 4 KiB each; receipt file capped at 64 KiB at write.
• Write: unique temp per write (last-backup-receipt.tmp-<pid>-<ts>), fsync, rename; stale temp sweep at write time.
• Failure semantics: sync failure never changes the backup's own outcome.

5. Snapshot projection: deploymentStateBridgeStore.mjs gains an additive optional maintenance.lastBackup section (schema-version compatible: old snapshots without it stay valid; consumers tolerate absence). The block is a validated allowlisted shape (never arbitrary parsed JSON), worst-case ~6 KiB — fits the 256 KiB snapshot cap with headroom recorded in the implementation. Outcomes vocabulary (stable, no consumer inference): absent file → section omitted; 'unreadable' ∈ 'corrupt' | 'oversize' | 'unsupported-version' with the receipt's finishedAt when parseable.
6. Lease semantics (stated, witnessed): the sync lifetime extends the exclusive-heavy backup lease; the lease remains held through the bounded-size receipt fsync/rename and releases afterward — no exact wall-clock lease-release bound is claimed; no second heavy task starts before that actual release (existing exclusivity — witness pinned).

Contract Ledger

Acceptance Criteria

• Default (empty command): zero behavior change; receipt records disabled
• Validation: each malformed key (type, bounds, partial/unknown placeholder, bad env name) → validation-failed receipt + no sync attempt
• Credential boundary: non-allowlisted orchestrator env absent from the child env; unset allowlisted name simply absent
• Hard completion: a SIGTERM-ignoring child receives SIGKILL within timeoutMs + killGraceMs (direct-child boundary); the receipt is written after direct-child completion and records terminatedVia: 'sigkill' (Emmy's probe shape becomes the witness)
• Receipt schema: not-run-backup-failed recorded when the backup fails before sync; all fields typed/nullable per the envelope
• Redact-before-bound: an allowlisted env value in stderr is *** in the store; tails ≤ 4 KiB; store ≤ 64 KiB
• Store: absent-before-first-run → section omitted; torn write → previous receipt intact; stale temp swept; corrupt/oversize/wrong-version → stable 'unreadable:<kind>' outcome
• Snapshot: old snapshot (no maintenance section) stays valid; new consumers tolerate absence; section fits the 256 KiB cap
• Lease: no second heavy task starts before the actual lease release (after bounded receipt fsync/rename); the TERM→KILL escalation timing is witnessed separately at the direct-child boundary; no exact wall-clock lease-release bound is claimed; receipt fields truthfully scope completion (completionScope: 'direct-child', descendants: 'unknown')
• Root SSOT: with NEO_BACKUP_PATH set to a custom root, bundle creation, retention sweep, receipt write, and snapshot read-back ALL resolve to that root (override witness)
• Bundle provenance: receipt + snapshot carry bundleName + bundleCompletedAt (null on pre-bundle failure); no absolute host path is projected; offHostSync.status: 'success' means exit 0 for that named bundle
• Owner boundary: a direct exported runBackup({bundleRoot: <disposable>}) call fires NO configured off-host command and writes NO deployment-global receipt; the CLI wrapper writes the failure receipt (not-run-backup-failed) around a thrown runBackup()
• Overlay order: the top-level AiConfig overlay is loaded before bundle, retention, receipt, and snapshot-root resolution (witness that would fail on the current line-162-before-212 order)
• Timing honesty: TERM→KILL bounded at the direct-child boundary; lease release claimed only as after receipt fsync/rename + process exit (no exact wall-clock bound)
• Unreadable projection: exact {status: 'unreadable', kind, finishedAt} shape for corrupt / oversize / unsupported-version
• Unit specs for all of the above (disposable repoDir contract; no live-store mutation)

Out of Scope

• First-boot restore automation (#15639, separately blocked); restore-path embedding (resolved as preserved-vector)
• A built-in S3/rsync/restic implementation (deployers own tooling); bundle encryption
• Descendant-process hard guarantees beyond the documented caveat
• Any supervisor/supervisor-capture contract change

Related

• #10844 (daily snapshot pipeline), #11649 (retention), #11724 (redeploy-safe persistence)
• Tenant-ingestion credential boundary (learn/agentos/cloud-deployment/TenantIngestionModel.md); ADR-0019

Live latest-open sweep: checked latest 20 open issues at creation (2026-07-21 ~16:45Z) and re-verified at each amendment; no equivalent found. A2A in-flight sweep: Emmy's intake cycles assign these folds to me; no competing claim.

Origin Session ID: d8a51237-4fcc-4171-8071-a391da0be361 (amended in session 72c8c42d-f18a-408c-97c8-aeb1f82dd276)

Retrieval Hint: "backup off-host sync executable argv envAllowlist hard completion SIGTERM SIGKILL receipt store schemaVersion deployment snapshot maintenance section"