LearnNewsExamplesServices
Frontmatter
id15639
titleCloud deployment: opt-in first-boot restore from latest backup bundle
stateOpen
labels
enhancementai
assignees[]
createdAtJul 21, 2026, 6:44 PM
updatedAtJul 22, 2026, 10:09 PM
githubUrlhttps://github.com/neomjs/neo/issues/15639
authorneo-kimi-phoebe
commentsCount3
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[x] 15740 Implement orchestrator-owned restore-empty-target action, [x] 15739 Amend ADR-0027 for target-set recovery runs, [ ] 15693 Implement orchestrator-governed restore-delta-merge, [x] 15692 Stream vector restores with bounded importer batches, [x] 15691 Validate embedding compatibility before restore mutation
blocking[]

Cloud deployment: opt-in first-boot restore from latest backup bundle

Open Backlog/active-chunk-8 enhancementai
neo-kimi-phoebe
neo-kimi-phoebe commented on Jul 21, 2026, 6:44 PM

Context

Cloud deployments already persist atomic backup bundles outside the service volumes, and the manual ai:restore path can consume them. A fresh host or intentionally reset volume still needs an operator to recognize the state, select a bundle, and start recovery.

The missing capability is an opt-in bootstrap selection policy. It must not become a second data-mutation controller.

Problem

The orchestrator cannot currently distinguish and record these bootstrap outcomes:

  1. restore disabled;
  2. stores are not fresh;
  3. no candidate bundle exists;
  4. a fresh deployment has a candidate bundle and should enter the governed recovery path.

Encoding that decision only in an operator runbook makes disaster recovery fragile. Directly spawning restore.mjs from startup would solve the trigger while bypassing the admission, audit, checkpoint, shadow/fence, and retry authority assigned to the recovery subsystem.

Authority Boundary

  • ADR-0014 assigns the cloud control plane to the orchestrator.
  • ADR-0019 governs the reactive AiConfig leaves; configuration is read from the injected provider SSOT.
  • ADR-0027 assigns persistent recovery mutation to orchestrator classification and DataRecoveryActuatorService.
  • #15691 owns bundle and embedding-contract validation before mutation.
  • #15692 owns bounded streaming vector import.
  • #15693 owns the admitted restore-delta-merge action, durable recovery-run state, shadow/fence/promotion, retry, and the terminal action receipt.

Therefore this ticket owns opt-in, freshness evidence, latest-candidate selection, request construction, and bootstrap receipt projection. It does not own restore execution.

Proposed Fix

Add a default-off orchestrator bootstrap selector:

  1. Add an ADR-0019-compliant orchestrator.backup.restoreOnFreshBoot config leaf, default false.
  2. After the persistent services are reachable, collect a seed-aware freshness snapshot for every restore target and enumerate complete backup bundles.
  3. When disabled, non-fresh, or without a candidate, emit a bounded bootstrap decision receipt and continue normal scheduling.
  4. When enabled, fresh, and a complete latest bundle exists:
    • build the validated bundle descriptor defined by #15691;
    • submit one classified restore-delta-merge request through the orchestrator recovery controller implemented by #15693;
    • never spawn restore.mjs or call a database importer directly;
    • keep data-consuming scheduled lanes ineligible until the action reaches a terminal receipt, while leaving the control plane diagnosable.
  5. Project the bootstrap decision plus the authoritative #15693 action reference/result into the deployment-state snapshot. The projection is observability, not a second recovery ledger.

Restore mode is deliberately not configurable here: first-boot recovery uses the governed merge action. Destructive replace semantics require separate architectural authority.

Acceptance Criteria

  • With restoreOnFreshBoot=false or absent, no recovery request is submitted and the deployment behaves as before.
  • Freshness is seed-aware and checks every persistent restore target; any non-fresh target prevents automatic submission.
  • No candidate bundle yields a clean skipped-no-bundle receipt.
  • Opted-in + fresh + complete latest bundle submits one classified restore-delta-merge request using the #15691 descriptor contract.
  • Persistent mutation occurs only through the #15693 recovery controller and #15692 bounded importer; bootstrap code contains no direct restore child spawn or importer call.
  • Data-consuming scheduled lanes wait for a terminal recovery receipt; failure/defer remains diagnosable and does not silently continue into mutation lanes.
  • The deployment-state snapshot exposes the bootstrap decision, selected bundle identity, recovery-run identity, terminal classification, and bounded redacted error metadata.
  • Bootstrap selection performs no embedding-provider call and has no re-embedding authority.
  • Unit coverage falsifies disabled, non-fresh, partially fresh, no-bundle, incomplete-bundle, exactly-one submission, failed/deferred action, and normal non-recovery boot paths.

Contract Ledger

Dimension Contract
Invocation One bootstrap evaluation after required persistent services are reachable and before data-consuming scheduled lanes become eligible.
Ownership This ticket owns selection and projection; #15693 owns admission and all persistent mutation.
Inputs Injected AiConfig, seed-aware freshness evidence, complete-bundle candidates, and the #15691 descriptor contract.
Outputs A bounded bootstrap decision receipt and, when submitted, a reference to the authoritative recovery-run receipt.
Benign outcomes skipped-disabled, skipped-not-fresh, and skipped-no-bundle continue normal scheduling.
Failure outcomes Invalid/incomplete descriptor, rejected/deferred admission, or failed action is explicit and keeps data-consuming mutation lanes ineligible.
Concurrency At most one recovery request per bootstrap evaluation; #15693 owns cross-restart idempotency and lease semantics.
Durability Recovery-run state is durable only in #15693; deployment-state is a projection and may be rebuilt.
Cancellation Shutdown prevents a new submission; an admitted action follows #15693 cancellation and settlement semantics.
Observability Decision reason, bundle identity, recovery-run identity, terminal classification, timing, and bounded redacted failure metadata.
Versioning Descriptor and receipt versions fail closed when unsupported; no implicit compatibility fallback.

Out of Scope

  • Bundle compatibility validation itself (#15691)
  • Streaming importer mechanics (#15692)
  • Recovery admission, mutation, checkpointing, shadow promotion, or retry (#15693)
  • Off-host bundle synchronization
  • Automatic restore of operational daemon state
  • Provider readiness probes or re-embedding
  • Destructive replace-mode bootstrap

Avoided Traps

  • Direct startup child spawn — creates a second mutation controller outside ADR-0027.
  • Provider-gated restore — historical vectors remain usable without an embedding-provider call; embeddings and re-embeddings stay orchestrator driven.
  • Single-store freshness — one empty store cannot prove the deployment is globally fresh.
  • Independent receipt/checkpoint store — the bootstrap surface projects the #15693 action result instead of inventing competing durable state.
  • Default-on or replace mode — unsafe when freshness evidence is wrong or partial.

Dependencies

Blocked by #15691, #15692, and #15693.

Related

  • #10844 — daily snapshot pipeline
  • #10871 — backup/restore parity
  • #11720 / ADR-0014 — cloud deployment architecture
  • ADR-0019 — AiConfig provider SSOT
  • ADR-0027 — orchestrator-governed recovery actions

Origin Session ID: d8a51237-4fcc-4171-8071-a391da0be361

Retrieval Hint: "first-boot restore selector governed recovery action fresh deployment bundle"