LearnNewsExamplesServices
Frontmatter
id15759
titlePersist orchestrator runtime state across cloud container recreation
stateClosed
labels
bugaitestingarchitecturebuild
assigneesneo-gpt-emmy
createdAtJul 23, 2026, 5:05 PM
updatedAtJul 23, 2026, 9:17 PM
githubUrlhttps://github.com/neomjs/neo/issues/15759
authorneo-gpt
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 23, 2026, 9:17 PM

Persist orchestrator runtime state across cloud container recreation

Closed Backlog/active-chunk-9 bugaitestingarchitecturebuild
neo-gpt
neo-gpt commented on Jul 23, 2026, 5:05 PM

Context

The canonical cloud Compose profile starts an orchestrator whose liveness, scheduler continuity, tenant-repo ingestion cursors, and recovery ledgers all depend on files below the orchestrator data directory. The current reference deployment persists SQLite, tenant-repo mirrors, Sandman handoff data, backup bundles, and model data, but it does not persist the orchestrator data directory itself.

Live latest-open sweep: checked the latest 20 open issues at 2026-07-23T15:05:16Z; no equivalent found. The recent all-state A2A claim sweep found no overlapping lane. Semantic and exact-history sweeps surfaced closed predecessors #11724, #11937, #12036, #11942, and #15604, but none added the missing orchestrator-state mount.

The Problem

ai/configBase.mjs declares orchestrator.dataDir as the owner of the daemon/task PID files, orchestrator.log, orchestrator-state.json, heavy-maintenance leases, tenant-repo revision state, and related recovery ledgers. TenantRepoSyncService stores tenant-repo-sync-revisions.json in that directory.

ai/deploy/docker-compose.yml nevertheless mounts no volume at /app/.neo-ai-data/orchestrator-daemon. A container recreate therefore discards:

  • task cadence and last-success/failure state;
  • tenant-repo lastIngestedRev, retry/backoff, and failure continuity;
  • recovery/heal/freeze ledgers;
  • diagnostic logs and process-supervision state.

The healthcheck itself reads /app/.neo-ai-data/orchestrator-daemon/orchestrator-state.json, so the reference topology relies on a file it leaves on the ephemeral container layer.

The documentation overclaims the shipped state: TenantIngestionModel.md says tenant-repo-sync-revisions.json is already backed up alongside other orchestrator state, and DeploymentCookbook.md says it survives a container restart. The canonical Compose file currently proves neither claim.

The Architectural Reality

  • AiConfig.orchestrator.dataDir is the path authority. Deployment wiring must bind that leaf to the mounted directory; consumers must continue reading the resolved leaf at use sites per ADR 0019.
  • The orchestrator is the sole owner/writer of this state. KB and MC containers do not need the volume.
  • Container-recreation persistence and off-host backup are separate claims. A named volume proves the first; the current logical backup bundle does not automatically prove the second.
  • Some files are durable continuity state, while PID/lease files may be process-epoch state that must be safely reconciled after restart. Persistence must not turn a stale PID or lease into a permanent startup block.

Structure-map ownership: existing ai/deploy/ Compose topology, ai/daemons/orchestrator/ state owners, and learn/agentos/cloud-deployment/ operator guides. No new subsystem or directory is required.

The Fix

  1. Add a dedicated named volume for the orchestrator data directory and mount it only into the orchestrator at /app/.neo-ai-data/orchestrator-daemon.
  2. Set NEO_AI_ORCHESTRATOR_DIR=/app/.neo-ai-data/orchestrator-daemon explicitly in the cloud profile so the AiConfig leaf, volume, and healthcheck share one path contract.
  3. Add a Compose contract test covering declaration, mount, sole-writer ownership, env binding, and healthcheck alignment.
  4. Add restart/recreate coverage for representative durable state, including orchestrator-state.json and tenant-repo-sync-revisions.json, plus stale PID/lease reconciliation.
  5. Correct the cloud guides and persistence table. State exactly what survives container recreation and do not call it off-host-backed-up unless the backup bundle is extended and proven separately.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback / Edge Case Docs Evidence
AiConfig.orchestrator.dataDir ADR 0019 + ai/configBase.mjs Resolve to the mounted cloud path through the existing env-bound leaf Local default remains unchanged Configuration + deployment cookbook Config/Compose contract test
Cloud orchestrator volume ai/deploy/docker-compose.yml Dedicated named volume mounted at /app/.neo-ai-data/orchestrator-daemon on orchestrator only Missing/unwritable mount fails readiness rather than silently using ephemeral storage Pipeline wiring persistence table Compose parse/static test + recreate witness
Task/revision/recovery state Existing orchestrator services Survive container recreation without losing continuity Stale PID/lease artifacts are reconciled safely at boot Tenant ingestion + recovery operations Restart fixtures for durable and process-epoch state
Backup claim Current logical backup bundle Documentation distinguishes named-volume durability from off-host backup No implication that a named volume is disaster recovery Backup/redeploy sections Doc assertion against the implemented bundle

Decision Record impact

Aligned with ADR 0014's cloud topology and ADR 0019's AiConfig Provider SSOT. No topology redraw is required; this repairs missing persistence wiring at the existing orchestrator boundary.

Acceptance Criteria

  • Canonical cloud Compose declares a dedicated orchestrator-state volume mounted only at /app/.neo-ai-data/orchestrator-daemon on the orchestrator.
  • NEO_AI_ORCHESTRATOR_DIR explicitly names that mount, and the liveness healthcheck resolves the same path contract.
  • orchestrator-state.json, tenant-repo-sync-revisions.json, retry/backoff state, and representative recovery ledger state survive a container recreate.
  • Persisted stale PID and lease files do not permanently block a restarted orchestrator; their existing stale-recovery semantics are covered.
  • Compose contract tests assert the volume declaration, mount, sole-writer ownership, env binding, and healthcheck alignment.
  • Cloud persistence docs distinguish container-recreation durability from off-host backup and remove the unsupported “already backed up” claim.
  • A live or CI container-recreate witness records representative before/after state hashes or values without exposing tenant identities or secrets.

Out of Scope

  • Persisting tenant-repo mirror contents beyond their existing cache volume.
  • Redesigning orchestrator state formats.
  • Treating runtime PID files as disaster-recovery artifacts.
  • Adding remote shell, Docker exec, or MCP mutation tools.
  • Changing deployment version/update authority.

Avoided Traps

  • Do not mount the whole .neo-ai-data tree into every service; preserve ownership and least privilege.
  • Do not hardcode a second path outside AiConfig or re-read the env var in consumers.
  • Do not call a Docker named volume an off-host backup.
  • Do not solve stale process artifacts by deleting the entire state directory on every boot; that recreates the continuity loss this ticket fixes.

Related

  • #11724 — broad redeploy-safe persistence predecessor; shipped backup/TLS persistence, not this state mount.
  • #11937 / PR #11939 — orchestrator healthcheck reads the state file but does not persist it.
  • #11942, #12036 — tenant-repo revision state and mirror-root lineage.
  • #15604 / PR #15753 — persists the Sandman handoff only.
  • #15748 / PR #15752 — fail-closed tenant-repo checkpoint behavior.

Origin Session ID: fc1a49c1-e30a-4e3a-960a-e0596367a4c1

Handoff Retrieval Hint: canonical cloud compose orchestrator state volume tenant-repo-sync-revisions container recreate persistence