A rebuild reports success after recreating half the plane

Context

Measured on the local Agent OS plane, 2026-08-19T12:44Z, while rebuilding it from a7b58e6a (2026-08-17) to dev's head 1bc70b56e1.

After docker compose -f docker-compose.yml -f docker-compose.local-agent-os.yml up -d --force-recreate --wait exited 0 and printed Healthy for everything it touched:

All four reported running healthy throughout. Half the plane stayed two days stale behind an exit-0 recreate, and the only reason it was noticed is that the revision was read per container from the image labels instead of trusted from the exit code.

Separating observation from inference: the revisions, the split, and the exit code are observed; the cause is read from the compose files (profiles) rather than inferred from behaviour.

The Problem

fleet-server declares profiles: [fleet] (ai/deploy/docker-compose.yml:632) and orchestrator declares profiles: [cloud] (:518). docker compose up without --profile omits profiled services from the set it operates on, and leaves already-running containers of those services alone. So the command completes successfully having recreated a strict subset of the plane, with nothing in its output naming what it skipped.

The working invocation needed the profiles of whatever happens to be running:

docker compose --profile cloud --profile fleet … up -d --force-recreate --wait fleet-server orchestrator

Nothing at the point of use says so, and the required set is not a constant — it is a function of the live stack, which the operator has to discover by listing containers and cross-referencing profile declarations across two compose files.

Why this is worse than a documentation gap. The failure returns success and the post-conditions look right: every service healthy, every image freshly built. A rebuild is exactly the moment someone believes the plane is current, so a half-rebuild converts an ordinary staleness problem into a false belief that it was already handled. The plane in question had been stale for two days and the substrate already records a prior instance of the same class — ai/deploy/docker-compose.local-agent-os.yml, above the auth secret: "every container running a revision 103 commits behind dev while the fixes sat merged."

What this ticket is NOT, because both were checked and discarded rather than assumed:

• Not the mutable-ref cache hazard. That is #16635, closed COMPLETED — ai/deploy/Dockerfile:20/:51 now refuse a non-SHA NEO_REF unless NEO_ALLOW_MUTABLE_REF=1. An unpinned build fails loudly instead of silently reusing the cached source layer. Verified present in the tree; the guard is working.
• Not a reporting gap. healthcheck publishes deployedRevision (observed 1bc70b56e13488… post-rebuild), and runtimeFreshness is honest about its scope — it emitted "Runtime config/schema identity matches the current checkout", and RuntimeFreshnessService excludes gitHead from status-driving fields deliberately, with a guard preventing a current verdict from claiming a source dimension it never measured. A draft of this ticket targeted that field; reading the module removed the claim.

The residual is narrow and specific: no single command rebuilds the whole running plane, and the one that looks like it does reports success after covering part of it.

The Architectural Reality

• ai/deploy/docker-compose.yml:518 — orchestrator, profiles: [cloud].
• ai/deploy/docker-compose.yml:632 — fleet-server, profiles: [fleet].
• ai/deploy/docker-compose.yml:666 / :711 — ingress (profiles: [ingress]) and local-model, the same shape; ingress was running during this incident and was likewise skipped, though it is caddy:2-alpine and carries no neo revision.
• ai/deploy/docker-compose.local-agent-os.yml:8 — name: "${NEO_LOCAL_AGENT_OS_PROJECT_NAME:-neo-local-agent-os}". The stable project name is why a rebuild from any checkout targets the same stack, and therefore why a whole-plane entrypoint is expressible at all.
• Image labels org.neomjs.image.requested-ref / org.opencontainers.image.revision — stamped at build, and the authority a per-container check reads. Emitted by #15775.

The Fix

Direction rather than prescription; the cheapest correct option depends on how much the deploy path should own.

Option A — a documented whole-plane entrypoint that derives the profile set from the containers currently running in the project, so the invocation cannot drift from the live stack. Removes the discovery step entirely.

Option B — a post-recreate assertion. After a rebuild, compare each running neo service's stamped revision against the revision just built and fail if any differs. Catches this class regardless of why a service was missed, including future profiles nobody has added yet, and matches the fail-closed posture #16635 established one layer down.

B is the stronger backstop because it does not enumerate the failure; A is the better ergonomics. They compose.

Contract Ledger Matrix

Decision Record impact

none.

Acceptance Criteria

• A documented rebuild path leaves no running neo service on a revision other than the one just built, with a profiled service running at the time.
• Red-proof: against the current tree, a probe that runs the documented rebuild and then asserts revision parity across every running neo service must FAIL, reproducing the 2-of-4 split. If it passes on main, it is not exercising the skip.
• Control: the same probe with a plane whose profiled services are not running must pass, so the arm cannot be satisfied by a rebuild that simply stops everything.
• Any assertion is on the revision each running container reports, never on command exit status — both the observed failure and its sibling in #16635 returned success.
• Negative control: a plane already at the built revision reports parity and raises nothing, so the check cannot become a permanently-firing alarm that gets muted.
• The deploy docs state that up omits profiled services, at the point where the rebuild is described.

Out of Scope

• #16635's mutable-ref cache hazard — closed COMPLETED and verified fixed in the tree.
• runtimeFreshness and the deliberate gitHead exclusion — correct as designed.
• Publishing a revision in the deployment-state snapshot — healthcheck already carries deployedRevision, and #16791 covered that surface.
• Automatic rebuilds or deploy-on-merge. This makes a partial rebuild impossible to mistake for a whole one; deciding when to deploy stays human.
• inspect_deployment returning 104 KB, over the tool-result cap — a real adjacent defect, separately ticketable.

Avoided Traps

• Trusting an exit code over a per-container read. Both this and #16635 return success. An AC written against exit status measures the wrong thing.
• Filing the cache hazard again. It is #16635, closed COMPLETED, and the guard is in the tree. My rebuild succeeded because I passed an explicit SHA; unpinned would have failed loudly, which is the fix working.
• Filing against a field name. A draft targeted runtimeFreshness for reporting current on a stale plane. It reported current for config/schema, which is what it measures and what it said.
• Enumerating the profiles as the fix. Naming cloud and fleet in a command fixes today's two and silently excludes the third profile someone adds later. Option B is preferred because it asserts the property rather than the list.

Related

• #16635 — the mutable-ref cache hazard, closed COMPLETED; the sibling failure mode in the same rebuild path, and the precedent for failing closed.
• #15774 / #15775 — made the packaged revision observable; this consumes those labels.
• #15787 — the open deploy-provenance holder.
• #16791 — the deployment-state snapshot's revision surface, closed.

Origin Session ID: fb387768-e68f-4a71-9b6a-3cf9ad4a9e7e

Retrieval Hint: query_raw_memories("docker compose up skips profiled services fleet cloud profile rebuild reports success half plane stale")