Surfaced while running a Claude-family divergence pass on Discussion #15758 (Agent OS cloud rollout authority) — see discussioncomment-17760990. Euclid's filing note flagged that Compose "does not forward NEO_REF"; re-verifying at origin/dev@6a172b90bb (98 non-merge commits after the filing head) showed the gap is a severed chain at both ends, and that it blocks the deployment motion currently in flight rather than only the future rollout design.
The lead-role priority map for 2026-07-24 contracts P0 as "the tenant deployment must CONSUME the fixes: exact-SHA rebuild → redeploy → RECEIPTS", on Discussion #15758's Option E floor (manual exact-SHA + stronger receipts). Option E cannot express its own minimum today. This ticket is the strict prerequisite that makes that floor performable and its receipts falsifiable, without pre-empting any open rollout-authority question.
This is the unclosed tail of #12150, which delivered pinned source acquisition but — per Discussion #15758's own prior-art sweep — left "the mutable-ref/cache edge" open and version promotion as operator configuration.
The Problem
A cloud stack can run stale code while every container reports healthy, and a rebuild/redeploy cycle can claim success without proving which source revision each service actually runs. The #15748–#15768 chain fixed four such alternate-reality defects (WAL drain defaulted off in cloud mode, error-bearing ingest advancing revisions, orchestrator state lost on recreate, runtime access bound to the wrong Compose project). Those were code and config defects — but the reason they survived in production undetected is the missing evidence half: nothing in the deployed artifact states what it is.
Two independent breaks, verified at 6a172b90bb:
(1) The revision cannot enter through the declarative path. All three Neo build blocks in ai/deploy/docker-compose.ymlalready carry an args: map — the plumbing exists and NEO_REF was simply omitted from every one:
docker compose build --build-arg NEO_REF=<sha> does reach the builds, so an exact-SHA rebuild is performable — but only from an operator's shell. It cannot live in the versioned desired state that learn/agentos/cloud-deployment/PipelineWiring.md recommends, and that file currently contains no mention of NEO_REF or --build-arg at all (verified by grep).
(2) The resolved revision cannot leave in the artifact.ai/deploy/Dockerfile documents NEO_REF as the reproducibility pin (Dockerfile:8), declares it (Dockerfile:11-13), and resolves it in the source-git stage via git fetch + git checkout --detach FETCH_HEAD (Dockerfile:19-23). But grep -c LABEL ai/deploy/Dockerfile → 0. No org.opencontainers.image.revision, no revision metadata of any kind on the final image. The resolved commit is known inside the build and then discarded.
Net: the Dockerfile's own documented contract — "pin to a tag/SHA for full reproducibility" — is unreachable from the deployment entrypoint, and even a correctly pinned build produces an artifact that cannot state its own identity.
The Architectural Reality
ai/deploy/Dockerfile:11-13 — ARG NEO_SOURCE=git / ARG NEO_REF=dev / ARG NEO_REPO_URL, declared before the first FROM, so they are available to FROM lines and to stages that re-declare them. The final stage (Dockerfile:53) does not re-declare NEO_REF, so a LABEL there needs an explicit ARG NEO_REF.
ai/deploy/Dockerfile:19-23 — source-git resolves the ref to a concrete commit; git rev-parse HEAD is available at that point. This is the only place the resolved SHA exists.
ai/deploy/Dockerfile:31 — FROM source-${NEO_SOURCE} AS source selects git vs local. The source-local stage (:25-27) is a bare COPY . . into node:24-alpine, which has no git binary — so a local build cannot resolve a SHA and must stamp an honest non-revision marker instead.
ai/deploy/Dockerfile:47 (COPY --from=source /neo ./) then :61 (COPY --from=builder /app ./) — anything written into /neo in the source stage flows to /app in the final image with no additional copy step.
ai/deploy/docker-compose.yml is the only compose file that builds from this Dockerfile (docker-compose.dev.yml and docker-compose.test.yml do not), so the change surface is bounded to one build path.
Mutable-dev requests are the dangerous case: Docker does not automatically invalidate a RUN layer when remote content changes, so re-running --build does not mechanically prove the branch was re-fetched. A requested ref and a resolved commit are therefore two distinct facts, and the artifact must be able to carry both.
The Fix
ai/deploy/docker-compose.yml — add NEO_REF: ${NEO_REF:-dev} to the build.args map of all three Neo services. The :-dev default is load-bearing: a bare ${NEO_REF} with the variable unset passes an empty string, which would override the Dockerfile's ARG NEO_REF=dev default and break git fetch origin "". With the default, an unset environment reproduces today's behaviour exactly.
ai/deploy/Dockerfile — stamp both facts:
final stage: ARG NEO_REF + LABEL org.opencontainers.image.revision="${NEO_REF}" (the requested identity; exact whenever the caller pins a SHA, which is what Option E's floor requires) plus LABEL org.opencontainers.image.source for the repo URL.
source-git: write the resolved commit to /neo/.neo-revision (git rev-parse HEAD), which reaches /app/.neo-revision in the final image via the existing copies. source-local: write an honest non-revision marker so the file is never absent and never mistakable for a real commit.
learn/agentos/cloud-deployment/PipelineWiring.md — document the exact-SHA invocation and the two verification commands, so the receipt is reproducible by a human operator without reading this ticket.
Contract Ledger Matrix
Target Surface
Source of Authority
Proposed Behavior
Fallback
Docs
Evidence
build.args.NEO_REF on kb-server / mc-server / orchestrator
forwards ${NEO_REVISION:-} to all three builds so the pipeline can resolve-once-then-pin
unset ⇒ empty, no revision asserted
PipelineWiring.md
docker compose config | grep NEO_REVISION
/app/.neo-revision
new file; ai/deploy/Dockerfile:19-23 is where the resolved commit exists
records the resolved commit (git mode) or an explicit local marker
file absent ⇒ pre-fix image
PipelineWiring.md
docker exec <svc> cat /app/.neo-revision
[AMENDED 2026-07-24 per PR #15776 cycle-1 review — @neo-gpt-emmy] The original prescription put the requested ref in org.opencontainers.image.revision. That is wrong: the OCI spec defines that key as the source-control revision of the packaged software, and D#15758's own framing makes a mutable channel policy input, never build identity. The shipped design therefore uses three non-overlapping surfaces — requested selector (vendor label), packaged revision (OCI label, caller-supplied, empty when not asserted), and the build-time resolved commit (/app/.neo-revision, always true and therefore primary. Requested-vs-resolved stays the diagnostic pair; the standard field no longer carries a fact it does not hold.
Decision Record impact
none — deliberately. This ticket adds build-arg plumbing and artifact metadata only. It does not choose a rollout authority, add a request surface, or widen runtime access, so it touches neither ADR-0014, ADR-0019, nor ADR-0026 authority. Discussion #15758's OQ1 (authority home) and OQ4 (which Neo diagnostic exposes provenance) remain fully open, and every row of its divergence matrix — including the do-nothing-automated row E — depends on this plumbing existing.
Acceptance Criteria
All three Neo build blocks in ai/deploy/docker-compose.yml declare NEO_REF: ${NEO_REF:-dev}.
With NEO_REF unset, docker compose config renders dev for all three services — proving the no-op default and that no existing deployment changes behaviour.
With NEO_REF=<full-sha> exported, docker compose config renders that SHA for all three services.
All three Neo build blocks also declare NEO_REVISION: ${NEO_REVISION:-}; with it unset, docker compose config renders an empty value for all three (no revision asserted).
ai/deploy/Dockerfile final stage emits org.neomjs.image.requested-ref, org.opencontainers.image.revision, and org.opencontainers.image.source.
[~] AC5[L3-partial — empty-case field-verified; SHA-case pending] — a built image reports the resolved revision in org.opencontainers.image.revision when NEO_REVISION is passed, and an empty value when it is not (never a channel name). Empty half field-verified 2026-07-24 (operator rehearsal run relayed by @neo-fable-clio): with NEO_REVISION unset the label read empty — the designed not-asserted state, not a defect. SHA half awaits the two-arg invocation, now with the operator.
AC6[L3-SATISFIED — field receipt 2026-07-24] — /app/.neo-revision exists in the built image and holds the resolved commit SHA in git mode. Verified on a real build: the read returned the exact 40-char SHA 3b36c2a323327125d5828478d88f700d95547b4e. Operator rehearsal run, relayed by @neo-fable-clio. The primary provenance fact passed on its first real-world read.
AC7[L3-deferred — operator handoff needed] — NEO_SOURCE=local builds succeed, stamp local-build (never empty, never a fabricated SHA), and leave org.opencontainers.image.revision empty.
learn/agentos/cloud-deployment/PipelineWiring.md documents the exact-SHA build invocation plus both verification commands.
AC9[L3-deferred — operator handoff needed] — an operator-side rebuild of the live tenant deployment produces a revision label matching the deployed SHA. Post-merge because it needs the deployment plane.
Deferral basis (Evidence Ladder). AC5/AC6/AC7/AC9 each require a real image build. No Docker daemon is reachable from the authoring sandbox, and CI's integration lane builds ai/deploy/docker-compose.test.yml, which does not use this Dockerfile — so exact-head CI green is a no-regression signal, never evidence for these four. @neo-fable-clio independently probed the host (colima stopped), confirming no agent sandbox can produce the receipt, and routed it operator-side. Owner: operator plane.
Out of Scope
Making a Neo MCP surface read the label.summarizeInspect (ai/daemons/orchestrator/services/DeploymentStateBridgeService.mjs:732-747) reduces Docker inspect to a configured image-name string plus process health, dropping image ID, repo digests, and labels — so the stamped label will not appear in the deployment snapshot until that is changed. Deliberately deferred: Discussion #15758OQ4 explicitly owns "which read-only Neo diagnostic exposes it," and the P0 verification path (docker inspect / docker exec) needs no Neo-surface change. Pre-empting OQ4 here would put an in-flight design decision inside a prerequisite PR.
Cohort identity, manifests, and desired/observed reconciliation. Divergence-window material on #15758.
An out-of-cohort evidence sink. Proposed there as Option H; the evidence reader currently living inside the update cohort is a separate structural problem.
Any rollout trigger, schedule, controller, or request surface. OQ1/OQ7 territory.
Rollback and state-safe recovery semantics. OQ5 / Option G territory.
docker-compose.dev.yml / docker-compose.test.yml — neither builds from this Dockerfile.
Avoided Traps
Labelling with the raw NEO_REF and calling it the deployed identity. Discussion #15758 names "passing mutable dev directly as the deployed identity" an invalid shape, and it is right. This ticket named that trap and then walked into it — the original prescription put the requested selector into org.opencontainers.image.revision, whose spec-defined meaning is the deployed identity. Caught by @neo-gpt-emmy at PR #15776 cycle 1, with the decisive instance being a NEO_SOURCE=local build labelling itself dev while /app/.neo-revision said local-build — the artifact contradicting itself. Recorded rather than silently corrected: getting the principle right is not the same as assigning it to the right field.
Deriving the LABEL value from the resolved commit. Tempting and not expressible: a LABEL needs a build ARG, and Docker cannot populate an ARG in a later stage from a file written in an earlier one. Attempting it would require buildx-specific machinery for a value the file already carries.
A bare ${NEO_REF} in compose. Passes an empty string when unset, overriding the Dockerfile default and breaking the fetch. The :-dev default is the whole safety margin on a live P0 deployment surface.
Shelling out to git in source-local.node:24-alpine has no git; and inferring a revision from a local working tree would fabricate provenance for a dev-iteration escape hatch. An explicit marker is the honest output.
Bundling the diagnostic read path to make the feature feel complete. That would import an open OQ into a prerequisite, and is the reason this ticket stays narrow.
Live latest-open sweep: checked latest 20 open issues at 2026-07-24T07:15:24Z; no equivalent found. A2A in-flight claim sweep (30 messages, all read-states) at 2026-07-24T07:15:33Z; no overlapping [lane-claim]/[lane-intent]. Structure-map gate: no new .mjs file and no relocation — both touched files are existing members of ai/deploy/, which owns deployment topology.
Context
Surfaced while running a Claude-family divergence pass on Discussion #15758 (Agent OS cloud rollout authority) — see discussioncomment-17760990. Euclid's filing note flagged that Compose "does not forward
NEO_REF"; re-verifying atorigin/dev@6a172b90bb(98 non-merge commits after the filing head) showed the gap is a severed chain at both ends, and that it blocks the deployment motion currently in flight rather than only the future rollout design.The lead-role priority map for 2026-07-24 contracts P0 as "the tenant deployment must CONSUME the fixes: exact-SHA rebuild → redeploy → RECEIPTS", on Discussion #15758's Option E floor (manual exact-SHA + stronger receipts). Option E cannot express its own minimum today. This ticket is the strict prerequisite that makes that floor performable and its receipts falsifiable, without pre-empting any open rollout-authority question.
This is the unclosed tail of
#12150, which delivered pinned source acquisition but — per Discussion #15758's own prior-art sweep — left "the mutable-ref/cache edge" open and version promotion as operator configuration.The Problem
A cloud stack can run stale code while every container reports healthy, and a rebuild/redeploy cycle can claim success without proving which source revision each service actually runs. The
#15748–#15768chain fixed four such alternate-reality defects (WAL drain defaulted off in cloud mode, error-bearing ingest advancing revisions, orchestrator state lost on recreate, runtime access bound to the wrong Compose project). Those were code and config defects — but the reason they survived in production undetected is the missing evidence half: nothing in the deployed artifact states what it is.Two independent breaks, verified at
6a172b90bb:(1) The revision cannot enter through the declarative path. All three Neo build blocks in
ai/deploy/docker-compose.ymlalready carry anargs:map — the plumbing exists andNEO_REFwas simply omitted from every one:build.argspresentNEO_REFkb-server(docker-compose.yml:50-54)TARGET_SERVER: knowledge-basemc-server(docker-compose.yml:107-111)TARGET_SERVER: memory-coreorchestrator(docker-compose.yml:184-188)SERVICE_ENTRYPOINT: ai/daemons/orchestrator/daemon.mjsdocker compose build --build-arg NEO_REF=<sha>does reach the builds, so an exact-SHA rebuild is performable — but only from an operator's shell. It cannot live in the versioned desired state thatlearn/agentos/cloud-deployment/PipelineWiring.mdrecommends, and that file currently contains no mention ofNEO_REFor--build-argat all (verified by grep).(2) The resolved revision cannot leave in the artifact.
ai/deploy/DockerfiledocumentsNEO_REFas the reproducibility pin (Dockerfile:8), declares it (Dockerfile:11-13), and resolves it in thesource-gitstage viagit fetch+git checkout --detach FETCH_HEAD(Dockerfile:19-23). Butgrep -c LABEL ai/deploy/Dockerfile→ 0. Noorg.opencontainers.image.revision, no revision metadata of any kind on the final image. The resolved commit is known inside the build and then discarded.Net: the Dockerfile's own documented contract — "pin to a tag/SHA for full reproducibility" — is unreachable from the deployment entrypoint, and even a correctly pinned build produces an artifact that cannot state its own identity.
The Architectural Reality
ai/deploy/Dockerfile:11-13—ARG NEO_SOURCE=git/ARG NEO_REF=dev/ARG NEO_REPO_URL, declared before the firstFROM, so they are available toFROMlines and to stages that re-declare them. The final stage (Dockerfile:53) does not re-declareNEO_REF, so aLABELthere needs an explicitARG NEO_REF.ai/deploy/Dockerfile:19-23—source-gitresolves the ref to a concrete commit;git rev-parse HEADis available at that point. This is the only place the resolved SHA exists.ai/deploy/Dockerfile:31—FROM source-${NEO_SOURCE} AS sourceselects git vs local. Thesource-localstage (:25-27) is a bareCOPY . .intonode:24-alpine, which has nogitbinary — so a local build cannot resolve a SHA and must stamp an honest non-revision marker instead.ai/deploy/Dockerfile:47(COPY --from=source /neo ./) then:61(COPY --from=builder /app ./) — anything written into/neoin the source stage flows to/appin the final image with no additional copy step.ai/deploy/docker-compose.ymlis the only compose file that builds from this Dockerfile (docker-compose.dev.ymlanddocker-compose.test.ymldo not), so the change surface is bounded to one build path.devrequests are the dangerous case: Docker does not automatically invalidate aRUNlayer when remote content changes, so re-running--builddoes not mechanically prove the branch was re-fetched. A requested ref and a resolved commit are therefore two distinct facts, and the artifact must be able to carry both.The Fix
ai/deploy/docker-compose.yml— addNEO_REF: ${NEO_REF:-dev}to thebuild.argsmap of all three Neo services. The:-devdefault is load-bearing: a bare${NEO_REF}with the variable unset passes an empty string, which would override the Dockerfile'sARG NEO_REF=devdefault and breakgit fetch origin "". With the default, an unset environment reproduces today's behaviour exactly.ai/deploy/Dockerfile— stamp both facts:ARG NEO_REF+LABEL org.opencontainers.image.revision="${NEO_REF}"(the requested identity; exact whenever the caller pins a SHA, which is what Option E's floor requires) plusLABEL org.opencontainers.image.sourcefor the repo URL.source-git: write the resolved commit to/neo/.neo-revision(git rev-parse HEAD), which reaches/app/.neo-revisionin the final image via the existing copies.source-local: write an honest non-revision marker so the file is never absent and never mistakable for a real commit.learn/agentos/cloud-deployment/PipelineWiring.md— document the exact-SHA invocation and the two verification commands, so the receipt is reproducible by a human operator without reading this ticket.Contract Ledger Matrix
build.args.NEO_REFonkb-server/mc-server/orchestratorai/deploy/docker-compose.yml:50-54,107-111,184-188(args maps exist; key absent)${NEO_REF:-dev}into each builddev, byte-identical to current behaviourgit diff+docker compose configrenderorg.neomjs.image.requested-reflabeldocker inspect --format '{{index .Config.Labels "org.neomjs.image.requested-ref"}}'org.opencontainers.image.revisionlabelgrep -c LABEL ai/deploy/Dockerfile→ 0 todayNEO_REVISION; empty = not asserted, never a channel namedocker inspect --format '{{index .Config.Labels "org.opencontainers.image.revision"}}'NEO_REVISIONbuild argai/deploy/docker-compose.ymlbuild-args maps${NEO_REVISION:-}to all three builds so the pipeline can resolve-once-then-pindocker compose config | grep NEO_REVISION/app/.neo-revisionai/deploy/Dockerfile:19-23is where the resolved commit existsdocker exec <svc> cat /app/.neo-revision[AMENDED 2026-07-24 per PR #15776 cycle-1 review — @neo-gpt-emmy] The original prescription put the requested ref in
org.opencontainers.image.revision. That is wrong: the OCI spec defines that key as the source-control revision of the packaged software, and D#15758's own framing makes a mutable channel policy input, never build identity. The shipped design therefore uses three non-overlapping surfaces — requested selector (vendor label), packaged revision (OCI label, caller-supplied, empty when not asserted), and the build-time resolved commit (/app/.neo-revision, always true and therefore primary. Requested-vs-resolved stays the diagnostic pair; the standard field no longer carries a fact it does not hold.Decision Record impact
none— deliberately. This ticket adds build-arg plumbing and artifact metadata only. It does not choose a rollout authority, add a request surface, or widen runtime access, so it touches neither ADR-0014, ADR-0019, nor ADR-0026 authority. Discussion #15758's OQ1 (authority home) and OQ4 (which Neo diagnostic exposes provenance) remain fully open, and every row of its divergence matrix — including the do-nothing-automated row E — depends on this plumbing existing.Acceptance Criteria
ai/deploy/docker-compose.ymldeclareNEO_REF: ${NEO_REF:-dev}.NEO_REFunset,docker compose configrendersdevfor all three services — proving the no-op default and that no existing deployment changes behaviour.NEO_REF=<full-sha>exported,docker compose configrenders that SHA for all three services.NEO_REVISION: ${NEO_REVISION:-}; with it unset,docker compose configrenders an empty value for all three (no revision asserted).ai/deploy/Dockerfilefinal stage emitsorg.neomjs.image.requested-ref,org.opencontainers.image.revision, andorg.opencontainers.image.source.[L3-partial — empty-case field-verified; SHA-case pending]— a built image reports the resolved revision inorg.opencontainers.image.revisionwhenNEO_REVISIONis passed, and an empty value when it is not (never a channel name). Empty half field-verified 2026-07-24 (operator rehearsal run relayed by @neo-fable-clio): withNEO_REVISIONunset the label read empty — the designed not-asserted state, not a defect. SHA half awaits the two-arg invocation, now with the operator.[L3-SATISFIED — field receipt 2026-07-24]—/app/.neo-revisionexists in the built image and holds the resolved commit SHA ingitmode. Verified on a real build: the read returned the exact 40-char SHA3b36c2a323327125d5828478d88f700d95547b4e. Operator rehearsal run, relayed by @neo-fable-clio. The primary provenance fact passed on its first real-world read.[L3-deferred — operator handoff needed]—NEO_SOURCE=localbuilds succeed, stamplocal-build(never empty, never a fabricated SHA), and leaveorg.opencontainers.image.revisionempty.learn/agentos/cloud-deployment/PipelineWiring.mddocuments the exact-SHA build invocation plus both verification commands.[L3-deferred — operator handoff needed]— an operator-side rebuild of the live tenant deployment produces a revision label matching the deployed SHA. Post-merge because it needs the deployment plane.Deferral basis (Evidence Ladder). AC5/AC6/AC7/AC9 each require a real image build. No Docker daemon is reachable from the authoring sandbox, and CI's integration lane builds
ai/deploy/docker-compose.test.yml, which does not use this Dockerfile — so exact-head CI green is a no-regression signal, never evidence for these four. @neo-fable-clio independently probed the host (colima stopped), confirming no agent sandbox can produce the receipt, and routed it operator-side. Owner: operator plane.Out of Scope
summarizeInspect(ai/daemons/orchestrator/services/DeploymentStateBridgeService.mjs:732-747) reduces Docker inspect to a configured image-name string plus process health, dropping image ID, repo digests, and labels — so the stamped label will not appear in the deployment snapshot until that is changed. Deliberately deferred: Discussion #15758 OQ4 explicitly owns "which read-only Neo diagnostic exposes it," and the P0 verification path (docker inspect/docker exec) needs no Neo-surface change. Pre-empting OQ4 here would put an in-flight design decision inside a prerequisite PR.docker-compose.dev.yml/docker-compose.test.yml— neither builds from this Dockerfile.Avoided Traps
NEO_REFand calling it the deployed identity. Discussion #15758 names "passing mutabledevdirectly as the deployed identity" an invalid shape, and it is right. This ticket named that trap and then walked into it — the original prescription put the requested selector intoorg.opencontainers.image.revision, whose spec-defined meaning is the deployed identity. Caught by @neo-gpt-emmy at PR #15776 cycle 1, with the decisive instance being aNEO_SOURCE=localbuild labelling itselfdevwhile/app/.neo-revisionsaidlocal-build— the artifact contradicting itself. Recorded rather than silently corrected: getting the principle right is not the same as assigning it to the right field.LABELvalue from the resolved commit. Tempting and not expressible: aLABELneeds a buildARG, and Docker cannot populate anARGin a later stage from a file written in an earlier one. Attempting it would require buildx-specific machinery for a value the file already carries.${NEO_REF}in compose. Passes an empty string when unset, overriding the Dockerfile default and breaking the fetch. The:-devdefault is the whole safety margin on a live P0 deployment surface.gitinsource-local.node:24-alpinehas nogit; and inferring a revision from a local working tree would fabricate provenance for a dev-iteration escape hatch. An explicit marker is the honest output.Related
Related: #12150 Related: #11733 Related: #15762 Related: #15767 Related: #15749 Related: Discussion #15758 Related: Discussion #15595
Live latest-open sweep: checked latest 20 open issues at 2026-07-24T07:15:24Z; no equivalent found. A2A in-flight claim sweep (30 messages, all read-states) at 2026-07-24T07:15:33Z; no overlapping
[lane-claim]/[lane-intent]. Structure-map gate: no new.mjsfile and no relocation — both touched files are existing members ofai/deploy/, which owns deployment topology.Origin Session ID: 92799c10-cb3b-4c01-a2b0-fd8552c3c02e
Retrieval Hint:
query_raw_memories("NEO_REF compose build arg OCI revision label deployed provenance"); Discussion #15758 commentdiscussioncomment-17760990; verified head6a172b90bb.