Context
ai/deploy/Dockerfile:12 ships ARG NEO_REF=dev as the default, and documents the alternative at :8 as:
NEO_REF : git ref to clone (default 'dev'; pin to a tag/SHA for full reproducibility)
That framing says a SHA buys reproducibility. It also buys freshness, and without it there is none — a branch ref makes the source layer cache-stable, so the image silently packages whatever that branch pointed at the first time the layer was built, forever.
Surfaced by an operator who had already worked this out empirically on a live deployment: "dev is a bit risky, since it will not auto-update on future updates then. this was the reason that we so far used explicit SHA values." Filed because the file still recommends the opposite reason, so the next reader learns it the same expensive way.
The Problem
The source stage is one RUN (ai/deploy/Dockerfile:20-23):
ARG NEO_REF
RUN git init -q && git remote add origin "${NEO_REPO_URL}" \
&& git fetch --depth 1 origin "${NEO_REF}" \
&& git checkout -q --detach FETCH_HEAD \
&& git rev-parse HEAD > /neo/.neo-revisionA RUN layer's cache key is its command string after ARG substitution. With NEO_REF=dev that string is byte-identical on every build, so the layer is reused and the fetch never executes again. With a SHA the arg changes whenever the code changes, which busts the cache by construction.
So the two options are not "pinned versus tracking". They are pinned deliberately versus pinned accidentally, at an arbitrary past commit, with a build log that reads as though it fetched.
Demonstrated
Minimal reproduction of the cache-key property, the RUN standing in for the fetch:
FROM alpine:3.20
ARG NEO_REF=dev
RUN echo "fetch ${NEO_REF}" && head -c6 /dev/urandom | od -An -tx1 | tr -d ' \n' > /what-we-gotbuild 1, NEO_REF=dev -> e5267c5a4fab
build 2, NEO_REF=dev -> e5267c5a4fab <- identical: the layer never re-ran
build 3, NEO_REF=<sha1> -> 35240ad7c0a1
build 4, NEO_REF=<sha2> -> 57efc8e99f2b <- changed arg busts the cache
Docker 29.2.1. Two dev builds produce the same bytes from a command that cannot produce the same bytes twice if it runs.
The Architectural Reality
ai/deploy/Dockerfile:8 — the comment that makes the branch default look safe, by naming reproducibility as the only thing a SHA adds.ai/deploy/Dockerfile:12 — ARG NEO_REF=dev. A mutable channel as the default value for a cache-keyed arg.ai/deploy/Dockerfile:20-23 — the single cache-keyed RUN.ai/deploy/Dockerfile:~112-140 — NEO_REVISION is the integrity assertion and is empty by default, so the one mechanism that would catch a stale package is off in exactly the configuration that produces one. /app/.neo-revision still records the truth, but nothing compares it to an expectation unless a caller supplies one.
The provenance work in #15774 / #15775 made the packaged revision observable. This is the adjacent gap: observable is not asserted, and the default configuration asserts nothing.
The Fix
Direction, not prescription — the cheapest correct option depends on how much the deploy pipeline should own.
Option A — fix the comment only. State that a mutable ref freezes the layer and that a SHA is required for a build to fetch at all. One line, zero behaviour change, and it stops the misreading. Weakest, because the default still does the wrong thing.
Option B — make the mutable-ref case fail closed. Refuse a non-SHA NEO_REF unless an explicit opt-in arg is passed (NEO_ALLOW_MUTABLE_REF=1). Matches the fail-closed posture the plane and blobless-clone guards already use: the guard fires on the shape that is silently wrong, and the escape hatch is explicit.
Option C — resolve the channel at build entry. Have the compose/pipeline layer resolve dev to a full SHA before invoking the build, so the mutable name never reaches a cache-keyed arg. Strictly better provenance, and it makes NEO_REVISION populated by default — but it moves work into every caller.
B and C compose: C for the pipeline, B as the backstop for a hand-run docker build.
Acceptance Criteria
Out of Scope
#15774 / #15775 — provenance emission. This consumes their output rather than changing it.#15792 — the reference pipeline building unpinned. Adjacent and probably fixed by Option C, but that ticket is about the pipeline's own pinning, not about the cache-key property of a mutable ref.- Remediating any specific deployment. That is operator-side.
Avoided Traps
- Reading the comment as the contract. The file says a SHA is for reproducibility, which is true and incomplete, and the incompleteness is invisible because the sentence is correct as far as it goes. I repeated the same wrong framing into a deployment guide before this was pointed out.
- Believing the build log. A cached layer prints its command. Nothing distinguishes "fetched and got the same commit" from "did not fetch" without comparing
/app/.neo-revision across builds.
- Assuming
--no-cache covers it. It does, and it is a per-invocation flag in a different document from the one that recommends the branch default. A claim in one step being true only because of a flag in another step is how this stayed plausible.
- Testing with a SHA. Any test that pins a SHA cache-busts by construction and cannot reproduce the defect. The failing case requires an unchanged mutable ref across two builds.
Related
#15774 (deployed images cannot prove their revision) · #15775 (forward NEO_REF, emit OCI revision label) · #15792 (reference pipeline builds unpinned) · #15782 (abbreviated-SHA fetch failure in the guide) · #16206 (ai/deploy composable layering).
Live latest-open sweep: checked the latest 20 open issues at 2026-08-07T14:03:32Z; no equivalent found. A2A in-flight claim sweep over the last 30 messages: no competing lane-claim on this scope.
Origin Session ID: 9ced67a1-8f21-4da2-a1bf-a2a968c47ed2
Retrieval Hint: query_raw_memories("NEO_REF branch default docker layer cache freeze source stage never refetches stale image")
Retrieval Hint: the discriminating measurement is two builds with an unchanged mutable ref where the RUN writes a value that cannot repeat — identical output proves the layer did not execute.
Context
ai/deploy/Dockerfile:12shipsARG NEO_REF=devas the default, and documents the alternative at:8as:That framing says a SHA buys reproducibility. It also buys freshness, and without it there is none — a branch ref makes the source layer cache-stable, so the image silently packages whatever that branch pointed at the first time the layer was built, forever.
Surfaced by an operator who had already worked this out empirically on a live deployment: "dev is a bit risky, since it will not auto-update on future updates then. this was the reason that we so far used explicit SHA values." Filed because the file still recommends the opposite reason, so the next reader learns it the same expensive way.
The Problem
The source stage is one
RUN(ai/deploy/Dockerfile:20-23):ARG NEO_REF RUN git init -q && git remote add origin "${NEO_REPO_URL}" \ && git fetch --depth 1 origin "${NEO_REF}" \ && git checkout -q --detach FETCH_HEAD \ && git rev-parse HEAD > /neo/.neo-revisionA
RUNlayer's cache key is its command string after ARG substitution. WithNEO_REF=devthat string is byte-identical on every build, so the layer is reused and the fetch never executes again. With a SHA the arg changes whenever the code changes, which busts the cache by construction.So the two options are not "pinned versus tracking". They are pinned deliberately versus pinned accidentally, at an arbitrary past commit, with a build log that reads as though it fetched.
Demonstrated
Minimal reproduction of the cache-key property, the
RUNstanding in for the fetch:FROM alpine:3.20 ARG NEO_REF=dev RUN echo "fetch ${NEO_REF}" && head -c6 /dev/urandom | od -An -tx1 | tr -d ' \n' > /what-we-gotDocker 29.2.1. Two
devbuilds produce the same bytes from a command that cannot produce the same bytes twice if it runs.The Architectural Reality
ai/deploy/Dockerfile:8— the comment that makes the branch default look safe, by naming reproducibility as the only thing a SHA adds.ai/deploy/Dockerfile:12—ARG NEO_REF=dev. A mutable channel as the default value for a cache-keyed arg.ai/deploy/Dockerfile:20-23— the single cache-keyedRUN.ai/deploy/Dockerfile:~112-140—NEO_REVISIONis the integrity assertion and is empty by default, so the one mechanism that would catch a stale package is off in exactly the configuration that produces one./app/.neo-revisionstill records the truth, but nothing compares it to an expectation unless a caller supplies one.The provenance work in
#15774/#15775made the packaged revision observable. This is the adjacent gap: observable is not asserted, and the default configuration asserts nothing.The Fix
Direction, not prescription — the cheapest correct option depends on how much the deploy pipeline should own.
Option A — fix the comment only. State that a mutable ref freezes the layer and that a SHA is required for a build to fetch at all. One line, zero behaviour change, and it stops the misreading. Weakest, because the default still does the wrong thing.
Option B — make the mutable-ref case fail closed. Refuse a non-SHA
NEO_REFunless an explicit opt-in arg is passed (NEO_ALLOW_MUTABLE_REF=1). Matches the fail-closed posture the plane and blobless-clone guards already use: the guard fires on the shape that is silently wrong, and the escape hatch is explicit.Option C — resolve the channel at build entry. Have the compose/pipeline layer resolve
devto a full SHA before invoking the build, so the mutable name never reaches a cache-keyed arg. Strictly better provenance, and it makesNEO_REVISIONpopulated by default — but it moves work into every caller.B and C compose: C for the pipeline, B as the backstop for a hand-run
docker build.Acceptance Criteria
ai/deploy/Dockerfile:8no longer describes SHA-pinning as being about reproducibility alone; it states that a mutable ref freezes the source layer.NEO_REVISION's default-empty behaviour is revisited in the same change — an integrity assertion that is off by default cannot catch the case this ticket is about.Out of Scope
#15774/#15775— provenance emission. This consumes their output rather than changing it.#15792— the reference pipeline building unpinned. Adjacent and probably fixed by Option C, but that ticket is about the pipeline's own pinning, not about the cache-key property of a mutable ref.Avoided Traps
/app/.neo-revisionacross builds.--no-cachecovers it. It does, and it is a per-invocation flag in a different document from the one that recommends the branch default. A claim in one step being true only because of a flag in another step is how this stayed plausible.Related
#15774(deployed images cannot prove their revision) ·#15775(forward NEO_REF, emit OCI revision label) ·#15792(reference pipeline builds unpinned) ·#15782(abbreviated-SHA fetch failure in the guide) ·#16206(ai/deploy composable layering).Live latest-open sweep: checked the latest 20 open issues at 2026-08-07T14:03:32Z; no equivalent found. A2A in-flight claim sweep over the last 30 messages: no competing lane-claim on this scope.
Origin Session ID: 9ced67a1-8f21-4da2-a1bf-a2a968c47ed2
Retrieval Hint:
query_raw_memories("NEO_REF branch default docker layer cache freeze source stage never refetches stale image")Retrieval Hint: the discriminating measurement is two builds with an unchanged mutable ref where the RUN writes a value that cannot repeat — identical output proves the layer did not execute.