LearnNewsExamplesServices
Frontmatter
id17069
titleNothing verifies a deployment''s live provider-lane shape against the elected envelope, so a forked compose with guessed values runs undetected
stateOpen
labels
bugaiarchitectureperformanceagent-os
assignees[]
createdAt10:22 PM
updatedAt10:25 PM
githubUrlhttps://github.com/neomjs/neo/issues/17069
authorneo-opus-vega
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]

Nothing verifies a deployment's live provider-lane shape against the elected envelope, so a forked compose with guessed values runs undetected

Open Backlog/active-chunk-15 bugaiarchitectureperformanceagent-os
neo-opus-vega
neo-opus-vega commented on 10:22 PM

Context

ai/deploy/docker-compose.provider-lanes.yml is explicit about its own contract:

Every resource value is required because #17024 elects the fixed-envelope allocation; this profile consumes that result and never guesses it.

It enforces that with fail-closed substitution — ${NEO_PROVIDER_LANE_EMBEDDING_SLOTS:?embedding slots required}, …TOTAL_CONTEXT_TOKENS:?, …UBATCH_TOKENS:?, …EMBEDDING_CPUS:? — and ships a validator (ai/scripts/diagnostics/providerLaneComposition.mjs) plus an election runner (ProviderLaneElectionRunner) that benchmarks candidate slot counts against the actual hardware. The intended per-slot floor is carried as NEO_PROVIDER_LANE_EMBEDDING_CONTEXT_TOKENS_PER_SLOT_REQUIRED (8192 in the shipped fixtures).

That protection is defeated by copying the file.

An external plane was observed on 2026-08-13 running a compose whose embedding-model block is byte-identical to the canonical one — same image digest, same entrypoint, same healthcheck — except that every :?-required value had been replaced with a literal:

LLAMA_ARG_CTX_SIZE=131072      # canonical: ${…TOTAL_CONTEXT_TOKENS:?}
LLAMA_ARG_N_PARALLEL=4         # canonical: ${…SLOTS:?}
LLAMA_ARG_BATCH=131072         # canonical: ${…BATCH_TOKENS:?}
LLAMA_ARG_UBATCH=32768         # canonical: ${…UBATCH_TOKENS:?}
cpus: "${EMBEDDING_CPU_LIMIT:-6.0}"   # canonical: ${…EMBEDDING_CPUS:?}

Against the shipped formula (total = slots × perSlot, perSlot = 8192), four slots prescribes 32,768 total context. The running plane had 131,072 — four times the elected shape, i.e. 32,768 tokens per slot. The engine reported n_slots = 4, n_ctx_slot = 32768 and a single 9,144-token embedding request measured ~11 minutes against a 300s client deadline. No embedding completed on that plane for weeks.

The Problem

The fail-closed :? guard protects exactly one thing: rendering this file without an elected envelope. It cannot protect against the far more likely operational path — a deployment that vendors its own compose and substitutes literals, whether by copy-paste, by templating, or because the deployment topology needs services the canonical overlay does not include.

Once that happens:

  • The election (#17024) never runs, so nothing is sized to the hardware.
  • The validator is never invoked — it is an optional local command, not a gate.
  • The runtime accepts whatever shape it is given and reports healthy. The provider answers, so liveness passes; the shape is simply wrong, and wrongness of this kind manifests as slow requests, which surface as provider timeouts rather than as configuration errors.

The system therefore had every fact required to detect this and never connected them. LLAMA_ARG_ENDPOINT_SLOTS: "true" is already set, so the engine's live slot and per-slot-context shape is queryable. The elected envelope is already a first-class artifact. Nothing compares them.

The consequence is the worst available failure mode: a misconfigured deployment is indistinguishable from a correctly-configured one under load, and the resulting timeouts are attributed to the provider. That attribution is why this survived weeks of investigation pointed at the wrong layer.

Architectural Reality

  • llama-server exposes its live configuration; n_slots and n_ctx_slot appear in its own startup log and via the slots endpoint the compose already enables.
  • The elected envelope is durable — it is the output of #17024 and the input this profile declares it consumes.
  • Health today answers "can the provider respond?". It does not answer "is the provider shaped the way this deployment elected?" — and only the second question catches a guessed envelope.
  • A forked compose is not an abuse to be prevented; it is the normal path for any deployment whose topology differs (ingress, extra services, different volumes). The guard has to live at runtime, not in the file.

The Fix (shape)

  1. At boot, read the embedding lane's live shape from the engine and compare it to the elected envelope (or, minimally, to CONTEXT_TOKENS_PER_SLOT_REQUIRED).
  2. On mismatch, degrade the health surface with a precise, actionable reason — naming observed vs expected per-slot context — rather than reporting healthy and letting the divergence express itself as provider timeouts hours later.
  3. Report the observed lane shape in the deployment-state snapshot regardless of match, so the value is visible without a shell on the host.
  4. Consider making the composition validator runnable against a rendered running deployment, not only against a local docker compose config — the deployments most likely to be wrong are the ones nobody renders locally.

Acceptance Criteria

  • Boot compares live embedding-lane slot/context shape against the elected envelope and records the comparison.
  • A per-slot context differing from the elected value degrades health with a reason naming both numbers.
  • The observed lane shape appears in the deployment-state snapshot on a healthy plane too.
  • A correctly-elected deployment shows no degradation (negative control).
  • Regression fixture: a lane rendered at 4× the elected per-slot context is flagged at boot, before any embedding request is attempted.

Out of Scope

  • Changing the election algorithm (#17024) or the shipped per-slot floor.
  • The healthcheck defect in the same block (#17063) — that one is byte-identical between canonical and fork, and is genuinely ours.
  • Admission ordering (#17062), ledger leaks (#17064), actuator deadlines (#17065).

Avoided Traps

  • "They should not have forked the compose." Forking is the normal path for a differing topology, and the canonical file is an overlay that does not stand alone. A contract enforced only by substitution syntax is enforced only for people who did not need to change anything.
  • "The validator already exists." It exists and is optional, local, and pre-deployment. Nothing requires it to have run, and nothing re-checks after a value drifts.
  • "Health was green, so detection worked." Health answered a narrower question than the one that mattered. The provider genuinely was alive; it was shaped wrong.

Related

  • #17024 — elects the fixed-envelope allocation this profile consumes
  • #17021 / ADR 0014 §2.8 — role-isolated provider composition
  • #17063 — the healthcheck in the same block, byte-identical between canonical and fork
  • #17062 · #17064 · #17065 · #17067 — the incident this would have caught at boot