LearnNewsExamplesServices
Frontmatter
id17073
titleThe canonical embedding lane never pins compute threads, so the engine spawns host-core-count spin threads inside its CPU quota
stateClosed
labels
bugaiperformanceagent-os
assigneesneo-opus-vega
createdAtAug 14, 2026, 12:17 AM
updatedAtAug 14, 2026, 2:48 AM
githubUrlhttps://github.com/neomjs/neo/issues/17073
authorneo-opus-vega
commentsCount1
parentIssue17072
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 14, 2026, 2:48 AM

The canonical embedding lane never pins compute threads, so the engine spawns host-core-count spin threads inside its CPU quota

Closed Backlog/active-chunk-15 bugaiperformanceagent-os
neo-opus-vega
neo-opus-vega commented on Aug 14, 2026, 12:17 AM

Context

ai/deploy/docker-compose.provider-lanes.yml requires every resource value fail-closed (:? substitution — CPUs, memory, slots, context, batch, ubatch) because "#17024 elects the fixed-envelope allocation; this profile consumes that result and never guesses it." Compute threads are the exception: nothing sets LLAMA_ARG_THREADS, so the engine guesses — from the wrong denominator.

At the pinned build, an unset LLAMA_ARG_THREADS resolves compute workers to the host's physical cores (common_cpu_get_num_math() — ~32 on the observed 32c/64t EPYC), and the HTTP pool separately defaults to max(n_parallel + 4, hardware_concurrency − 1) (63 there). Container CPU quotas (deploy.resources.limits.cpus) change neither answer — only cpusets would. The observed plane's 98 threads therefore decompose as ~32 compute + 63 HTTP + service threads: ~5.3× compute oversubscription inside a 6-cpu quota, plus a needlessly host-sized HTTP pool. (Corrected 2026-08-14 per Euclid's source-cited peer review — the original body over-read the process listing as "~64 compute / ~10×"; engine-source receipts in the correction comment.) Measured consequences from the plane's own ledger:

  • tiny health-canary embeds averaged 12.8 s; "successful" canaries took 205–246 s — minutes for ~10-token inputs;
  • the lane held ~591% of its 600% quota around the clock while storing almost nothing — compute spin-workers at ggml graph barriers convert quota into scheduler/cache thrash that reads as utilization (the 63 HTTP workers are a separate blocking pool and are not claimed as spinners; per-thread CPU attribution was not captured, so the clean before/after throughput is the falsifier for how much was overhead);
  • every downstream instrument (canary latencies, probe outcomes, wall-time tables) was contaminated, which mis-aimed two prior root-cause analyses at capacity and at model size (see epic #17072).

The deployment-side fork has pinned threads as a mitigation; the canonical template still ships the defect to every future deployment.

The Problem

A lane whose thread counts derive from host hardware violates the template's own never-guess contract on the values that govern whether the CPU allocation does work or does overhead. Oversubscribed compute spinners are not a tuning nit: the lane presented as saturated around the clock while storing almost nothing — the exact "frozen cores" failure the lane split exists to prevent. How much of the burn was pure overhead is settled by the acceptance falsifier (clean before/after throughput and return-to-idle), not asserted from the process listing.

Architectural Reality

  • LLAMA_ARG_THREADS / LLAMA_ARG_THREADS_HTTP exist as env-mapped args at the pinned engine build (server-b10380).
  • The election (#17024) already elects the lane's CPU allocation; threads must derive from that result via the same fail-closed substitution style as every sibling value.
  • ai/scripts/diagnostics/providerLaneComposition.mjs already direct-checks service env and pushes bounded error codes into the receipt's existing errors field (precedent: slots-endpoint-disabled) — the mechanical guard lands there with no receipt-shape change, so durable receipts and the election runner are unaffected.
  • The Ollama chat lane has the same derivation class engine-side with no env to pin it — a documented gap, out of scope here (its heavy producers are being shaped in #17046).

The Fix (shape)

  1. Canonical lane env: LLAMA_ARG_THREADS: ${NEO_PROVIDER_LANE_EMBEDDING_THREADS:?…} (fail-closed like every sibling) plus a pinned LLAMA_ARG_THREADS_HTTP, so /health stays answerable under full compute load.
  2. Analyzer rules (bounded error codes, existing errors channel): embedding lane missing/non-integer LLAMA_ARG_THREADS → error; threads exceeding ceil(lane cpus) → error; missing LLAMA_ARG_THREADS_HTTP → error.
  3. A lane comment documenting the quota-vs-cpuset engine behavior, so the next compose fork keeps the pin instead of rediscovering the incident.

Acceptance Criteria

  • Rendering the canonical provider-lanes profile without the threads value fails closed at substitution, like every sibling resource var.
  • analyzeProviderLaneComposition rejects a composition whose embedding service lacks LLAMA_ARG_THREADS or whose value exceeds ceil(lane cpus), with distinguishing error codes (unit arms).
  • LLAMA_ARG_THREADS_HTTP is pinned and asserted.
  • Negative control: the updated valid fixture (threads == elected CPUs) passes analysis unchanged.
  • The chat-lane thread-derivation gap is documented at the lane with this ticket family referenced.

Out of Scope

  • The healthcheck payload defect in the same service block (#17063 — sibling, separate leaf).
  • Runtime live-shape verification against the elected envelope (#17069).
  • Ollama engine-side thread control (no env exists; tracked as a documented gap).

Avoided Traps

  • "The quota already limits CPU." It limits time, not thread count; oversubscribed spin workers burn the whole quota on synchronization, and the waste presents as work.
  • "Use nproc inside the container." nproc reads host affinity; under a quota (no cpuset) it still answers 64.
  • "Fix the forks." The template is the generator; every unfixed render seeds the next incident.

Contract Ledger (backfilled 2026-08-14 per cross-family review RA-1 on PR #17074)

Surface Producer Consumer Derivation Refusal posture
NEO_PROVIDER_LANE_EMBEDDING_CPUS (existing elected input, now dual-consumed) #17024 election → closed deploymentInputs env export compose deploy.resources.limits.cpus and LLAMA_ARG_THREADS compute threads ≡ the elected CPU allocation, byte-identical (single authority; integer by the election contract) render fails closed without it; analyzer rejects threads !== cpuCores (embedding-threads-cpu-mismatch) and non-integer values (embedding-threads-unpinned) — a fractional allocation fails the integer gate and forces an explicit decision
LLAMA_ARG_THREADS_HTTP (template constant 4) canonical template llama-server base HTTP pool (the engine may add dynamic workers under load; this is not a total cap) fixed small base pool; deliberately not elected — no per-host variance in need analyzer rejects missing (embedding-http-threads-unpinned) or > EMBEDDING_HTTP_THREADS_MAX (embedding-http-threads-oversized) — the host-derived hw−1 default cannot pass

No new deployment input exists: the election actor's --env-file /dev/null + closed deploymentInputs export carries this contract unchanged, proven by the closed-input render arm in the composition spec.

Related

  • #17063 — sibling defect in the same service block
  • #17069 — runtime shape verification (consumes this pin as part of the envelope)
  • #17024 — the election whose result the pin must consume
  • Epic #17072 — incident family + operational doctrine

Live latest-open sweep: checked latest 20 open issues + recent A2A claims at creation time; no equivalent found (closest: #17063, deliberately separate leaf).

tobiu closed this issue on Aug 14, 2026, 2:48 AM
tobiu referenced in commit ec09ec4 - "feat(ai): provider-lane probes go liveness-only and compute threads pin to the elected allocation (#17063, #17073) (#17074) on Aug 14, 2026, 2:48 AM