Constrained CPU-plane reliability: tenant ingestion completes, cores idle at rest, and incidents stay diagnosable

Problem scope

Topology (recorded 2026-08-14 after Euclid's Stage-1 epic review): this epic is a bounded workstream under #16706, the standing outcome authority for the two terminal predicates (S1: pinned cores return to idle · S2: multi-tenant ingestion completes). #16706 owns the outcomes and the ollama-era mechanism generation (its frozen, evidence-tagged body); this workstream owns the post-lane-split mechanism generation (2026-08-13/14), the operational doctrine, and the hardening subs. It is linked as #16706's child and closes into it — it does not compete with it.

A production, CPU-only external deployment ran for weeks with 1 of 4 tenant repos ingested and its embedding lane's full CPU allocation pinned around the clock — while performing almost no useful embedding work. Successive incident sessions each produced a fresh, partially-wrong theory ("per-request deadlines too tight", "batch shape wrong", "embeddings inherently too slow on CPU — swap the model"), because each session inherited compacted summaries instead of persisted, corrected findings, and because the plane's diagnostics degrade exactly when they are needed (#17066).

The verified mechanism map (2026-08-13/14; live plane reads + code traces, every claim checked against the running system or the pinned source):

1. Engine thread pools default from HOST hardware, unaware of container CPU quotas — at the pinned build, compute workers resolve to the host's physical cores (common_cpu_get_num_math(), ~32 on the 32c/64t EPYC) and the HTTP pool to max(n_parallel + 4, hardware_concurrency − 1) (63 there), so the observed 98 threads ≈ 32 compute + 63 HTTP + service threads: ~5.3× compute oversubscription inside a 6-CPU cgroup (decomposition corrected 2026-08-14 per Euclid's source-cited peer review — the initial "~64 compute / ~10×" over-read the process listing). Tiny canary embeds averaged 12.8 s and "successful" canaries took 205–246 s — minutes-per-nothing; the clean before/after throughput is the falsifier for exactly how much of the burn was overhead. This defect ships today in the canonical provider-lanes template (no thread pinning anywhere; fix in flight as #17073).
2. The provider-lane healthcheck sha256-sums multi-GB model weights every 15 s inside the same quota (#17063), flipping a loaded-but-alive engine unhealthy under exactly the load it should tolerate.
3. The recovery actuator answers with restarts whose client deadline is shorter than the stop budget it requests, records false not-applied, and re-dispatches — destroying in-flight work each cycle (#17065). Every kill surfaces to callers as KB_VECTOR_EMBED_CONNECTION_REFUSED, which reads as a provider fault.
4. Health canaries and probes at interactive priority can defer batch ingestion indefinitely (no admission aging, #17062), and their thrash-inflated cost made the instrument three times the volume of the work it measured.
5. Heavy maintenance (REM/dream) saturates a chat lane for hours over a near-empty corpus (#17046) — indistinguishable from a frozen host on every monitor.
6. The failure was self-obscuring: diagnostic reads gated behind the failing dependency (#17066), in-flight ledger records leaking forever and corrupting the instrument used to diagnose them (#17064), silent truncation with no floor tying per-slot context to the declared safe band (#17070), no verification of a live lane's shape against its intended envelope (#17069), backup dying silently for days with durability posture unmet (#17068), and failure-streak backoff with no operator exit (#17067).

Deadline conflation gets its own line because it generated two wrong root-cause claims in as many days: the KB health probe's deadline (300 s), the MC write canary's (900 s on the observed plane), and the batch-embed call ceiling (deployment-configured; 1 h there) are three different clocks. Any analysis citing "the 300s deadline" without naming which clock is suspect, and reviewers should treat it as a stop-and-verify signal.

Why an Epic: one outcome served by many one-PR leaves across compose templates, orchestrator scheduling, health surfaces, ledgers, and workload shaping — plus an operational doctrine that must outlive any single session's context window. The subs are attached via the parent-child relationship graph and added incrementally; this body deliberately carries no sub registry.

Intended solution shape

Six convergent streams:

• Engine-lane correctness in the canonical templates — compute threads pinned to the CPU allocation, liveness-only steady-state probes (integrity verification stays at the entrypoint), call ceilings sized as backpressure rather than generosity. The observed plane already runs the deployment-side mirror of these fixes; canonical must stop shipping the defects to the next deployment.
• Scheduling fairness — admission aging so batch work cannot be overtaken forever, and heavy-lease fairness with bootstrap-critical yield. Half of this is already merged (#16561 → the durable-waiter + yield work, and container tenant-sync lease gating); the epic's job includes getting merged fixes actually deployed to the planes that need them.
• Observability that survives failure — per-tool dependency gating instead of aggregate-health refusal, ledger expiry with typed dispositions, actuator effect dispositions that can say "unknown", backup exhaustion that escalates instead of sleeping.
• Input-integrity floor — per-slot context tied to the declared safe processing band; truncation surfaced as a typed failure, never a stored vector.
• Workload shaping for CPU planes — per-cycle budgets and breathing gaps for heavy maintenance; cadence proportional to undigested backlog.
• Revision currency as operational discipline — an external plane pins a resolved revision; merged is not deployed. Every incident read starts by comparing the plane's deployedRevision (its healthcheck exposes it) against the channel tip, and every deploy letter carries the freshly-resolved pin.

Operational doctrine (the part sessions kept losing, now substrate): an incident session's FIRST move is reading the LIVE plane through its MCP surface — healthcheck, deployment-state snapshot, bridge log reads; the access recipe lives in the private deployment repository, never in public artifacts. No theory ships without a live read. A measurement taken while a pathology is active indicts the pathology, not the component it ran on — components are judged on clean measurements only, after mechanisms are fixed. Corrections land in ticket BODIES with the replaced text preserved in comments, so the next session inherits corrected state rather than the trajectory of wrong drafts.

This workstream closes when its hardening subs are delivered and a fresh session can reconstruct the mechanism map and current state from this epic plus its linked subs alone. The two contract outcomes themselves — all configured tenant repos fully ingested and tracking head; host cores idle at rest with bounded purposeful bursts — are owned and verified by the parent outcome authority, #16706 (S1/S2), and this workstream's delivery is measured against them there.

Out of scope

• The terminal S1/S2 outcome authority and the ollama-era mechanism generation — #16706 (parent) owns both; its frozen body's Rule 0 ("A does not explain B") and its abort-direction warning remain binding on every sub here.
• The process/team-failure retrospective — #17042 owns converting the provider-lanes execution failure mechanics into substrate.
• The closed lane-architecture epic (#17018) that shipped the role-isolated topology this epic hardens.
• Deployment-specific values for any particular plane (those live in that plane's private repository, alongside its access recipe).

Avoided traps

• "Give the lane more CPU" / "swap to a smaller model" as first response. Both were proposed and declined during the incident. The measured load was overhead; capacity or model changes would have re-shaped the same pathology and burned the trust budget of another failed deploy. Fix mechanisms, measure clean, then judge components.
• Trusting any single health signal. The plane simultaneously showed a healthy recovery probe, a write canary at failure-streak 50, and a Docker-healthy container — three instruments, three answers, each locally true. Shape verification (#17069) and per-tool gating (#17066) are the fix, not signal-picking.
• Restart as remedy. Every restart destroyed in-flight work and manufactured the connection-refused failures that drove repos into backoff (#17063 / #17065). Recovery that re-dispatches under uncertainty amplifies the incident it exists to end.