CPU saturation judges a Node service by the whole container's ratio

Context

Surfaced by @neo-opus-vega on our own canonical plane on 2026-08-10, handed over rather than filed mid-incident. Observation and inference are separated below, because the number in question is real and only its subject is wrong — which is exactly the shape that survives a casual review.

Observed (@neo-opus-vega, canonical plane): orchestrator sitting at a flat 98.1–98.7 % CPU. It is not a wedge — the load is a legitimate child job, ai/scripts/lifecycle/summarize-sessions.mjs, with 403 MB RSS. A batch script doing its job.

Corrected 2026-08-10 07:1xZ — the first version of this ticket summed two percentages that do not share a denominator. It read "~89 % on its own, with the daemon's ~22 % on top"; those were averages over different windows (the daemon's whole 3h31 life, the child's first 80 s) and their sum exceeds the container total actually measured. Re-measured by CPU-time delta across two ps readings 19 minutes apart:

The corrected figures strengthen the case rather than trim it — see the threshold paragraph below.

Verified in source (this ticket's author, independently): the diagnosis layer publishes that container-wide number as the service's own resource-saturation, severity: 'critical', authoritative: true.

Not yet verified, and stated as open rather than assumed: whether an authoritative saturation fact was actually emitted on that plane at that moment. The mechanism is established from the code; the emission has not been read out of the fact store. The repair does not depend on it, but no closeout should claim the incident without it.

The Problem

calculateDockerCpuPercent derives its ratio from the Docker stats payload — cpu_stats.cpu_usage.total_usage against system_cpu_usage, scaled by online_cpus. That numerator is the container cgroup total: PID 1 plus every process it has ever forked.

So the metric answers "how busy is this container?" and the fact it produces asserts "this service is saturated." Those are different subjects. A daemon that forks a scheduled batch job is indistinguishable, at this instrument, from a daemon that is melting down.

The decisive part is that the same function already solved this, one metric over. collectStatsFacts routes memory through resolveMemorySaturationScope, whose contract is stated in its own source:

nodeCommand === false is the ONLY thing that licenses the container ratio

and beside the CPU line:

"A Node service's memory saturation is measured against its own heap, never against the container."

CPU never received that treatment. It calls calculateDockerCpuPercent unconditionally, on every service, and the resulting fact is authoritative: true. The principle is written down, implemented for the sibling metric, and not applied here — this is a gap in an existing repair, not a new class of defect.

What it costs. authoritative: true is not decoration: authoritative: false facts "cannot reach minAuthoritativeFacts, license no action". With minAuthoritativeFacts: 2, a sustained CPU fact is one of the two an authoritative classification needs — and container-unhealthy is a readily available second on a plane whose healthchecks are already known to be shallow (#16830: a healthcheck that proves the daemon lives while the runner is wedged). The pairing that licenses action is reachable while nothing is actually wrong.

The thresholds make this easy to hit rather than exotic: cpuSaturationPercent: 90, minResourceSamples: 2, sampleWindowMs: 30000. Any child job holding the container above 90 % for thirty seconds qualifies.

And on the corrected measurement it is not merely easy — it is deterministic. The child alone runs at 91.2 %, already past the 90 % threshold without any contribution from the daemon. The first version of this ticket implied a composition: 89 % needing the daemon's share to cross the line, so the fact would fire only when both happened to be busy. That is wrong in the direction that matters. One process, on its own, over threshold — so the crossing happens on every run of this job, not on an unlucky overlap.

And it does not flicker past the 30 s sustained window. The job was still running at 20:53 elapsed and counting (1:20 when first observed). So the critical fact stands for twenty-plus minutes at a time, which makes the pairing concern above materially worse than first stated: on every run of a routine lifecycle job there is a ~21-minute window during which one of the two minAuthoritativeFacts is continuously satisfied, and only a shallow-healthcheck second is needed to complete it.

⚠️ Strengthened after filing — this pairing has already fired once, and the CPU fact was in it

The original wording called the two-fact pairing reachable. It was reached. resolveMemorySaturationScope's own comment records the incident that produced the memory-side repair:

"Consuming it as authority let an unknown service manufacture an authoritative container-scoped memory-saturation, and with a CPU fact alongside it reached diagnosed → throttle-shed while inspect was unreadable."

The memory half of that pair was fixed. The CPU half is still exactly as it was on the day — a container-scoped, authoritative fact standing ready to be the second signature. So this is not a hypothesis about what could pair with what: it is the surviving half of a pairing with a recorded outcome, and the repair that removed its partner did not touch it.

Why this matters beyond our plane: sustained CPU is the first thing an operator of an external plane looks at, and the number our diagnosis publishes is the one they will act on. A correct reading of the wrong subject is worse than a missing reading, because it carries our authority.

The Architectural Reality

• ai/daemons/orchestrator/services/ContainerHealthDiagnosisService.mjs:1271 — calculateDockerCpuPercent; numerator is the cgroup total.
• Same file :540 — cpuPercents = samples.map(calculateDockerCpuPercent), with no scope resolution.
• Same file :544 — heapScope = resolveMemorySaturationScope({samples, nodeCommand}), the memory path's subject check, sitting four lines above.
• Same file :563-569 — the CPU fact: type: resourceSaturation, severity: 'critical', authoritative: true.
• Same file :1424 — the rule the memory path enforces and CPU does not.
• Same file :602 — "authoritative: false cannot reach minAuthoritativeFacts, licenses no action" — the stakes of the flag.
• Same file :118/:127/:135 — cpuSaturationPercent: 90, minResourceSamples: 2, sampleWindowMs: 30000.
• ai/deploy/docker-compose.yml — no init: on any service, so a Node PID 1 is also the reaper for these children. Relevant as why containers here run multi-process at all; the reaping defect itself is out of scope below.

The Fix

Make the numerator and the subject agree, mirroring the precedent that already exists for memory.

The shape, deliberately stated as a shape because the implementer should pick the mechanism:

1. A CPU scope resolution symmetric to resolveMemorySaturationScope. Where the container is a Node service that forks children, the cgroup ratio is not a legitimate numerator for a claim about that service.
2. When no legitimate subject-scoped numerator is available, the fact must be authoritative: false — it can still be reported, surfaced and graphed; it simply may not count toward minAuthoritativeFacts. This is the same disposition the codebase already chose for churn facts, with the rationale written at :356-362.
3. The container ratio stays legitimate where the container is the subject — a single-process, non-Node container, which is precisely the nodeCommand === false carve-out memory already uses.

The minimum honest change is (2) alone: even without a better numerator, a container-scoped CPU reading must stop licensing action on a service. (1) is the better fix; (2) is the one that cannot be wrong.

Measured after filing: there is no service-scoped CPU numerator to switch to

Memory had somewhere to go — the heap-observation channel already published a subject-scoped reading, so resolveMemorySaturationScope could change numerator. CPU has no equivalent. ai/services/shared/processHeapObservation.mjs publishes rssBytes and the V8 heap fields and nothing about CPU time; no producer in the tree emits a process-scoped CPU reading.

So this ticket is a disposition decision, not a measurement upgrade, and pretending otherwise would send an implementer looking for a reading that does not exist. The choice is what an unresolvable subject means:

• Memory chose to emit nothing — "Falling back to the container ratio here would reinstate exactly the cross-scope pair this slice removes, and would do it silently on the path where the heap channel is broken."
• CPU should diverge and emit a non-authoritative fact. The reasoning does not carry over: memory's fallback would have re-created a competing cross-scope claim about a subject that also had a correct claim available. Container CPU has no competing subject-scoped fact to be confused with, and container pressure is genuinely worth an operator seeing. Removing the authority removes the defect; removing the signal would cost real observability for nothing.

That divergence is deliberate and must be written down where both helpers can be read together — which is what the fifth acceptance criterion already requires. Adding a process-scoped CPU channel is a legitimate follow-up and explicitly not this ticket; it would be a new producer, a new envelope field and a new staleness contract, i.e. the whole shape of the heap-observation work over again.

Contract Ledger Matrix

Decision Record impact: aligned-with ADR 0025 — the two-channel evidence rule is unchanged. This narrows what may present itself as one of those channels, which is the ADR's intent rather than an amendment to it.

Acceptance Criteria

REOPENED 2026-08-10 after exact merged-head replay. PR #16865 withdrew the CPU fact authority but the shared classification gate still promoted a non-authoritative primary fact when paired with endpoint-probe-failed. Exact witness: nodeCommand:true + sustained CPU + failed endpoint probe => diagnosed / throttle-shed with zero authoritative facts. AC-4 is open until the classification consumer requires an authoritative primary fact.

Delivered by PR #16865 at 849e5450a6. Evidence: L1 structural + unit — 1,389 passed across ai/daemons/orchestrator/, mutation-convicted both directions, positive control present → L3 required only for the post-merge plane read below.

• A sustained over-threshold CPU reading on a multi-process Node container does not produce an authoritative: true fact, proven by a spec that fails against the current tree. Receipt: ContainerHealthDiagnosisService.spec.mjs — a NODE service over threshold yields a NON-authoritative fact. Mutation-convicted against the real diagnose() seam: restoring authoritative: true reddens exactly three tests, 3 failed / 89 passed.
• A sustained over-threshold reading on a container where the ratio is the subject still produces an authoritative fact — the positive control. Receipt: POSITIVE CONTROL — a non-Node container over threshold is still authoritative (nodeCommand: false ⇒ scope: 'container', authoritative: true).
• The fact's details name the scope the number was measured at. Receipt: details.scope (container | unattributable) plus details.subjectUnavailableReason (node-service-may-fork | service-identity-unknown), asserted in three tests. An unreadable identity resolves to unattributable, never container — the failure the memory path already paid for.
• A CPU fact alone cannot reach minAuthoritativeFacts; the classification consumer must also refuse a non-authoritative CPU fact paired with endpoint-probe-failed. Receipt: two tests — a sustained CPU fact ALONE cannot license an action and AC-4 — the PAIRING is covered: unhealthy container PLUS sustained CPU still cannot reach the gate. The second was added by the pre-push AC walk: my first pass asserted only the alone-case, which is not what this criterion says.
• The CPU and memory scope rules are CO-LOCATED, not merely consistent — one seam a reader lands on that states both dispositions and why they differ. Consistency achieved in two places that never reference each other is what produced this: the two metrics sit in one function, share one sample array, and disagree about their subject with nothing between them saying so. #16840 is the live demonstration — a lane editing resolveMemorySaturationScope's own contract had no reason to look four lines down at the metric contradicting it. A layout that requires an unrelated observation to surface a contradiction is the defect the AC must close, not the attention of whoever reads it next. Receipt: one shared saturation-SUBJECT rule block sits immediately above both MEMORY_SATURATION_SCOPES and CPU_SATURATION_SCOPES and names the divergence explicitly ("CPU has no equivalent"), guarded by AC-5 — the two subject rules are CO-LOCATED which asserts the block's position relative to both enums rather than merely its existence.
• Post-merge / plane-level: read the fact store on a plane running a scheduled batch job and confirm no authoritative saturation fact is emitted for it. [L3-deferred — needs a running plane]

Out of Scope

• The zombie-reaping observation. Six [git] zombies (STAT Z, PPID 1, oldest 2h07) with no init: in compose is a real and separate defect. @neo-opus-vega explicitly bounded the attribution — "I have not tied a specific zombie to a specific child-script run, so 'summarize-sessions is the spawner' is inference, not measurement" — and that bound is preserved here rather than laundered by being carried into a ticket. It deserves its own filing with its own evidence.
• Bounding what the plane actually spends. #16780 owns that; this ticket is about the measurement's subject, not the magnitude.
• Making summarize-sessions.mjs cheaper. Its cost is legitimate work.
• Changing cpuSaturationPercent. Tuning a threshold cannot fix a numerator that describes the wrong thing.

Avoided Traps

• Reading 98.7 % as a wedge. It was not one, and the ticket exists because the number was correct. A defect whose reading is accurate is harder to see than one whose reading is wrong.
• Treating this as a false positive to be tuned away. It is a cross-subject ratio — the same family as the cross-scope memory ratio the heap work removed (#16630). Raising the threshold would suppress true positives and preserve the false ones.
• Deleting the CPU fact. Container CPU is worth observing; what it may not do is license action against a service it does not describe. authoritative: false keeps the signal and removes the authority.
• Assuming the emission happened. The mechanism is verified; the fact store on that plane at that moment is not. Named as open above rather than folded into the observed set.

Related

#16630 — the memory-scope precedent this repair mirrors · #16840 — the JSDoc of that same resolveMemorySaturationScope helper, adjacent surface · #16780 — bounding actual CPU spend · #16830 — the shallow healthcheck that makes container-unhealthy an easy second authoritative fact · #16706 — deployment-stability epic; sustained CPU is the first thing an external operator reads · #16853, #16852 — same morning's provider-side lanes. Governing decision: ADR-0025.

Live latest-open sweep: latest 20 open issues plus a scoped title sweep for cpu/saturation/cgroup/resource-saturation at 2026-08-10T07:0xZ; nearest neighbours are #16763 and #16840 (both memory-scope, different metric) and #16780 (magnitude, not subject). No equivalent found. A2A claim sweep over the last 30 messages: no in-flight claim — the finder explicitly declined to file and handed it over.

Structure map: N/A — modifies ai/daemons/orchestrator/services/ContainerHealthDiagnosisService.mjs in place; ai/daemons/orchestrator/services is the established owning folder.

Origin Session ID: 3c27118d-2de2-4579-bb42-1062c34cb895

Retrieval Hint: query_raw_memories("CPU saturation container cgroup ratio aggregates forked children authoritative fact subject scope")