Provenance. This body was rewritten 2026-08-11 23:5x to cut a book of false trails, and that rewrite also (a) inverted the deployment fact and (b) deleted the root-cause analysis and its rationale — both restored below. The full pre-rewrite findings ledger (541 lines) is preserved in repo history: resources/content/issues/chunk-14/issue-16706.md at data-sync commit fc1b9db590. This body is now frozen except for release-outcome updates. Every row is tagged [MEASURED], [SOURCE-READ], [FALSIFIED] or [CORRECTED]; do not implement against anything untagged.
What matters
Exactly two symptoms are being tracked. Everything else in this ticket's history is instrumentation about them.
S1 — four CPU cores are pinned and never recover, with zero model work in flight.
S2 — multi-tenant KB ingestion has never once succeeded end-to-end.
They are stated separately on purpose. A single-cause story has been asserted for them four times and falsified four times.
Release state — 2026-08-12
[CORRECTED] The fix set was never on the external plane. The deployed revision — read from the field that owns it, healthcheck.deployedRevision — is 3f9f8343a8 (2026-08-10), which predates the entire #16706 fix set. The previous body's claim that the recovery was "already deployed and did not act" was false and is retracted (2026-08-12 07:46Z); the half-night lane it created ("why did #16884 not fire") was chasing a phantom.
[SOURCE-READ] The cumulative release is merged: private !57 (2026-08-12). It pins neo dev @ fcc0f8149422d83f481969dfd866480b96c40a8f — 74 commits past the deployed revision — and ships as one client letter: copy one compose file (embedding batch default 5), set one .env pin, docker compose build + up -d --force-recreate --wait. Volumes preserved; never down -v. The force-recreate replaces the model container, clearing the currently-wedged runner at deploy time.
Awaiting: the client executes the letter → ordinary outcome verification only (S1: CPU settles to idle and stays there under normal producers; S2: lastIngestedRev non-null and documents surviving recreation). No experiments, probes, or staged tests on the external plane.
[MEASURED] The same fix set ran on our canonical plane at 271bb132d8 (10 commits before the pin, all fix commits aboard) with 3/3 tenant repos durably ingested, zero consecutive failures (deployment-state snapshot 2026-08-12T08:32Z). Our plane is being moved to the exact release pin for parity.
Root cause — [MEASURED] 2026-08-11 13:40Z from the orchestrator container log
A single KB tenant-ingestion embedding batch exceeds the 30-minute provider timeout and retries 5×. That is the 400%.
- One batch ("batch 7") takes >1,800,000 ms; retried 5× ⇒ ~2.5 h continuous embedding for one batch.
- Every retry re-buys the identical work (#16963 → fixed by #16964).
- It holds the single
OLLAMA_NUM_PARALLEL=1 slot, so the KB canary's 30-second probe can never acquire it ⇒ probe-timeout streak 27 ⇒ the KB is health-gated out of its own tools.
- The two waiting repos die at the concurrency gate (39 and 46 consecutive failures).
- A warm loop fires into the saturated slot every ~5–11 min at
num_ctx 32768.
Why a batch takes >30 minutes when one embed takes 150 ms: batch size, not hardware. [SOURCE-READ] The running host resolves the batch size empty → shipped default 50; measured on comparable CPU inference, a 14-chunk batch took 8m25s — 50 puts a single batch at the ceiling, and a batch that cannot finish stores nothing. The release compose defaults it to 5.
The upstream anchor — ollama/ollama#11889, with its honest bounds
[MEASURED] The provider does not observe cancellation while it waits: abandoned work keeps computing. The frozen witness aborted a node:http.request({signal}) — socket already destroyed — and the runner still held 397–400% CPU with zero established sockets. At NUM_PARALLEL=1 the runner slot is then dead to all work. Live confirmation from host telemetry 2026-08-12 09:19Z: the pinned task is the embedding runner at 401% CPU with 101h50m accumulated CPU time over a ~25h container life — four cores for its entire life — while the chat runner idles at 0%.
[CORRECTED] Demotion recorded 2026-08-11: the specific orphan chain first proposed (docker healthcheck timeout: 45s → client disconnect → orphan) is false — the canary is a long-lived in-process loop, not a per-healthcheck dispatch; no transport close was demonstrated on that path. #11889 is the mechanism class, not a proven per-request chain on that plane.
⛔ Do NOT wire the abort signal through #embedOllama to provider.embed. It is the unsafe direction. [MEASURED] An early abort STRANDS the runner at 397–400% (this ticket's own witness). settleCallerWhileProviderContinues — return the caller, let the provider finish — is the design, not the bug. Three PRs died on the opposite premise: #16956, #16957, #17003.
Rule 0 — two facts
|
fact |
status |
| A |
Ingestion once made durable progress: KB count 0 → 50, one repo complete, outstanding: 0. |
[MEASURED] |
| B |
Four CPU cores pegged continuously, every Neo dispatcher idle. |
[MEASURED], live as of 2026-08-12 |
A does not explain B. Retracted three times by three maintainers; do not re-couple them.
The fix ledger — what ships in the release pin
S1 (CPU):
ddd2f6a89d #16942 — Ollama.stream() gets a cancellation surface (stop abandoning work un-cancelled; the #11889 answer on our side)fcabb64595 #16996 — no native-Ollama timeout handoff across tenant repos10a29b1a79 #16978 — provider timeouts are not retried in-cycleaccfdb0c1a #16943 — native Ollama admission enforced and reported874e6ab26b #16950 — a resident model is no longer reported missing forever (kills the warm loop)73eb465793 #16916 — provider-readiness deadlines actually reach the services
S2 (ingestion):
31b8452d50 #16889 — a repo with nothing to ingest completes instead of backing off foreverd25c397def #16964 — a failed vector write does not re-buy identical embeddingseff78e326e #16959 — tenant repo waiters stay behind active work9dc2a098d7 #16899 — embedding-probe policy is reachable by deployments63da0eabdf #16812 — Memory Core accepted-response bounded after WAL
Deployment wiring (private repo, rides the same letter): embedding batch default 5; kb healthcheck --expected-status; chat fallback gemma4:26b; probe deadlines.
Open hardening, explicitly NOT gating this release: #16695, #16780 (its merged slice #16959 IS aboard), #16853.
Falsified trails — do not re-run these
Each cost hours. Each is closed by evidence, not by opinion.
| # |
trail |
why it is dead |
| 1 |
model-tag mismatch (model vs model:latest) drives a warm loop that burns the cores |
[CORRECTED] the original kill-reason ("the fix #16950 is already deployed and the symptom persists") rested on the false deployment fact. The trail stays dead for a better reason: the measured burner is the batch-occupancy chain above, and #16950 ships in the release pin |
| 2 |
oversized embedding batch (50 chunks/call) |
[CORRECTED] this trail was half-right and wrongly buried: dev pins 5, but the running host resolves empty → 50. The batch ceiling is part of the root cause; the release closes it |
| 3 |
the embedding timeout was too low → raise 300s → 1800s |
[MEASURED] same failure at smaller timeouts; raising a ceiling because reality exceeded it is not a fix, and the 1800s budget made every downstream breaker too slow to fire |
| 4 |
the warm-provider heal loop drives the CPU |
[MEASURED] 1987 of 3053 heal events are deferred, and the provider log shows zero requests arriving |
| 5 |
our 30-minute timeout abandons a request the provider keeps computing |
[MEASURED] the runner is pinned from t=0, before any request could have been abandoned |
| 6 |
a 240MB WAL proves a held reader starves checkpointing |
[MEASURED] log fell 11475→3425 while the file stayed 240.5MB; the size was a high-water mark |
| 7 |
docker timeout: 45s disconnects a client → orphaned provider work |
[SOURCE-READ] the canary is a long-lived boot loop inside the server, not a per-healthcheck dispatch; the client never vanished |
| 8 |
abort + socket teardown (Connection: close) releases the runner |
[SOURCE-READ] the frozen witness aborted with the socket already destroyed and the runner still held 397–400% with zero sockets |
| 9 |
restore the retired stuck-runner liveness canary |
[SOURCE-READ] retired for cause, zero callers; at NUM_PARALLEL=1 busy and stuck fail a completion canary identically; it probed chat while the burner is embedding |
The pattern across 1–9: a mechanism was proposed from a remembered phrase, not from the source or the plane.
Standing rules for this ticket
- No new diagnostic tooling. Every measurement above came from probes that already existed; the deployment-state snapshot carries a 32KB log tail per container (
services[].logs.text) — the root cause sat in that field all day. Read it before proposing anything.
- Tag every claim —
[MEASURED] / [SOURCE-READ] / [FALSIFIED] / [CORRECTED] — or it does not go in this body.
- A mechanism may not be implemented until an acceptance test for it fails on the deployed revision.
- Never validate our own uncertainty on the external plane: no quarantines, staged tests, or probe runs there; one cumulative revision, one deployment, outcome verification only.
- This body is frozen except for release-outcome updates. Findings go to comments; only a confirmed outcome (or a falsification of a row above) edits the body, and the edit must cite its instrument.
What matters
Exactly two symptoms are being tracked. Everything else in this ticket's history is instrumentation about them.
S1 — four CPU cores are pinned and never recover, with zero model work in flight. S2 — multi-tenant KB ingestion has never once succeeded end-to-end.
They are stated separately on purpose. A single-cause story has been asserted for them four times and falsified four times.
Release state — 2026-08-12
[CORRECTED]The fix set was never on the external plane. The deployed revision — read from the field that owns it,healthcheck.deployedRevision— is3f9f8343a8(2026-08-10), which predates the entire #16706 fix set. The previous body's claim that the recovery was "already deployed and did not act" was false and is retracted (2026-08-12 07:46Z); the half-night lane it created ("why did #16884 not fire") was chasing a phantom.[SOURCE-READ]The cumulative release is merged: private !57 (2026-08-12). It pins neodev@fcc0f8149422d83f481969dfd866480b96c40a8f— 74 commits past the deployed revision — and ships as one client letter: copy one compose file (embedding batch default 5), set one.envpin,docker compose build+up -d --force-recreate --wait. Volumes preserved; neverdown -v. The force-recreate replaces the model container, clearing the currently-wedged runner at deploy time.Awaiting: the client executes the letter → ordinary outcome verification only (S1: CPU settles to idle and stays there under normal producers; S2:
lastIngestedRevnon-null and documents surviving recreation). No experiments, probes, or staged tests on the external plane.[MEASURED]The same fix set ran on our canonical plane at271bb132d8(10 commits before the pin, all fix commits aboard) with 3/3 tenant repos durably ingested, zero consecutive failures (deployment-state snapshot 2026-08-12T08:32Z). Our plane is being moved to the exact release pin for parity.Root cause —
[MEASURED]2026-08-11 13:40Z from the orchestrator container logA single KB tenant-ingestion embedding batch exceeds the 30-minute provider timeout and retries 5×. That is the 400%.
OLLAMA_NUM_PARALLEL=1slot, so the KB canary's 30-second probe can never acquire it ⇒ probe-timeout streak 27 ⇒ the KB is health-gated out of its own tools.num_ctx 32768.Why a batch takes >30 minutes when one embed takes 150 ms: batch size, not hardware.
[SOURCE-READ]The running host resolves the batch size empty → shipped default 50; measured on comparable CPU inference, a 14-chunk batch took 8m25s — 50 puts a single batch at the ceiling, and a batch that cannot finish stores nothing. The release compose defaults it to 5.The upstream anchor —
ollama/ollama#11889, with its honest bounds[MEASURED]The provider does not observe cancellation while it waits: abandoned work keeps computing. The frozen witness aborted anode:http.request({signal})— socket already destroyed — and the runner still held 397–400% CPU with zero established sockets. AtNUM_PARALLEL=1the runner slot is then dead to all work. Live confirmation from host telemetry 2026-08-12 09:19Z: the pinned task is the embedding runner at 401% CPU with 101h50m accumulated CPU time over a ~25h container life — four cores for its entire life — while the chat runner idles at 0%.[CORRECTED]Demotion recorded 2026-08-11: the specific orphan chain first proposed (docker healthchecktimeout: 45s→ client disconnect → orphan) is false — the canary is a long-lived in-process loop, not a per-healthcheck dispatch; no transport close was demonstrated on that path. #11889 is the mechanism class, not a proven per-request chain on that plane.⛔ Do NOT wire the abort signal through
#embedOllamatoprovider.embed. It is the unsafe direction.[MEASURED]An early abort STRANDS the runner at 397–400% (this ticket's own witness).settleCallerWhileProviderContinues— return the caller, let the provider finish — is the design, not the bug. Three PRs died on the opposite premise: #16956, #16957, #17003.Rule 0 — two facts
complete,outstanding: 0.[MEASURED][MEASURED], live as of 2026-08-12A does not explain B. Retracted three times by three maintainers; do not re-couple them.
The fix ledger — what ships in the release pin
S1 (CPU):
ddd2f6a89d#16942 —Ollama.stream()gets a cancellation surface (stop abandoning work un-cancelled; the #11889 answer on our side)fcabb64595#16996 — no native-Ollama timeout handoff across tenant repos10a29b1a79#16978 — provider timeouts are not retried in-cycleaccfdb0c1a#16943 — native Ollama admission enforced and reported874e6ab26b#16950 — a resident model is no longer reported missing forever (kills the warm loop)73eb465793#16916 — provider-readiness deadlines actually reach the servicesS2 (ingestion):
31b8452d50#16889 — a repo with nothing to ingest completes instead of backing off foreverd25c397def#16964 — a failed vector write does not re-buy identical embeddingseff78e326e#16959 — tenant repo waiters stay behind active work9dc2a098d7#16899 — embedding-probe policy is reachable by deployments63da0eabdf#16812 — Memory Core accepted-response bounded after WALDeployment wiring (private repo, rides the same letter): embedding batch default 5; kb healthcheck
--expected-status; chat fallbackgemma4:26b; probe deadlines.Open hardening, explicitly NOT gating this release: #16695, #16780 (its merged slice #16959 IS aboard), #16853.
Falsified trails — do not re-run these
Each cost hours. Each is closed by evidence, not by opinion.
modelvsmodel:latest) drives a warm loop that burns the cores[CORRECTED]the original kill-reason ("the fix #16950 is already deployed and the symptom persists") rested on the false deployment fact. The trail stays dead for a better reason: the measured burner is the batch-occupancy chain above, and #16950 ships in the release pin[CORRECTED]this trail was half-right and wrongly buried:devpins 5, but the running host resolves empty → 50. The batch ceiling is part of the root cause; the release closes it[MEASURED]same failure at smaller timeouts; raising a ceiling because reality exceeded it is not a fix, and the 1800s budget made every downstream breaker too slow to firewarm-providerheal loop drives the CPU[MEASURED]1987 of 3053 heal events aredeferred, and the provider log shows zero requests arriving[MEASURED]the runner is pinned from t=0, before any request could have been abandoned[MEASURED]logfell 11475→3425 while the file stayed 240.5MB; the size was a high-water marktimeout: 45sdisconnects a client → orphaned provider work[SOURCE-READ]the canary is a long-lived boot loop inside the server, not a per-healthcheck dispatch; the client never vanishedConnection: close) releases the runner[SOURCE-READ]the frozen witness aborted with the socket already destroyed and the runner still held 397–400% with zero sockets[SOURCE-READ]retired for cause, zero callers; atNUM_PARALLEL=1busy and stuck fail a completion canary identically; it probed chat while the burner is embeddingThe pattern across 1–9: a mechanism was proposed from a remembered phrase, not from the source or the plane.
Standing rules for this ticket
services[].logs.text) — the root cause sat in that field all day. Read it before proposing anything.[MEASURED]/[SOURCE-READ]/[FALSIFIED]/[CORRECTED]— or it does not go in this body.