Embedding work is not bounded by pending items, so an empty corpus can sustain four cores indefinitely

Context

An external Agent OS plane was observed at 2026-08-09T12:06Z with its model container holding 398.6% CPU sustained (~4 cores) and 12.5 GB resident, with a ~24 h uptime and that load present throughout. On the same plane, at the same instant:

• the Knowledge Base collection is empty — ingestion has never produced a document on that deployment;
• the Memory Core corpus is small — kilobytes-to-megabytes of turns, not a backlog;
• the host is otherwise idle: load 5.54 on 64 cores (7.4%), memory 10% of 252 GB, no disk pressure.

The observation is a contradiction, and the contradiction is the finding. Embedding work is a function of items to embed. Zero documents plus a small turn corpus is close to zero work. Four cores of continuous inference for a day is not explained by anything the corpus contains — so the consumption is not being driven by the workload, and nothing in Neo notices.

The Problem

There is no invariant anywhere tying provider consumption to work in flight. Every existing surface answers an adjacent question:

• #16770 (provider-stage attribution) will say which operation stage is calling the provider — necessary, and not sufficient: it reports attribution, never a bound. A stage burning four cores over an empty corpus is a perfectly well-attributed row.
• #14154 covers the inverse failure — the embedder becoming un-resident mid-sync (404 / model evicted). This is a model that is resident and never stops working.
• #16566 covers tenant ingestion failing at the embed stage. That explains an empty KB. It does not explain, and does not bound, continued CPU after the failure.
• #16630 / #16763 bound heap; nothing bounds provider compute against pending work.
• #16012 covers retry amplification across attempts; a single request that never completes is a different shape and is not covered.

So a deployment can burn its entire provider budget producing nothing, indefinitely, and every diagnostic surface reports normally.

Corrected 2026-08-09, before any implementation — the original hypothesis was falsified by an existing ticket

This section originally proposed that embedding SUCCEEDS and persistence fails, so each sweep re-embeds the same set forever. That is contradicted by evidence already recorded on #16566: the observed plane's tenant ingestion reports KB_VECTOR_EMBED_FAILED with lastIngestedRev: null — the pre-first-ingest stage. The embed does not succeed. The retraction is kept rather than quietly rewritten because the wrong version was filed, and because it was filed by skipping the duplicate sweep that would have caught it.

What the evidence actually supports. #16566 additionally records, on the canonical plane, a kbSync re-embed that took the heavy-maintenance slot and still held it 13 hours later, starving both tenant-repo-sync and REM consolidation. So the shape is not a converging loop that stores nothing; it is embedding work that does not terminate, while the corpus it would have produced stays empty.

That reframes this ticket's contribution and narrows it honestly:

• #16566 owns the ingestion failure itself — why embed fails, the starvation it causes, and the reporting that hides it. All of that is in scope there and out of scope here.
• What no ticket owns is the BOUND. Nothing ties provider consumption to work in flight, so an embedding path that never terminates is indistinguishable from one making progress, from every surface we have. Four cores against zero pending items is not reported as anomalous by anything — not by attribution (#16770 says which stage, never whether it should be running), not by health, not by the deployment snapshot.
• And nothing declares the concurrency. How many embedding requests Neo may have in flight against one provider is emergent from how many callers exist, so a plane can multiply its own contention with no Neo surface reporting it — which is the difference between one stuck request and four.

The discriminator remains the same instrument and is still worth building: work submitted per sweep against items selected, and a repeat-submission counter over a window. It distinguishes "does not terminate" from "legitimately busy" without needing to know which of the two is true first.

The Fix

Direction, not premature mechanism.

1. Bound provider work by pending work. A sweep's embedding submissions must be a function of the items it selected. A sweep that embeds more than it selected, or that selects the same items it selected last time, is a defect the system can see.
2. Make repeat-embedding of identical content detectable. A content-identity counter over a bounded window, exposed on the existing metrics observer — never a new tool. Re-embed ratio is the discriminator between "the loop does not converge" and "the load is legitimate".
3. Give the embedding request an effective deadline, and prove it is effective. A request that cannot complete must be abandoned rather than held; a held request is indistinguishable from progress from every surface we currently have.
4. Declare the concurrency. How many embedding requests may be in flight against one provider is a declared, bounded, reported number — not an emergent property of how many callers happen to exist.
5. Alarm on the disproportion, not on the absolute. "Four cores" is not a defect; "four cores against zero pending items" is. The condition is a ratio, and it is exactly the one an operator cannot compute today.

Acceptance Criteria

Progress ledger (updated 2026-08-10, second revision). This ticket is delivered through narrow leaves — one PR per AC — because no single PR can honestly Resolves a list this wide. Ticks below carry the delivering leaf so a reader can tell shipped work from remaining work without opening every linked PR. A tick means the evidence exists and is reviewed; the State column is the authority on whether it has landed.

AC	Leaf	PR	State
AC-4 deadline	#16822	#16823	merged
AC-7 (memory-pending half)	#16835	#16836	merged
AC-1 zero-submission control	#16866	#16867	merged (69aaeabcd1)
AC-3 re-embed ratio + exposure	#16866	#16867	merged (69aaeabcd1)
AC-6 negative control	#16866	#16867	merged (69aaeabcd1)
AC-5 concurrency	#16861	#16862	merged (7ef07a7ee3)
AC-2 non-convergence	#16875	#16876	open — RC addressed at 19553863d1, re-review pending
AC-7 (KB half)	—	—	partially closed by #16867 — see AC-7
AC-8	—	—	ticks when the rest land

Successor tickets opened by @neo-gpt-emmy from exact-head review, recorded here so they are not lost between PRs: #16879 (the non-convergence guard must bind the deployed VectorService.embed() selector — addressed at 19553863d1, hers to close) and #16880 (getOllamaEmbeddingAdmission() has no production consumer, and observeUnqueuedProviderActivity records genuinely-queued work as not-applicable — the "declared but inert" shape AC-5's ninth criterion existed to prevent, one layer up, missed on my own surface).

• A fixture with zero pending items proves the embedding path issues zero provider submissions — the direct control for the observed state. — #16866 / PR #16867. Asserted at the provider boundary, not the return value: a function can report embedded: 0 while still having called out, and it is the call that burns the core. The WAL drain half was already covered — drainCycle's AC1 spec pins collection.addCalls at 1 across a second, zero-pending cycle.

  V-B-A note (2026-08-10): the memory WAL drain path already satisfies this — drainWalOnce guards embedBatch behind if (drainable.length > 0), so zero pending issues zero submissions. This AC is a control, not a repair, and it must be written against the KB sync path (VectorService.embedChunks), which is where the observed empty-corpus load lived. Recording it so the fixture is not written against the path that was never broken.

• A fixture where persistence fails proves the next sweep does not re-select and re-embed the identical set unboundedly; the non-convergence is detected and reported rather than repeated.

  Delivered by #16875 / PR #16876, open. Ticks on merge.

  The first attempt at this leaf was a vacuous guard, and that is worth carrying forward rather than only fixing. It ran its own sweep loop and supplied its own selected set, so a correct detect-and-stop implementation kept it green — demonstrated by mutation, not argued: under a detect-and-stop mutation the old spec passed 2/2 while the repaired one failed. The repair binds both production selectors, because VectorService.embed() carries an inline existingIds filter and selectResumableChunks is a second implementation of the same rule used only by the shadow-swap path. That duplication is an open production concern this leaf does not fix.

• Re-embed ratio over a bounded window is exposed on the existing observer, and a fixture with deliberately repeated content moves it above 1 while a clean run holds it at 1. — #16866 / PR #16867, merged 69aaeabcd1. Both halves verified against the branch rather than against the PR title: ratio is pinned at exactly 1 for a clean run and > 1 (and < 1.01) for honestly duplicated work, and reembedRatio is a required property of the existing observer response in openapi.yaml, sitting beside providerActivity and walDrain rather than behind a new tool.

  Ticked 2026-08-10, unticked the same day, and now re-ticked on a different mechanism. The history is retained deliberately, because a reader who sees only the current state cannot tell a criterion that was never claimed from one that was claimed, withdrawn, and re-earned.

  The first tick relied on a metrics surface that was process-local and structurally blind to the KB producer: TextEmbeddingService is a per-process singleton, so VectorService recorded into kb-server's window while the observer read mc-server's, and kb-server never imports MemoryCoreRecorderService at all. @neo-gpt found it; I reverted the wiring and unticked.

  The repair replaces the process-local window with a shared SQLite ledger already mounted into both recorder processes: KBRecorderService and MemoryCoreRecorderService write through independent connections, and the Memory Core reads one transactionally coherent interval. @neo-gpt authored it; @neo-gpt-emmy reviewed and approved it. Her [RETROSPECTIVE] states the general form better than my original note did: a coherent process-local window is still invisible at the deployed consumer.

• An embedding request that never completes is abandoned at a stated deadline, asserted by a fixture that would hang without it. The deadline is a declared config leaf read at its use site, not a constant. — #16822 / PR #16823, merged. Intake landed on a narrower shape than the wording implies and the difference is recorded in this ticket's 2026-08-09 comment: the request-level deadline already existed as openAiCompatible.batchEmbeddingTimeoutMs; what had none was the operation, which is the thing that holds the lease.

• In-flight embedding concurrency is declared and reported; a fixture proves the cap is enforced rather than advisory. — #16861 / PR #16862, merged 7ef07a7ee3. The V-B-A partly falsified this AC's premise: openAiCompatible was already serialized by its post queue (cap 1 by construction, merely undeclared), while ollama had no admission control at all — it reached the provider through observeUnqueuedProviderActivity, which observes and does not admit. So the delivery declares the existing cap and builds the missing one, rather than adding a second limiter. Enforcement is proven by a cap-of-2 control, without which accidentally-serial code satisfies a cap-of-1 assertion and the suite certifies a cap that does nothing.

  Two release blockers were found at exact-head review and repaired in-branch rather than through an RC round (9bc05dda52, authored by @neo-gpt): a caller aborted while queued stayed hung behind the occupied slot, and the numeric leaf admitted two requests for a declared cap of 1.5. As PR author I falsified that repair independently rather than inheriting its author's verdict — the witness was validated RED on the pre-repair head and is GREEN at 9bc05dda52. Formal review is routed to a second GPT seat because the head carries the reviewer's own commit.

• Negative control: a legitimately large ingestion is NOT flagged. A ratio alarm that fires on real work is worse than none, because it will be disabled within a week. — #16866 / PR #16867, merged 69aaeabcd1. A 500-document ingest with one honestly-duplicated snippet lands just above 1, and the test pins the reported key set so the module cannot start adjudicating. The instrument reports; the judgement is deliberately not its.

• The disproportion is computable from the public surface alone — an operator can answer "is this provider load explained by pending work?" without shell access or log parsing.

  Three of four quadrants delivered as of 2026-08-10. Deliberately still unticked, and the missing quadrant is named rather than estimated. This AC needs a numerator (repeated work) and a denominator (pending work), on both producers:

  	Memory Core	Knowledge Base
  pending (denominator)	✅ walDrain, #16836 merged	❌ nothing
  repetition (numerator)	✅ reembedRatio, #16867	✅ reembedRatio, #16867

  #16867 closes the KB repetition gap that this note previously recorded as open: KBRecorderService.recordEmbeddingSubmissions stamps source: 'knowledge-base' into the shared ledger, so the ratio now spans both producers rather than one.

  What remains is narrower and better specified than "the KB half": KB ingestion publishes no pending-work figure at all. Verified rather than assumed — the recorder writes identity rows only, and walDrain is the memory WAL projection alone. So on the ingestion side an operator can see that work repeats but not how much work was outstanding, which is the exact ratio this ticket exists to make computable. This is the next leaf and it is currently unowned.

• Coverage fails against today's code and passes against the repair.

  Satisfied per-leaf so far (#16822, #16835, #16861, #16866 each carry a red-proof), but this AC is about the whole ticket and cannot tick until the remaining ACs land.

Out of Scope

• Which model is selected, or how many model instances a deployment chooses to run — this bounds and reports Neo's own submissions, it does not pick a topology.
• The embedder-eviction root cause (#14154) — the inverse condition.
• Why tenant ingestion fails at the embed stage, the starvation it causes, and its reporting honesty — all #16566. That ticket is the primary owner of the ingestion failure and its 13-hour slot-holding re-embed; this one owns only the missing BOUND and the undeclared concurrency. If the bound turns out to be better placed inside #16566's scheduling-fairness AC, this ticket should be closed into it rather than run in parallel.
• Retry amplification across attempts (#16012).
• Provider-stage attribution itself (#16770) — a prerequisite this ticket consumes rather than duplicates.

Avoided Traps

• Filing without the duplicate sweep. This ticket's original central hypothesis was falsified by a ticket that was already open, because the sweep was skipped on a self-assessed constraint that was never measured. The sweep is the cheap half; the wrong hypothesis is the expensive half.
• Reading CPU as throughput. Sustained inference is evidence of submissions, never of progress. The observed plane had maximal CPU and zero documents.
• Treating an empty corpus as "nothing has run yet". It is equally consistent with "everything has run, repeatedly, and stored nothing" — and those need opposite responses.
• Alarming on absolute load. The provider is supposed to be busy during ingestion. Only the ratio to pending work separates the two.
• Assuming the concurrency figure. The deployment topology is the operator's; what is missing is that Neo neither declares nor reports its own in-flight count, so contention it causes is invisible to it.
• Fixing this with a timeout alone. A deadline stops one stuck request; it does not stop a loop that completes each cycle successfully and persists nothing.
• Ticking an AC from a PR title. AC-3 was re-ticked only after reading the branch: the openapi.yaml required-property list, and the ledger spec's clean-run-equals-1 / duplicated-work-exceeds-1 pair. AC-7 was left unticked by the same check — the KB recorder was read and publishes no pending figure, so the tempting "the KB half is done now" was falsified before it reached this list.
• Observing across a process boundary with a process-local instrument. AC-3's first delivery was coherent, tested, and unreadable at the deployed consumer, because the recorder and the observer are different OS processes. A cross-process observation needs a cross-process substrate.

Related

#16770 (provider-stage attribution — the prerequisite instrument) · #14154 (embedder eviction, the inverse) · #16566 (tenant ingestion failing at embed) · #16549 / #16208 (Knowledge Base durability and store loss — candidate persistence half) · #16012 (retry amplification) · #16630 / #16763 (heap ceilings) · #16706 (deployment-readiness tracker)

Origin Session ID: a641ddac-565a-4fc8-adc1-6c25629bddb7

Retrieval Hint: embedding CPU burn empty corpus re-embed ratio pending items unbounded provider work convergence