LearnNewsExamplesServices
Frontmatter
id13928
titleqwen3-embedding continuous CPU burn on a cloud deployment — root unconfirmed (re-ingest loop vs CPU backlog vs wedged inference)
stateClosed
labels
bugaimodel-experience
assigneesneo-gpt
createdAtJun 23, 2026, 3:34 PM
updatedAtJun 23, 2026, 5:15 PM
githubUrlhttps://github.com/neomjs/neo/issues/13928
authorneo-opus-grace
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 23, 2026, 5:15 PM

qwen3-embedding continuous CPU burn on a cloud deployment — root unconfirmed (re-ingest loop vs CPU backlog vs wedged inference)

Closed v13.1.0/archive-v13-1-0-chunk-6 bugaimodel-experience
neo-opus-grace
neo-opus-grace commented on Jun 23, 2026, 3:34 PM

Confirmed symptom (operator-verified via ollama ps on a real cloud deployment, 2026-06-23): the qwen3-embedding model pins ~100% CPU (4 cores; 113 CPU-hours over 4 days) on a CPU-only host, keep_alive Forever, context 40960. The chat/REM burn-stop (#13919/#13922) does NOT address this — different model, different path.

The ambiguity

Caller network is LOW (knowledge container ~9Kb/3Kb TX, orchestrator ~648b) while embedding burns 100% continuously. A high-frequency request flood would show heavy caller TX; it does not. So the root is ambiguous between:

  1. Re-ingestion loop: tenant-repo-sync wakes every 60s (sweepCadenceMs) + per-repo re-sync every 30min (tenantRepoSyncMs); if ingestion dedup does not skip re-embedding unchanged content, that is a continuous re-embed.
  2. CPU-bound backlog: a large first-ingest grinding slowly (13GB embedder on CPU is slow); legitimate but may never drain.
  3. Wedged or oversized single embedding: one inference spinning the runner (best fits the low caller traffic).

Disambiguating data (deployment host access)

  • model container logs: a stream of embedding requests (root 1) vs silence around one long computation (root 3)
  • knowledge container logs: is KB actively re-ingesting
  • request rate: flood (ingestion) vs single long computation (wedged/backlog)

Likely fixes once root is known

  • Root 1: ensure re-sync dedup skips already-embedded unchanged content (no re-embed loop); verify the contentHash/upsert path actually short-circuits embedding.
  • Root 2/3: embedding-path backpressure + per-input size cap; treat an embedding input exceeding context 40960 as skip-not-grind.
  • Cross-cutting: the #13923 eval-attribution probe deployed would auto-classify busy-vs-stuck remotely; #13914 would expose the logs/health without host access.

Relates: #13914 (remote observability), #13923 (eval attribution instrument), #13860 (diagnostics epic). Separate: the memory container is unhealthy/down on the same deployment — needs its own logs, distinct root.

tobiu referenced in commit 4032501 - "fix(ai): guard local embedding inputs (#13928) (#13929)" on Jun 23, 2026, 5:15 PM
tobiu closed this issue on Jun 23, 2026, 5:15 PM