LearnNewsExamplesServices
Frontmatter
id13923
title[detect] Per-model throughput + chat-vs-embedding load-attribution probe (ollama eval metrics)
stateClosed
labels
enhancementaiarchitecturemodel-experience
assigneesneo-gpt
createdAtJun 23, 2026, 2:11 PM
updatedAtJun 23, 2026, 3:36 PM
githubUrlhttps://github.com/neomjs/neo/issues/13923
authorneo-opus-grace
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 23, 2026, 3:36 PM

[detect] Per-model throughput + chat-vs-embedding load-attribution probe (ollama eval metrics)

Closed v13.1.0/archive-v13-1-0-chunk-6 enhancementaiarchitecturemodel-experience
neo-opus-grace
neo-opus-grace commented on Jun 23, 2026, 2:11 PM

Parent epic: #13860 (diagnostics) · Grounds: ADR-0025 §2.4 (false-positive-safe detect) + AC-4 (model canaries classify contention-first). Sibling of the detect→diagnose producer ticket.

Problem

When the local-model (ollama) container pins its CPU cap, neo cannot attribute the load — chat (gemma) vs embedding (qwen3-embedding) — nor measure per-model throughput, nor tell a busy runner from a resident-but-stuck one. Today's ollama diagnostics check residency / readiness / loaded-context only (providerReadinessHelper.mjs, #13900 / #13885 / #13851); none reads tokens/sec. This blindness directly blocked live cloud-fire triage: we could not answer "is it the chat or the embedding model burning 4 cores?" — and that answer decides whether the REM burn-stop (#13918/#13919) even applies.

The signal is already available

Ollama returns eval_count / eval_duration (+ prompt_eval_count / prompt_eval_duration) per call → tokens/sec is derivable; /api/ps lists loaded models + state. neo simply never captures or surfaces it.

Scope

A diagnostics probe that:

  1. samples per-loaded-model tokens/sec from ollama eval metrics over a window;
  2. attributes shared-cap consumption to chat vs embedding (which model is eating the CPU/parallel slots);
  3. flags resident-but-~zero-eval-progress (stuck runner) distinct from genuinely busy;
  4. surfaces the result for #13914 (observability) and as the contention fact feeding the ADR-0025 §2.4 detect model.

Acceptance Criteria

  • Per-model tokens/sec sampled from ollama eval metrics over a window.
  • Chat-vs-embedding load attribution on the shared cap.
  • Stuck (resident, ~0 eval progress over the window) flagged distinct from busy.
  • Exposed via the diagnostics / observability surface (#13914).
  • No new config reader — extend an existing leaf() (ADR-0019).

This is the instrument for ADR-0025 §1's motivating failure class: "local-model saturates its CPU cap, embedding starves behind chat on the shared cap" — and is immediately useful for cloud triage even before the full detect→diagnose→act loop exists.

tobiu referenced in commit 96c4bd8 - "feat(ai): expose Ollama eval attribution (#13923) (#13927)" on Jun 23, 2026, 3:36 PM
tobiu closed this issue on Jun 23, 2026, 3:36 PM