Attribute model-provider load to operation stages

Context

Neo can currently answer three adjacent questions, but not the one an operator needs when a shared model endpoint is busy:

• providerReadinessHelper exposes native Ollama eval attribution by loaded model and broad role (chat versus embedding) from ai/services/graph/providerReadinessHelper.mjs:1856-1882; that was the scope of #13923 / PR #13927.
• get_memory_core_tool_metrics exposes whole MCP-call duration, unfinished calls, and recent slow completions from MemoryCoreRecorderService.getMemoryCoreToolMetrics(); #16723 / PR #16724 deliberately stop at the tool boundary.
• get_rem_pipeline_state records REM cycle and phase wall-clock timing from #12068 / #12088; it does not cover Knowledge Base or Memory Core consumers and does not split provider queue wait from provider execution.

The provider calls already carry part of the missing vocabulary. Local chat requests share the process-wide InteractiveBatchQueue in ai/provider/buildChatModel.mjs:8-17, and consumers pass diagnostic operationLabel values such as ask_knowledge_base synthesis, session summarization, and miniSummary generation. Embedding calls accept a bounded operationLabel and create an internal {phase, startedAt} record in ai/services/memory-core/TextEmbeddingService.mjs:44-76,1059-1066,1134-1141.

None of those labels becomes a bounded diagnostic ledger for successful calls. The chat queue stores only {task, priority, resolve, reject} and exposes no enqueue/start/complete times (ai/provider/InteractiveBatchQueue.mjs:26-99). TextEmbeddingService currently emits its bounded timing only on abort (ai/services/memory-core/TextEmbeddingService.mjs:180-193). Several embedding consumers also omit a source-specific label, including Knowledge Base query embedding and ingestion batches.

The Problem

When a local or remote provider is saturated, current diagnostics can identify the busy model role or a slow outer MCP tool, but cannot identify the originating operation stage.

That distinction is not cosmetic:

• an ask request performs query embedding before it can synthesize; with no references it performs no synthesis at all;
• add_memory can acknowledge durable WAL capture before later embedding drain and mini-summary work, so its MCP duration is not the duration of that provider work;
• tenant ingestion can issue large embedding batches independently of interactive queries;
• the embedding write canary is lifecycle-owned after #16222; a health read is not the canary's causal owner;
• REM graph calls, session summaries, and mini-summaries share chat-provider capacity but have different priority and recovery semantics.

A CPU graph, provider role, or tool-duration maximum therefore cannot answer “which Neo operation is consuming the model endpoint?” Guessing from temporal overlap can blame the wrong subsystem and lead to the wrong tuning or recovery action.

The Architectural Reality

• buildChatModel(...).generateContent() is the provider-agnostic chat boundary. Local OpenAI-compatible and Ollama calls enter one InteractiveBatchQueue; Gemini bypasses that queue (ai/provider/buildChatModel.mjs:95-223).
• InteractiveBatchQueue owns admission order. It is the only truthful place to separate queue wait from task execution for local chat calls.
• TextEmbeddingService.embedText() and embedTexts() own the embedding-provider boundary, including the OpenAI-compatible interactive/batch queues and native Ollama/Gemini calls.
• Existing free-form operationLabel values are diagnostic prose. Some contain session or asset identifiers in graph paths, so they are not safe as a retained grouping key.
• MemoryCoreRecorderService and KBRecorderService already use the shared Memory Core SQLite artifact for bounded operational telemetry. get_memory_core_tool_metrics is the existing public diagnostic observer; a second MCP tool is unnecessary.
• ollamaEvalAttribution remains the provider/model-role signal. This ticket adds the missing originating-stage axis; it does not replace or reinterpret that signal.
• REM's existing per-phase state remains the cycle-level authority. Provider activity supplements it with queue/execution timing and covers non-REM consumers.

The Fix

Extend the existing provider and recorder boundaries with a bounded, redacted operation-stage ledger.

1. Add a stable, low-cardinality operationStage value alongside the existing human-readable operationLabel. The stage is source-owned and allowlisted; the raw label is never persisted or used as an aggregation key.
2. Instrument InteractiveBatchQueue / buildChatModel so each chat call records enqueue, provider-start, and settle boundaries. Strip operationStage and queue-control fields before the provider request.
3. Instrument TextEmbeddingService at its real provider boundaries, preserving its existing interactive/batch semantics while recording the same lifecycle.
4. Persist only the bounded structural row needed for attribution: {activityId, service, operationStage, role, provider, model, priority, enqueuedAt, startedAt, completedAt, queueDisposition, queueWaitMs, executionMs, success, failureStage}.
5. Extend get_memory_core_tool_metrics with a NEW providerActivity projection containing bounded aggregates, in-flight rows, and recent completed rows. Reuse its effective lookback/limit semantics where applicable; disabled or unavailable telemetry must be explicit.
6. Cover the current high-value stage families with stable keys: Knowledge Base query embedding, ask synthesis, tenant-ingestion embedding, Memory Core WAL-drain embedding, mini-summary, session summary, REM Tri-Vector/topology work, and embedding canaries. Unclassified callers report unknown; the observer never infers a stage from timing, model name, or raw label.
7. Keep the read path model-free and side-effect free. Telemetry must not change queue order, provider routing, retry policy, health status, or recovery decisions.

For providers with no Neo-owned queue, such as remote Gemini, queueDisposition is not-applicable and queueWaitMs is null; zero would falsely claim a measured queue interval.

Contract Ledger Matrix

Decision Record impact

aligned-with ADR 0025. This is a false-positive-safe diagnostic producer/read surface and adds no recovery action or diagnosis class. No ADR amendment is expected.

If implementation needs a retention or cap setting, it is also aligned-with ADR 0019: add a declarative leaf to the canonical config owner and read it at the recorder use site. This ticket does not authorize provider-selection, endpoint-routing, or hidden-default changes.

Acceptance Criteria

• A deterministic local-chat fixture proves queue wait and provider execution are recorded separately while interactive-before-batch ordering remains unchanged.
• A remote-provider fixture reports queueDisposition: 'not-applicable' and queueWaitMs: null, not a fabricated zero.
• An ask fixture with references emits distinct query-embedding and synthesis stages; an empty-reference fixture emits query embedding only.
• Deferred Memory Core work is attributed to WAL-drain embedding and mini-summary stages rather than to the earlier add_memory tool call.
• Tenant-ingestion embedding, session summary, REM graph work, and embedding-canary fixtures each resolve to a stable low-cardinality stage.
• Unclassified provider callers report operationStage: 'unknown'; no stage is inferred from temporal overlap, model name, tool name, or free-form operationLabel.
• The observer exposes bounded aggregate, in-flight, and recent-completion projections with explicit unavailable/partial status.
• Rows distinguish role, provider, model, service, and priority without exposing endpoint URLs or credentials.
• Prompts, inputs, results, vectors, raw operation labels, and tenant/repo/user/agent/session/asset identities cannot appear in storage or the public projection.
• Existing get_memory_core_tool_metrics fields, REM phase telemetry, provider role attribution, queue behavior, retries, health verdicts, and recovery decisions remain unchanged.
• OpenAPI/service parity, focused provider/queue/recorder tests, and config-template SSOT gates pass.
• L3 post-merge: under two concurrent known workloads, the public observer distinguishes their operation stages and separates queue wait from provider execution without shell/log inference.

Out of Scope

• Selecting Gemma, Gemini, or another model.
• Choosing one versus multiple model instances, parallel embedding lanes, or shared versus isolated endpoints.
• Ask/mini-summary reasoning-effort control, owned by #16768.
• Model eviction/root-cause work, owned by #14154.
• Retry-amplification or contention policy changes, covered by #16012 and existing bounded-retry work.
• Container recovery/control-loop changes, owned by #16766.
• Replacing ollamaEvalAttribution, REM run-state telemetry, or MCP tool-call telemetry.
• Persisting prompts, outputs, token content, embeddings, or caller identity.
• Adding a new MCP tool.

Avoided Traps

• Whole tool duration equals provider work. Deferred/background work breaks that equivalence.
• Chat versus embedding equals causal ownership. It identifies a model role, not the Neo operation using it.
• Raw operationLabel as a metric key. Current graph labels can contain session/asset identifiers and create unbounded cardinality.
• Zero queue wait for a provider with no Neo queue. That asserts a measurement that did not happen.
• Log parsing as the contract. Logs may remain useful evidence, but the observer needs structured bounded data.
• A new tool for each diagnostic. The existing metrics observer owns this operational family.
• Telemetry changes scheduling. Observation must not alter queue, routing, retry, health, or recovery semantics.

Related

Related: #13923 #13927 #16723 #16724 #12068 #12088 #12090 #16222 #16768 #14154 #16012 #16766

Creation-Gate Record

• Structure map: existing owners are ai/provider, TextEmbeddingService, the KB/MC recorders, and the existing Memory Core metrics observer; no new public server or tool is needed.
• Source V-B-A: the local chat queue has no lifecycle timestamps; successful embedding calls have no retained stage ledger; the current public observer groups whole calls by tool; provider readiness groups Ollama samples by model/role.
• Prior art: Knowledge Base retrieval surfaced REM phase timing (#12068/#12088) and chat-versus-embedding attribution (#13923), both preserved as adjacent authorities rather than treated as this missing cross-service provider-stage axis.
• Duplicate sweep: latest-created open issues, all-state GitHub title/body searches, synced issue/discussion artifacts, and the latest 30 A2A messages were checked immediately before creation. No equivalent ticket or competing claim was found.
• External deployment observations were deliberately excluded; this ticket rests only on current Neo source and shipped contracts.

Origin Session ID: 0473b65d-090b-412a-a926-b90e5851f58a

Retrieval Hint: model provider operationStage queue wait execution ask synthesis WAL drain tenant ingestion REM telemetry #13923 #16723 #12088