OpenAI-compatible timeout bypasses the tenant-run circuit

Context

PR #16996 deliberately fixes the native-Ollama form of #16995: a provider timeout opens the tenant-run circuit synchronously before native Ollama admission hands its slot to another queued repository. Its exact approved head is 7774c072a20d3478f03297f1600a69e94e2a5f03.

The scope boundary is correct, but the review exposed a sibling defect in the OpenAI-compatible embedding lane. This is a follow-up rather than a widening of #16996.

Prior art is load-bearing. #12814 / PR #12818 established that local providers share one observable timeout contract even when their transports differ. The current authority is ai/provider/createTimeoutError.mjs, which owns both PROVIDER_TIMEOUT and OPENAI_COMPATIBLE_REQUEST_TIMEOUT.

A live latest-open sweep checked the newest 20 open issues plus the latest 30 all-state A2A messages on 2026-08-11; no equivalent ticket or lane was found. Exact GitHub searches for OpenAI-compatible tenant timeout, circuit, queue, handoff, onProviderTimeout, and OPENAI_COMPATIBLE_REQUEST_TIMEOUT returned no match.

The Agent OS structure-map gate was attempted during the predecessor intake and failed with Cannot create a string longer than 0x1fffffe8 characters; source ownership here was therefore established from exact origin/dev@668253ec2d, PR #16996's exact head, and the prior-art commits.

The Problem

OpenAI-compatible embeddings already run through a process-wide serialized queue. When the underlying request reaches its final socket timeout:

1. the transport rejects with OPENAI_COMPATIBLE_REQUEST_TIMEOUT and destroys the client request;
2. #drainOpenAiCompatiblePostQueue() rejects that task; and
3. the loop immediately selects the next queued task.

The tenant-run onProviderTimeout control is not carried into this queue and is never invoked at that handoff boundary. A second tenant repository can therefore dispatch in the same sweep immediately after the first repository times out. Destroying the client request is not proof that every OpenAI-compatible backend has stopped server-side compute.

Copying the native-Ollama if (timeout) { try hook } finally { release } branch into the OpenAI-compatible drain would fix one symptom by creating a second policy implementation. That is not acceptable: timeout identity and timeout-before-handoff ordering have already drifted into repeated code lists and provider-specific branches.

The Architectural Reality

The provider paths share a contract, not an admission implementation:

• Native Ollama uses a configurable cap plus waiter queue. Caller settlement can precede provider settlement, so admission remains owned by the provider promise.
• OpenAI-compatible uses a single-lane interactive-first embedding queue. A signal can synchronously remove a never-dispatched queued task, and the queue owns final retry exhaustion before advancing.
• Gemini is a remote, unqueued provider. With tenant concurrency greater than one, another repository may already be in flight before the first timeout. The local queued-provider guarantee cannot be claimed there.

The reusable unit is therefore narrow:

1. one shared typed timeout predicate;
2. one shared safe synchronous timeout notification; and
3. provider-specific invocation at each real admission-settlement boundary before admission advances.

InteractiveBatchQueue is not a drop-in replacement for both embedding paths. It is a single-lane chat scheduler and lacks the OpenAI embedding queue's queued-abort removal plus Ollama's provider-settlement ownership. Forcing the queues into one class would erase the cancellation semantics repaired by #16869.

The Fix

1. Extend the existing timeout SSOT in ai/provider/createTimeoutError.mjs with one provider-neutral predicate for exactly:
   • PROVIDER_TIMEOUT;
   • OPENAI_COMPATIBLE_REQUEST_TIMEOUT;
   • ETIMEDOUT; and
   • ESOCKETTIMEDOUT.
2. Replace the repeated four-code classifiers in touched consumers, including VectorService and the embedding drain cycle, with that predicate.
3. Add one shared ordered helper in the local embedding service that:
   • uses the shared predicate;
   • invokes the caller-owned onProviderTimeout(error) synchronously before admission advances;
   • does not own or synthesize the tenant-run circuit controller or circuit-open reason;
   • contains a callback error without replacing the source provider error when the callback has already opened its circuit; and
   • explicitly makes no dispatch-suppression guarantee when the callback throws before opening its caller-owned circuit.
4. Use that exact helper from both:
   • native Ollama's provider-promise rejection, before slot release; and
   • OpenAI-compatible's final queue-task rejection, before the drain can select another task.
5. Thread the already-normalized onProviderTimeout control through OpenAI-compatible single and batch embeddings. Fire it only after the queue task's internal contention/unload retries have exhausted, never for an intermediate retry.
6. Keep the run-scoped circuit, distinct circuit-open error, and later-run reset in TenantRepoSyncService. Do not change public MCP payloads.

Contract Ledger

Decision Record impact

None. This completes the shared timeout contract established by #12814 / PR #12818 and composes the existing tenant-run circuit. It does not add a provider, service, public protocol, or configuration authority.

Acceptance Criteria

• AC-1 — one common implementation: native Ollama and OpenAI-compatible call the same timeout-notification helper and the same timeout predicate. No provider-specific copy of timeout-code matching, post-open hook containment, or timeout-before-handoff ordering is added.
• AC-2 — OpenAI-compatible production composition: with two tenant repositories, tenant concurrency 2, and one OpenAI-compatible queue lane, repository A reaches the provider while B is genuinely queued; A's final typed timeout opens the circuit and B makes zero provider calls in that run.
• AC-3 — distinct outcomes: A retains its real OpenAI-compatible provider-timeout outcome; B receives KB_VECTOR_EMBED_PROVIDER_CIRCUIT_OPEN.
• AC-4 — ordering has teeth: with B genuinely queued behind A, synchronous circuit opening keeps B at zero provider calls; if onProviderTimeout returns before opening the circuit and defers the caller-owned abort by one microtask without awaiting or otherwise stalling the drain, the queue advances and B reaches the provider. Awaiting inside the drain is not a valid falsifier because it also pauses queue advancement.
• AC-5 — Ollama parity: the existing #16996 cap-1/two-repo witness remains green through the shared helper. Removing either provider's shared-helper call turns its paired control red.
• AC-6 — final failure only: OpenAI-compatible contention/unload retry paths do not open the circuit on an intermediate failure; only the final logical queue-task timeout does.
• AC-7 — negative matrix: caller abort, circuit-open, generic provider error, provider-busy/model-load error, and timeout-shaped store/write errors do not open or re-open the provider circuit.
• AC-8 — post-open hook containment: a callback that synchronously opens its caller-owned circuit and then throws cannot replace A's source timeout, leak/stall the queue, undo queued-task removal, or permit B to dispatch. A callback that throws before opening the circuit is explicitly outside this guarantee; the shared helper does not invent fallback circuit authority.
• AC-9 — honest settlement: no path claims server-side settlement, aborts already-dispatched work through this circuit, or releases native Ollama admission from caller settlement.
• AC-10 — fresh run: a later tenant sweep creates a fresh circuit and resumes B from durable Knowledge Base state.
• AC-11 — no false provider neutrality: Gemini remains outside this queued-local-provider contract, and source/JSDoc name the boundary explicitly.

Out of Scope

• Unifying the native Ollama and OpenAI-compatible queue implementations.
• Adding a Gemini tenant circuit or a provider-neutral concurrency claim.
• Cancelling already-dispatched transport work or claiming server compute has stopped.
• Changing public MCP schemas or adding AiConfig leaves.
• Adaptive batch sizing, deployment tuning, or plane-global admission policy.

Avoided Traps

• Copying the Ollama branch. A second timeout-code list and hook wrapper guarantees future drift.
• Forcing queue reuse. The two queues have different capacity, abort, priority, and provider-settlement semantics.
• Using provider telemetry as policy. createProviderActivityLifecycle is deliberately best-effort and may swallow failures; admission control cannot live there.
• Opening the circuit in Vector/Tenant catches. The queue can already have admitted B before those catches run.
• Inventing fallback circuit authority. The embedding service owns notification ordering, not the tenant-run controller or circuit-open reason; it must not synthesize B's outcome when a caller callback fails before opening the circuit.
• Firing on intermediate retries. The queue task, not each transport attempt, owns final failure.
• Collapsing A into B. The dispatched provider timeout and the never-dispatched circuit-open outcome are different facts.
• Treating socket destruction as compute settlement. The client-side transport outcome does not establish the remote/local backend's execution state.

Related

Predecessor: #16995 / PR #16996
Shared timeout-contract precedent: #12814 / PR #12818
Caller-vs-provider settlement repair: PR #16869
Incident ledger: #16706
Plane-global admission and observability: #16780

Origin Session ID: 019fe5e8-b963-7e93-8762-c8e4af16bdec

Retrieval Hint: OpenAI compatible embedding timeout queue handoff tenant circuit shared predicate synchronous notification no duplicate Ollama helper