One repo timeout dispatches the next queued embed in the same sweep

Context

An external deployment exposed a bounded-attempt chain that stays active for hours. At the 2026-08-11 live read, tenantRepoSync had remained running since 08:48Z while one Knowledge Base embedding batch exhausted five 1,800,000ms attempts and a later batch continued retrying. PR #16978 / ticket #16973 correctly made one provider timeout terminal for the current VectorService sweep.

That repair is necessary, but its boundary is one repository. The tenant task still runs repositories concurrently. With tenant concurrency 2 and native-Ollama admission cap 1, repository A can own the provider slot while repository B waits inside the same process.

Live latest-open sweep: checked the newest 25 open issues plus the latest 30 all-state A2A messages at 2026-08-11T16:54Z; no equivalent filed or in-flight lane was found. Exact GitHub searches for tenant-repo timeout/admission/same-sweep terms also returned no match.

The Agent OS structure-map gate was attempted before filing and failed with Cannot create a string longer than 0x1fffffe8 characters; source ownership was therefore resolved from exact origin/dev@728dab196f imports and call sites instead.

The Problem

Native Ollama admission releases its slot and wakes the oldest waiter as soon as the provider promise rejects. A timeout-class rejection reaches that release boundary before it has propagated through:

TextEmbeddingService → VectorService → fail-soft IngestionService summary → TenantRepoSyncService outcome classification.

Consequently, repository B can leave the queue and dispatch immediately after repository A times out, even though that timeout does not prove A's server-side runner stopped. #16978 prevents same-repository retry/batch amplification, but it cannot stop the next repository already queued in the same runTask.

This is a task-wide admission-order defect. It is not evidence of one unbounded promise: each observed provider attempt had a finite deadline. It is the sequence of individually bounded offers that keeps the provider continuously occupied and prevents the Knowledge Base sweep from converging.

The Architectural Reality

At origin/dev@728dab196f:

• TenantRepoSyncService.syncTenantRepos() constructs one semaphore per run, defaults concurrencyLimit to 2, and awaits all per-repo jobs through Promise.all (ai/daemons/orchestrator/services/TenantRepoSyncService.mjs, the per-run gate and repo aggregation around lines 1760–1767 and 2447–2448).
• The pull lane invokes KnowledgeBaseIngestionService.ingestSourceFiles(..., viaMcp: false) around lines 2086–2097.
• IngestionService.embedChunkGroups() converts a provider failure to an error-bearing, fail-soft summary rather than rejecting the tenant task (ai/services/knowledge-base/IngestionService.mjs:367–402).
• #16978 makes a provider timeout throw out of VectorService.embedChunks() after one offer while preserving already-upserted batches as the durable resume boundary (ai/services/knowledge-base/VectorService.mjs:637–661, 738+).
• TextEmbeddingService.#embedOllama() attaches slot release to both provider-settlement arms and immediately wakes the next waiter (ai/services/memory-core/TextEmbeddingService.mjs:1404–1415, 1537–1545).
• Queued caller abort is already a supported, honest path: it removes never-dispatched work and closes its provider-activity row at the queue stage. Post-dispatch caller abort deliberately does not claim provider settlement.

The circuit therefore belongs to the tenant run plus the native-Ollama admission settlement boundary. A high-level flag set only after ingestSourceFiles() returns is too late: the next waiter has already been woken.

The Fix

Create one task-scoped provider-timeout circuit for each TenantRepoSyncService.runTask():

1. Thread its cancellation/admission signal through the internal tenant-ingestion path to native Ollama admission without adding an AiConfig leaf or changing external MCP schemas.
2. When a provider-phase timeout wins, open the circuit before the provider slot hands off to another waiter.
3. Reject/defer every queued or not-yet-dispatched repository in that run using existing bounded Knowledge Base timeout vocabulary.
4. Keep already-dispatched provider work observed until its real provider promise settles; do not close its socket, release its slot on caller settlement, or claim server cancellation.
5. A later scheduler run gets a fresh, closed circuit and resumes from the already-persisted chunk IDs.

Contract Ledger

Decision Record impact

None. This composes the existing tenant-task lifecycle, typed provider-timeout classification, native-Ollama admission, and durable resume contracts. It does not introduce a new service, config authority, or public protocol.

Acceptance Criteria

• AC-1 — production composition: with two tenant repositories, tenant concurrency 2, and native Ollama cap 1, repository A reaches the provider while B is genuinely queued; A rejects with a typed provider timeout and B makes zero provider calls in that run.
• AC-2 — ordering has teeth: the timeout circuit opens before slot handoff. A mutation that restores wake-before-circuit makes B dispatch and turns the named test red.
• AC-3 — honest settlement: A's already-dispatched provider activity remains in flight until its provider promise settles; the circuit never closes the transport, releases admission from caller settlement, or records false provider completion.
• AC-4 — structural task completion: both repository jobs settle as bounded complete/deferred outcomes, task state clears, and the tenant task's semaphore/lease lifecycle completes.
• AC-5 — resume: a later scheduler run starts with a fresh circuit, dispatches B, and does not re-buy A's durable prefix.
• AC-6 — non-timeout controls: a timeout-shaped Chroma upsert/write error and an ordinary non-timeout provider error do not open the provider-timeout circuit.
• AC-7 — compatibility: non-tenant native-Ollama callers and the existing pre-dispatch/queued/post-dispatch abort semantics remain unchanged.

Out of Scope

• Choosing the deployment's embedding batch size or implementing adaptive halving (#16972). A deliberately small batch remains the immediate deployment mitigation.
• The broader plane-global admission declaration still open under #16780 AC-5.
• Dead-process provider-activity reconciliation (#16987).
• Generic task-holder deadlines for Git or Chroma operations; a bare Promise.race() or early semaphore release is explicitly rejected.
• Terminating already-running provider work or forwarding a caller abort into the native transport (#16853).
• Clearing work already executing on a deployed plane; this prevents the next same-run offer after the repair is deployed.

Avoided Traps

• Inline adaptive retry. Dispatching a smaller batch immediately after timeout can queue it behind work whose server-side settlement is unknown.
• Flagging after ingestion returns. The native admission handoff has already happened by then.
• Early slot release. A caller timeout is not provider settlement.
• Whole-task wall-clock timeout. Slow but progressing ingestion is valid and must retain its durable prefix.
• Calling running:true proof of one hung await. It proves only that start was persisted and no durable terminal transition was observed.

Related

Parent incident: #16706
Predecessor repair: #16973 / PR #16978
Provider cancellation contract: #16853
Plane-global admission/observability: #16780
Dead-generation activity rows: #16987
Adaptive batch policy: #16972

Retrieval Hint: tenant repo timeout circuit native Ollama admission cap 1 queued second repo provider settlement wake before defer