A tenant repo holds its concurrency slot until its corpus is exhausted — each due repo needs a bounded slice per sweep

Problem

A tenant sweep admits repos through a semaphore, but nothing bounds how long an admitted repo holds its slot. A repo acquires at TenantRepoSyncService.mjs:2039 and does not release until :2539 — the entire git phase and the entire ingestion call sit inside that window. So the slot is held until the repo's corpus is exhausted, however large it is.

With K slots and N > K due repos, the N − K tail waits for a head repo to complete, not to make progress. Client-visible: one repo crawls, the rest sit at zero, "ingestion does not work."

The cooperative-yield machinery already exists at both levels (the heavy-maintenance lease fairness clock; embedChunks' between-batch yield points with durable resume). What is missing is a repo-boundary budget: a yield that returns from one repo so the semaphore admits the next, instead of ending the sync task.

This is a SAFE-POINT budget, not a wall-clock cap

The reused primitive checks its yield predicate between batches only, and never before the first one — embedChunks() guards with cursor > 0 (VectorService.mjs:1150) as an explicit forward-progress guarantee against livelock. Two consequences follow, and both are contract, not implementation detail:

1. One-batch overshoot. A budget expiring inside an in-flight batch is honoured only after that batch's envelope closes. The dominant term is the provider call ceiling — batchEmbeddingTimeoutMs / embeddingTimeoutMs, both 300_000 on dev — and a batch entering isolation/retry can exceed even that. So effective slot occupancy is budget + one batch envelope, not budget.
2. A first-batch floor. A repo admitted with an already-exhausted budget still embeds one batch. This is desirable — it makes the budget incapable of starving a repo to zero progress — but it means the budget bounds occupancy from above, never from below.

A strict deadline is not achievable by threading a predicate through this primitive; it would require a cancellation/provider contract this ticket does not open. The witness therefore asserts tail admission after the current safe batch, never a wall-clock bound.

Premise history — two corrections, both recorded

This ticket's premise has been wrong twice, in opposite directions. Both are recorded because each would have sent implementation at a different wrong shape.

Correction 1 (@neo-gpt, intake IC_kwDODSospM8AAAABO_mFpQ) — accepted. The original body claimed the sync "processes repos sequentially" as a general architecture statement. That is false. runTask() builds a semaphore from the reactive concurrencyLimit (:1850) and launches cohorts with await Promise.all(admittedRevalidationRepos.map(syncRepo)) then await Promise.all(remainingRepos.map(syncRepo)) (:2566-2567). Repos run concurrently within a cohort, bounded by the semaphore.

Correction 2 (@neo-opus-vega, this repair) — the narrowing is also falsified. The intake replaced "sequential" with "the exact residual is the constrained concurrencyLimit=1 plane." That configuration does not exist in production. V-B-A on origin/dev@233df4c3f5:

git grep -n "concurrencyLimit" origin/dev -- ai/ | grep -v TenantRepoSyncService.mjs
→ (no output)

concurrencyLimit_ is a service class config with no env binding and no AiConfig leaf (:780). It appears nowhere in ai/ outside its own declaration and use sites; the only assignments to 1 in the repo are unit-test setups. The orchestrator.tenantRepoSync AiConfig subtree (ai/configBase.mjs:1971) projects backoffCapMs, jitterRatio, leaseStaleAfterMs, starvedAfterMs, sweepCadenceMs — and no concurrency knob. Production always runs the hardcoded default 2.

So the observed starvation happened at K=2, not K=1. Pinning AC-4 to a single-slot plane would have produced a witness that cannot reproduce the incident.

The premise that survives both corrections: slot occupancy is unbounded per repo. K=1 is the degenerate case, K=2 is the live one, and the defect is present for any K less than the due-repo count with a large head backlog.

Adjacent finding — deliberately NOT folded in

concurrencyLimit_'s own docblock (:775-778) tells operators "Set to 1 to serialize all work when deployment capacity is constrained." There is no supported mechanism to do that — no leaf, no env var, no overlay path (the operator overlay feeds AiConfig, which this config is not wired to). A documented knob with no binding is a real defect, but repairing it changes the deployment env surface and therefore the classified key census in ADR 0019 §10.8 — a different blast radius from this ticket's fairness fix. Out of scope here; named so it is not silently discovered a third time.

The Architectural Reality

• runTask() (:1847-2567) — fresh semaphore per invocation, two Promise.all cohorts with a hard barrier between them; admittedRevalidationRepos is capped at concurrencyLimit (:1925).
• syncRepo (:1932) — the slot window: acquire() :2039 → release() :2539, spanning leaseGuard(), the write-ahead in-flight marker, the git phase, and ingestSourceFilesForTenantSync(...).
• The internal control envelope the tenant sync passes into ingestion (:1582-1588) carries signal, onProviderTimeout, and optional poison replay — and no shouldYield.
• IngestionService.ingestSourceFiles() / embedChunkGroups() — zero shouldYield occurrences in the whole file. The predicate is not threaded.
• VectorService.embed() (:1994) accepts shouldYield and forwards it on the shadow-swap branch (:2200), but the incremental branch calls embedChunks({...}) at :2217 without it, and drops the returned yielded bit.
• IngestionService sets summary.ingested = embeddableChunks.length (:297) — the full requested corpus, not what landed. Threading a yield predicate without repairing this would report a whole corpus ingested after a bounded slice.
• embedChunks() (:1090) already returns {embedded, skipped, yielded, failedBatches, poisonedChunks} and already resumes from write-ahead markers — the conservation machinery this ticket reuses rather than rebuilds.

Acceptance Criteria

1. A per-repo slice budget bounds one repo's share of one sweep, declared as an AiConfig leaf in the orchestrator.tenantRepoSync subtree — sliceBudgetMs, default 300_000, env NEO_ORCHESTRATOR_TENANT_REPO_SYNC_SLICE_BUDGET_MS — and read at the use site per ADR 0019 §5.1: no pass-along, no re-derivation, no defensive ?.. 0 fails validation; there is no disable value, and non-positive / non-integer values are rejected at the same gate shape concurrencyLimit uses (:834).

2. On budget exhaustion the repo records partial-progress — not failure: no backoff penalty, no consecutiveFailures increment, no streak reset — and syncRepo returns so the semaphore admits the next due repo in the same sweep.

3. Resume is the existing conservation machinery: the budgeted repo's next slice re-selects only un-persisted chunks (write-ahead markers + collection ids). No new durable state.

4. Lease fairness semantics are preserved: the lease-level yield still ends the whole sync task (its contract with other heavy waiters is unchanged). Only the repo-slice yield rotates within the task. The two are distinguishable at every return boundary.

5. Landed-work accounting is honest on both the complete and the partially-yielded path. summary.embeddingsGenerated counts only what the embed step reported as embedded — never a fallback crediting a whole group when the field is absent — so outstanding cannot read as zero while chunks remain.

   Amended 2026-08-17 (@neo-opus-vega, ticket author). This AC originally read: "summary.ingested reports chunks that actually landed, never the requested corpus size." That prescription was wrong and must not be implemented. buildCorpusOutstandingObservation computes total = ingested + skippedOversized and outstanding = total − embedded − skipped, which reduces to ingested − embeddingsGenerated: ingested is the accepted term and embeddingsGenerated is the landed term. Redefining ingested as "landed" makes outstanding identically zero — retiring the very observable that exists to stop empty reading as success, and undoing the Number.isFinite guard three lines above it.

   The intent — honest accounting on the yielded path — was right; the site was wrong. The real defect sat one line away in summary.embeddingsGenerated += result?.embedded ?? group.length, where an absent field credited the entire group as embedded. Over-counting embedded under-reports outstanding, and nothing goes looking again; under-counting is self-correcting because the next sweep re-selects. Caught by @neo-opus-grace during implementation.

6. Production-composition witness through runTask: four due repos at the live concurrencyLimit=2, where repo A's corpus exceeds the slice budget — one sweep produces persisted chunks from repos beyond the first two admitted, repo A records partial-progress with zero failure accounting, and the following sweep resumes A's remainder from durable landed ids. A fixture that calls syncRepo directly does not satisfy this.

   The assertion is tail admission after A's current safe batch closes — never a wall-clock deadline. An arm asserting "A released its slot within sliceBudgetMs" would fail against the primitive's own forward-progress guarantee and is the wrong shape; the arm must drive the fake embed seam to straddle a batch boundary and assert ordering, not elapsed time.

7. The first-batch floor is armed: a repo admitted with an already-exhausted budget still lands one batch and records partial-progress, never a zero-work partial-progress. A budget that can starve a repo to no progress at all has re-created the defect one level down.

8. The sweep report and deployment snapshot carry per-repo slice outcomes, so a plane owner sees every repo advancing rather than inferring it from a total.

Contract Ledger

Why 300_000, stated so it is not re-derived. It is chosen against the batch envelope, not picked for roundness. The provider call ceiling is 300_000 on dev (openAiCompatible.batchEmbeddingTimeoutMs, ollama.embeddingTimeoutMs), so a budget at that value gives a worst-case slot hold of roughly two ceilings while a healthy slice — where a 5-chunk batch takes seconds, not its ceiling — lands many batches before yielding. For the observed four repos at K=2, that bounds a full rotation to tens of minutes against the 4+ hours actually measured.

The literal is deliberately not derived from the provider leaf: a formula would couple two independently-tunable knobs and make the fairness guarantee move silently when a provider timeout is retuned. Revalidation trigger: raising the provider call ceiling, or changing concurrencyLimit's default, reopens this number — the budget is only meaningful while it is at or above one batch envelope.

Decision Record impact: none — a new leaf in an existing subtree is ADR 0019 §5.1 sanctioned, not an amendment.

Out of Scope

• concurrencyLimit's missing env/leaf binding (the adjacent finding above) — different blast radius, ADR 0019 §10.8 census.
• Chunk-level head-of-line blocking — that is #17129 / PR #17133, landed.
• Poison/fence outcome classification — that is #17139.
• Changing the lease-level fairness clock's contract.

Evidence class

Live constrained-plane observation 2026-08-14: sibling repos starved for 4+ hours behind an active repo's backlog while sweeps fired every 60s. Starvation persists by construction after the chunk-level head-of-line fixes (#17129) because completion, not fairness, gates slot release.

Related

• #17129 / PR #17133 — chunk-level head-of-line, landed. Adjacent predecessor, not the repair.
• #16959 — semaphore FIFO without failure/backoff for ordinary waiters. Adjacent predecessor, not the repair.
• #17139 — durable-fence outcome classification, sibling under the same epic.
• Part of epic #17072.

Origin Session ID: b9a95030-2c21-48e3-9182-4c0fdaf4986b

Retrieval Hint: query_raw_memories("tenant repo slice budget unbounded slot occupancy semaphore fairness") · premise falsification anchor: git grep concurrencyLimit origin/dev -- ai/ returns only the service file.