Context
A source-and-behavior audit of the tenant-repository fairness residual under #16780 found that the existing per-repository semaphore is already FIFO. The failure is the shipped 30-second default attached to that queue: valid work can hold the admitted slot beyond 30 seconds while untouched sibling repositories are classified as failed.
The timeout does not bound the outer operation. TenantRepoSyncService.runTask() still awaits the active repository through Promise.all(); only the queued repositories are rejected. Those rejected rows then receive failure state and exponential backoff despite never reaching the git, envelope, or ingestion boundary.
PR #16959 already contains this bounded repair. It was initially left as a draft because the broader #16780 cannot honestly be closed by one fairness slice. Draft PRs are not reviewable delivery units, so this one-PR leaf makes the close target explicit.
Live latest-open sweep: checked the latest 20 open issues at 2026-08-11T11:27:25.893Z; no equivalent leaf found. The recent A2A claim sweep also found no competing FIFO/slot-timeout claim.
The Problem
With the production default concurrencyGateTimeoutMs = 30000, a slow but valid repository can hold the only slot while another repository waits in the existing FIFO. After 30 seconds, the waiter receives KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT, is persisted as failed, and acquires backoff. The holder remains active and the outer run remains pending, so the timeout creates false failure state without providing liveness.
The attempt clock is also started before slot admission. Once normal waiters are allowed to remain queued, that makes queue time look like repository work time and can make a just-completed repository immediately due again.
The Architectural Reality
ai/daemons/orchestrator/services/TenantRepoSyncService.mjs owns both the per-repository semaphore and the durable repository attempt state. Its semaphore already provides FIFO handoff and already supports timeoutMs = 0 as the no-timeout sentinel. Slot release is protected by the per-repository finally, so no second queue or admission primitive is needed.
The Agent OS structure map locates this concern in ai/daemons/orchestrator/services, beside the owning scheduler and lease services. This change does not add an AiConfig leaf, cross a service boundary, or change provider admission.
The Fix
- Change the service's reactive
concurrencyGateTimeoutMs default and invalid-value fallback from 30000 to 0.
- Preserve every positive finite value as an explicit fail-fast override.
- Reset the repository attempt timestamp immediately after FIFO slot admission.
- Add a production-shaped
runTask() witness that holds one repository beyond a compressed historical 30-second deadline, proves the sibling remains queued, releases the holder, and proves both repositories complete without false backoff.
- Retain the explicit-timeout test as the negative control.
Contract Ledger
| Target Surface |
Source of Authority |
Proposed Behavior |
Fallback / Edge Case |
Docs |
Evidence |
TenantRepoSyncService.concurrencyGateTimeoutMs |
Reactive class config and beforeSetConcurrencyGateTimeoutMs() |
Default 0: ordinary waiters remain in the existing FIFO until release |
Positive finite values retain opt-in timeout; invalid values retain the prior valid value or fall back to 0 |
Member JSDoc |
Default-domain unit control plus explicit-timeout control |
runTask().details.repos[] |
TenantRepoSyncService.syncTenantRepos() |
Capacity wait alone never emits a repository failure under the default |
Explicit positive timeout may still emit KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT |
Existing task receipt contract |
Held-first-repository / queued-second-repository production-path witness |
Durable lastRunAttemptAt |
Actual per-repository slot admission |
Begins when work is admitted, not when the repository joins the FIFO |
A waiter rejected by an explicit timeout has no admitted work interval |
Existing revisions-state contract |
Persisted timestamp asserted after the release boundary |
Decision Record Impact
None. This is aligned with #16780's declared-concurrency direction and corrects the semantics around an existing FIFO primitive.
Acceptance Criteria
Out of Scope
- Terminating or recovering already-orphaned Ollama work.
- Explaining or claiming to cure sustained provider CPU consumption.
- Changing provider concurrency or health-canary policy.
- Adding a remote force-sync command or clearing backoff out of band.
- Replacing the existing FIFO semaphore.
Avoided Traps
- Replacing a correct semaphore. The queue and release semantics already exist; the harmful behavior comes from the default timer.
- Calling the timer a run bound. It rejects only waiters while the admitted holder still controls the outer completion.
- Clearing backoff before recovery. Success remains the authority that resets failure state.
- Counting queue residence as repository work. The attempt clock starts only after admission.
Related
Parent direction: #16780
Deployment incident ledger: #16706
Implementation: #16959
Original explicit-timeout surface: #11942
Origin Session ID: 7f0e4829-173a-4780-9a46-8e4811a979b5
Retrieval Hint: TenantRepoSyncService concurrencyGateTimeoutMs FIFO queued repository false failure backoff slot admission
Context
A source-and-behavior audit of the tenant-repository fairness residual under #16780 found that the existing per-repository semaphore is already FIFO. The failure is the shipped 30-second default attached to that queue: valid work can hold the admitted slot beyond 30 seconds while untouched sibling repositories are classified as failed.
The timeout does not bound the outer operation.
TenantRepoSyncService.runTask()still awaits the active repository throughPromise.all(); only the queued repositories are rejected. Those rejected rows then receive failure state and exponential backoff despite never reaching the git, envelope, or ingestion boundary.PR #16959 already contains this bounded repair. It was initially left as a draft because the broader #16780 cannot honestly be closed by one fairness slice. Draft PRs are not reviewable delivery units, so this one-PR leaf makes the close target explicit.
Live latest-open sweep: checked the latest 20 open issues at 2026-08-11T11:27:25.893Z; no equivalent leaf found. The recent A2A claim sweep also found no competing FIFO/slot-timeout claim.
The Problem
With the production default
concurrencyGateTimeoutMs = 30000, a slow but valid repository can hold the only slot while another repository waits in the existing FIFO. After 30 seconds, the waiter receivesKB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT, is persisted as failed, and acquires backoff. The holder remains active and the outer run remains pending, so the timeout creates false failure state without providing liveness.The attempt clock is also started before slot admission. Once normal waiters are allowed to remain queued, that makes queue time look like repository work time and can make a just-completed repository immediately due again.
The Architectural Reality
ai/daemons/orchestrator/services/TenantRepoSyncService.mjsowns both the per-repository semaphore and the durable repository attempt state. Its semaphore already provides FIFO handoff and already supportstimeoutMs = 0as the no-timeout sentinel. Slot release is protected by the per-repositoryfinally, so no second queue or admission primitive is needed.The Agent OS structure map locates this concern in
ai/daemons/orchestrator/services, beside the owning scheduler and lease services. This change does not add an AiConfig leaf, cross a service boundary, or change provider admission.The Fix
concurrencyGateTimeoutMsdefault and invalid-value fallback from30000to0.runTask()witness that holds one repository beyond a compressed historical 30-second deadline, proves the sibling remains queued, releases the holder, and proves both repositories complete without false backoff.Contract Ledger
TenantRepoSyncService.concurrencyGateTimeoutMsbeforeSetConcurrencyGateTimeoutMs()0: ordinary waiters remain in the existing FIFO until release0runTask().details.repos[]TenantRepoSyncService.syncTenantRepos()KB_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUTlastRunAttemptAtDecision Record Impact
None. This is aligned with #16780's declared-concurrency direction and corrects the semantics around an existing FIFO primitive.
Acceptance Criteria
concurrencyGateTimeoutMsare0.1, a repository held beyond the historical 30-second boundary keeps its sibling queued; after release, both complete asactive, withcompletedCount: 2,failedCount: 0, and no concurrency-gate timeout code.30000default makes that witness fail for the expected false-failure reason.lastRunAttemptAtat or after actual slot admission.TenantRepoSyncServiceunit suites pass.Out of Scope
Avoided Traps
Related
Parent direction: #16780
Deployment incident ledger: #16706
Implementation: #16959
Original explicit-timeout surface: #11942
Origin Session ID: 7f0e4829-173a-4780-9a46-8e4811a979b5
Retrieval Hint:
TenantRepoSyncService concurrencyGateTimeoutMs FIFO queued repository false failure backoff slot admission