LearnNewsExamplesServices
Frontmatter
id17158
titleTenant-sync concurrency knobs cannot be set by any deployment
stateClosed
labels
bugaiarchitectureai-generatedagent-os
assigneesneo-opus-vega
createdAtAug 15, 2026, 11:19 AM
updatedAtAug 22, 2026, 9:21 PM
githubUrlhttps://github.com/neomjs/neo/issues/17158
authorneo-opus-vega
commentsCount0
parentIssue17411
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[x] 17132 A tenant repo holds its concurrency slot until its corpus is exhausted — each due repo needs a bounded slice per sweep
blocking[]
closedAtAug 22, 2026, 9:21 PM

Tenant-sync concurrency knobs cannot be set by any deployment

Closed Backlog/active-chunk-16 bugaiarchitectureai-generatedagent-os
neo-opus-vega
neo-opus-vega commented on Aug 15, 2026, 11:19 AM

Context

Found on 2026-08-15 while repairing #17132's premise. That ticket claimed the tenant sweep "processes repos sequentially"; @neo-gpt's intake correctly falsified it and re-narrowed the residual to "the constrained concurrencyLimit=1 plane". Running the falsifier on that replacement is what surfaced this: there is no concurrencyLimit=1 plane, because there is no way to reach one.

Recorded in #17132's Out of Scope at the time, and broadcast as claimable, rather than folded into a fairness fix with a different blast radius.

The Problem

TenantRepoSyncService declares two reactive class configs whose docblocks explicitly instruct operators to change them:

  • concurrencyLimit_ (:780) — "Set to 1 to serialize all work when deployment capacity is constrained. Set higher when network/CPU headroom permits."
  • concurrencyGateTimeoutMs_ (:791) — "A positive value remains an explicit fail-fast override."

Neither can be set by any deployment. V-B-A on origin/dev:

git grep -n "concurrencyLimit" origin/dev -- ai/ | grep -v TenantRepoSyncService.mjs
→ (no output)

Both appear only inside their own declaring file. They are Neo class configs, not AiConfig leaves: no leaf() declaration, no env binding, no entry in the orchestrator.tenantRepoSync subtree (ai/configBase.mjs:1971, which projects backoffCapMs, jitterRatio, leaseStaleAfterMs, starvedAfterMs, sweepCadenceMs and no concurrency knob). The operator overlay (ai/config.mjs) feeds AiConfig, which these are not wired to, so that route does not reach them either. Every assignment to a non-default value in the repository is a unit-test setup.

Production therefore always runs the hardcoded concurrencyLimit: 2 and concurrencyGateTimeoutMs: 0.

The class comment at :103-104 states the intended contract — "a fresh semaphore is created per call from the current reactive concurrencyLimit / concurrencyGateTimeoutMs config values, so live config edits take effect" — and the per-call freshness is genuinely implemented. It is the "config edits" half that has no surface. The machinery for live tunability was built; the binding was never attached.

Why this is more than tidiness: a constrained plane is exactly where an operator needs to serialize tenant-repo work, and the docblock tells them how. Following that instruction is currently impossible, and nothing reports the omission — the value silently stays at 2. This is the class of defect where the code, the documentation, and the operator all disagree with no error anywhere.

The Architectural Reality

  • concurrencyLimit_: 2 (:780) and concurrencyGateTimeoutMs_: 0 (:791) — the declarations, with @reactive.
  • Validation already exists and stays: :834 rejects non-positive-integer concurrencyLimit (0 would create a never-acquirable semaphore); :849 rejects non-finite/negative concurrencyGateTimeoutMs (0 is meaningful there).
  • Two consumer methods, not one — this is the part a single-site fix would miss:
    • refreshTenantRepoAccessReadiness() (:1022) builds its own semaphore at :1038.
    • runTask() builds one at :1851-1852 and additionally slices the revalidation cohort at :1925.
  • The sanctioned shape already exists in this same file. runTask resolves a sibling deployment value as a default parameter reading the resolved leaf at the use site:
      globalCadenceMs = AiConfig.data.orchestrator.intervals.tenantRepoSyncMs   // :1126
    That is ADR 0019 §5.1 (read at the use site, no pass-along, no alias, no defensive ?.) and it keeps per-call injection available to callers and specs without mutating the shared AiConfig singleton — the safety-critical B4 antipattern ADR 0019 §4 exists to prevent. Roughly a dozen spec sites currently set TenantRepoSyncService.concurrencyLimit = N directly; the parameter path is what lets them keep isolating by construction.

The Fix

Give both values the globalCadenceMs treatment:

  1. Declare two leaves in ai/configBase.mjs under the existing orchestrator.tenantRepoSync subtree.
  2. Consume them as default parameters at both call sites, reading the resolved leaf at the use site.
  3. Retire the class configs so there is exactly one declaration per value.

Contract Ledger

Target surface Source of authority Behavior Failure / fallback Evidence
orchestrator.tenantRepoSync.concurrencyLimit ai/configBase.mjs leaf(2, 'NEO_ORCHESTRATOR_TENANT_REPO_SYNC_CONCURRENCY_LIMIT', 'number') — default preserves today's behavior exactly Non-positive / non-integer rejected at the existing :834 gate shape, which moves rather than duplicates leaf-projection + invalid-value arms
orchestrator.tenantRepoSync.concurrencyGateTimeoutMs ai/configBase.mjs leaf(0, 'NEO_ORCHESTRATOR_TENANT_REPO_SYNC_CONCURRENCY_GATE_TIMEOUT_MS', 'number') Non-finite / negative rejected at the existing :849 gate; 0 stays meaningful (FIFO wait, not fail-fast) leaf-projection + invalid-value arms
runTask() consumption TenantRepoSyncService:1851-1852, :1925 Default parameters reading the resolved leaves, mirroring globalCadenceMs (:1126); explicit arguments still win Absent argument ⇒ leaf value ⇒ current behavior at the shipped defaults per-call-override arm proving injection still works
refreshTenantRepoAccessReadiness() consumption TenantRepoSyncService:1038 Same resolution at its own semaphore A fix that lands only in runTask leaves this site on a deleted config — the arm exists to catch exactly that second-call-site arm
Class configs retired TenantRepoSyncService:780, :791 Removed; one declaration per value Specs migrate to the parameter path — they MUST NOT mutate the shared AiConfig singleton (ADR 0019 §4 / B4) check-aiconfig-test-mutation stays green
Declared-path census ai/scripts/lint/config-leaf-parity.json → ai/config.template.mjs list Two new paths added in the same PR The parity collector reads name: leaf( declarations; a leaf added without its census entry fails the lint config-template-ssot lint green

Decision Record impact: aligned-with ADR 0019. This removes an unbound config pair in favour of the §5.1 sanctioned form; no ADR clause is amended or challenged.

Two mechanical constraints an implementer will otherwise discover late:

  1. config-leaf-parity.json enumerates orchestrator.tenantRepoSync.* paths individually (verified: five entries today). Two more must be added in the same PR.
  2. $behaviorBindingProjection.clockSuffixes includes _TIMEOUT_MS, so the gate-timeout env name falls under the behavior-binding-clock projection and must satisfy it.

Acceptance Criteria

  • Both values are AiConfig leaves under orchestrator.tenantRepoSync, with defaults identical to today's class-config values, so a deployment that sets nothing is byte-for-byte unchanged in behavior.
  • Setting each env var changes the resolved value that the semaphore is actually constructed with — asserted through the public path, not by reading the leaf back.
  • Both consumer methods resolve the leaves: runTask() and refreshTenantRepoAccessReadiness(). An arm covers the second site specifically.
  • Explicit per-call arguments still override the resolved leaf, and specs use that path — check-aiconfig-test-mutation stays green and no spec mutates the shared singleton.
  • The existing validation gates (:834, :849) still reject their invalid values, including the concurrencyLimit: 0 never-acquirable-semaphore case, and concurrencyGateTimeoutMs: 0 remains a valid FIFO-wait value rather than being swept up as "non-positive".
  • config-leaf-parity.json carries both new declared paths; the config-template-SSOT lint is green.
  • The class configs at :780 / :791 are gone — one declaration per value, no alias, no pass-along, no re-derivation.

Out of Scope

  • Setting either key in canonical or dev Compose. The defaults are correct; this ticket restores the ability to override, not a new deployment posture. Leaving Compose untouched also keeps the ADR 0019 §10.8 30-key Compose census unchanged (verified: that census counts Compose env keys, not leaf count).
  • Per-repo slice fairness — #17132, in flight with @neo-gpt. This ticket does not bound how long a repo holds its slot; it restores the ability to change how many slots exist.
  • Any change to semaphore semantics, cohort ordering, or the FIFO contract.
  • The other unbound configs elsewhere in ai/ — if a census is wanted, that is its own sweep, not a rider here.

Avoided Traps

  • Keeping the class config and defaulting it from the leaf. That is two declarations for one value — ADR 0019 A6/B5 — and the two can disagree. The class configs are retired, not backed.
  • Fixing only runTask. Two methods build semaphores. The refreshTenantRepoAccessReadiness site is the easy miss, and a fix that removes the class config while missing it breaks that path outright.
  • Migrating specs by mutating AiConfig. The dozen TenantRepoSyncService.concurrencyLimit = N spec sites are the reason the parameter path matters. Rewriting them as singleton mutations would trade this defect for the #12335 orphan-bleed class that ADR 0019 §4 calls safety-critical.
  • Treating 0 uniformly. It is invalid for concurrencyLimit (never-acquirable semaphore) and meaningful for concurrencyGateTimeoutMs (wait in FIFO). One shared "reject non-positive" rule would silently remove the gate-timeout default.

Sequencing — blocked by #17132, deliberately

@neo-gpt claimed #17132 at 2026-08-15T08:37Z and is implementing it now. That work and this one collide on four surfaces, two of them shared literal lists where a merge conflict is also a semantic one:

  • ai/configBase.mjs — his sliceBudgetMs leaf lands in the same orchestrator.tenantRepoSync object literal these two leaves belong in.
  • ai/scripts/lint/config-leaf-parity.json — the same declared-path list, which both changes must extend.
  • TenantRepoSyncService.mjs — his control-envelope and syncRepo changes, against this ticket's :780 / :791 / :1038 / :1851 / :1925.
  • TenantRepoSyncService.spec.mjs — both migrate spec setup.

His claim is earlier, so this ticket waits rather than racing him. Filed now instead of held as a note because it has already survived one round as an out-of-scope paragraph and that is how findings evaporate. It is assigned and blocked, not parked.

Related

  • #17132 — where this was found and recorded as out of scope; sibling under the same epic; blocks this ticket.
  • #17072 — parent epic (constrained CPU-plane reliability).
  • ADR 0019 — learn/agentos/decisions/0019-aiconfig-reactive-provider-ssot.md, §5.1 sanctioned form, §4 the B4 danger, §10.8 the Compose census.

Part of epic #17072.

Origin Session ID: 5cd926fa-77e1-4309-8bbf-ca563ab07403

Retrieval Hint: query_raw_memories("tenant sync concurrencyLimit no env binding documented knob no mechanism") · falsification anchor: git grep concurrencyLimit origin/dev -- ai/ returns only TenantRepoSyncService.mjs.

tobiu referenced in commit afd083a - "feat: tenant-sync concurrency knobs become AiConfig leaves (#17158) (#17551) on Aug 22, 2026, 9:21 PM
tobiu closed this issue on Aug 22, 2026, 9:21 PM