Serialize manual tenant-repo sync with the periodic scheduler

Context

The cloud tenant-repo lane exposes two entry paths over the same mirrors and tenant-repo-sync-revisions.json state:

• the long-running orchestrator's periodic TenantRepoSyncService invocation;
• the operator CLI node ./ai/scripts/maintenance/syncTenantRepos.mjs, including the scoped full-replay mode added by #15748.

The CLI is documented as the recovery/bootstrap path, but it does not participate in the daemon process's task-state guard.

Live latest-open sweep: checked the latest 20 open issues at 2026-07-23T15:11:38Z; no equivalent found. The recent all-state A2A claim sweep found no overlapping lane. Semantic and exact-history sweeps surfaced the original scheduler, heavy-maintenance lease, and manual CLI tickets, but no cross-process serialization contract for these two invocations of the same lane.

The Problem

syncTenantRepos.mjs creates an in-memory TaskStateService. TenantRepoSyncService.runTask() checks only the injected service's running state. The live orchestrator and a separately launched CLI therefore cannot see each other's active tenant-repo sync.

Each process then:

1. reads the whole revisions JSON document;
2. mutates its in-memory copy while processing selected/all repos;
3. writes the whole document back.

A manual replay can race a periodic sweep and produce last-writer-wins loss:

• the CLI may erase a periodic update for another repo;
• the periodic sweep may erase the CLI's successful replay checkpoint;
• failure/backoff increments may disappear;
• a crash during a non-atomic whole-file write may leave unreadable state.

The existing in-process per-repo semaphore controls work inside one invocation. It does not provide cross-process exclusion or atomic manifest commit.

The Architectural Reality

• This is same-lane serialization, not a request to merge tenant-repo-sync with the separate local-only kbSync lane.
• The revision manifest is one deployment-wide whole-document state file. Until it becomes a transactional per-repo store, the simplest correctness boundary is one cross-process tenant-repo-sync writer at a time.
• Existing heavy-maintenance lease primitives provide stale-owner/token/recovery precedent, but tenant-repo sync needs a dedicated lease identity so it does not accidentally serialize unrelated maintenance beyond the required correctness boundary.
• The periodic scheduler should defer cleanly when an operator replay owns the lease; that is not an ingestion failure and must not increment repo failure/backoff.
• The CLI needs a bounded held/busy outcome and deterministic exit code rather than silently racing or waiting forever.
• The lease and manifest live under the orchestrator state owner and therefore depend on durable cloud state wiring (#15759) for recreate continuity.

Structure-map ownership: existing orchestrator lease primitives, TenantRepoSyncService, the manual maintenance script, tenant-repo error taxonomy, and existing unit/integration suites. Reuse sibling primitives; no new subsystem or directory is required.

The Fix

1. Add one dedicated cross-process tenant-repo-sync lease acquired by every periodic and manual invocation before reading mirrors/revision state or starting repo work.
2. Reuse the existing tokenized stale-owner/recovery discipline where applicable; do not implement a boolean lock with permanent-dead-owner failure.
3. Define bounded contention behavior:
   • periodic invocation: deferred/skipped with a stable non-failure reason and no repo backoff mutation;
   • manual CLI: bounded wait or immediate busy result with stable code and documented exit code.
4. Hold the lease through the final revision-manifest commit and release it in all normal/error paths.
5. Write tenant-repo-sync-revisions.json atomically (temporary sibling file + fsync/rename or equivalent proven primitive) so a process crash cannot expose a partially written JSON document.
6. Surface only bounded lease owner class/timestamps/reason codes in diagnostics; never expose PIDs, host paths, repo identities, or command lines publicly.
7. Add a real cross-process or deterministic two-writer witness proving no lost update between periodic and CLI paths.

Contract Ledger Matrix

Decision Record impact

Aligned with ADR 0014's orchestrator-owned tenant-repo lane and existing heavy-maintenance lease precedent. No ADR amendment: this closes a same-lane concurrency defect without coupling tenant-repo sync to local kbSync.

Acceptance Criteria

• Periodic and manual tenant-repo sync acquire the same dedicated cross-process lease before reading or mutating revision state.
• Two processes cannot concurrently perform repo work or write tenant-repo-sync-revisions.json.
• A held lease makes the periodic invocation defer/skip with a stable non-failure reason and does not change any repo's consecutiveFailures, lastRunAttemptAt, or checkpoint.
• A held lease gives the manual CLI a bounded wait or immediate busy result with stable code, deterministic exit code, and updated help text.
• Lease release occurs on success, returned error-bearing ingestion, thrown error, and process-normal shutdown paths.
• Stale/crashed lease owners recover safely; a stale lease cannot block the lane forever, and token mismatch cannot release a new owner's lease.
• Revision-manifest writes are atomic; injected failure before replacement leaves the previous complete JSON readable.
• A deterministic two-writer or real child-process test proves that disjoint repo updates cannot overwrite each other between manual and periodic paths.
• Full replay retains #15752 fail-closed semantics while serialized: old checkpoint remains until an error-free replay commits.
• Public diagnostics expose only bounded contention/recovery status and timestamps; no PID, host path, repo identity, command line, credential, or raw error leaks.
• The lock is dedicated to tenant-repo sync and does not couple the lane to local-only kbSync or unrelated maintenance work.

Out of Scope

• Replacing the JSON manifest with SQLite or a distributed database.
• Serializing tenant-repo sync against unrelated kbSync, Dream, backup, or defrag lanes.
• Adding a remote replay MCP actuator.
• Changing per-repo in-process concurrency limits.
• Solving deployment state persistence itself (#15759).

Avoided Traps

• Do not rely on in-memory task state for cross-process exclusion.
• Do not reuse the global heavy-maintenance lease if that over-serializes unrelated work; reuse its primitive semantics with a dedicated lease identity.
• Do not treat operator ownership as an ingestion failure or advance backoff.
• Do not release a lease by deleting a path without token ownership validation.
• Do not add a lease while leaving whole-file writes crash-partial.

Related

• #11790, #11942, #12036 — tenant-repo scheduler/manual-path and persistence lineage.
• #11503 / #10088 — heavy-maintenance lease/mutex precedent.
• #15748 / PR #15752 — scoped full replay and fail-closed checkpoint advancement.
• #15759 — persistent orchestrator state volume.
• #15761 — automatic legacy checkpoint revalidation.

Origin Session ID: fc1a49c1-e30a-4e3a-960a-e0596367a4c1

Handoff Retrieval Hint: syncTenantRepos manual CLI periodic scheduler cross-process lease revision manifest lost update atomic write