LearnNewsExamplesServices
Frontmatter
id11790
titletenant-repo-sync scheduler lane — periodic + manual
stateClosed
labels
enhancementaiarchitecture
assigneesneo-opus-ada
createdAtMay 22, 2026, 11:35 PM
updatedAtJun 7, 2026, 7:14 PM
githubUrlhttps://github.com/neomjs/neo/issues/11790
authorneo-opus-ada
commentsCount2
parentIssue11731
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[x] 11789 Diff-to-ingest envelope builder for tenant-repo sync
blocking[x] 11791 Operator docs + health/telemetry for tenant-repo ingestion
closedAtMay 25, 2026, 7:48 AM

tenant-repo-sync scheduler lane — periodic + manual

Closed v13.0.0/archive-v13-0-0-chunk-13 enhancementaiarchitecture
neo-opus-ada
neo-opus-ada commented on May 22, 2026, 11:35 PM

Context

Sub 5 of Epic #11731 (Server-side tenant-repo ingestion for cloud Agent OS deployments), graduated from Discussion #11782. This sub is the trigger layer — the tenant-repo-sync lane that drives sub 3 (GitMirror) + sub 4 (envelope builder).

The Problem

The persistent-mirror refresh cycle needs a driver. Discussion #11782 OQ2 resolved: periodic refresh + a manual run path first; webhook is a later accelerator (a webhook-first design reintroduces per-repo tenant wiring and is explicitly deferred).

The Architectural Reality

A new cloud-deployable tenant-repo-sync Orchestrator scheduler lane + TenantRepoSyncService — distinct from the local-only primary-dev-sync lane. ADR 0014's §2.1 lane taxonomy is amended (sub 1 / #11740) to classify tenant-repo-sync as cloud-deployable. File placement: structural-pre-flight at implementation — candidate ai/daemons/services/TenantRepoSyncService.mjs (sibling of PrimaryRepoSyncService.mjs), wired into TaskDefinitions.mjs as a serviceTask lane.

The Fix

  • A TenantRepoSyncService consuming GitMirror (sub 3) + the envelope builder (sub 4): iterate configured tenant repos → clone-if-missing / fetch → diff → ingest.
  • A new tenant-repo-sync periodic Orchestrator lane: per-repo cadence with jitter/backoff.
  • A manual/operator run path (CLI or MCP op) to force a refresh of one or all configured repos.
  • ADR 0014 lane-taxonomy: tenant-repo-sync = cloud-deployable (handed to sub 1 / #11740).

Acceptance Criteria

  • TenantRepoSyncService iterates configured tenant repos through the sub-3 / sub-4 pipeline.
  • Periodic tenant-repo-sync Orchestrator lane with per-repo cadence + jitter/backoff.
  • A manual/operator run path.
  • Lane classified cloud-deployable in the ADR 0014 taxonomy (coordinated with sub 1 / #11740).
  • structural-pre-flight applied to new .mjs placements.

Out of Scope

  • Webhook-driven triggering — deferred as a later accelerator.
  • GitMirror (sub 3) and the envelope builder (sub 4).

Contract Ledger

Per the Contract Completeness Gate for tickets introducing public or consumed surfaces. #11790 introduces the tenant-repo-sync Orchestrator scheduler lane and TenantRepoSyncService, consuming sub 2 (TenantRepoAccessContract, #11787), sub 3 (GitMirror, #11788), sub 4 (envelope builder, #11789), and sub 1's cloud-deployable lane classification (#11740 / ADR 0014 amendment). Contract surface is enumerated below.

Target Surface Source of Authority Proposed Behavior Fallback / Edge Case Docs Evidence
TenantRepoSyncService class Epic #11731 + ADR 0014 tenant-repo pull-ingestion amendment extends Base, singleton with reactive configs for enablement, concurrency limit, and cadence/backoff policy; exposes syncAllTenantRepos() and syncOneTenantRepo({tenantId, repoSlug}). Disabled service -> all syncs no-op with operator-visible status; service init failure -> lane records a stable failure and skips the tick without taking down the Orchestrator. JSDoc + TenantIngestionModel.md reference Unit tests with stubbed GitMirror + envelope builder: per-repo iteration and disabled/init-failure gates
tenant-repo-sync Orchestrator lane ADR 0014 amendment + #11740 lane classification Add a serviceTask entry in TaskDefinitions.mjs, classify it as cloud-deployable, and wire a deployment-aware enable toggle. Cloud profile defaults enabled when tenant repos exist; local Neo-maintainer profile defaults disabled unless explicitly enabled. Env override disables the lane; missing tenant-repo config makes the tick a no-op, not an error. TaskDefinitions.mjs / Orchestrator JSDoc + cloud deployment docs Unit test: task definition exists; Orchestrator getter/toggle chooses the correct deployment default and env override
Per-repo iteration (syncAllTenantRepos) #11790 AC1 + #11787 TenantRepoAccessContract config Iterate KnowledgeBaseIngestionService.getTenantConfig().tenantRepos[]; for each entry: clone-if-missing, fetch, build bootstrap/incremental envelope, and ingest. Per-repo failures are isolated. Empty tenantRepos[] -> no-op; one repo failure logs/returns a per-repo error while remaining repos continue; failed repo revision state remains unchanged. JSDoc Unit test: three repo configs, middle fetch fails, other two complete, failed repo state unchanged
Manual operator run path #11790 AC3 Provide a manual operator surface (CLI or MCP op) to refresh one configured repo or all repos. CLI path should follow existing commander-backed script patterns if selected. Missing required selector -> parser/schema rejects; unknown tenant or repo -> stable KB_TENANT_REPO_SYNC_* error; no configured repos -> no-op with visible log/summary. TenantIngestionModel.md operator-run section Unit test for parser/schema accept/reject; dispatch test stubbing TenantRepoSyncService
Per-repo cadence + jitter/backoff #11790 AC2 + Orchestrator scheduling pure-function pattern Pure scheduling function computes next-due state from last sync attempt, configured cadence, deterministic repoSlug/tenant salt jitter, and failure backoff. First bootstrap wave is jittered; repeated failure backs off without starving healthy repos; success resets backoff. JSDoc on scheduling function Unit test: jitter deterministic per repoSlug/tenant; backoff grows on consecutive failures and resets on success
Revision-state persistence (lastIngestedRev writeback) #11789 envelope-builder handoff contract After a successful ingestSourceFiles() summary, service persists lastIngestedRev = envelope.headRevision for that {tenantId, repoSlug} through the selected tenant-sync state surface. Persistence failure leaves revision unchanged and returns/logs KB_TENANT_REPO_SYNC_MANIFEST_UPDATE_FAILED; next tick re-detects the same diff and retries idempotently. JSDoc on state handoff Unit test: successful sync advances revision; persistence failure leaves revision unchanged and next run sees same diff
Concurrency-limit gate #11790 implementation discipline Reactive concurrencyLimit config caps simultaneous git/ingest work to avoid multi-tenant CPU/network exhaustion. concurrencyLimit: 1 serializes; invalid or omitted limit falls back to a conservative default. JSDoc Unit test: five repos with limit 2 never exceed two in-flight syncs
Cross-lane interaction with kbSync ADR 0014 tenant-repo amendment tenant-repo-sync is separate from local-only kbSync; no repointing, shared lock, or shared local-checkout assumption. If both lanes write to the KB collection, existing tenant/repo scoping prevents cross-tenant leakage; lane state remains independent. TenantIngestionModel.md operator note + ADR 0014 anti-pattern reinforcement Unit/concurrency test: both lanes' state independent; no cross-pollution of revision or manifest state
Stable error codes Cross-sub consistency KB_TENANT_REPO_SYNC_* prefix for service/scheduler/manual-run errors, e.g. TENANT_NOT_FOUND, REPO_NOT_CONFIGURED, SYNC_FAILED, MANIFEST_UPDATE_FAILED, CONCURRENCY_GATE_TIMEOUT; callers branch on code, not message prose. All surfaced errors use the #11787/#11788 redaction path for credential-bearing hints or git stderr before logging/returning. JSDoc on error codes Unit test: each error path returns the expected stable code and redacts injected fake secrets

Behavioral invariants:

  • Service consumes #11787, #11788, and #11789 contracts without redesigning them; this lane is integration and scheduling, not another API layer.
  • The enable toggle must match ADR 0014's cloud-deployable classification: cloud tenant deployments can run it; the local Neo-maintainer profile does not run it by default.
  • Per-repo failure isolation is load-bearing: one tenant repo's git/ingest failure does not block other repos.
  • Revision state advances only after successful ingestion; a failed persistence step leaves state unchanged so retries remain idempotent.
  • Jitter is deterministic, not random per process, so test runs and operator diagnosis are reproducible.

Out of contract scope:

  • GitMirror clone/fetch/diff -> sub 3 #11788.
  • Envelope builder bootstrap/incremental/force-push fallback -> sub 4 #11789.
  • TenantRepoAccessContract no-secret persistence -> sub 2 #11787.
  • Lane classification authority in ADR 0014 -> sub 1 #11740.
  • Webhook-driven triggering -> deferred per Discussion #11782 OQ2.
  • Cross-deployment KB-collection conflict resolution -> out of scope; tenant/repo scoping handles isolation.

Related

  • Parent epic: #11731.
  • Origin Discussion: #11782 (OQ2).
  • ADR: 0014 (lane taxonomy — amended by sub 1 / #11740).

Origin Session ID

39185c66-a107-46ea-b0bf-eb4fa1137257

tobiu closed this issue on May 25, 2026, 7:48 AM
tobiu referenced in commit d21b41b - "feat(orchestrator): concurrency-limit gate for tenant-repo-sync (#11942 AC2) (#11960) on May 25, 2026, 10:45 AM
tobiu referenced in commit 1bfee42 - "docs(day0): rewrite Milestone 6 to first-class pull mode (#12052) (#12053) on May 27, 2026, 12:58 AM