LearnNewsExamplesServices
Frontmatter
id13757
titlefix(orchestrator): #13358 inherited-token compatible-pair race skips kb-sync syncDatabase → no embedding since 06-17 (regression)
stateClosed
labels
bugaiarchitecture
assigneesneo-opus-grace
createdAtJun 21, 2026, 1:27 PM
updatedAtJun 21, 2026, 3:05 PM
githubUrlhttps://github.com/neomjs/neo/issues/13757
authorneo-opus-grace
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 21, 2026, 3:05 PM

fix(orchestrator): #13358 inherited-token compatible-pair race skips kb-sync syncDatabase → no embedding since 06-17 (regression)

Closed v13.1.0/archive-v13-1-0-chunk-5 bugaiarchitecture
neo-opus-grace
neo-opus-grace commented on Jun 21, 2026, 1:27 PM

Sub of #13755. Confirmed regression, traced to code.

Root cause

#13358 (06-15) added DEFAULT_COMPATIBLE_HEAVY_MAINTENANCE_TASK_PAIRS = [['kbSync','memory-summary-backfill']], letting them run concurrently by passing the parent's lease token to the child via NEO_HEAVY_MAINTENANCE_LEASE_INHERITED_TOKEN.

withHeavyMaintenanceLease (HeavyMaintenanceLeaseService.mjs:526-552):

  • inherited token matches active lease → runs the task (intended concurrency). ✅
  • inherited token set but doesn't matchfalls through to acquireHeavyMaintenanceLease → if another heavy task holds the lease, returns {status:'held'}the task never runs. ⚠️

memory-summary-backfill is short — it usually releases the lease before the kb-sync child finishes booting (~148ms to connect ChromaDB). By then the inherited token is stale → fall-through → another task holds the lease → kb-sync skips syncDatabase: a 148ms "completed" no-op where syncDatabase's first log ("Starting full database synchronization") never fires.

Evidence

  • kb-sync embedded daily through 06-16 (06-15: 43 batches, 06-16: 6) → zero 06-17→06-20 → 2566-chunk accumulated delta when force-run 06-21.
  • #13358 is the most-recent INHERITED_TOKEN change; timing tracks the decline exactly.
  • 06-17 log: Starting kb sync 06:12:25.686 → completed successfully 06:12:25.834 (148ms), ChromaDB connected, no sync body, no "⏸️ Deferred".

Consequence

New items never embed → not in graph → Golden Path recommends months-old #9xxx → swarm-wide derailment.

Fix

  1. Remove ['kbSync','memory-summary-backfill'] from DEFAULT_COMPATIBLE_HEAVY_MAINTENANCE_TASK_PAIRS → serialize again (restores the pre-#13358 behavior that worked for weeks). The concurrency optimization can be re-introduced later with a race-free handshake.
  2. Harden withHeavyMaintenanceLease: a set-but-mismatched inherited token must not silently fall through to a task-skip — log it (the race becomes visible) so a future recurrence is observable, not a silent multi-day stall.

AC

  • kb-sync runs syncDatabase (the full embed) on its cadence even right after memory-summary-backfill held the lease.
  • A mismatched inherited token emits an observable log, not a silent no-op.
  • Spec: MaintenanceBackpressureService no longer treats the kbSync/backfill pair as compatible; regression test for the skip.
  • PrimaryRepoSyncService.mjs:564 (same INHERITED_TOKEN read) audited for the same skip.
tobiu referenced in commit dd286a5 - "fix(orchestrator): serialize kbSync/miniSummary — inherited-token race skipped kb-sync embedding for days (#13757) (#13762) on Jun 21, 2026, 3:05 PM
tobiu closed this issue on Jun 21, 2026, 3:05 PM