LearnNewsExamplesServices
Frontmatter
id13755
title[Epic] Orchestrator heavy-lease: deferred/held runs log as ''completed'' while real kb-sync/REM/golden-path embedding stalls for days → new items never reach the graph
stateClosed
labels
bugepicaiarchitecture
assigneesneo-opus-grace
createdAtJun 21, 2026, 12:44 PM
updatedAtJun 25, 2026, 2:31 AM
githubUrlhttps://github.com/neomjs/neo/issues/13755
authorneo-opus-grace
commentsCount10
parentIssuenull
subIssues
13767 Dream skipped outcomes must not mark successful runs
13777 Propagate miniSummary backfill progress outcomes
13784 kb-sync child-outcome propagation: emit + capture deferred outcomes (C.2 sibling of #13777, sub of #13755)
13844 fix(ai): scope Golden Path computed candidate-pool to ISSUE + DISCUSSION vectors (non-actionable-type pollution slice of #13750)
13960 Resolve orchestrator state from canonical data root
13962 Rewrite stale Sandman handoff when Golden Path has no candidates
13971 REM backlog drains only at hourly steady-state cadence
13978 Golden Path query failures must not false-green
subIssuesCompleted8
subIssuesTotal8
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJun 25, 2026, 2:31 AM

[Epic] Orchestrator heavy-lease: deferred/held runs log as 'completed' while real kb-sync/REM/golden-path embedding stalls for days → new items never reach the graph

Closed v13.1.0/archive-v13-1-0-chunk-5 bugepicaiarchitecture
neo-opus-grace
neo-opus-grace commented on Jun 21, 2026, 12:44 PM

Problem (VBA'd from the live orchestrator log, 2026-06-21)

The orchestrator's single heavy-maintenance lease lets short, frequent high-value tasks starve the long-window ones, and deferred attempts are logged/counted as completions — so the stall is invisible until the backlog explodes.

Evidence:

  • kb-sync last actually embedded on 2026-06-16. Then a ~4-day gap (06-17→06-20): it logged "Starting knowledge base sync" 5–18×/day but never embedded — every attempt deferred.
  • The forced 06-21 sync found 2566 chunks to add/update + 1513 to delete (accumulated). Healthy syncs (06-14/06-15) found 0–293.
  • Deferral counts in the log: kb-sync 502, session-summary 440, REM 381, golden-path 131 (github-sync 10).
  • The frequent kb-sync "completed successfully" lines were deferrals masked as completions (lease held by REM / session-summary / backup / github-sync), which reset freshness without doing the work.

Correction of my earlier wrong framings (all operator-corrected):

  • ❌ "Golden path frozen 18 days" — it regenerates hourly (my stale clone misled me).
  • ❌ "kb-sync ran fine ~25×/48h" — those were deferrals; real embedding gapped 06-17→06-20.
  • ❌ "low-value chat-model churn" — session/turn summaries + tri-vector extraction are high-value; the problem is scheduling, not value.
  • ✅ Actual: one lease + many high-value contenders → long-window embedding/extraction tasks preempted before completion + deferred-runs-masked-as-completed.

Consequence

New issues/PRs/sessions never get embedded → not in the graph → the Golden Path navigates stale, old, meta-heavy data → the swarm self-selects #9xxx tickets. Operator had to manually kill the chat-model processes to free the lease + VRAM so embedding could finally run.

Why my prior fixes did not solve it

#13593 (backup-prio-0 + staleness fair-picker), #13666 (bound summary lease-hold), #13609 (fairness ADR), #13139 (chat-model lane), #13684 (wire sync) — all addressed rotation fairness, none of:

  1. Deferred ≠ done. A deferred/no-op attempt must NOT reset staleness or count as a run (confirm whether the deferral path updates lastRunAt — if it does, that single bug defeats the whole fair-picker).
  2. Protect long-window tasks. kb-sync embedding / REM / golden-path need an uninterrupted window; a short heavy task must not preempt the next cycle indefinitely.
  3. Model-residency. Keep the embedding model resident for embedding tasks; co-schedule by model family so chat↔embedding don't thrash (the 06-21 Model unloaded during the forced sync).

Solution axes (subs — ACs live in each sub, added incrementally)

  1. Deferred-run accounting — deferral does not reset freshness / count as completion; staleness reflects real work since, not attempt since. (Likely the highest-leverage single fix.)
  2. Long-window protection + model-residency co-scheduling — reserve windows for embedding/extraction tasks; group by model; don't evict a model mid-run.
  3. Net-progress + idle (absorbs #13624) — ~0-progress tasks back off; orchestrator idles when the reachable backlog is drained.
  4. Golden-path ranking de-bias (absorbs #13750) — recency / release-label boost + stale-meta decay so new release issues surface even at low structural weight.

Avoided traps

  • Staleness-fairness alone (ADR-0022 / #13593) is insufficient: deferred-runs-masked-as-completed + long-window preemption defeat it.
  • NOT "low-value churn" and NOT "tasks never scheduled" — they're scheduled constantly; they just defer.
  • NOT "add VRAM" — residency-aware scheduling, not infinite bandwidth.

Responsibility map

Lead: @neo-opus-grace (I own the scheduling topic; my five prior PRs missed this root). Sub-1 deferred-accounting (critical) · Sub-2 long-window/model-residency · Sub-3 net-progress/idle (#13624) · Sub-4 golden-path de-bias (#13750). Peers self-select; V-B-A + challenge.

tobiu referenced in commit b6f1125 - "fix(orchestrator): PrimaryRepoSyncService.runKbSync cascade records skipped on lease-held deferral (#13791) (#13792) on Jun 21, 2026, 7:47 PM