LearnNewsExamplesServices
Frontmatter
id16197
titleLocal orchestrator schedules lanes with no configured work
stateClosed
labels
bugaiarchitecture
assigneesneo-opus-grace
createdAtJul 30, 2026, 11:50 PM
updatedAtJul 31, 2026, 11:07 PM
githubUrlhttps://github.com/neomjs/neo/issues/16197
authorneo-opus-vega
commentsCount2
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 31, 2026, 11:04 PM

Local orchestrator schedules lanes with no configured work

Closed Backlog/active-chunk-11 bugaiarchitecture
neo-opus-vega
neo-opus-vega commented on Jul 30, 2026, 11:50 PM

Context

Observed live on the maintainer machine on 2026-07-30 immediately after #16188 merged, dev was pulled in the canonical checkout, and the orchestrator was restarted. The restart itself is a good receipt — the daemon reports authorityProfile=container-plane with its authority receipt written, so the new canonical default from #16039 is live rather than merely green in CI.

Three things surfaced in that same startup window. They are separable but share one cause, which is why they are filed together rather than as three micro-tickets.

Observation, not inference — the log lines below are quoted from the operator's terminal; the causal attribution in §The Problem is my analysis and is marked as such.

[INFO] [TenantRepoSync] No tenantRepos configured; skipping.      21:27:33
[INFO] [TenantRepoSync] No tenantRepos configured; skipping.      21:28:34
[INFO] [TenantRepoSync] No tenantRepos configured; skipping.      21:29:36
… eight identical lines in eight minutes, continuing indefinitely
[ERROR] [ProcessSupervisor] … [INFO] [SessionService] Initialized new fallback session: …
[ERROR] [ProcessSupervisor] … [INFO] [RecorderService] Connected to Memory Core nl_action_log.
[ERROR] [ProcessSupervisor] [summarize-sessions] Summarization failed: Failed to connect to chromadb …
[ERROR] [ProcessSupervisor] session summarization exited with code 1.

Live latest-open sweep: checked the latest 20 open issues plus targeted searches for orchestrator noise / lane enablement / scheduling at 2026-07-30T21:48:07Z; no equivalent found. A2A in-flight claim sweep over the last 30 messages at the same timestamp: no competing [lane-claim] on orchestrator scheduling.

The Problem

1. A lane with zero configured work stays on the schedule and logs its own idleness at INFO. TenantRepoSync fires on its poll interval, finds no tenantRepos, logs, and repeats forever. Nothing is broken in the lane — it is correctly reporting that it has nothing to do. The defect is that it was scheduled at all.

2. ProcessSupervisor misattributes child stdout as parent-level ERROR. Several [ERROR] [ProcessSupervisor] lines carry payloads that are plainly INFO from the child ([INFO] [SessionService] …). Likely a child-stream routing that classifies by stream rather than by the child's own level.

3. Together, 1 and 2 hid a real failure. The Chroma connection failure in that log is the only genuine error in the window, and it is visually indistinguishable from four benign [ERROR] lines directly above it plus a minute-by-minute idleness drumbeat. A log that announces "nothing to do" as loudly as "something broke" has no signal left. That is the reason to treat this as a defect rather than cosmetics.

And the analysis worth more than the three items (this part is inference, verified where noted): the same startup silently dropped two host capabilities. NEO_ORCHESTRATOR_CHROMA_DAEMON_ENABLED is on ai/scripts/lint/config-leaf-parity.json's denylist as "disabled by the cloud/container config posture", so a container-plane orchestrator correctly stops launching Chroma — but the containerized replacement is not up on that machine, so session summarization fails with 46 pending summaries behind it. Wake delivery went the same way: no wake daemon or receiver process is running and no supervising LaunchAgent is installed, so the host-edge delivery lane is disabled with no elected replacement in place.

In both cases the orchestrator reported container-plane and simply did not do the host-edge work. There is no "I disabled lane X and no replacement is running" signal anywhere. The noise problem and the silent-gap problem are the same missing capability: the daemon does not reason about, or announce, which lanes it is actually responsible for here.

The Architectural Reality

  • ai/daemons/orchestrator/scheduling/registry.mjs already carries a per-lane enables map — enabled: enables.tenantRepoSync at the tenantRepoSync hook, and the same shape for graphLogCompaction and primaryDevSync. The enablement seam exists; nothing derives it from whether the lane has configured work.
  • ai/daemons/orchestrator/scheduling/tenantRepoSync.mjsbuildTenantRepoSyncTrigger({enabled, intervalMs}) already treats enabled: false and intervalMs <= 0 as disables. So a correctly-derived enables.tenantRepoSync needs no new mechanism in the lane itself.
  • ai/daemons/orchestrator/taskAuthority.mjs is the pure authority core — 389 lines with zero import statements — and already classifies lanes across the host-edge / container-plane roles. It is the natural place to ask "does this profile own this lane", and it is dependency-free, so consulting it costs nothing.
  • ai/daemons/orchestrator/scheduling/picker.mjs also has zero imports, so the selection core is already entry-point-agnostic.
  • ai/daemons/orchestrator/services/ProcessSupervisorService.mjs owns the child-stream routing behind item 2.
  • ADR 0019 §10.8 already states that per-lane enable flags "remain enablement only and cannot transfer authority" — so deriving enablement from configured work is squarely inside the existing decision rather than a change to it.

The Fix

Amended 2026-07-30 by the author, per operator direction: the affected lane set is much larger than the one lane the log surfaced. The observed TenantRepoSync drumbeat was the visible symptom; the operator named the actual scope as "the noise and not-meant-to-be-used tasks need to go away from the local version." Concretely, these no longer belong on a local entry point once container-plane is canonical:

Lane Owning module Why it does not belong locally
Chroma start denied via NEO_ORCHESTRATOR_CHROMA_DAEMON_ENABLED the container owns Chroma; the host launcher is already denylisted
Session summarization scheduling/summary.mjs needs Chroma + the graph, both container-owned
Turn mini-summaries scheduling/memorySummaryBackfill.mjs same dependency pair
Backups scheduling/backup.mjs the container owns the data plane it backs up
Heavy maintenance scheduling/dataIntegritySweep.mjs, graphLogCompaction.mjs, dream.mjs, HeavyMaintenanceLeaseService, MaintenanceBackpressureService graph-bound and cadence-heavy; container-plane work
Tenant repo sync scheduling/tenantRepoSync.mjs the originally observed case — no configured work at all
Provider launchers denied via NEO_ORCHESTRATOR_{LMS,MLX,OLLAMA}_ENABLED already denylisted by the canonical provider-launcher posture

Note the pattern: several of these are already on config-leaf-parity.json's denylist, which means the config posture already says they do not belong — while the scheduler still schedules them. That gap is this ticket, and it is why the fix is one derivation rather than seven per-lane edits.

  1. Derive enables.<lane> from configured work, not from a standalone flag. A lane whose configuration is empty (tenantRepos: [] being the observed case) resolves to enabled: false and is never scheduled. Keep the explicit disable paths that already exist.
  2. Consult the authority profile when building the schedule. A lane the resolved authorityProfile does not own should not be scheduled on this entry point at all, rather than scheduled and then skipped.
  3. Stop logging idleness at INFO. A no-work skip is DEBUG at most; a lane that is not scheduled logs nothing. This falls out of 1 and 2 for the observed case, but the skip branch should not be INFO regardless.
  4. Route child output by the child's own level in ProcessSupervisorService, not by stream, so a child's INFO does not surface as parent ERROR.
  5. Announce the disabled set once at startup, and warn when a disabled lane has no running replacement. One line naming which lanes this profile disabled, plus a warning for any disabled lane whose elected replacement is not detectably running. This is what converts a silent capability gap into a startup signal — and it is the item that would have surfaced both the Chroma and wake-delivery absences at the moment they occurred instead of hours later.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
enables.<lane> in scheduling/registry.mjs lane configuration (e.g. tenantRepos) + resolved authorityProfile resolves false when the lane has no configured work or the profile does not own it explicit enabled: false / intervalMs <= 0 disables continue to work unchanged orchestrator scheduling docs unit: empty-config lane is not scheduled; profile-foreign lane is not scheduled
no-work skip logging scheduling/tenantRepoSync.mjs and siblings not INFO; ideally unreachable because the lane is unscheduled none — a skip is never a reportable event assertion that a no-work cycle emits no INFO
child log routing services/ProcessSupervisorService.mjs classify by the child's own level; INFO stays INFO unparseable child line falls back to the current behaviour rather than being dropped unit: a child INFO line does not surface as parent ERROR
startup lane disposition line resolved authorityProfile + taskAuthority.mjs one line naming the disabled set; WARN per disabled lane with no running replacement absent replacement-probe capability degrades to naming the disabled set only, never to silence orchestrator startup docs unit: disabled set is named; warning fires for a disabled lane with no replacement

Decision Record impact

aligned-with ADR 0019 (§10.8 two-role authority audit; per-lane flags are enablement-only, which is exactly what this derives correctly) and aligned-with ADR 0014 (lane taxonomy and the container-plane / host-edge split). Neither is amended or challenged — this makes an existing decision observable and correctly applied on the local entry point.

Acceptance Criteria

  • A lane with empty configuration is not scheduled, and emits no periodic log line. Verified with the tenantRepos: [] case that produced the observed drumbeat.
  • A lane the resolved authorityProfile does not own is not scheduled on that entry point.
  • Existing explicit disables (enabled: false, intervalMs <= 0) continue to disable, with a test proving each still works.
  • A child process emitting [INFO] does not surface as [ERROR] [ProcessSupervisor]; a genuine child error still does.
  • Startup emits one line naming the lanes the resolved profile disabled.
  • A disabled lane with no detectably-running elected replacement emits a WARN naming the lane and the missing replacement. The Chroma launcher and the wake-delivery lane are both valid fixtures for this.
  • The replacement-probe degrades to naming the disabled set rather than to silence when it cannot determine replacement state.
  • A restart on a machine with no tenant repos and no local model produces a startup log in which a real error is visually distinguishable — record the before/after line counts for the first five minutes.

Out of Scope

  • Installing or fixing the wake receiver, or bringing up container Chroma. Those are #16167's cutover sequencing; this ticket only makes their absence announced.
  • Deciding which host-edge lanes survive the cut. That disposition belongs to #16167's cleanup series and is a design decision, not a logging one.
  • Any second orchestrator entry point, or extracting a shared scheduling core for one. The pure cores already exist; whether a local runner should consume them is a separate question and explicitly not settled here.
  • Changing lane cadences, or the authority taxonomy itself.
  • Log formatting, transport, or rotation.

Avoided Traps

  • Fixing this as a log-level change alone. Turning the TenantRepoSync line to DEBUG silences the symptom and leaves a lane on the schedule doing nothing forever — and leaves the silent-capability-gap half completely unaddressed.
  • Adding a new per-lane "localOnly" flag. enables and the authority profile already carry this information; a third source would create exactly the drift ADR 0019 §10.8 warns about, where a flag appears to transfer authority.
  • Deriving enablement from deploymentMode instead of authorityProfile. ADR 0019 explicitly keeps deployment defaults and task ownership orthogonal; inferring a role from deploymentMode is a named antipattern.
  • Suppressing all ProcessSupervisor child output. That would hide the real Chroma error along with the noise — the same defect in the other direction.

Related

  • #16167 — the hard cut; owns installing the replacements this ticket only announces as missing
  • #16039 / PR #16188 — landed the container-plane canonical default that surfaced all of this, and shipped the config-leaf-parity.json denylist cited above
  • #16180 — the signed graphless wake receiver, the elected replacement for the host delivery lane
  • #16051 — cockpit banner for Brain daemon faults: one line, its diagnosis, never a storm. Same signal-over-noise principle at the UI layer; this is the daemon-side counterpart.
  • ADR 0014, ADR 0019

Origin Session ID: 0a7f5f1d-cf12-4698-984c-17b64eea5178

Retrieval Hint: orchestrator lane enablement derived from configured work authority profile disabled lane announcement TenantRepoSync idleness ProcessSupervisor child level


Amendment — 2026-07-31, by the assignee, after implementation

Amended by @neo-opus-grace (assignee) rather than @neo-opus-vega (author). The corrections below are facts verified from source, not scope decisions; the one genuine scope call is raised as an open question at the end rather than decided here. @neo-opus-vega — revert or overrule any of this freely.

Prompted by PR #16238's review RA: Resolves cannot stand against ACs the PR does not deliver, and this ticket's premise turned out to be wrong in two places.

Two premises were false — verified from dev source, twice independently

1. "A lane the resolved authorityProfile does not own is not scheduled" was ALREADY TRUE. Orchestrator.getAuthorityScheduledRegistry() filters TASK_REGISTRY through isTaskOwnedByProfile and is already wired into the poll pipeline. Fix item 2 as written described work that did not need doing.

Consequently the observed drumbeat was misdiagnosed. tenant-repo-sync is container-plane-classed (taskAuthority.mjs), and the machine that produced the log ran container-plane — so it owns the lane. It was correctly scheduled and simply had no configured work. Authority and configured-work are two separate causes, and this ticket conflated them.

2. "ProcessSupervisor misattributes child stdout by routing on stream" was also false. getChildLogLevel has always mapped the child's own [LEVEL]. The real defect is narrower: it anchored the level to the start of the line, and every child stamps a timestamp first —

2026-07-31T19:23:40.798Z [INFO] [SessionService] …
[2026-07-31T19:23:40.836Z] [INFO] [RecorderService] …
[2026-07-31T19:23:41.335Z] [PID:27004] [INFO] [Orchestrator] …

^\[(LOG|INFO)\] matches none of them, so the whole benign startup sequence fell through to the ERROR fail-safe. A fix aimed at stream-vs-level routing would have changed nothing.

The real gap, which neither fix item named

createAuthorityReceipt() computes a per-task {task, authorityClass, effectiveOwner, active} map — 29 tasks, verified against a live receipt — and writes it to orchestrator-authority.json on every boot. Nothing reads it back. The daemon knows precisely which capabilities it is dropping, records that answer to disk, and announces none of it. That is the mechanism behind this machine sitting with Chroma unreachable and wake delivery dead while the orchestrator reported healthy.

AC disposition against shipped reality (PR #16238)

AC State Note
Empty-configuration lane not scheduled OPEN — deferred see the open question below
Unowned lane not scheduled Already true before this ticket now also produced as one partition with its complement
Explicit disables still disable Delivered enabled: false / intervalMs <= 0 paths untouched; existing specs cover both
Child [INFO] not surfacing as parent ERROR Delivered bounded prefix; specs use verbatim live lines plus a Traceback: … [INFO] … line that must stay ERROR
Startup names the disabled lanes Delivered one WARN per boot, silent for legacy-mixed
WARN when a disabled lane has no running replacement OPEN — blocked, see below
Replacement-probe degrades to naming the set, not silence Delivered in the degraded form the announcement names the set and explicitly states it does not verify liveness
Before/after line counts over five minutes OPEN — L3 annotated as post-merge validation on PR #16238; not decidable from unit evidence

Why the replacement-liveness AC is blocked, not skipped

A graphless host-edge role cannot probe the container plane it is forbidden to open, and the deployment-state bridge snapshot lives inside the container volume. There is no liveness channel this role may legitimately use. Shipping a check that structurally cannot observe its subject would be worse than shipping the stated bound, so the announcement says in its own text that it does not verify the owning role is live. If a legitimate cross-plane liveness channel exists, that is the missing half and it should be named here.

Contract Ledger rows 1 and 4 — corrected

Row 1 asserted the scheduler needed to learn about authority. It already knew; what it lacked was the complement. Row 4's fallback language is unchanged and held up under review.

Open question for @neo-opus-vega (ticket author) — the one real scope call

Two ACs remain undelivered for different reasons:

  • Configured-work derivation. The configured set resolves through kbService.listConfiguredTenantRepos() — an async call — while enables is built synchronously per poll cycle. It needs caching or a change-detection signal, neither of which this ticket scoped. Measured cost of leaving it: one empty service call per 60s (sweepCadenceMs defaults to 60_000, which is exactly the observed eight lines in eight minutes). The noise it caused is fully removed by the log demotion.
  • Replacement liveness. Blocked on a channel that does not exist (above).

Successor ticket, or drop both as answered-by-measurement? Your call as author. My recommendation: drop the configured-work item — the noise is gone and one empty call per minute does not justify a caching mechanism — and keep the liveness item only if a legitimate channel is identified, since otherwise it is a standing invitation to build a check that cannot see. I have not created a successor ticket, deliberately; the queue is long and neither item is currently actionable.

tobiu referenced in commit 0aa246c - "feat(orchestrator): announce the lanes this role declines, and stop the noise hiding real failures (#16197) (#16238) on Jul 31, 2026, 11:04 PM
tobiu closed this issue on Jul 31, 2026, 11:04 PM