Context
Live external-plane evidence (2026-08-13, deployment-state snapshots via the MC surface): the orchestrator's tenant-repo-sync scheduler froze for 90+ minutes (lastRunAt pinned at 07:56:31 on a 30-minute cadence / 60-second sweep) while the orchestrator container reported healthy — its healthcheck watches the state-file mtime, which the deployment-state bridge thread keeps fresh, so a dead scheduler inside a live process is invisible to Docker.
The wedge location was proven by probe divergence: the orchestrator's own embedding recovery probe timed out (EMBEDDING_PROBE_TIMEOUT) while kb-server's identical probe against the same provider passed. One process's serial embed dispatch was stuck; the provider was fine.
Timeline: a sync attempt failed instantly at 07:56 (KB_VECTOR_EMBED_FAILED recorded in the same second the run started — an unserviceable call, not a slow one); the self-heal system restarted the embedding provider container at 08:10; the orchestrator's queue never recovered afterward. A candidate mechanism — a request whose socket died in the provider restart without surfacing an error — should have been released by the 60-minute batch ceiling, and was not, which points at a call on this path that carries no deadline at all (or an await that a transport error never rejects).
This is the third layer of the same disease class in one incident family: engine (abandoned work never dies) → client process queue (serial dispatch starved by a hung head) → scheduler (starved by the queue). Each layer's lesson is identical: every call on the embed path must carry a deadline, and every abort must propagate to whatever is awaiting it.
The Fix (shape)
- Audit every await on the orchestrator's embedding dispatch path (recovery probe, tenant-sync ingest calls, WAL-adjacent embeds) for missing or unreachable deadlines; wrap the transport so socket death and container restarts reject promptly.
- The serial dispatch queue gets a queue-head watchdog: a head older than the maximum legitimate ceiling is failed and evicted, loudly, rather than starving the process forever.
- The scheduler's liveness becomes observable independently of the state-file mtime (the bridge keeping the file fresh must not mask a dead sync loop) — a
lastRunAt-age field on the snapshot plus a degraded healthcheck contribution when it exceeds the cadence by a bound.
Acceptance Criteria
Related
#16972 (batch retry sizing — sibling, distinct defect) · #16706 (external-plane recovery epic) · #17044 (self-heal futility circuit breaker — same incident family) · #17042 (retrospective mechanics)
Origin Session ID: 944f029a-1fab-46a1-9d11-a9fadd003db8
Context
Live external-plane evidence (2026-08-13, deployment-state snapshots via the MC surface): the orchestrator's tenant-repo-sync scheduler froze for 90+ minutes (
lastRunAtpinned at 07:56:31 on a 30-minute cadence / 60-second sweep) while the orchestrator container reported healthy — its healthcheck watches the state-file mtime, which the deployment-state bridge thread keeps fresh, so a dead scheduler inside a live process is invisible to Docker.The wedge location was proven by probe divergence: the orchestrator's own embedding recovery probe timed out (
EMBEDDING_PROBE_TIMEOUT) while kb-server's identical probe against the same provider passed. One process's serial embed dispatch was stuck; the provider was fine.Timeline: a sync attempt failed instantly at 07:56 (
KB_VECTOR_EMBED_FAILEDrecorded in the same second the run started — an unserviceable call, not a slow one); the self-heal system restarted the embedding provider container at 08:10; the orchestrator's queue never recovered afterward. A candidate mechanism — a request whose socket died in the provider restart without surfacing an error — should have been released by the 60-minute batch ceiling, and was not, which points at a call on this path that carries no deadline at all (or an await that a transport error never rejects).This is the third layer of the same disease class in one incident family: engine (abandoned work never dies) → client process queue (serial dispatch starved by a hung head) → scheduler (starved by the queue). Each layer's lesson is identical: every call on the embed path must carry a deadline, and every abort must propagate to whatever is awaiting it.
The Fix (shape)
lastRunAt-age field on the snapshot plus a degraded healthcheck contribution when it exceeds the cadence by a bound.Acceptance Criteria
lastRunAtage > cadence × N) is visible on the deployment-state snapshot and degrades the orchestrator's health contribution.Related
#16972 (batch retry sizing — sibling, distinct defect) · #16706 (external-plane recovery epic) · #17044 (self-heal futility circuit breaker — same incident family) · #17042 (retrospective mechanics)
Origin Session ID: 944f029a-1fab-46a1-9d11-a9fadd003db8