Context
Diagnosing a stalled tenant-repo ingestion on a remote deployment, using only the MCP tools that deployment exposes. Four repos were registered; none had ingested. The deployment-state snapshot reported the failure count but not a single fact about the failure, and the one component that holds the error text is excluded from the bridge that would surface it. The diagnosis was therefore impossible remotely — not difficult, impossible.
The Problem
1. Failures are counted, never characterised. The tenantRepoSync record reports, per repo:
status: "not-due" · lastIngestedRev: null · consecutiveFailures: 3
lastErrorCode: null · lastSourceErrorCode: null
Three failures, no code, no reason, no message. lastErrorAt carries a timestamp, so the state machine knows a failure happened and when — it simply discards what. Two dedicated fields exist for the purpose and are always null.
Worse, the repos then enter backoff and report not-due, which reads like "nothing to do" rather than "wedged after repeated failure." An operator watching the snapshot sees a healthy-looking idle scheduler.
2. The component holding the error excludes itself from observation. bridgeDiagnostics.runtimeAccess.allowedServices is ["chroma", "kb-server", "mc-server", "local-model"]. The orchestrator publishes the bridge, can read logs for four services, and is not one of them. So the error text — which exists only in orchestrator logs — is unreachable through the very interface built for remote diagnosis. Closing it requires host shell access, which defeats the purpose of the bridge.
3. A related mislead: get_ingestion_progress reported errorCount: 0 with all timestamps null, because the failure happens during repo acquisition, before ingestion starts. Technically accurate, and it reads as "no errors."
None of this is a regression — per-failure cause capture and orchestrator self-observability never existed.
The Architectural Reality
The snapshot was designed to answer "is it running?" and answers that well. It was not designed to answer "why did it stop?" — and a scheduler with backoff needs the second question, because backoff makes a wedged lane indistinguishable from an idle one at a glance.
The self-exclusion is a defensible default read too literally: the allow-list exists so the bridge cannot be turned into arbitrary container control, and the orchestrator was omitted because it is the subject rather than a target. But read-only log access to itself is not privilege escalation — it already has the socket. The safe shape is a distinct read-only self-observation capability rather than adding itself to the general lifecycle allow-list.
The Fix (one PR)
- Populate
lastErrorCode / lastSourceErrorCode with a bounded, secret-free classification on every failure path, plus a short safe reason string.
- Distinguish backoff-suppressed from not-due in
status, so a wedged lane cannot present as idle.
- Grant the orchestrator read-only self-observation (logs/inspect) through the bridge without adding it to lifecycle-operable services.
- Make
get_ingestion_progress distinguish "no ingestion has been attempted" from "attempted, failed before starting."
Acceptance Criteria
Out of Scope
- Fixing any particular ingestion failure (see the corpus-identity ticket for one concrete cause).
- Retry/backoff policy tuning — this is about reporting the state, not changing it.
Related
- #15798 (Local Runtime Parity) — parent epic
Context
Diagnosing a stalled tenant-repo ingestion on a remote deployment, using only the MCP tools that deployment exposes. Four repos were registered; none had ingested. The deployment-state snapshot reported the failure count but not a single fact about the failure, and the one component that holds the error text is excluded from the bridge that would surface it. The diagnosis was therefore impossible remotely — not difficult, impossible.
The Problem
1. Failures are counted, never characterised. The
tenantRepoSyncrecord reports, per repo:Three failures, no code, no reason, no message.
lastErrorAtcarries a timestamp, so the state machine knows a failure happened and when — it simply discards what. Two dedicated fields exist for the purpose and are always null.Worse, the repos then enter backoff and report
not-due, which reads like "nothing to do" rather than "wedged after repeated failure." An operator watching the snapshot sees a healthy-looking idle scheduler.2. The component holding the error excludes itself from observation.
bridgeDiagnostics.runtimeAccess.allowedServicesis["chroma", "kb-server", "mc-server", "local-model"]. The orchestrator publishes the bridge, can read logs for four services, and is not one of them. So the error text — which exists only in orchestrator logs — is unreachable through the very interface built for remote diagnosis. Closing it requires host shell access, which defeats the purpose of the bridge.3. A related mislead:
get_ingestion_progressreportederrorCount: 0with all timestamps null, because the failure happens during repo acquisition, before ingestion starts. Technically accurate, and it reads as "no errors."None of this is a regression — per-failure cause capture and orchestrator self-observability never existed.
The Architectural Reality
The snapshot was designed to answer "is it running?" and answers that well. It was not designed to answer "why did it stop?" — and a scheduler with backoff needs the second question, because backoff makes a wedged lane indistinguishable from an idle one at a glance.
The self-exclusion is a defensible default read too literally: the allow-list exists so the bridge cannot be turned into arbitrary container control, and the orchestrator was omitted because it is the subject rather than a target. But read-only log access to itself is not privilege escalation — it already has the socket. The safe shape is a distinct read-only self-observation capability rather than adding itself to the general lifecycle allow-list.
The Fix (one PR)
lastErrorCode/lastSourceErrorCodewith a bounded, secret-free classification on every failure path, plus a short safe reason string.status, so a wedged lane cannot present as idle.get_ingestion_progressdistinguish "no ingestion has been attempted" from "attempted, failed before starting."Acceptance Criteria
lastErrorCodeand, where applicable,lastSourceErrorCode; asserted per path, with a test that fails if a path leaves them null.statusdistinguishes backoff suppression from genuine not-due, and the record carries when the lane will next attempt.lifecycleOperationstargets — asserted both ways: readable, and not restartable.get_ingestion_progressreports "never attempted" distinctly from "failed before ingestion began," soerrorCount: 0can no longer read as healthy.Out of Scope
Related