LearnNewsExamplesServices
Frontmatter
id16056
titleIngestion failures must record their cause, and the orchestrator must be observable through its own bridge
stateClosed
labels
enhancementai
assigneesneo-opus-vega
createdAtJul 27, 2026, 9:29 PM
updatedAtJul 28, 2026, 3:10 AM
githubUrlhttps://github.com/neomjs/neo/issues/16056
authorneo-opus-vega
commentsCount2
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 28, 2026, 3:10 AM

Ingestion failures must record their cause, and the orchestrator must be observable through its own bridge

Closed Backlog/active-chunk-10 enhancementai
neo-opus-vega
neo-opus-vega commented on Jul 27, 2026, 9:29 PM

Context

Diagnosing a stalled tenant-repo ingestion on a remote deployment, using only the MCP tools that deployment exposes. Four repos were registered; none had ingested. The deployment-state snapshot reported the failure count but not a single fact about the failure, and the one component that holds the error text is excluded from the bridge that would surface it. The diagnosis was therefore impossible remotely — not difficult, impossible.

The Problem

1. Failures are counted, never characterised. The tenantRepoSync record reports, per repo:

status: "not-due" · lastIngestedRev: null · consecutiveFailures: 3
lastErrorCode: null · lastSourceErrorCode: null

Three failures, no code, no reason, no message. lastErrorAt carries a timestamp, so the state machine knows a failure happened and when — it simply discards what. Two dedicated fields exist for the purpose and are always null.

Worse, the repos then enter backoff and report not-due, which reads like "nothing to do" rather than "wedged after repeated failure." An operator watching the snapshot sees a healthy-looking idle scheduler.

2. The component holding the error excludes itself from observation. bridgeDiagnostics.runtimeAccess.allowedServices is ["chroma", "kb-server", "mc-server", "local-model"]. The orchestrator publishes the bridge, can read logs for four services, and is not one of them. So the error text — which exists only in orchestrator logs — is unreachable through the very interface built for remote diagnosis. Closing it requires host shell access, which defeats the purpose of the bridge.

3. A related mislead: get_ingestion_progress reported errorCount: 0 with all timestamps null, because the failure happens during repo acquisition, before ingestion starts. Technically accurate, and it reads as "no errors."

None of this is a regression — per-failure cause capture and orchestrator self-observability never existed.

The Architectural Reality

The snapshot was designed to answer "is it running?" and answers that well. It was not designed to answer "why did it stop?" — and a scheduler with backoff needs the second question, because backoff makes a wedged lane indistinguishable from an idle one at a glance.

The self-exclusion is a defensible default read too literally: the allow-list exists so the bridge cannot be turned into arbitrary container control, and the orchestrator was omitted because it is the subject rather than a target. But read-only log access to itself is not privilege escalation — it already has the socket. The safe shape is a distinct read-only self-observation capability rather than adding itself to the general lifecycle allow-list.

The Fix (one PR)

  • Populate lastErrorCode / lastSourceErrorCode with a bounded, secret-free classification on every failure path, plus a short safe reason string.
  • Distinguish backoff-suppressed from not-due in status, so a wedged lane cannot present as idle.
  • Grant the orchestrator read-only self-observation (logs/inspect) through the bridge without adding it to lifecycle-operable services.
  • Make get_ingestion_progress distinguish "no ingestion has been attempted" from "attempted, failed before starting."

Acceptance Criteria

  • Every tenant-repo-sync failure path sets lastErrorCode and, where applicable, lastSourceErrorCode; asserted per path, with a test that fails if a path leaves them null.
  • Recorded reasons are redacted — a clone failure must not surface a credential from a remote URL. Asserted with a token-bearing URL fixture.
  • status distinguishes backoff suppression from genuine not-due, and the record carries when the lane will next attempt.
  • The orchestrator's own logs are readable through the bridge, read-only, without appearing in lifecycleOperations targets — asserted both ways: readable, and not restartable.
  • get_ingestion_progress reports "never attempted" distinctly from "failed before ingestion began," so errorCount: 0 can no longer read as healthy.
  • Reviewer-checkable: from a remote MCP client alone, a wedged ingestion lane can be diagnosed to a cause without host shell access. That is the whole point of the ticket and should be the test.

Out of Scope

  • Fixing any particular ingestion failure (see the corpus-identity ticket for one concrete cause).
  • Retry/backoff policy tuning — this is about reporting the state, not changing it.

Related

  • #15798 (Local Runtime Parity) — parent epic