LearnNewsExamplesConnect

Cloud Deployment Troubleshooting

A first real deployment of the Agent OS behind a reverse proxy crosses several layers — the MCP SDK's Host validation, the proxy's Accept / auth handling, the identity-header contract, and the MCP handshake itself. Each layer fails with its own error, and the fix differs per layer. This guide is the connect-error ladder: each error → which layer you just cleared → the fix.

Complements the Day-0 Tutorial (the happy path) and Client Authentication (the PAT login). Reach for this when a connection does not work.

The connect-error ladder

Work top-to-bottom — each error means you cleared the layer above it.

-32000 Invalid Host: <your-host>

Layer: the MCP SDK's DNS-rebinding protection, which defaults to a localhost-only Host allowlist. Behind a reverse proxy the public Host header reaches the server unchanged and is rejected before any MCP handling — for every client, not just browsers.

Fix (either):

  • Configure the server's Host allowlist to include your public hostname: set NEO_PUBLIC_URL=https://mcp.<your-host> (its hostname is auto-allowed) and/or NEO_MCP_ALLOWED_HOSTS=<comma-separated hostnames>. (localhost stays allowed — the container healthcheck needs it.)
  • Or rewrite the upstream Host to localhost at the proxy (Caddy: header_up Host localhost on the reverse_proxy block). This is the no-rebuild interim fix; the allowlist config is the durable one.

Not Acceptable: Client must accept text/event-stream

Layer: you reached the Streamable HTTP MCP endpoint, not a web page; the client omitted the required Accept header.

Fix: send Accept: application/json, text/event-stream on every request.

HTTP 302 → /oauth2/sign_in

Layer: a reverse-proxy auth gate (e.g. oauth2-proxy) sits in front of the MCP server; your request has no valid session, so it redirects to sign-in.

Fix: authenticate past the proxy. For the durable headless path see Client Authentication (GitLab-PAT Bearer); for first bring-up you can open the test path (see Test vs production profile).

-32000 Unauthorized: Missing proxy identity header

Layer: you cleared the proxy, and the server is in trustProxyIdentity mode — it expects the proxy to inject a validated identity header (e.g. X-PREFERRED-USERNAME) and strips any client-supplied identity header (anti-spoof). No injected identity → rejected.

Fix: have the proxy inject the identity header for authenticated requests (oauth2-proxy forwards the upstream identity; Caddy via forward_auth + header_up). For a no-auth test path, set NEO_AUTH_TRUST_PROXY_IDENTITY=false and inject a static identity at the proxy, or run the open test profile.

-32000 Bad Request: Server not initialized

Layer: you are authenticated and talking to the MCP server — but called a tool (e.g. tools/list) before the MCP initialize handshake.

Fix: do the handshake first — initialize → capture the Mcp-Session-Id response header → notifications/initialized → then tool calls (carrying the session id). See Verify from outside below.

A body of event: message / data: {…}

Not an error. That is normal SSE response framing within Streamable HTTP — a successful initialize returns event: message plus a data: line carrying the JSON-RPC result (including serverInfo).

Deployment gotchas

The Caddyfile is baked into the image

A Caddyfile (or any config copied in at image-build time) is not picked up by a container restart. After editing it, rebuild:

NEO_DEPLOY_PROJECT_NAME=<project> docker compose up -d --build caddy

A plain restart re-runs the old baked config.

Container ports are network-internal

In a typical compose deployment only the ingress (Caddy) publishes a host port; the MCP servers listen on the internal compose network. A host-side curl http://127.0.0.1:<port> therefore finds nothing — expected, not a failure. Test from inside the network instead:

NEO_DEPLOY_PROJECT_NAME=<project> docker compose exec <server> node ./ai/scripts/diagnostics/mcpHealthcheck.mjs --url http://127.0.0.1:<port>

…or go through the ingress on the public URL.

Two auth layers — do not fight the wrong one

A proxied deployment commonly has two auth layers, and an error at one is easily mistaken for the other:

  1. The reverse-proxy gate (e.g. oauth2-proxy cookie / OAuth) — decides whether a request reaches the MCP server at all (error: 302 → sign_in).
  2. The MCP server's identity (trustProxyIdentity header or a Bearer token) — decides who the authenticated request is (errors: Missing proxy identity header, 401).

Identify which layer an error came from (via the ladder) before changing any config.

dependency <svc> failed to start after enabling NEO_AUTH_MODE=gitlab-pat

Symptom: after switching a compose deployment to NEO_AUTH_MODE=gitlab-pat, docker compose up leaves kb-server or mc-server unhealthy, and dependent services report dependency <svc> failed to start.

Confirm it is the bearer healthcheck case, not a boot crash: send a tokenless initialize request through the public ingress. A clean 401 with WWW-Authenticate: Bearer means the MCP server is up and enforcing auth; a 502, timeout, or connection-refused result points at a process/proxy boot failure instead.

Cause: the in-container healthcheck runs mcpHealthcheck.mjs against the same authenticated /mcp route. In gitlab-pat mode that self-probe must also send a bearer token. If NEO_MCP_HEALTHCHECK_TOKEN is unset or empty, the healthcheck gets 401, the container never reaches service_healthy, and compose dependency waits abort.

Fix: set NEO_MCP_HEALTHCHECK_TOKEN=<read_user bearer> in the deployment .env (a GitLab PAT or OAuth access token with the same read_user validation surface as other MCP clients), then recreate the affected services so the environment is reloaded:

NEO_DEPLOY_PROJECT_NAME=<project> docker compose up -d --force-recreate kb-server mc-server orchestrator

This is an environment-only repair; no image rebuild is required.

inspect_deployment says No Docker container found for compose service ...

Layer: the MCP server is reachable and the deployment-state bridge snapshot is fresh, but the orchestrator's runtime-access holder cannot resolve one or more allowlisted Compose services through Docker labels.

Read snapshot.bridgeDiagnostics before changing service code:

  • runtimeAccess.enabled: false means the bridge is intentionally not using a runtime handle. Set NEO_ORCHESTRATOR_RUNTIME_ACCESS_ENABLED=true only when B1 runtime diagnostics/recovery are intended.
  • reason: broad-service-lookup-failure means most or all configured services failed at the observation layer. Treat this as bridge configuration first, not as four independent service outages.
  • compose-project-unavailable means runtime access was enabled without a project identity. For the canonical stack, set NEO_DEPLOY_PROJECT_NAME=<project> and recreate the orchestrator; the Compose file binds that one value to both Docker project labels and the runtime holder.
  • compose-service-no-match means Docker returned no container for the label filter shown in the error details. Verify both com.docker.compose.project=<project> and com.docker.compose.service=<service>. Align NEO_ORCHESTRATOR_RUNTIME_ACCESS_ALLOWED_SERVICES and NEO_DEPLOYMENT_STATE_BRIDGE_ALLOWED_SERVICES with Docker service labels, not container names, and do not pass a standalone -p that conflicts with NEO_DEPLOY_PROJECT_NAME.
  • compose-service-ambiguous means more than one container matched a service label inside the configured project. Resolve the duplicate/scaled target; project binding must not be weakened to select one arbitrarily.
  • compose-project-mismatch or compose-service-mismatch means a Docker response did not prove the exact requested label pair. The holder rejects the target before inspect, logs, stats, or restart and does not expose the foreign label value.
  • docker-socket-unavailable or docker-socket-forbidden means the orchestrator cannot read the runtime socket. Mount /var/run/docker.sock into the orchestrator with suitable permissions, or disable runtime access explicitly when that deployment should not expose B1 diagnostics.
  • The default diagnostic set observes sibling services, not the orchestrator container itself. When orchestrator logs/state are needed for a cloud incident, add the Compose service label to both NEO_ORCHESTRATOR_RUNTIME_ACCESS_ALLOWED_SERVICES and NEO_DEPLOYMENT_STATE_BRIDGE_ALLOWED_SERVICES (for the bundled compose file, use orchestrator).

The diagnostic intentionally exposes only non-secret config, service keys, and the label filter shape. It does not enumerate arbitrary containers or expose Docker, shell, restart, or daemon-control routes through public MCP tools.

Test profile vs production auth profile

Verify the wiring before real auth is enabled, but keep the two profiles distinct:

  • Test / bring-up profile: open the Caddy path (no oauth2-proxy gate) and either NEO_AUTH_TRUST_PROXY_IDENTITY=false or a static identity injected at the proxy. Proves ingress + handshake without per-user auth. Not for production.
  • Production profile: proxy gate on + a real per-user credential — trustProxyIdentity with an injected identity, or NEO_AUTH_MODE=gitlab-pat Bearer (see Client Authentication).

Verify from outside (the handshake)

Confirm a deployment end-to-end with curl — no MCP client needed. MCP initialize is a short sequence:

URL="https://mcp.<your-host>/mc/mcp"
AUTH=(-H "Authorization: Bearer ${NEO_MCP_TOKEN}")   # or your proxy's auth, per profile

<h1 class="neo-h1" data-record-id="17">1. initialize → expect HTTP 200 + an <code>mcp-session-id</code> response header + serverInfo</h1>

curl -sS -i -X POST "$URL" "${AUTH[@]}" \
  -H "Accept: application/json, text/event-stream" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"curl","version":"1.0"}}}'

<h1 class="neo-h1" data-record-id="18">2. read the Mcp-Session-Id header from step 1, then:</h1>

SID="<mcp-session-id from step 1>"

<h1 class="neo-h1" data-record-id="19">3. notifications/initialized (carry the session id)</h1>

curl -sS -X POST "$URL" "${AUTH[@]}" \
  -H "Accept: application/json, text/event-stream" -H "Content-Type: application/json" \
  -H "Mcp-Session-Id: ${SID}" \
  -d '{"jsonrpc":"2.0","method":"notifications/initialized","params":{}}'

<h1 class="neo-h1" data-record-id="20">4. tool calls now work, e.g. tools/list</h1>

curl -sS -X POST "$URL" "${AUTH[@]}" \
  -H "Accept: application/json, text/event-stream" -H "Content-Type: application/json" \
  -H "Mcp-Session-Id: ${SID}" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}'

A healthy Streamable HTTP server can return event: message SSE framing carrying the JSON-RPC results. A Server not initialized error means step 1's handshake was skipped.

There is no auth-free GET /health liveness endpoint today — the server exposes only the authenticated /mcp route — so external liveness checks run the initialize handshake above (or rely on the container's internal healthcheck).

First query returns nothing — the empty-KB gap

A freshly deployed Knowledge Base can be healthy but empty: healthcheck reports count: 0 and queries return nothing until an ingest writes chunks. Treat that as an ingestion-state question, not a Chroma readiness question.

For push-mode deployments, trigger the deployment's ingestion entry point once, then query again.

For a pull-mode deployment, read tenantRepoSync.config.bootstrap before treating an empty repo count as intentional:

Bootstrap status Meaning Operator action
missing The optional kb-config.yaml file is not mounted. Confirm graph/AiConfig is the intended authority, or mount the bootstrap file. This state does not degrade diagnostics.
empty The file was readable but contained no YAML document. Add the intended tenants: mapping, or keep the file empty when another tier is authoritative. This state does not degrade diagnostics.
loaded The file matched the {tenants: {...}} contract. Compare tenantCount, effective repoCount, and tierCounts to the intended deployment. A zero tenant count is valid.
read-failed The file exists or was addressed, but the service could not read it. Fix mount visibility, ownership, or permissions, then refresh the deployment snapshot.
parse-failed The file was readable but was not valid YAML. Validate and correct YAML syntax, redeploy the file, then refresh the snapshot.
invalid-shape YAML parsed, but the top level was not an object containing an object-valued tenants mapping. Restore the documented tenants: bootstrap shape, then refresh the snapshot.

The three failure states degrade the config diagnostic without discarding safely resolved graph or AiConfig fallback repos. The snapshot deliberately omits paths, YAML content, tenant/repository identities, clone URLs, credentials, tokens, stacks, and raw filesystem/parser messages.

For pull-mode deployments with configured tenantRepos[], call inspect_deployment or get_deployment_state_snapshot and inspect the tenantRepoSync section before taking manual action. It distinguishes disabled, true no-configured-repos, not-due, running, completed, failed, and degraded/unreadable state without exposing credentials or raw logs. A degraded config-read error means the graph/YAML/default config resolver could not prove the effective tenantRepos; treat that differently from a real empty config. If the task is configured but has not advanced, use the stable reason code and per-repo hashed state there to decide whether to wait for the next due sweep, fix credentials/config, or run node ./ai/scripts/maintenance/syncTenantRepos.mjs --repo-slug <slug> inside the orchestrator container. When lastErrorCode is KB_TENANT_REPO_SYNC_SYNC_FAILED, check lastSourceErrorCode: KB_GITMIRROR_CREDENTIAL_REF_INVALID, KB_GITMIRROR_CLONE_FAILED, or KB_GITMIRROR_FETCH_FAILED points to the credential/ref/upstream access path before generic ingestion debugging. KB_VECTOR_EMBED_FAILED means the repository fetch and envelope reached the ingestion layer but vector writes failed; correct the embedding path before retrying. KB_INGEST_STORE_UNREACHABLE is the transport sibling of that code: ingestion started but could not reach the vector store at all, so treat it as store availability (restarting, saturated, wrong host/port) rather than as an embedding-path defect — retrying is appropriate once the store answers, whereas KB_VECTOR_EMBED_FAILED will keep failing until the embedding path itself is corrected. A bare KB_INGEST_FAILED now means only that the thrown error carried no bounded code; the underlying message is deliberately not projected, so escalate on the surrounding lane state rather than expecting the code to narrow further.

credentialRef: none is genuinely anonymous: GitMirror does not consult the orchestrator account's Git config, URL rewrites, helpers, .netrc, SSH config, agent, or default keys. Configure env:, file:, or ssh: explicitly when the remote is private. SSH endpoints should carry their non-secret login name (for example ssh://git@host/org/repo.git); the key remains in credentialRef. GitMirror preserves TOFU continuity in <mirrorRoot>/.gitmirror-ssh/known_hosts, not in the host user's home.

Read tenantRepoSync.accessReadiness before waiting for cadence. ready means every enabled repo passed the process-local credential check and bounded remote capability probe; degraded means at least one proved failure; unknown means the current process has not produced valid evidence yet (commonly immediately after restart); not-required means there are no enabled required repos. Inspection reads cached evidence and never triggers network work.

Access code Meaning Operator action
KB_TENANT_REPO_ACCESS_CREDENTIAL_INVALID The env secret, secret file, or SSH key is missing, empty, unreadable, or its reference shape is invalid. Align the configured reference with the orchestrator's mounted env/file/key, then let the next sweep re-resolve it.
KB_TENANT_REPO_ACCESS_TIMEOUT The bounded ls-remote probe exceeded its deadline. Check upstream latency, firewall/proxy behavior, and DNS; do not infer token scope from a timeout.
KB_TENANT_REPO_ACCESS_TRANSPORT_FAILED Git reported a network/DNS/connection-class failure. Restore egress and name resolution from the orchestrator network.
KB_TENANT_REPO_ACCESS_DENIED_OR_NOT_FOUND The remote did not grant readable repository capability, or intentionally hid a missing repository behind the same response. Verify repository membership/reach and read permission for this specific repo; do not assume one token reaches sibling groups/namespaces.
KB_TENANT_REPO_ACCESS_REF_NOT_FOUND The repo was readable but the configured ref was not advertised. Correct branchRef or publish the intended branch/tag.
KB_TENANT_REPO_ACCESS_REF_UNVERIFIED A raw commit SHA was not an advertised ref tip, so ls-remote could prove repo access but not commit reachability. Let the normal clone/fetch resolve the SHA; do not rewrite the ref solely from this unknown result.
KB_TENANT_REPO_ACCESS_EVIDENCE_EXPIRED The cached proof is older than its bounded lifetime. Wait for the next sync sweep to re-probe; inspection does not launch network work.
KB_TENANT_REPO_ACCESS_PROBE_FAILED / KB_TENANT_REPO_ACCESS_PROBE_UNAVAILABLE The probe could not produce a certifying category. Verify the orchestrator and GitMirror code are from one current image cohort, then inspect service health.
KB_TENANT_REPO_ACCESS_SYNC_FAILED A later authoritative clone/fetch failed after or without the cached probe. Use lastSourceErrorCode and the normal GitMirror acquisition runbook; clone/fetch supersedes older probe evidence.

Current releases accept a tenant-repo ingest as successful only when the returned summary contains an array-valued, empty errors field. Under checkpoint contract v2, a manifest-bearing bootstrap, non-linear fallback, manual full replay, or legacy revalidation must also persist an attempt-bound graph receipt for a positive safe-integer ingested or deleted count. The one zero-effect exception is checkpoint recovery: when the matching receipt is still unacknowledged because the prior local checkpoint write failed or crashed, the next run settles it before repeating KB mutation. An acknowledged receipt cannot be reused by a later manual full replay. The receipt is pull-internal; ingest_source_files cannot submit or receive it. Otherwise KB_TENANT_REPO_SYNC_EMPTY_MATERIALIZATION preserves the last known-good revision. Incremental no-op remains valid, as does a full reconciliation whose only effect is deletion. A failed attempt keeps the last known-good revision. If the deployment was upgraded after an older release had already advanced its checkpoint on an error-bearing or zero-effect v1 summary, the stored head has no current success proof. The periodic lane classifies that repo as checkpointStatus: pending and performs one bounded null-base replay automatically. Only concurrencyLimit legacy checkpoints are admitted per one-minute scheduler sweep; failures become checkpointStatus: failed, preserve the old head, and retry through normal backoff.

Inspect tenantRepoSync.checkpointRevalidation for pending, failed, complete, uninitialized, and unsupported counts. The aggregate is unavailable when repository enumeration, revision-state reading, or persisted-marker validation fails, because an all-zero projection cannot prove an empty checkpoint set. Per-repo rows use hashed identities and expose the same checkpointStatus without clone URLs, refs, credentials, or raw errors. A future/unsupported contract marker requires upgrading this runtime. A present but malformed marker is invalid persisted state. Both fail closed and are never silently downgraded; malformed state makes the aggregate unavailable rather than fabricating a per-repo classification.

After correcting the underlying failure, ordinary periodic retry is sufficient. To accelerate one known repo rather than wait for its cadence/backoff, use the scoped operator override:

node ./ai/scripts/maintenance/syncTenantRepos.mjs --full --repo-slug <slug>

--full is rejected without an explicit repo selector. It does not delete the stored checkpoint: a failed or fresh zero-effect replay preserves it, while an error-free replay with a positive ingest/delete effect advances it to the current head and writes the current success-contract marker. A zero-effect retry can advance when it settles the matching unacknowledged receipt from an interrupted post-ingest checkpoint commit before repeating Knowledge Base mutation, and also when the manifest authoritatively declares no content and the matching receipt's attempt is already committed — repeating --full on an unchanged, legitimately-empty repo therefore succeeds every time instead of failing from the second run onward. On a manifest that declares content, a zero-effect replay still cannot advance the checkpoint.

The CLI first acquires the deployment-wide heavy-maintenance lease, then the same cross-process tenant-repo-sync lease as the daemon's periodic sweep. Exit code 4 is the bounded busy result for either holder class. A global deferral writes Deferred: heavy-maintenance lease held by <owner> to stderr; a same-lane deferral returns KB_TENANT_REPO_SYNC_LEASE_HELD, so use the diagnostic rather than the shared code to choose a retry cadence. Global holders such as Dream/REM or backup can legitimately run up to the configured heavy-maintenance stale bound (six hours by default); wait for the named task instead of retrying on the shorter tenant-sync cadence. A periodic sweep deferred by the narrower lease reports a skipped outcome with no backoff mutation. Crashed lease owners recover automatically on the next attempt (pid-liveness). A live tenant sync renews its lease every max(5s, TTL/3), so leaseStaleAfterMs only ever expires an owner that stopped renewing (fully wedged or gone); a run that loses its lease anyway fails with KB_TENANT_REPO_SYNC_LEASE_LOST at its next work fence — before further git, ingest, or manifest work — leaving checkpoints, backoff state, and the new owner's lease untouched, so the failure is safe to retry once the concurrent run finishes. Lease transitions themselves (recovery, release, renewal) serialize through a sibling …lease.json.lifecycle-guard directory; an abandoned guard from a hard crash self-heals after ~10 seconds and needs no operator action.

Manifest writes are atomic (temp sibling + fsync + rename), so upgrades from this release forward cannot produce a torn tenant-repo-sync-revisions.json. Should a manifest from an older release (or exotic filesystem damage) fail the strict read anyway, deleting the file is the safe last resort: it costs one full re-ingestion pass, because every head then classifies uninitialized and replays from a null base under the normal bounded admission — never hand-edit it instead.

If the deployment snapshot is fresh but the tool returns status: degraded with reason: snapshot-section-missing or snapshot-producer-metadata-missing, fix the bridge producer before debugging repo credentials or embeddings. Those reasons mean the public KB/MC server can read the snapshot file, but the orchestrator bridge that wrote it is older than the current diagnostic contract or omitted a required top-level section such as tenantRepoSync. Recreate the orchestrator with the current image/config, then re-run the public tool and only continue pull-mode ingestion debugging once schemaDiagnostics.status is available.

See also