Cloud-deploy troubleshooting: gitlab-pat healthcheck-token 401 entry

Context

A live cloud deployment (2026-06-16) enabled NEO_AUTH_MODE=gitlab-pat and hit a documented-but-still-surprising failure: right after the bearer switch went live, docker compose up left the kb/mc containers unhealthy and dependents reported dependency kb-server failed to start. Root cause (V-B-A'd against the live endpoint): the bearer switch is working — a tokenless initialize correctly returns 401 WWW-Authenticate: Bearer — but the in-container healthcheck (mcpHealthcheck.mjs) now hits that same gate and 401s because NEO_MCP_HEALTHCHECK_TOKEN is empty, so the containers never reach service_healthy and the orchestrator depends_on / up dependency-wait abort.

#12990 already charted this seam — but only as a proactive comment on the ai/deploy/docker-compose.yml healthcheck blocks (plus .env.example marking the token REQUIRED). That comment was present in the deployment that hit this, and the operator still shipped an empty token and lost a debugging round-trip. The proactive comment is necessary-but-insufficient: there is no symptom-indexed entry an operator finds by searching the error they actually see.

The Problem

learn/agentos/cloud-deployment/Troubleshooting.md covers host-allowlist 401s, the auth-identity 401, the exec-healthcheck command, and the "healthy but empty KB" case — but not the "auth enabled → self-probe 401 → unhealthy → dependency failed to start" failure. An operator who enables gitlab-pat and is staring at the unhealthy-stack symptom has nothing to search for. The #12990 compose comment only helps the operator who reads the compose proactively before deploying — not the one already looking at the failure.

The Architectural Reality

• ai/mcp/server/shared/services/AuthService.mjs — setupGitlabPat installs requireBearerAuth app-wide with no health exemption (the supported posture; an exemption would weaken the gate, explicitly rejected in #12990).
• ai/scripts/diagnostics/mcpHealthcheck.mjs — the in-container probe; sends Authorization: Bearer ${NEO_MCP_HEALTHCHECK_TOKEN} only when that env is set.
• ai/deploy/docker-compose.yml — kb/mc healthcheck blocks (carry the #12990 seam comment); orchestrator depends_on: {kb-server: service_healthy, mc-server: service_healthy}, so an unhealthy probe cascades to dependency ... failed to start.
• learn/agentos/cloud-deployment/Troubleshooting.md — the symptom-indexed operator doc that lacks this entry.

The Fix

Add one troubleshooting entry to learn/agentos/cloud-deployment/Troubleshooting.md (and a one-line cross-ref from Day0Tutorial.md at the gitlab-pat enable step), keyed on the strings the operator actually sees:

• Symptom: after enabling NEO_AUTH_MODE=gitlab-pat, docker compose up leaves kb/mc unhealthy; dependents report dependency <svc> failed to start.
• Confirm it is this (not a boot crash): a tokenless initialize through the ingress returns a clean 401 WWW-Authenticate: Bearer (server is up and enforcing auth). A crash would be 502 / connection-refused.
• Cause: the in-container mcpHealthcheck.mjs self-probe now needs a bearer; NEO_MCP_HEALTHCHECK_TOKEN is unset/empty.
• Fix: set NEO_MCP_HEALTHCHECK_TOKEN=<read_user PAT> in .env, then docker compose -p <project> up -d --force-recreate (env-only; no rebuild).

Register any new nav node in learn/tree.json per the public-nav rule.

Contract Ledger

Decision Record impact

none — doc-only prose, aligned-with the cloud-deployment topology (ADR 0014); no gates, no behavior change, no loaded-byte growth (read-on-demand learn/ doc).

Acceptance Criteria

• Troubleshooting.md has a symptom-indexed entry keyed on dependency ... failed to start / unhealthy-after-enabling-gitlab-pat, with the 401-vs-502 confirm step, the cause, and the env-only fix.
• Day0Tutorial.md cross-references it at the gitlab-pat enable step.
• If a new nav node is added, learn/tree.json is updated (public-nav lint).
• Generic neo-deployment framing only (no client/deployment-specific names).

Out of Scope

• Any AuthService health-exemption / auth-free liveness endpoint (weakens the gate; rejected in #12990; the token route is the supported shape).
• Changing the #12990 compose comment or .env.example (already correct; this is the complementary symptom-doc surface).
• New mechanical gates / lint.

Related

• Builds on #12990 (charted the seam in the compose comment + .env.example; this adds the symptom-indexed troubleshooting surface it deliberately left).
• Auth contract: #12378 / #12377. Aligned-with ADR 0014 (Cloud Deployment Topology).

Release classification

post-release (documentation hardening — non-blocking; v13 shipped). Boardless.

Origin Session ID: 0b27b21a-2146-4976-945f-f1682c6a1c9c

Retrieval Hint: query_raw_memories "gitlab-pat healthcheck token 401 containers unhealthy dependency failed to start"; canonical doc learn/agentos/cloud-deployment/Troubleshooting.md.

Live latest-open sweep: checked latest 30 open issues at ~2026-06-16 09:4x UTC; no equivalent open ticket (closest #13403/#13391 unrelated; #12990 closed + complementary). A2A claim sweep: mailbox empty. KB ticket-semantic sweep: confirms the NEO_MCP_HEALTHCHECK_TOKEN troubleshooting case is not documented.