Plane PAT validation survives a GitHub authenticated-API outage

Context

Measured live, 2026-08-17 (#17271 L1 session): a plane redeploy landed at 14:03:45Z — one minute into GitHub's "Partial System Outage" (SAML/OIDC authentication explicitly impacted). The freshly-started mc/kb/fleet-server containers sat unhealthy for ~2 hours: cold PAT-validation caches forced every admission through GitHub's /user, which answered 503 the entire window. The previous containers had ridden the same outage invisibly on warm in-process caches. Observation vs inference separated: the 503s, the healthcheck failures, and the /rate_limit 200s are observed (receipts below); the cache-warmth explanation for the old containers' survival is inferred from the mechanism, not directly observed.

Key measurement (16:13Z, same token, seconds apart):

• GET https://api.github.com/user → 503 (the validation endpoint)
• GET https://api.github.com/rate_limit (authenticated) → 200 (token validity provable)

Fleet blast radius: all agent MCP sessions dead (mailbox, memory saves, A2A), cockpit plane reads degraded, docker compose --wait deploys blocked on unhealthy targets. One upstream endpoint held the whole local plane hostage.

The Problem

ai/mcp/server/shared/services/AuthService.mjs already has the right cache primitives — successful validations cached by SHA-256 token hash for auth.patCacheTtlSeconds, failures never cached (:742-750 documents the pattern; :183 wires the same naked-401 bearer path against GitHub /user). Two gaps turn a provider incident into a plane outage:

1. The cache is in-process only. A container restart during a provider outage = total auth death. The incident's exact shape: redeploy → cold cache → hostage.
2. No endpoint fallback for the outage class. /user resolves the SUBJECT (login); when it is down, token VALIDITY is still provable via /rate_limit (200/401 distinguish valid/invalid) — the plane never asks.

The Architectural Reality

• Owning substrate (structure-map cited): ai/mcp/server/shared/services/ — AuthService.mjs is the admission authority for the plane's MCP ingress.
• Plane-boundary clarification (coordinate with @neo-opus-ada): this is explicitly NOT in tension with #17283/#17285 ("GitHub reads are host-edge by design", GitHub logic stripped from MC). Provider-PAT admission already lives plane-side by design — this ticket hardens the existing verify path; it adds zero GitHub READ logic to the plane.
• Healthchecks (ai/scripts/diagnostics/mcpHealthcheck.mjs) authenticate through the same path, so their verdict inherits the same single-endpoint coupling.

The Fix (fail-closed preserved)

1. Disk-backed validation cache under the plane data root: token-hash → {subject, verifiedAt}, TTL-bounded, written only on a SUCCESSFUL /user validation. Restart-safe. No token material on disk (hash only).
2. Stale-while-revalidate under provider outage: if /user is 5xx/unreachable AND /rate_limit with the same token answers 200 AND a disk-cached subject exists within its declared stale window → admit with the cached subject, mark the session stale-validated (observable, honest). A 401 from either endpoint stays a hard refusal — invalid token ≠ provider outage.
3. Healthcheck degrades instead of dying: when the ONLY fault is provider-validation staleness, report degraded (the plane serves; truth labels carry the degradation) — --expected-status healthy,degraded already tolerates it deploy-side.

Decision Record impact

aligned-with ADR 0014 (plane admission) — no authority change; adds outage-resilience to the existing verify path. If ADR 0014 pins single-endpoint validation semantics, upgrade this line to amends before implementation.

Contract Ledger Matrix

Acceptance Criteria

• Container restart during a simulated /user 503 (valid token, warm disk cache) → plane admits, session marked stale-validated, healthcheck degraded not unhealthy (red→green witness). DELIVERED in PR #17772: cold-process arm + consumed healthcheck-staleness composition.
• Invalid/revoked token refused in both modes (401 path unchanged, fail-closed). DELIVERED: /user 401 evicts both tiers; /rate_limit probe is 401-only-evicting (shared-egress 403 stays unresolved); allowlist policy reapplies on disk admission.
• Cache file: hash-only, plane data root, documented TTL + stale window, no token material. DELIVERED: hash-keyed atomic store; subject normalized to id/login/name (+scopes+timestamp) with a sentinel test; TTL/grace reuse existing leaves; location explicit via NEO_AUTH_PAT_DISK_CACHE_PATH, bound per-service in the canonical profiles.
• stale-validated visible on the operator surface (who_is_online / cockpit truth labels), never silent. PARTIALLY DELIVERED in PR #17772: AuthInfo validationState at the transport boundary + loud per-admission logs + memory-core healthcheck composes a consumed degraded signal naming mode/identity/since. The who_is_online-row / cockpit-label RENDERING sliver is [L2-deferred — owned by #17781].
• ADR-0019 discipline: new config leaf reactive-provider-SSOT-conformant (no env reads, no defensive ?., no hidden defaults). DELIVERED: declarative empty-default leaf; SSOT lint green incl. parity snapshot.

Out of Scope

GitHub READ surfaces (census, tickets — host-edge per #17283/#17285) · GitLab-path parity (same pattern applies; separate ticket if wanted) · offline-first auth redesign.

Related

#17271 (measured incident, receipts) · #17283 / #17285 (plane/host-edge separation — boundary named above, coordinate with @neo-opus-ada) · #16087 (pinned-deploy discipline that surfaced the cold-start class) · ADR 0014 · ADR-0019 (config leaf discipline)

Live latest-open sweep: latest 20 open checked 2026-08-17T16:19Z, no equivalent; A2A claim window (last 60 min) clean; KB semantic sweep clean (top hit #16677 MCP-wedge liveness — different fault class).

Origin Session ID: 7ee47ccf-d1c7-469d-a75e-15cebf3b5ea5

Retrieval Hint: query_raw_memories("PAT validation 503 rate_limit stale-while-revalidate plane hostage GitHub outage")