Cloud-deployment observability MCP tool (logs, health, recovery-runs)

Parent: Epic #13860 (diagnostics / observe half). Operator-directed (2026-06-23): "a new epic sub for a new mcp tool is needed. cloud only (7 docker containers) versus local mode is key… the goal is not to create a black box. you must be capable to PROOF that it works once deployed."

Release classification: boardless — agent-OS deployment-observability work. Boardless is the default for current agent-OS topics: the v13 release board (Project 12) is closed; board #13 "Agent Harness" is harness-scoped (not agent-OS); v13.1 milestone attachment is optional. Consistent with the boardless immune-system epics #13860 / #13874 / #13889.

Context

The deployment immune system (diagnostics #13860 / ADR-0025 + recovery #13874 / ADR-0026) can detect, diagnose, heal, and escalate — but nothing lets an agent or operator SEE what it did, remotely, in a deployed cloud stack. The recovery-run ledger (#13881) writes local JSONL to the orchestrator container's disk (ai/services/memory-core/helpers/recoveryRunStateStore.mjs → appendRecoveryRunState = fs.appendFile per <recoveryRunId>.jsonl); no MCP tool surfaces it, container logs, diagnosis events, or sampled container metrics.

Verified 2026-06-23: the only deployment-observability MCP operations are healthcheck / get_sqlite_holder_diagnostics / get_memory_core_tool_metrics (all MC-internal) + neural-link get_console_logs (browser). So the immune-system subs declare proof steps ("feed a diagnosis on a live orchestrator and verify the ledger") that are unexecutable from anywhere but a shell on the box. That gap is what let a B0-actuator PR (#13912) pass review as "post-merge smoke" — there is no remote smoke.

Live evidence (operator): a cloud model container's ollama runner wedged at ~400% CPU for ~93h (the canonical detect target) while reporting healthy; the memory container unhealthy; orchestrator up 3 days healing nothing — and the only way it was seen was the operator reading glances by hand. That is the black box this ticket closes. Goal: provable-in-deployment.

The Problem

A deployed Agent OS is a multi-container stack. In cloud, the Memory Core MCP server runs inside the memory container; an agent connects only to it. The orchestrator, model, knowledge, and chroma are separate containers — their logs, health, resource metrics, and the orchestrator's recovery-run ledger are unreachable across the container boundary. The immune system can act, but its actions are invisible → unprovable → a black box.

Cloud vs local is the load-bearing axis (operator):

• Cloud (primary — the black box): ~7 containers (Neo: model, orchestrator, knowledge, chroma, memory; + infra runners). Cross-container observability requires either the docker socket or data the orchestrator publishes to a shared SSOT. This is where the tool earns its keep.
• Local (secondary — NOT a black box): single host, direct filesystem + process access; docker logs / glances / the JSONL ledger are already reachable. Local support is best-effort; the design must NOT optimize for it at the cloud case's expense.

The Architectural Reality

• Recovery-run ledger: ai/services/memory-core/helpers/recoveryRunStateStore.mjs — local JSONL on the orchestrator container's disk; appendRecoveryRunState / readRecentRecoveryRunStates are filesystem-local.
• MCP tool surface: ai/mcp/server/memory-core/openapi.yaml — no log / recovery / diagnosis / metrics retrieval operation exists.
• Docker socket: per ADR-0026 §2.3 the runtime socket is the orchestrator's privilege (B1 actuator, #13884); ai/deploy/docker-compose*.yml mounts no socket today.
• Graph SSOT: query_hybrid_graph / get_node / search_nodes already serve graph nodes remotely from memory-core — the natural home for structured immune-system events.

The Fix

A new read-only memory-core MCP tool — proposed inspect_deployment — returning, for the cloud stack:

• per allowlisted Neo container: health state, sampled CPU/mem, recent log tail (bounded);
• the recovery-run ledger + diagnosis events (so the loop is provable remotely).

Two data paths (the cloud design crux):

1. Structured immune-system state (recovery-runs, diagnoses, sampled health) → orchestrator daemons publish to the Memory Core graph SSOT via the existing A2A/WAL/graph ingestion (the #13889 path); inspect_deployment reads the graph. No new privilege.
2. Raw container logs + live docker stats → require the docker socket, which lives on the orchestrator per ADR-0026. Recommended shape: the orchestrator exposes a read-only, allowlisted observability endpoint (docker logs / docker stats for the named Neo services); inspect_deployment proxies to it. Socket stays on one container; agent-facing surface stays on memory-core.

The one open design fork:

→ (a) recommended; (c) is the no-privilege subset that can ship first if the socket grant needs more cross-family review.

Contract Ledger Matrix

Decision Record impact

depends-on ADR-0026 (the docker-socket privilege envelope — a read-only observability use; allowlist mirrors B1's) and ADR-0025 (detect/observe). Likely amends ADR-0026 to record the read-only observability use + the orchestrator-proxy shape (carry as an OQ for the cross-family re-poll). No ADR superseded. Structural placement follows existing patterns (memory-core tool handler + orchestrator service) — sibling-lift, no novel directory.

Acceptance Criteria

• inspect_deployment on memory-core returns, for the cloud multi-container stack, per-allowlisted-container health + sampled CPU/mem + a bounded recent log tail.
• Returns the recovery-run ledger + diagnosis events, read from the graph SSOT (not local JSONL), so the immune loop is provable remotely.
• Cross-container log/stat retrieval is read-only + allowlisted to the named Neo services, riding ADR-0026's socket envelope (no new independent privilege; socket stays on the orchestrator, proxied per option (a)).
• Cloud is the primary, tested reality; local is explicitly best-effort (direct access already exists).
• Proof AC (L3): an agent calls inspect_deployment against a live cloud deployment and can SEE a wedged/unhealthy container's state + any recovery-runs — i.e. the deployment is no longer a black box. (Reproduces today's model-wedged / memory-unhealthy incident through the tool.)
• Read-only; no synchronous model hit; bounded payload (log tail capped); tool description ≤ §5.3 budget with no internal cross-refs.

Out of Scope

• The actuator itself (B1, #13884) — this is read-only observability.
• The diagnostics detector/producer (the unfiled #13860 detect/diagnose subs) — this tool READS what the producer emits; the re-sequence that files those is tracked separately.
• The graph-routing of immune-system events, if it grows beyond a small seam — split to a sibling sub.
• Rich local-mode tooling (low priority — local is not a black box).

Avoided Traps

• Don't mount a second docker socket on memory-core — keep it on the orchestrator per ADR-0026; proxy (option a).
• Don't make this a write/actuator surface — read-only.
• Don't ship unbounded log dumps — cap the tail, budget the payload.
• Don't build local-first — the black box is cloud; local already has direct access.
• Don't let "post-merge smoke" stand in for proof — the proof AC must run through this tool against a live cloud deployment.

Related

Parent: #13860. Related: #13874 (recovery — recovery-run readout), #13884 (B1 docker-socket — shares the privilege envelope), #13881 (recovery-run ledger — the data source), #13889 (graph/WAL ingestion — the publish path), #13899 / #13901 (Neural Link bridge stdio-log routing — adjacent log-handling, distinct scope), ADR-0025, ADR-0026.

Origin Session ID: acbc412a-8060-422c-b0dc-05f929da50e8

Handoff Retrieval Hints: cloud deployment observability MCP tool inspect_deployment black box recovery-run readout; immune system proof remote logs docker socket orchestrator proxy graph SSOT