Problem scope
A deployed Agent OS runs as a multi-container stack (Memory Core, Knowledge Base, the local model server, vector store, orchestrator). The OS has no deployment-wide health → heal → escalate loop:
- The orchestrator's existing watchdogs (
remConsolidationLivenessWatchdog, embedDrainLivenessWatchdog, the lease watchdogs) are per-task liveness — they watch the orchestrator's own work, not sibling-container health.
ProcessSupervisorService supervises in-process spawned children (PID-level), not sibling containers.- The container runtime's
restart: unless-stopped only fires on exit — an unhealthy-but-running container (a degraded service whose probe trips while it is still functionally serving) is never auto-restarted.
So a deployment can degrade — resource contention, a probe-failing-but-functional container, or config-drift where an intended env override silently isn't applied — and nothing detects it, heals it, or even alerts. It stays degraded until a human notices and intervenes by hand. For a self-evolving Agent OS, that is a missing immune function.
Intended solution
A new orchestrator child service — ContainerHealthService, sibling to ProcessSupervisorService under ai/daemons/orchestrator/services/ — extending supervision from in-process children to sibling containers, running a bounded health loop:
Detect — periodically collect real per-container signals (memory %, CPU load, container-health state, and env-var/config correctness) across the deployment's services. Decide on resource/contention reality, not a single health probe: a service's probe can legitimately false-fail (e.g. a model-dependent canary while the model is saturated) while the service is functionally serving — so a probe trip alone must not trigger a restart of a working container.
Act (tiered, bounded) — restart (transient fault) → throttle / shed load (contention) → escalate-with-diagnosis (config-drift needing a redeploy: page, don't loop). Rate-limited, N-attempt-capped, lifecycle-only actions (config/restart, never arbitrary code — the two-worlds safety boundary), reversible.
The actuator (greenfield — no container-control precedent in ai/). For the orchestrator (itself a container) to restart a sibling, it needs a container-runtime handle. That privilege is powerful, so safety must live in a constrained action set (restart + a few known config tweaks, nothing else), not unrestricted runtime access. This is the design-first decision.
Placement — inside the orchestrator container (accepted single-point-of-failure: if the orchestrator dies, no heal — a pragmatic floor, not a flaw).
Decomposition (design-first; subs linked natively, added incrementally)
The actuator + heal-safety bounds + the false-positive-safe detect model are a design/ADR sub that gates the implementation subs (cross-family review mandatory). Implementation then decomposes into detect (the container-health model), act (the bounded actuator), and deploy (actuator wiring) — filed and linked as the design lands, so this body never hardcodes a stale sub-list.
Avoided traps
- Restart-on-probe-fail — restarting a functionally-working container because a health probe tripped = self-inflicted outage. Decide on resource reality + multiple signals, not a single probe.
- Over-privileged actuator — a runtime handle grants total container control; safety is the constrained action set, not the access.
- Resilience mistaken for a config/model problem — a deployment can be correctly configured and still degrade under contention; this is a heal gap, not a config/model swap.
- Healer thrash — an unbounded heal loop can make an incident worse than the fault; rate-limit + N-cap + escalate-on-repeat.
Decision Record impact
Required — new subsystem + a privileged actuator; the ADR is produced by the design/ADR sub.
Precedents (codebase)
ProcessSupervisorService (in-process supervision — the structural analog), remConsolidationLivenessWatchdog / embedDrainLivenessWatchdog (the liveness-watchdog pattern — task-liveness, generalized here to container-health), HealthService (the health-probe / detect-signal source).
Related
#13839 (alarm dispatcher for a watchdog — the escalation pattern), #13435 (in-container healthcheck decoupling — a detect-signal surface), #13624 / #13755 (orchestrator watchdog epics — per-task; this is container-health).
Origin Session ID
2439c28b-245e-425a-85a7-ca7ee4aa0336
Problem scope
A deployed Agent OS runs as a multi-container stack (Memory Core, Knowledge Base, the local model server, vector store, orchestrator). The OS has no deployment-wide health → heal → escalate loop:
remConsolidationLivenessWatchdog,embedDrainLivenessWatchdog, the lease watchdogs) are per-task liveness — they watch the orchestrator's own work, not sibling-container health.ProcessSupervisorServicesupervises in-process spawned children (PID-level), not sibling containers.restart: unless-stoppedonly fires on exit — an unhealthy-but-running container (a degraded service whose probe trips while it is still functionally serving) is never auto-restarted.So a deployment can degrade — resource contention, a probe-failing-but-functional container, or config-drift where an intended env override silently isn't applied — and nothing detects it, heals it, or even alerts. It stays degraded until a human notices and intervenes by hand. For a self-evolving Agent OS, that is a missing immune function.
Intended solution
A new orchestrator child service —
ContainerHealthService, sibling toProcessSupervisorServiceunderai/daemons/orchestrator/services/— extending supervision from in-process children to sibling containers, running a bounded health loop:Detect — periodically collect real per-container signals (memory %, CPU load, container-health state, and env-var/config correctness) across the deployment's services. Decide on resource/contention reality, not a single health probe: a service's probe can legitimately false-fail (e.g. a model-dependent canary while the model is saturated) while the service is functionally serving — so a probe trip alone must not trigger a restart of a working container.
Act (tiered, bounded) — restart (transient fault) → throttle / shed load (contention) → escalate-with-diagnosis (config-drift needing a redeploy: page, don't loop). Rate-limited, N-attempt-capped, lifecycle-only actions (config/restart, never arbitrary code — the two-worlds safety boundary), reversible.
The actuator (greenfield — no container-control precedent in
ai/). For the orchestrator (itself a container) to restart a sibling, it needs a container-runtime handle. That privilege is powerful, so safety must live in a constrained action set (restart + a few known config tweaks, nothing else), not unrestricted runtime access. This is the design-first decision.Placement — inside the orchestrator container (accepted single-point-of-failure: if the orchestrator dies, no heal — a pragmatic floor, not a flaw).
Decomposition (design-first; subs linked natively, added incrementally)
The actuator + heal-safety bounds + the false-positive-safe detect model are a design/ADR sub that gates the implementation subs (cross-family review mandatory). Implementation then decomposes into detect (the container-health model), act (the bounded actuator), and deploy (actuator wiring) — filed and linked as the design lands, so this body never hardcodes a stale sub-list.
Avoided traps
Decision Record impact
Required — new subsystem + a privileged actuator; the ADR is produced by the design/ADR sub.
Precedents (codebase)
ProcessSupervisorService(in-process supervision — the structural analog),remConsolidationLivenessWatchdog/embedDrainLivenessWatchdog(the liveness-watchdog pattern — task-liveness, generalized here to container-health),HealthService(the health-probe / detect-signal source).Related
#13839 (alarm dispatcher for a watchdog — the escalation pattern), #13435 (in-container healthcheck decoupling — a detect-signal surface), #13624 / #13755 (orchestrator watchdog epics — per-task; this is container-health).
Origin Session ID
2439c28b-245e-425a-85a7-ca7ee4aa0336