Context
#14030 covers the backup reliability gap exposed by the Memory Core corruption incident: the canonical backup failed loudly, but no alert/escalation reached operators for days. The AC1 design has now converged across source reads and A2A with Vega: Vega owns the producer-core / scheduling-overdue lane from #14055 / PR #14056, while this ticket owns the two existing-service wiring pieces in the ProcessSupervisor and RecoveryActuator surfaces.
Live latest-open sweep: checked the latest 30 open GitHub issues at 2026-06-26T04:42:32Z plus targeted search backup failure diagnosis escalation ProcessSupervisor RecoveryActuator; no equivalent hook+sink ticket found. Closest open tickets are #14055 (producer-core helpers), #14053 (backup restorability AC2), and #14048 (empty-parity backup integrity).
A2A in-flight sweep: checked the latest 30 A2A messages at 2026-06-26T04:42:32Z. Vega explicitly confirmed this split in MESSAGE:855f3aad-7f07-4ac3-a081-ec5865806441; no competing claim for the ProcessSupervisor/RecoveryActuator hook+sink slice was present.
The Problem
backup.mjs already exits non-zero on failure, and the orchestrator already observes that child-process failure. The gap is downstream of observation: the failure is written into health/task status but not escalated through the recovery diagnosis substrate. That turns a fail-loud backup into a silent health record, recreating the multi-day detection lag this epic is supposed to remove.
The Architectural Reality
ai/daemons/orchestrator/services/ProcessSupervisorService.mjs:177 defines recordTaskOutcome(taskName, status, details = null) and currently forwards only to HealthService.recordTaskOutcome().- The child-process failure paths converge on
recordTaskOutcome(..., 'failed', ...): spawn failure at ProcessSupervisorService.mjs:466, runtime error at :506, success-hook failure at :540, and non-zero exit at :545. The backup task reaches that path because ai/daemons/orchestrator/taskDefinitions.mjs:260 defines backup as a spawned backup.mjs process task.
ai/services/memory-core/helpers/recoveryRunStateStore.mjs:42 admits targetIdentity.kind = 'supervised-task', and createRecoveryDiagnosisEvent() returns typed recovery-diagnosis events at :146.ai/daemons/orchestrator/services/RecoveryActuatorService.mjs:385 deliberately allows page only for deploy-target; supervised-task actions are restart or warm-provider. Therefore apply('backup', 'page') is structurally wrong, and this ticket must add a narrow diagnosis escalation sink rather than routing backup through the deploy-target action model.RecoveryActuatorService.finishAction() already appends recovery-run state and records ledger/task outcome around RecoveryActuatorService.mjs:642, so the new sink should reuse that ledger shape for an escalated diagnosis without calling executeTargetAction().
The Fix
- Add a narrow
RecoveryActuatorService.escalateDiagnosis(diagnosisEvent, options = {}) path that validates a recovery-diagnosis event with details.actionClass === 'escalate', records an escalated recovery-run ledger entry, and dispatches/logs the operator page signal without executing restartSupervisedTask(), pageDeployTarget(), or any deploy-target action.
- Wire
ProcessSupervisorService.recordTaskOutcome() for failed backup task outcomes to build/receive the supervised-task diagnosis and pass it to the actuator sink. If PR #14056 has merged, consume its producer-core helper; if not, keep this PR from duplicating that helper and use only the existing recovery event contract required for the hook+sink slice.
- Keep the overdue scheduler half out of this ticket. Vega owns that lane after PR
#14056 merges.
- Add focused unit coverage proving that failed backup outcomes escalate exactly once through the diagnosis sink, unsupported/non-escalate diagnoses are rejected or no-op safely, and no privileged recovery action is executed.
Contract Ledger Matrix
| Target Surface |
Source of Authority |
Proposed Behavior |
Fallback |
Docs |
Evidence |
RecoveryActuatorService.escalateDiagnosis(diagnosisEvent, options = {}) |
ADR-0026 recovery actuator boundary; existing RecoveryActuatorService.finishAction() recovery-run ledger |
Accept only escalation-class diagnosis events, append an escalated recovery-run state, and dispatch/log the operator escalation signal without executing target actions |
Invalid diagnosis returns a typed rejected/no-op outcome and records no privileged action |
JSDoc on the new public method plus test names documenting the no-action guarantee |
Unit test asserts ledger/status and spies prove executeTargetAction, restartSupervisedTask, and pageDeployTarget are not called |
ProcessSupervisorService.recordTaskOutcome('backup', 'failed', details) hook |
ProcessSupervisorService.recordTaskOutcome() is the central task outcome sink; all backup process failures converge there |
Failed backup outcomes create/consume a supervised-task recovery-diagnosis with details.actionClass = 'escalate' and call the actuator sink once |
HealthService recording remains best-effort even if escalation throws; the supervisor loop does not crash |
Method JSDoc updated to mention health + escalation side effects |
Unit test covers non-zero backup exit/failure details and HealthService still called |
recovery-diagnosis event for supervised-task:backup |
createRecoveryDiagnosisEvent() and RECOVERY_TARGET_IDENTITY_KINDS in recoveryRunStateStore.mjs |
Preserve the diagnosis target as supervised-task:backup with task-failure evidence and no deploy-target coercion |
If the producer helper is unavailable, do not invent a new target kind; use the existing event factory only |
Existing recovery-run state docs plus local JSDoc |
Unit test asserts event target identity and action class |
Decision Record impact
Aligned with ADR-0025 and ADR-0026. This ticket uses the existing recovery diagnosis / recovery actuator split and does not amend either ADR.
Acceptance Criteria
Out of Scope
- Scheduling overdue detection from
scheduling/backup.mjs; Vega owns that after PR #14056 merges.
- Backup restorability checks from
#14053 and empty-parity handling from #14048.
- Backup retention SLA / detect cadence work from
#14026.
- Editing
backup.mjs; the failure signal already exists at the spawned process boundary.
- Any automatic backup restart, retry, or data-recovery actuator.
Avoided Traps
- Do not use
emitConsumerFriction; that path is embedding/model-friction-specific and requires context-limit metadata that backup failures do not have.
- Do not make
backup a deploy target just to page it.
- Do not stack on PR
#14056 in a non-dev branch; if the producer helper is needed and not merged, leave the hook integration bounded rather than duplicating its full producer-core logic.
Related
Refs #14030
Related: #14055, #14056, #14053, #14048, #13999
Origin Session ID: 35f83031-f1a6-41a7-9c3b-089b87307db9
Handoff Retrieval Hints: query_raw_memories("#14030 AC1 backup failure ProcessSupervisorService RecoveryActuatorService escalateDiagnosis"); A2A anchors: MESSAGE:e7260839-adac-45ed-9b69-4a220c6d1e4a, MESSAGE:855f3aad-7f07-4ac3-a081-ec5865806441; source anchors: ProcessSupervisorService.recordTaskOutcome, RecoveryActuatorService.finishAction, createRecoveryDiagnosisEvent.
Context
#14030covers the backup reliability gap exposed by the Memory Core corruption incident: the canonical backup failed loudly, but no alert/escalation reached operators for days. The AC1 design has now converged across source reads and A2A with Vega: Vega owns the producer-core / scheduling-overdue lane from#14055/ PR#14056, while this ticket owns the two existing-service wiring pieces in the ProcessSupervisor and RecoveryActuator surfaces.Live latest-open sweep: checked the latest 30 open GitHub issues at 2026-06-26T04:42:32Z plus targeted search
backup failure diagnosis escalation ProcessSupervisor RecoveryActuator; no equivalent hook+sink ticket found. Closest open tickets are#14055(producer-core helpers),#14053(backup restorability AC2), and#14048(empty-parity backup integrity).A2A in-flight sweep: checked the latest 30 A2A messages at 2026-06-26T04:42:32Z. Vega explicitly confirmed this split in
MESSAGE:855f3aad-7f07-4ac3-a081-ec5865806441; no competing claim for the ProcessSupervisor/RecoveryActuator hook+sink slice was present.The Problem
backup.mjsalready exits non-zero on failure, and the orchestrator already observes that child-process failure. The gap is downstream of observation: the failure is written into health/task status but not escalated through the recovery diagnosis substrate. That turns a fail-loud backup into a silent health record, recreating the multi-day detection lag this epic is supposed to remove.The Architectural Reality
ai/daemons/orchestrator/services/ProcessSupervisorService.mjs:177definesrecordTaskOutcome(taskName, status, details = null)and currently forwards only toHealthService.recordTaskOutcome().recordTaskOutcome(..., 'failed', ...): spawn failure atProcessSupervisorService.mjs:466, runtime error at:506, success-hook failure at:540, and non-zero exit at:545. The backup task reaches that path becauseai/daemons/orchestrator/taskDefinitions.mjs:260definesbackupas a spawnedbackup.mjsprocess task.ai/services/memory-core/helpers/recoveryRunStateStore.mjs:42admitstargetIdentity.kind = 'supervised-task', andcreateRecoveryDiagnosisEvent()returns typedrecovery-diagnosisevents at:146.ai/daemons/orchestrator/services/RecoveryActuatorService.mjs:385deliberately allowspageonly fordeploy-target;supervised-taskactions arerestartorwarm-provider. Thereforeapply('backup', 'page')is structurally wrong, and this ticket must add a narrow diagnosis escalation sink rather than routing backup through the deploy-target action model.RecoveryActuatorService.finishAction()already appends recovery-run state and records ledger/task outcome aroundRecoveryActuatorService.mjs:642, so the new sink should reuse that ledger shape for anescalateddiagnosis without callingexecuteTargetAction().The Fix
RecoveryActuatorService.escalateDiagnosis(diagnosisEvent, options = {})path that validates arecovery-diagnosisevent withdetails.actionClass === 'escalate', records anescalatedrecovery-run ledger entry, and dispatches/logs the operator page signal without executingrestartSupervisedTask(),pageDeployTarget(), or any deploy-target action.ProcessSupervisorService.recordTaskOutcome()for failed backup task outcomes to build/receive the supervised-task diagnosis and pass it to the actuator sink. If PR#14056has merged, consume its producer-core helper; if not, keep this PR from duplicating that helper and use only the existing recovery event contract required for the hook+sink slice.#14056merges.Contract Ledger Matrix
RecoveryActuatorService.escalateDiagnosis(diagnosisEvent, options = {})RecoveryActuatorService.finishAction()recovery-run ledgerescalatedrecovery-run state, and dispatch/log the operator escalation signal without executing target actionsexecuteTargetAction,restartSupervisedTask, andpageDeployTargetare not calledProcessSupervisorService.recordTaskOutcome('backup', 'failed', details)hookProcessSupervisorService.recordTaskOutcome()is the central task outcome sink; all backup process failures converge thererecovery-diagnosiswithdetails.actionClass = 'escalate'and call the actuator sink oncerecovery-diagnosisevent forsupervised-task:backupcreateRecoveryDiagnosisEvent()andRECOVERY_TARGET_IDENTITY_KINDSinrecoveryRunStateStore.mjssupervised-task:backupwith task-failure evidence and no deploy-target coercionDecision Record impact
Aligned with ADR-0025 and ADR-0026. This ticket uses the existing recovery diagnosis / recovery actuator split and does not amend either ADR.
Acceptance Criteria
backupprocess-task outcome recorded throughProcessSupervisorService.recordTaskOutcome()triggers a supervised-task recovery diagnosis escalation.RecoveryActuatorService.escalateDiagnosis()records anescalatedrecovery-run ledger entry while preservingtargetIdentity: {kind: 'supervised-task', id: 'backup'}.apply('backup', 'page'), and does not execute deploy-target paging/action machinery.Out of Scope
scheduling/backup.mjs; Vega owns that after PR#14056merges.#14053and empty-parity handling from#14048.#14026.backup.mjs; the failure signal already exists at the spawned process boundary.Avoided Traps
emitConsumerFriction; that path is embedding/model-friction-specific and requires context-limit metadata that backup failures do not have.backupa deploy target just to page it.#14056in a non-dev branch; if the producer helper is needed and not merged, leave the hook integration bounded rather than duplicating its full producer-core logic.Related
Refs #14030
Related: #14055, #14056, #14053, #14048, #13999
Origin Session ID: 35f83031-f1a6-41a7-9c3b-089b87307db9
Handoff Retrieval Hints:
query_raw_memories("#14030 AC1 backup failure ProcessSupervisorService RecoveryActuatorService escalateDiagnosis"); A2A anchors:MESSAGE:e7260839-adac-45ed-9b69-4a220c6d1e4a,MESSAGE:855f3aad-7f07-4ac3-a081-ec5865806441; source anchors:ProcessSupervisorService.recordTaskOutcome,RecoveryActuatorService.finishAction,createRecoveryDiagnosisEvent.