Propagate miniSummary backfill progress outcomes

Context

During the live Agent OS stability proof for #13755, read-only diagnostics showed that memory-summary-backfill can report a successful child process while preserving deferred residual work. The current health/task-state layer then records a fresh success timestamp, which makes the orchestrator look healthier than the memory pipeline actually is.

Release classification: ON the board — this blocks the current Agent OS stability proof because the heavy-maintenance scheduler needs net-progress evidence, not just exit-code evidence.

Evidence already posted on the parent lane:

• https://github.com/neomjs/neo/issues/13755#issuecomment-4762307496 — KB embedding recovered, but REM/WAL drain was not proof-complete.
• https://github.com/neomjs/neo/issues/13755#issuecomment-4762312885 — miniSummary backfill runs can complete with deferred residuals.
• https://github.com/neomjs/neo/issues/13755#issuecomment-4762317683 — source mechanism: child outcome JSON exists but the supervisor discards stdout.
• https://github.com/neomjs/neo/issues/13755#issuecomment-4762322259 — implementation seam and focused test anchor.

The Problem

MemoryService.backfillMiniSummaries() already returns a structured outcome with processed, updated, deferred, missingContent, and runBudgetHit. The lifecycle child prints that JSON on stdout and exits 0, including fail-soft deferred-row cases. ProcessSupervisorService.runTask() currently spawns children with stdout ignored and only stderr captured, so the orchestrator cannot distinguish these exit-0 cases:

• real progress: some rows updated or missing-content rows intentionally moved out of the retry set;
• partial progress: some rows updated, some deferred;
• no useful progress: every attempted row deferred;
• lease/content temporarily unavailable: the run should be observed as skipped/deferred, not completed.

The result is a false-green task envelope: markCompleted() updates lastSuccessAt, and HealthService.recordTaskOutcome(..., 'completed') receives only {reason, code, completedAt}. A full-deferred run can therefore satisfy the scheduler's visible success path while the backlog stays stuck.

The Architectural Reality

Verified against current source on 2026-06-21:

• ai/services/memory-core/MemoryService.mjs:1631-1749 documents and returns {processed, updated, deferred, missingContent, runBudgetHit} from backfillMiniSummaries().
• ai/scripts/lifecycle/backfill-memory-summaries.mjs delegates to MemoryService.backfillMiniSummaries() and emits the structured result on stdout for successful runs.
• ai/daemons/orchestrator/services/ProcessSupervisorService.mjs:333-421 owns child process execution, health outcome recording, and success/failure task-state writes.
• ProcessSupervisorService.mjs:355 currently uses stdio: ['ignore', 'ignore', 'pipe'], discarding child stdout globally.
• ProcessSupervisorService.mjs:407-411 calls onSuccess?.(), taskStateService.markCompleted(taskName), and recordTaskOutcome(taskName, 'completed', ...) for any child exit code 0.
• ai/daemons/orchestrator/services/TaskStateService.mjs:190-216 separates markCompleted() from markSkipped(); skipped is the correct terminal for no-progress/deferred outcomes that should not refresh lastSuccessAt.
• ai/daemons/orchestrator/scheduling/memorySummaryBackfill.mjs:143-241 owns the existing no-progress backoff hook, but that hook only runs as an arg-less onSuccess callback and cannot consume child outcome details today.

The Fix

Add an opt-in, bounded stdout JSON capture path to ProcessSupervisorService.runTask() and enable it first for the memory-summary-backfill task definition.

Expected shape:

1. Task definitions can opt into child stdout capture, for example with a small captureStdoutJson / outcomeCapture option.
2. ProcessSupervisorService buffers stdout only for opted-in tasks, with a hard byte limit and parse-failure fallback.
3. Parsed child outcome details are merged into the task outcome details recorded through HealthService.recordTaskOutcome().
4. memory-summary-backfill maps structured outcomes into task-state semantics:
   • progress (updated > 0 or intentionally handled missingContent > 0) stays completed and records counts;
   • all-deferred/no-progress (processed > 0 && updated === 0 && missingContent === 0 && deferred > 0) records skipped with a reason such as all-deferred and must not call markCompleted();
   • lease-held/content-store-unreachable/deferred-only cases remain visible as skipped/deferred rather than completed;
   • malformed/oversized stdout does not break unrelated tasks and records a bounded parse warning where appropriate.

Contract Ledger Matrix

Decision Record impact

aligned-with ADR 0022. This does not change the heavy-maintenance scheduling fairness model; it adds the stall-observability and task-state truth needed for that model to be falsifiable in production.

Acceptance Criteria

• ProcessSupervisorService.runTask() supports bounded stdout JSON capture for explicitly opted-in task definitions only.
• Existing supervised tasks without the opt-in keep their current stdout-ignored behavior.
• The memory-summary-backfill task definition opts into child outcome capture.
• A child exit 0 with {"processed":8,"updated":2,"deferred":6,"missingContent":0,"runBudgetHit":false} records completed with the parsed counts.
• A child exit 0 with {"processed":6,"updated":0,"deferred":6,"missingContent":0,"runBudgetHit":false} records skipped with reason all-deferred and does not call markCompleted() / refresh lastSuccessAt.
• Lease-held or content-store-unreachable deferred-only output remains visible as skipped/deferred, not completed.
• Malformed or oversized stdout for an opted-in task fails soft with bounded diagnostic detail and does not affect non-opted tasks.
• Focused unit coverage lands in the existing orchestrator/service test area; no broad Playwright/default test run is required.

Out of Scope

• Changing miniSummary generation prompts, model selection, or provider timeout policy.
• Reworking the heavy-maintenance picker or lease compatibility policy.
• Changing the inherited-token wrapper behavior covered by #13764.
• Changing Golden Path ranking, KB sync, or REM digestion logic.
• Making stdout parsing global for all orchestrator children.

Avoided Traps

• Do not treat exit code 0 as progress. A child can exit cleanly after deferring every attempted row.
• Do not make fail-soft row deferrals process failures. The child should be allowed to exit 0; the supervisor should classify the outcome accurately.
• Do not parse stdout globally. The right surface is a bounded opt-in for tasks with an explicit JSON contract.
• Do not rely only on the existing no-progress hook. It is useful scheduler state, but it currently cannot carry the child outcome across the process boundary.

Duplicate / Claim Sweep

Live latest-open sweep: checked the latest 20 open GitHub issues immediately before creation at 2026-06-21T14:50:20Z. The live set included #13776, #13772, #13763, #13755, #13753, #13751, #13750, #13746, #13744, #13740, #13739, #13736, #13700, #13679, #13652, #13624, #13623, #13618, #13613, and #13600; no equivalent open leaf existed.

Targeted live GitHub searches for memory-summary-backfill stdout outcome, ProcessSupervisorService stdout JSON, and child stdout ProcessSupervisorService surfaced only the parent epic #13755 as open. The exact/content sweep surfaced adjacent closed context, especially #13462 for session-summary child stdout observability and #12943 for miniSummary no-progress backoff, but neither owns this miniSummary child-outcome propagation gap.

A2A in-flight sweep: checked the latest 30 A2A messages immediately before creation at 2026-06-21T14:50:20Z. No peer [lane-claim] or [lane-intent] overlapped this exact implementation scope.

Semantic KB sweep and team-memory sweep found prior context but no equivalent open ticket or active claim.

Related

Parent: #13755 Related: #13624, #13764, #13462, #12943

Origin Session ID: 69f79662-2fbe-403a-a124-78bca1abdb16

Handoff Retrieval Hints:

• Retrieval Hint: "memory-summary-backfill ProcessSupervisorService stdout JSON all-deferred markSkipped lastSuccessAt"
• Retrieval Hint: "miniSummary false-green completed exit 0 deferred residual health outcome"
• Source seam: MemoryService.backfillMiniSummaries() returns counts, backfill-memory-summaries.mjs prints them, ProcessSupervisorService.runTask() currently discards stdout and completes on exit 0.

Authored by GPT-5 (Codex Desktop) as @neo-gpt.