Data Sync failures are silent for days — add a staleness alarm

Context

Tonight the swarm learned — only because the operator noticed — that the Data Sync Pipeline has not landed anything in eight days. The root cause has an owner (#15744, App identity + ruleset bypass). The silence has no owner, and it is the separate gap: 60+ consecutive scheduled failures, 383 stranded ticket-markdown files, and nothing alarmed.

Independently verified before filing (V-B-A): last successful run 2026-07-17T03:20:56Z; the latest 10 scheduled runs are all failure, including 2026-07-25T22:03Z tonight — the pipeline is failing right now, quietly.

Premise correction (2026-07-26, Ada's premise-check — verified before adoption): the 383 uncommitted ticket-markdown files are NOT a pipeline effect. GENERATED_DATA_PATHS excludes resources/content/** (the pipeline regenerates it into the pages clone and discards; the neo copy advances only via hand-authored chore: ticket sync commits — never github-actions[bot]). Two things lapsed on 2026-07-17: the last successful pipeline run AND the last hand-authored sync — and the second had no owner at all, not even a failing cron. A run-status-only alarm would therefore certify the corpus backlog forever once the pipeline goes green. The watchdog carries BOTH axes: run-status AND committed-corpus age (resources/content/** last-commit age on dev, measured through the API — never a working-tree mtime, which can read current in the exact episode it must catch).

The Problem

The Data Sync Pipeline (.github/workflows/, scheduled) degrades silently: consecutive failures produce no swarm-visible signal. Worse, the swarm's own knowledge ingestion rides this pipeline — the stranded ticket-markdown corpus is the input an in-Agent-OS watchdog would watch with, so a watchdog living inside the Agent OS is partially blind to exactly this degradation. The principle is the graduated D#15904 lesson: a quiet failure fails silently; observability must ship before trust. Tonight's wake-envelope work hit the same shape from the other side (a planted plugin that never wrote, indistinguishable from a working one until the log sink was probed).

The Architectural Reality

• The pipeline is a GitHub Actions cron (Data Sync Pipeline, workflow id 259117761) — a GitHub-side surface whose health is observable via the Actions API independently of the pipeline itself and of the Agent OS.
• Sibling prior art, different pipeline: #11942 / #11791 built per-repo failure backoff + a quarantined status for the Agent-OS-side TenantRepoSyncService — same principle (consecutive-failure state made explicit), wrong host for this gap.
• A watchdog needs only actions:read + issues:write — no ruleset bypass, no coupling to #15744's App identity. It ships independent of the root-cause fix.

The Fix

A small scheduled staleness watchdog (new workflow + one script, sibling precedent buildScripts/util/*.mjs):

1. On a schedule (offset from the pipeline's own cron), query the Data Sync Pipeline's recent runs via the GitHub API.
2. Compute consecutive-failure count and last-success age.
3. On threshold breach (proposed: ≥3 consecutive failures OR last success >24h), open one standing alarm issue — or update the existing open one — naming the latest failed run id and the streak length. Idempotent: N breaches, one issue.
4. On recovery, close the standing issue with a comment linking the recovering run.

Contract Ledger Matrix

Decision Record impact: none — operational observability; aligned with the D#15904 silent-channel principle, no ADR surface.

Acceptance Criteria

• AC-1: The watchdog computes consecutive-failure count + last-success age for the Data Sync Pipeline from the Actions API, verified by a workflow_dispatch dry run with a forced-threshold override.
• AC-2: On breach it opens exactly one standing alarm issue; a second breach while it is open updates the same issue (verified: two consecutive forced breaches → one issue).
• AC-3: On recovery it closes the standing issue with the recovering run linked.
• AC-4: Threshold-edge logic has unit evidence (0 failures, exactly-at-threshold, recovery transition) in the canonical unit tree.
• AC-5 (post-merge): the first real breach — or a manual workflow_dispatch demonstration — produces the alarm issue on the live tracker.
• AC-6: the corpus axis measures the last resources/content/** commit on the COMMITTED default branch via the API (48h threshold, strictly-past boundary), breaches independently of run-status, and never lets a green run close the alarm while the corpus axis is breaching — spec-covered.

Out of Scope

• The sync failure's root cause (#15744 owns it: App identities + ruleset bypass actor).
• Extending the watchdog to other GitHub Actions pipelines — the computation takes the workflow id as input (WATCHDOG_WORKFLOW), so that extension is config, not code.
• Cloud / multi-tenant deployments (challenged by the operator 2026-07-26 — tracked in the successor ticket): a cloud Agent OS running data syncs for MULTIPLE tenants is a different surface, and this GitHub-side watchdog does not reach it. The blindness argument inverts there: an Agent OS watching tenant syncs is not watching its own ingestion input, so an in-deployment check is safe. The successor consumes the per-tenant staleness state that already exists (#11942's consecutiveFailures backoff + lastIngestedRev in tenant-repo-sync-revisions.json) and raises the same one-standing-alarm-per-episode shape keyed per (deployment, tenant).
• Agent-OS-side telemetry or A2A-push escalation for THIS surface — the GitHub issue is the alarm surface; richer escalation can follow once the data path heals.
• Any change to the Data Sync Pipeline itself.

Avoided Traps / Gold Standards Rejected

• An Agent-OS-internal watchdog — partially blind to its own degraded input (the stranded corpus is its ingestion source). The alarm must live on a surface independent of the thing it watches.
• Per-failure issue spam — one standing issue per breach episode, updated in place.
• Coupling to #15744's App identity — actions:read + issues:write on the default token suffices; the watchdog must not wait on the root-cause fix to become useful (it would have caught this episode on day one).

Related

• #15744 (root cause: Data Sync App identity) · #15750 · #15751
• #11942 / #11791 (sibling consecutive-failure state + quarantine mechanism, Agent-OS side)
• D#15904 (graduated: wake-vs-mailbox separation; the silent-channel principle this ticket applies to CI)

Live latest-open sweep: checked latest 20 open issues at 2026-07-25T23:0xZ; no equivalent found. A2A in-flight sweep: no claim on this scope; Ada's #15744 broadcast explicitly surfaced the silence gap as unowned. KB semantic sweep: closest are #11942/#11791 (tenant-repo sync backoff — different pipeline, cited as prior art).

Origin Session ID: 318916f0-3f6b-4f1c-b0d2-ee16e2dd8af0

Retrieval Hint: "data sync pipeline silent failure staleness watchdog" · gh run list --workflow=259117761