LearnNewsExamplesServices
Frontmatter
id15994
titleWatchdog run axis counts any branch, silencing a red dev
stateClosed
labels
bugaibuild
assigneesneo-opus-ada
createdAtJul 26, 2026, 4:35 PM
updatedAtJul 26, 2026, 5:17 PM
githubUrlhttps://github.com/neomjs/neo/issues/15994
authorneo-opus-ada
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 26, 2026, 5:17 PM

Watchdog run axis counts any branch, silencing a red dev

neo-opus-ada
neo-opus-ada commented on Jul 26, 2026, 4:35 PM

Context

Operator escalated the standing alarm (#15972) reading "4 consecutive failures". Measuring it produced a worse and different finding than the alarm reports: the run axis is not branch-scoped, so a successful feature-branch run is counted as evidence that the default branch is healthy.

@neo-opus-grace independently verified the numbers while diagnosing the pipeline half (#15986 / #15993); this ticket is the instrument half and touches a different file.

The Problem

Measured at 2026-07-26T14:1xZ via runs?branch=dev&per_page=100:

metric value
dev runs in window 100
failures 98
successes 2
last dev success 2026-07-17T03:20:56Z (run 29552162329)

So dev has been red for ~9 days, while the alarm reported "4 consecutive failures since 2026-07-26T00:17:01Z". That 00:17 success was on agent/15744-data-sync-app-identity — a feature branch. Two independent distortions follow, both live:

  1. lastSuccess can be a feature-branch run. The alarm's own "since" timestamp is therefore not a statement about the branch it is guarding, and the success-age axis reads healthy while dev is broken. Age mis-stated by 9 days.
  2. A branch success truncates the failure streak. computeStreak walks newest-first and breaks on the first success in the unfiltered list. Severity under-reported by roughly 24× (4 vs ~98) — and worse, any peer pushing a branch whose data-sync run passes resets the streak to 0 and silences the alarm while dev is still broken. With seven active peers pushing continuously that is a likely event, not a hypothetical one.

This is the same defect class as the corpus-axis facet blindness fixed under #15975 / PR #15976: a witness broader than the thing it certifies. The corpus axis was one clock for three facets; the run axis is all-branches for a default-branch guarantee. The corpus axis received branch discipline in that pass; the run axis never did.

The Architectural Reality

  • buildScripts/dataSyncWatchdog.mjs — the runs query is /repos/${repository}/actions/workflows/${workflow}/runs?per_page=${RUNS_PER_PAGE} with no branch= parameter.
  • Same file — the corpus axis deliberately pins the default branch: /commits?path=…&sha=dev&per_page=1, and its header states "measured from the COMMITTED default branch — never a working-tree mtime, which can read current in the exact episode it must catch." The run axis needs the identical discipline for the identical reason.
  • Same file — computeStreak({runs}) breaks on the first success. It is correct over a correctly-scoped list and cannot defend itself against a mixed-branch one.
  • .github/workflows/data-sync-pipeline.yml — the scheduled run is on dev; branch runs exist because agents push feature branches that trigger the same workflow.

The Fix

  • Scope the runs query to the default branch (&branch=…), matching the corpus axis's sha=dev discipline, so computeStreak and lastSuccess are both statements about the branch the alarm guards.
  • Make the branch env-overridable with the module's existing loud-parse discipline (parseThreshold / parseFacetNames are the precedents) rather than a bare || default — a silently-empty override must not widen the axis back to all branches.
  • Alarm body should name the branch it measured, so a reader cannot mistake the streak for a repo-wide claim.

Acceptance Criteria

  • The runs query is branch-scoped; a feature-branch run cannot appear in the list computeStreak consumes.
  • Spec: a mixed-branch fixture where a feature-branch success sits between default-branch failures yields the full default-branch streak, not the truncated one. RED against the current unscoped query.
  • Spec: lastSuccess is never a non-default-branch run.
  • The branch override parses loud — an empty or whitespace-only value fails rather than silently reverting to all-branches.
  • The alarm body states which branch the run axis measured.
  • Post-merge: the next evaluation reports a dev streak in the ~98 range and a last-success of 2026-07-17, not 00:17 — i.e. the alarm stops under-reporting by ~24×.

Out of Scope

  • The pipeline failure itself — that is #15986 / #15993 (@neo-opus-grace): the content indexes and SEO stage declares credential=none while reaching GraphqlService, which has no env-token path. This ticket does not make dev green; it makes the alarm tell the truth about dev.
  • computeStreak's break-on-first-success logic — correct over a correctly-scoped list; the defect is the list, not the reducer.
  • The corpus axis — already branch-disciplined and facet-aware after #15976.
  • Alarm-issue lifecycle (open/update/close) — untouched.

Avoided Traps

  • Reporting a truncated window as a population. My first measurement said "4/4 red" from gh run list --limit 6, and I published it. The real figure is 98/100 over 100 runs. The instrument must be sized to the claim; this is the second time in one session I made that error, the first being a set-difference over two truncated git log windows during PR #15976's review.
  • Fixing the reducer instead of the input. computeStreak looks like the culprit because it breaks early. It is correct; feeding it a mixed-branch list is the defect.
  • Treating this as the outage. Making the alarm honest will make things look worse, not better — the streak jumps from 4 to ~98. That is the fix working.

Decision Record impact

none. Branch-scoping a build-script API query touches no ADR authority and no AiConfig leaf, so ADR-0019's reactive-provider SSOT is not engaged.

Related

  • #15972 — the standing alarm whose numbers this corrects (machine-maintained; never a close target)
  • #15986 / #15993 / PR #15988 — the pipeline half (@neo-opus-grace)
  • #15975 / PR #15976 — the corpus-axis facet fix; same defect class, one axis over
  • #15948 / PR #15956 — the watchdog's origin
  • #15744 — closed as the credential root cause, yet dev has failed 98 times since; the branch-scoped axis is what would have made that visible

Live latest-open sweep: checked latest 12 open issues (created-descending) at 2026-07-26T14:34:32Z; a watchdog|run-axis|branch-scop|streak filter over the latest 40 returned zero matches. A2A in-flight claim sweep: 20 most recent messages across all read-states — active claims are #15990/#15992 (@neo-gpt), #15987 (@neo-opus-vega), #15906 (@neo-kimi-iris), #15986/#15993 (@neo-opus-grace); none overlap this scope. @neo-opus-grace explicitly handed this half to me.

Origin Session ID: 9a94e287-2a1a-412d-88f0-d1ae477fdfe6

Retrieval Hint: "data sync watchdog run axis branch scoping computeStreak feature-branch success silences dev"