Data Sync exhausts its stale-head budget on an active merge day

Context

Observed on the first scheduled run after #17148 merged — run 31877195197, which did publish (f87cff0c5e, ending a ~22h corpus freeze). It published on attempt 2 of 2, the last available attempt.

This is a second-order consequence of #17148, not a defect in it: that fix lengthened emission by running the stages the pipeline used to skip by aborting at stage 3 of 7. A longer emission window is more likely to be overtaken by dev.

The Problem

runDataSyncPipeline defaults maxAttempts = 2 (buildScripts/dataSyncPipeline.mjs). When origin/dev advances during emission, the attempt is discarded and re-emitted from the new head; exhausting both attempts throws "Data Sync stopped cleanly because dev advanced during all 2 emission attempts" — and publishes nothing.

Measured on the run above:

Emission is ~18 minutes and the budget is 2. On a morning where merges land every ~18 minutes, the pipeline can burn both attempts and freeze the corpus again — for a reason unrelated to any credential or stage failure. One more merge during attempt 2 and this run would have published nothing.

The Architectural Reality — measured, because the obvious mitigation is wrong

My first instinct was "publish the corpus before the slow DevIndex enrichment, shrinking the window to minutes". The measurement falsifies that. Per-stage decomposition of attempt 1:

The DevIndex enrichment is only 20% of the window. The corpus stage alone is 71%, and it must precede any publish — it generates the thing being published. So reordering to publish before enrichment saves at most ~3m36s of 18m, roughly a 20% narrowing. Worth having, nowhere near "shrink to minutes".

Recording this because the intuition is attractive and wrong, and the next person will have it too.

The Fix — candidates, none obviously dominant

1. Raise maxAttempts. Simplest and directly targets the observed exhaustion. Cost: each extra attempt is ~18 minutes of runner time on a pipeline that runs hourly, so 3 attempts can approach the schedule interval.
2. Publish corpus + content-indexes before the intake stages. Measured ~20% window reduction. Also makes publication structurally independent of enrichment, which is the same separation #17148 established at the failure axis. Requires confirming content indexes and SEO does not consume DevIndex output.
3. Re-emit only what the new head invalidated rather than the whole corpus. Largest win — it attacks the 71% — and by far the most complex; needs a per-facet dependency on the diff between heads.
4. Accept it and monitor. The watchdog already alarms on corpus staleness, and a stale-head exhaustion is self-correcting on the next hourly run. This is a legitimate answer if exhaustion turns out to be rare.

Deciding between these needs a frequency measurement nobody has: how often does dev advance twice inside 18 minutes? One observation is not a rate.

Acceptance Criteria

• The dev-advance rate during emission windows is measured over a representative period (a merge-heavy weekday and a quiet one), not inferred from this single run.
• A decision between candidates 1–4 is recorded with that rate as its basis, including an explicit "accept and monitor" option.
• If a code change lands, a stale-head exhaustion no longer silently publishes nothing: the failure names dev-advance as the cause distinctly from a stage or credential failure.
• Any claimed window reduction is stated as a measured figure with its per-stage decomposition, not a nominal one.

Out of Scope

• #17148 / #17150 / #17162 — the failure-isolation work. This is a timing interaction, not a failure path.
• Making the corpus stage itself faster. That is the 71% and the real prize, but it is a different investigation with a different risk profile.

Avoided Traps

• "Publish before enrichment shrinks the window to minutes." Falsified above: enrichment is 20%, the corpus stage is 71% and is unavoidable pre-publish. I proposed this before measuring and it was wrong.
• Treating one observation as a rate. This run exhausted attempt 1 on a morning with unusually dense merges. That is an existence proof, not a frequency.
• Reading the publish as a clean success. It landed on the last available attempt; a single additional merge would have produced a silent no-publish.

Related

• #17148 / PR #17149 — lengthened emission by removing the early abort; the cause of the widened window.
• #17131 — the standing alarm; a stale-head exhaustion presents there as continued corpus staleness with no failing stage.

Retrieval Hint: "data sync stale-head budget exhausted; emission 18min vs dev advance rate; corpus stage is 71% of the window"

⚖️ Ada · @neo-opus-ada · Claude Opus 5 · Claude Code