Summary

Changes the DevIndex Spider step in .github/workflows/data-sync-pipeline.yml from a 3-iteration bash loop to a single invocation. Reduces GraphQL rate-limit budget consumption in the hourly CI pipeline, which was contributing to downstream failures in the same workflow (see companion PR #10114 / ticket #10112 for the diagnostic-layer work).

The Change

-      - name: Run DevIndex Spider (3x Loop)
+      - name: Run DevIndex Spider
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        run: |
-          for i in 1 2 3; do
-            echo "Starting Spider Run $i/3..."
-            npm run devindex:spider -- --strategy random
-          done
+        run: npm run devindex:spider -- --strategy random

One step, no other workflow changes.

Why Now

The 3x loop made sense during bootstrapping when discovery throughput (finding top-50k candidates) was the bottleneck. The DevIndex is now at the saturation state the architecture was designed around:

• users.jsonl at the 50k maxUsers cap
• threshold.json raised accordingly — Updater's Meritocracy filter aggressively prunes below-threshold candidates
• tracker.json sits near cap (~49,649), kept in sync via Cleanup's pruning (per learn/guides/devindex/data-factory/DataHygiene.md)

At saturation, extra spider iterations produce marginal new qualified users relative to their API cost. Concretely:

• Low Pending backlog (< 2000): all 3 runs proceed fully → 3× core + search rate-limit burn for little net intake
• High Pending backlog (≥ 2000): Spider.mjs:97 Backpressure Valve (config.spider.maxPendingUsers: 2000) aborts runs 2 and 3, but each still costs state-loading cycles
• In both regimes, Updater throughput (~800/hour) is the actual bottleneck. Discovery oversupply queues candidates that mostly get threshold-rejected when Updater reaches them.

Downstream Relationship

Companion PR #10114 (ticket #10112) restored diagnostic visibility on LabelService failures in the same pipeline. The prevailing hypothesis — spider loop's 3× burn pushes subsequent GraphQL calls (labels, release, tickets index generation) into 429 territory — becomes empirically testable once #10114 lands and this PR removes the pressure. Two independent channels, drained in parallel.

What Stays Untouched

• Spider.mjs code — Backpressure Valve logic unchanged, no config tuning
• DevIndex.services.Cleanup — Garbage Collector + 30-day Penalty Box TTL unchanged
• Updater throughput / Meritocracy threshold — no change
• CI cadence (hourly cron) — no change to on.schedule

This is purely CI-step tuning.

Test Plan

• YAML syntax confirmed valid (one-step replacement, preserved indentation)
• Post-merge: observe next data-sync-pipeline run — total workflow duration, GraphQL rate-limit remaining at end-of-run, whether labels.mjs step completes cleanly
• Post-merge (if #10114 has landed): confirm/refute the rate-limit-429 hypothesis via the now-visible real GraphQL error in any subsequent failure

Related

• Companion: #10114 / #10112 (diagnostic restoration; this PR removes the pressure, that PR restores the visibility)
• Future candidate (not filed): tracker.json (~49,649) vs users.jsonl (~50,000) count gap — 351 users exist without tracker entries, which would prevent re-update. Noted as a separate observation to confirm before filing; deliberately NOT scoped here

Avoided Traps

• Touching Spider.mjs or the Backpressure Valve config — architectural primitives are correct, this is purely a CI frequency knob
• Reducing CI workflow cadence (hourly cron) — orthogonal concern; adjust later only if saturation dynamics shift
• Bumping Updater throughput — separate large change with its own rate-limit exposure

Resolves #10113

🤖 Generated with Claude Code