Reduce DevIndex Spider CI loop from 3x to 1x per pipeline invocation

The Problem

.github/workflows/data-sync-pipeline.yml currently runs the DevIndex Spider three times per CI invocation:

for i in 1 2 3; do
    echo "Starting Spider Run $i/3..."
    npm run devindex:spider -- --strategy random
done

This was valuable during the bootstrap phase when discovery throughput was the bottleneck. At current saturation, it wastes the GitHub GraphQL rate-limit budget and contributes to downstream failures in the same pipeline (see companion ticket for LabelService diagnostic restoration).

The Architectural Reality

The DevIndex is now at the saturation state the architecture was designed around:

• users.jsonl is at the maxUsers cap (50,000)
• threshold.json has raised the contribution bar accordingly (DevIndex.services.Updater's Meritocracy filter aggressively prunes below-threshold candidates per learn/guides/devindex/data-factory/Updater.md)
• tracker.json sits near cap (~49,649), kept in sync with users.jsonl by DevIndex.services.Cleanup per learn/guides/devindex/data-factory/DataHygiene.md §1

The Spider has a Backpressure Valve (Spider.mjs:97, config.spider.maxPendingUsers: 2000) that halts a run when the Pending backlog exceeds 2000. But the valve triggers after a run's state loading — each spider invocation still consumes some API budget even when backpressure aborts it, and the 2nd and 3rd loop iterations may fire a full discovery run if the Updater has drained Pending below 2000 between iterations.

Why 3x No Longer Earns Its Cost

• Low Pending backlog (< 2000): all 3 runs proceed fully → 3× the rate-limit burn on core + search buckets for marginal additional discovery
• High Pending backlog (≥ 2000): runs 2 and 3 abort via backpressure, but still cost state-loading cycles
• In both regimes, the Updater's throughput (~800/hour per learn/guides/devindex/data-factory/DataHygiene.md) is the actual bottleneck. Discovery oversupply yields new Pending entries that wait in queue, most of which then get threshold-rejected when Updater reaches them. Net new qualified users from runs 2-3 is low relative to their API cost.

Downstream Damage

The companion ticket (LabelService error propagation) is triggered by data-sync-pipeline failures where labels.mjs gets an opaque "Invalid response from GH_LabelService" error. Hypothesis: the spider loop's 3× GraphQL burn pushes subsequent GraphQL calls in the same workflow (labels, release, tickets index generation) into rate-limit territory. This ticket addresses the root cause of that pressure.

Fix

Change the workflow step from the 3-iteration bash loop to a single invocation:

- name: Run DevIndex Spider
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  run: npm run devindex:spider -- --strategy random

Avoided Traps

• Touching config.spider.maxPendingUsers or Backpressure Valve logic. The architectural primitives at Spider.mjs:97 and DevIndex.services.Cleanup handle saturation correctly; they don't need new mechanisms. This ticket is purely a CI frequency tuning.
• Reducing spider schedule frequency. The CI workflow cadence (i.e. how often the pipeline itself runs) is orthogonal; tuning it is a separate concern if needed after observing post-merge behavior.
• Modifying Updater throughput or Meritocracy threshold. Increasing Updater drain rate would be another way to let 3x-discovery land useful candidates, but that's a significantly larger change with its own rate-limit exposure. Out of scope.

Acceptance Criteria

• .github/workflows/data-sync-pipeline.yml runs npm run devindex:spider -- --strategy random once per pipeline invocation
• No other changes to spider invocation, strategy, env, or step ordering

Post-Merge Validation

• Observe next data-sync-pipeline run: total workflow duration, GraphQL rate-limit remaining at end-of-run, whether the labels.mjs step completes cleanly
• If LabelService companion ticket has landed: its new clear-error logging will confirm whether rate-limit 429 was the actual prior failure mode

Related

• Companion: LabelService error propagation ticket (diagnostic restoration; the two drain independent channels — this removes pressure, that restores visibility)
• Future candidate if invariant check flags a real issue: tracker.json (~49,649) vs users.jsonl (~50,000) count gap — 351 users exist without tracker entries, which would prevent re-update. Deliberately NOT scoped here; needs confirmation before filing

Origin Session ID

07f601dc-353a-44d2-a373-18da2a0d305a