LearnNewsExamplesServices
Frontmatter
id9227
titleUnify DevIndex Workflows and implement Spider backpressure
stateClosed
labels
enhancementaibuild
assigneestobiu
createdAtFeb 21, 2026, 3:32 PM
updatedAtFeb 21, 2026, 3:36 PM
githubUrlhttps://github.com/neomjs/neo/issues/9227
authortobiu
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtFeb 21, 2026, 3:34 PM

Unify DevIndex Workflows and implement Spider backpressure

Closed v12.0.0 enhancementaibuild
tobiu
tobiu commented on Feb 21, 2026, 3:32 PM

The DevIndex currently uses two independent GitHub Action workflows (devindex-spider.yml and devindex-updater.yml) that run on competing 30-minute cron schedules. This leads to intermittent Git merge conflicts when both jobs attempt to modify and push tracker.json simultaneously. Furthermore, uncontrolled Spider discovery can bloat the tracker backlog.

Tasks:

  1. Unify Workflows: Combine the Spider and Updater into a single, sequential pipeline (.github/workflows/devindex-pipeline.yml) that runs hourly. This guarantees atomic commits and zero git conflicts.
  2. Backpressure Valve: Enhance the Spider to check the number of "pending" users (lastUpdate: null) in tracker.json. If the backlog exceeds config.spider.maxPendingUsers (e.g., 2000), the Spider should abort its discovery run to let the Updater catch up.
  3. Restore 3x Loop: With the workflows unified and backpressure in place, the Spider can safely be restored to execute 3 discovery loops per hour instead of 1.
  4. Cleanup: Remove the obsolete individual workflows.
tobiu added the epic label on Feb 21, 2026, 3:32 PM
tobiu added the ai label on Feb 21, 2026, 3:32 PM
tobiu added the build label on Feb 21, 2026, 3:32 PM
tobiu referenced in commit d106ed9 - "build(devindex): Unify GH Actions into single pipeline and add Spider backpressure (#9227)" on Feb 21, 2026, 3:33 PM
tobiu assigned to @tobiu on Feb 21, 2026, 3:34 PM
tobiu
tobiu Feb 21, 2026, 3:34 PM

Input from Gemini 3 Pro:

✦ I have implemented the Unified Pipeline and Spider Backpressure as requested.

  • Created .github/workflows/devindex-pipeline.yml to replace the two independent cron jobs, orchestrating the 3x Spider Loop followed immediately by the Updater within a single, sequential job.
  • Removed devindex-spider.yml and devindex-updater.yml to eliminate Git merge conflicts and double-running.
  • Modified apps/devindex/services/config.mjs to add maxPendingUsers: 2000 to the spider configuration.
  • Modified apps/devindex/services/Spider.mjs run() method to check the number of users with lastUpdate: null. If this backlog equals or exceeds the maxPendingUsers cap, the Spider now immediately aborts its run, forcing it to sleep and letting the Updater consume the existing queue.

These changes have been committed, rebased over the latest remote background updates, and pushed to dev. Closing issue.

tobiu closed this issue on Feb 21, 2026, 3:34 PM
tobiu removed the epic label on Feb 21, 2026, 3:36 PM
tobiu added the enhancement label on Feb 21, 2026, 3:36 PM