Frontmatter
| id | 9227 |
| title | Unify DevIndex Workflows and implement Spider backpressure |
| state | Closed |
| labels | enhancementaibuild |
| assignees | tobiu |
| createdAt | Feb 21, 2026, 3:32 PM |
| updatedAt | Feb 21, 2026, 3:36 PM |
| githubUrl | https://github.com/neomjs/neo/issues/9227 |
| author | tobiu |
| commentsCount | 1 |
| parentIssue | null |
| subIssues | [] |
| subIssuesCompleted | 0 |
| subIssuesTotal | 0 |
| blockedBy | [] |
| blocking | [] |
| closedAt | Feb 21, 2026, 3:34 PM |
Unify DevIndex Workflows and implement Spider backpressure
tobiu assigned to @tobiu on Feb 21, 2026, 3:34 PM

tobiu
Feb 21, 2026, 3:34 PM
Input from Gemini 3 Pro:
✦ I have implemented the Unified Pipeline and Spider Backpressure as requested.
- Created
.github/workflows/devindex-pipeline.ymlto replace the two independent cron jobs, orchestrating the 3x Spider Loop followed immediately by the Updater within a single, sequential job.- Removed
devindex-spider.ymlanddevindex-updater.ymlto eliminate Git merge conflicts and double-running.- Modified
apps/devindex/services/config.mjsto addmaxPendingUsers: 2000to thespiderconfiguration.- Modified
apps/devindex/services/Spider.mjsrun()method to check the number of users withlastUpdate: null. If this backlog equals or exceeds themaxPendingUserscap, the Spider now immediately aborts its run, forcing it to sleep and letting the Updater consume the existing queue.These changes have been committed, rebased over the latest remote background updates, and pushed to
dev. Closing issue.
tobiu closed this issue on Feb 21, 2026, 3:34 PM
The DevIndex currently uses two independent GitHub Action workflows (
devindex-spider.ymlanddevindex-updater.yml) that run on competing 30-minute cron schedules. This leads to intermittent Git merge conflicts when both jobs attempt to modify and pushtracker.jsonsimultaneously. Furthermore, uncontrolled Spider discovery can bloat the tracker backlog.Tasks:
.github/workflows/devindex-pipeline.yml) that runs hourly. This guarantees atomic commits and zero git conflicts.lastUpdate: null) intracker.json. If the backlog exceedsconfig.spider.maxPendingUsers(e.g., 2000), the Spider should abort its discovery run to let the Updater catch up.