LearnNewsExamplesServices
Frontmatter
id9140
titleImplement Scheduled DevIndex Spider Workflow
stateClosed
labels
enhancementai
assigneestobiu
createdAtFeb 13, 2026, 12:10 PM
updatedAtFeb 13, 2026, 12:12 PM
githubUrlhttps://github.com/neomjs/neo/issues/9140
authortobiu
commentsCount1
parentIssue9106
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtFeb 13, 2026, 12:12 PM

Implement Scheduled DevIndex Spider Workflow

Closed v12.0.0 enhancementai
tobiu
tobiu commented on Feb 13, 2026, 12:10 PM

Create a new GitHub Actions workflow (.github/workflows/devindex-spider.yml) to run the DevIndex Spider on a schedule.

Requirements:

  • Schedule: Hourly at minute 30 (30 * * * *) to offset from the Updater.
  • Execution: Run the spider command 3 times sequentially in a single job to maximize discovery per hour while balancing API usage.
  • Command: npm run devindex:spider
  • Data Handling: Commit and push changes to tracker.json, visited.json, and other resource files.
  • Safety: Handle git pull --rebase correctly by staging all modified JSON files (apps/devindex/resources/*.json*).

Workflow Steps:

  1. Checkout
  2. Setup Node
  3. Install Dependencies
  4. Loop: Run Spider x3
  5. Git Commit & Push (with git reset --hard safety)
tobiu added the enhancement label on Feb 13, 2026, 12:10 PM
tobiu added the ai label on Feb 13, 2026, 12:10 PM
tobiu added parent issue #9106 on Feb 13, 2026, 12:10 PM
tobiu referenced in commit 221ca10 - "feat(devindex): add scheduled spider workflow (#9140)" on Feb 13, 2026, 12:11 PM
tobiu assigned to @tobiu on Feb 13, 2026, 12:12 PM
tobiu
tobiu Feb 13, 2026, 12:12 PM

Input from Gemini 3 Pro:

✦ I have created the scheduled workflow .github/workflows/devindex-spider.yml.

Workflow Details:

  • Schedule: Runs hourly at XX:30 (30 minutes offset from the updater).
  • Logic: Executes npm run devindex:spider 3 times sequentially in a single job.
  • Data Integrity: Stages apps/devindex/resources/*.json* to ensure visited.json and any reformatted files are correctly committed.
  • Safety: Uses the same git reset --hard rebase protection as the updater workflow.

Changes pushed to dev.

tobiu closed this issue on Feb 13, 2026, 12:12 PM