LearnNewsExamplesServices
Frontmatter
id8937
titleFeat: Discovery Engine - The Spider
stateClosed
labels
enhancementai
assigneestobiu
createdAtFeb 1, 2026, 5:43 PM
updatedAtFeb 7, 2026, 5:29 PM
githubUrlhttps://github.com/neomjs/neo/issues/8937
authortobiu
commentsCount1
parentIssue8930
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtFeb 7, 2026, 5:29 PM

Feat: Discovery Engine - The Spider

Closed v12.0.0 enhancementai
tobiu
tobiu commented on Feb 1, 2026, 5:43 PM

This task focuses on implementing the "Spider" discovery mechanism to expand the contributor index beyond the top repositories.

Requirements

  1. Spider Logic: Update gh_discovery.mjs to traverse the following graph of the initial "seed" candidates.
  2. Depth Control: Implement a depth limit (e.g., 1 or 2 layers) to prevent infinite crawling.
  3. Filtration: Apply a "Pulse Check" (e.g., >50 contributions in current year) before performing a "Deep Scan" on new candidates to optimize API usage.
  4. Deduplication: Ensure users are not scanned twice.

Acceptance Criteria

  • The script discovers and indexes high-contribution users who may not be in the top 50 repositories but are followed by top contributors.
  • The resources/data.json file is populated with these new candidates.
tobiu added the enhancement label on Feb 1, 2026, 5:43 PM
tobiu added the ai label on Feb 1, 2026, 5:43 PM
tobiu added parent issue #8930 on Feb 1, 2026, 5:43 PM
tobiu assigned to @tobiu on Feb 1, 2026, 5:44 PM
tobiu
tobiu Feb 7, 2026, 5:27 PM

Input from Gemini 3 Pro:

✦ Implemented the Spider (Discovery Engine) service:

  • Spider.mjs: Crawls high-star repositories and fetches top contributors.
  • Integration: Wired into Manager.mjs via the spider command.
  • Logic: Filters bots, respects blacklists, and updates the visited.json graph to avoid cycles.
  • Verification: Ran a successful crawl discovering 284 new candidates.
tobiu referenced in commit 2b15098 - "feat: Discovery Engine - The Spider (#8937) on Feb 7, 2026, 5:28 PM
tobiu closed this issue on Feb 7, 2026, 5:29 PM