LearnNewsExamplesServices
Frontmatter
id9096
titleFeat: GitHub API Secondary Rate Limit Handling & Concurrency Tuning
stateClosed
labels
enhancementai
assigneestobiu
createdAtFeb 10, 2026, 9:09 PM
updatedAtFeb 11, 2026, 3:02 PM
githubUrlhttps://github.com/neomjs/neo/issues/9096
authortobiu
commentsCount1
parentIssue8930
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtFeb 10, 2026, 9:22 PM

Feat: GitHub API Secondary Rate Limit Handling & Concurrency Tuning

Closed v12.0.0 enhancementai
tobiu
tobiu commented on Feb 10, 2026, 9:09 PM

Objective

Enhance the stability of the DevRank Updater service by implementing smarter handling for GitHub's Secondary Rate Limits (403 Forbidden / Abuse Detection) and tuning concurrency.

Problem

The Updater is experiencing sporadic bursts of 403 errors despite having plenty of primary API quota remaining (e.g., 4700/5000). This indicates we are triggering GitHub's "Abuse Detection" or "Secondary Rate Limit" due to high concurrency (10 parallel requests) or complex GraphQL queries.

Tasks

  1. Tune Concurrency: Reduce Updater.mjs concurrency from 10 to 5. This is a safer baseline for complex GraphQL fetching.
  2. Smart 403 Handling: Update GitHub.mjs:
    • Inspect headers on 403. If x-ratelimit-remaining > 0, treat it as a Secondary Limit.
    • Implement a specific backoff strategy for Secondary Limits (e.g., wait 30s-60s) instead of the standard short retry.
    • Log a distinct warning ("Abuse Detection triggered") to differentiate from quota exhaustion.

References

tobiu added the enhancement label on Feb 10, 2026, 9:09 PM
tobiu added the ai label on Feb 10, 2026, 9:09 PM
tobiu added parent issue #8930 on Feb 10, 2026, 9:09 PM
tobiu referenced in commit 699b488 - "feat: Enhance GitHub API Concurrency and Rate Limit Handling (#9096)" on Feb 10, 2026, 9:20 PM
tobiu
tobiu Feb 10, 2026, 9:21 PM

Input from Gemini 3 Pro:

✦ I have implemented the concurrency tuning and smart rate limit handling.

Changes

  1. Updater.mjs: Set concurrency to 8. This is a balanced middle ground between the previous aggressive 10 and conservative 5.
  2. GitHub.mjs: Implemented smart 403 detection. If x-ratelimit-remaining > 0 (indicating secondary limit/abuse detection), the service now logs a warning and backs off for 10 seconds before retrying. This allows the pipeline to "dance" with the API limits rather than crashing or crawling.

This should stabilize the data enrichment process while maintaining high throughput.

tobiu closed this issue on Feb 10, 2026, 9:22 PM
tobiu assigned to @tobiu on Feb 11, 2026, 3:02 PM