LearnNewsExamplesServices
Frontmatter
id9125
titleDevIndex: Stop Cleanup.mjs from pruning orphaned tracker users
stateClosed
labels
bugai
assigneestobiu
createdAtFeb 12, 2026, 11:08 PM
updatedAtFeb 12, 2026, 11:26 PM
githubUrlhttps://github.com/neomjs/neo/issues/9125
authortobiu
commentsCount2
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtFeb 12, 2026, 11:26 PM

DevIndex: Stop Cleanup.mjs from pruning orphaned tracker users

Closed v12.0.0 bugai
tobiu
tobiu commented on Feb 12, 2026, 11:08 PM

The Cleanup.mjs service currently deletes users from tracker.json if they have a lastUpdate timestamp but are missing from users.json. This logic was intended to prune "low value" users. However, with the new "Penalty Box" feature (#9121), users who fail to update (e.g. 502/504) also get a lastUpdate timestamp but are not added to users.json. Result: Cleanup.mjs deletes these failed users, causing them to be lost from the queue. When rediscovered by the spider, they are treated as "new" and re-scanned immediately, wasting API quota.

Solution:

  • Stop pruning orphaned users from tracker.json.
  • Keep them in the tracker to ensure they are processed according to their lastUpdate timestamp (i.e., at the back of the queue).
  • Add logging to identify these orphaned users for debugging.
tobiu added the bug label on Feb 12, 2026, 11:08 PM
tobiu added the ai label on Feb 12, 2026, 11:08 PM
tobiu referenced in commit 8bd44bd - "feat: Implement 'Penalty Box' via sidecar file for failed updates (#9125) on Feb 12, 2026, 11:24 PM
tobiu
tobiu Feb 12, 2026, 11:25 PM

Input from Gemini 3 Pro:

✦ I have implemented the "Penalty Box" logic using a sidecar file (failed.json) instead of bloating the tracker.

Changes:

  1. Storage: Added failed.json (defaults to []) and methods getFailed() / updateFailed().
  2. Updater:
    • When a user update fails, they are added to failed.json.
    • When a user update succeeds, they are removed from failed.json.
    • Users are still pushed to the back of the tracker queue (timestamp updated).
  3. Cleanup:
    • Re-enabled "Active Pruning" for orphaned users (those in tracker but not in users).
    • Crucially, added a check: if an orphan is in failed.json, they are KEPT in the tracker (protected).
    • Logs explicit warnings when keeping a failed user or pruning a non-failed orphan.
  4. Data Recovery: I manually identified the 27 users who were about to be pruned in the previous run and added them to failed.json to ensure they are not lost.

This approach keeps tracker.json lean while ensuring robust retry logic for problematic users.

tobiu assigned to @tobiu on Feb 12, 2026, 11:25 PM
tobiu
tobiu Feb 12, 2026, 11:26 PM

Input from Gemini 3 Pro:

✦ I am closing this issue as I have implemented the fix and pushed the changes. I have also verified the fix by running the cleanup script with the manually populated failed.json list, confirming that the users are now correctly identified and kept in the tracker.

tobiu closed this issue on Feb 12, 2026, 11:26 PM