LearnNewsExamplesServices
Frontmatter
id9796
titleImplement MD5 Hash Deltas for Autonomous Ingestion Pipeline
stateClosed
labels
enhancementai
assigneestobiu
createdAtApr 8, 2026, 9:16 PM
updatedAtApr 8, 2026, 9:17 PM
githubUrlhttps://github.com/neomjs/neo/issues/9796
authortobiu
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtApr 8, 2026, 9:17 PM

Implement MD5 Hash Deltas for Autonomous Ingestion Pipeline

Closedenhancementai
tobiu
tobiu commented on Apr 8, 2026, 9:16 PM

Context

The autonomous REM sleep ingestion pipeline relied on file mtimeMs to detect structural changes. However, CI/CD checks or scripts often "touch" files, updating their modification timestamps without changing their contents. This causes the pipeline to falsely assume a structural mutation has occurred, resulting in massive redundant hashing flows.

Implementation

  1. Delta Hash Strategy: Replaced simple timestamp checks within FileSystemIngestor.mjs with Native Node crypto MD5 checks. Now, both mtimeMs and hash are cached in the Graph DB. When mtimeMs shifts, the ingestor calculates the MD5 hash and compares it; if identical, it gracefully skips graph bloat.
  2. Issue Embedding Hardening: Assured that logic within DreamService.mjs relies correctly on existing cache comparison to not falsely flag needsEmbedding = true for identically re-synced open issues.
  3. Optimized Logging: Elevated Dream milestones to INFO and reduced repetitive node upsert spam to DEBUG levels.
tobiu added the enhancement label on Apr 8, 2026, 9:16 PM
tobiu added the ai label on Apr 8, 2026, 9:16 PM
tobiu assigned to @tobiu on Apr 8, 2026, 9:16 PM
tobiu referenced in commit 5d2e893 - "feat(ai): Implement MD5 Hash Deltas for Autonomous Ingestion Pipeline (#9796)" on Apr 8, 2026, 9:16 PM
tobiu
tobiu Apr 8, 2026, 9:16 PM

Input from Gemini 3.1 Pro (Antigravity):

✦ Implemented and tested properly natively via MD5 SQLite verification.

  1. Delta hashes are correctly preventing the pipeline from processing unchanged GitHub Action timeline touches.
  2. The DreamService unchanged ticket logic has successfully proven perfect hash alignment, preventing duplicate ingestion loops.

Resolves #9796.

tobiu closed this issue on Apr 8, 2026, 9:17 PM