LearnNewsExamplesServices
Frontmatter
id9869
titleUnified Data Sync Pipeline (DevIndex, Portal Indexes, and SEO)
stateClosed
labels
enhancementaibuild
assignees[]
createdAtApr 10, 2026, 7:32 PM
updatedAtApr 10, 2026, 7:35 PM
githubUrlhttps://github.com/neomjs/neo/issues/9869
authortobiu
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtApr 10, 2026, 7:35 PM

Unified Data Sync Pipeline (DevIndex, Portal Indexes, and SEO)

Closedenhancementaibuild
tobiu
tobiu commented on Apr 10, 2026, 7:32 PM

Overview

The DevIndex automation pipeline originally only scraped opt-in/opt-out statuses and generated the users.jsonl output. However, the neomjs/portal application relies on an array of generated indices (tickets.json, labels.json, releases.json) and raw Markdown ticket content (resources/content/issues and issue-archive) to natively render live tracking without needing a full framework version bump. Additionally, SEO files like sitemap.xml and llms.txt should represent the live state of the site.

Task

Rename and harden the existing devindex-pipeline.yml into a fully unified data-sync-pipeline.yml.

Architecture Additions:

  1. Rebuild Indexes & SEO: Automatically loop in buildScripts/docs/index/*.mjs and buildScripts/docs/seo/generate.mjs to keep our knowledge index and LLM map fresh.
  2. Content Synchronization: Ensure the raw markdown ticket contents (issues, issue-archive) are correctly copied to the neomjs/pages GitHub Pages root so that the Portal Ticket Explorer does not hit a 404 when querying newly created tickets.
  3. Atomicity: The pipeline will commit all locally generated indexes to the neo repo, and propagate them symmetrically to neomjs/pages via a single bot push.

Avoided Pitfalls

Using cp -r for resources/content/* initially poses a problem since a ticket moving from issues to issue-archive would not be natively deleted in temp_pages. We avoid ghost collisions by running a clean rm -rf on the target directories inside the clone, copying fresh, and using git add -A node_modules/neo.mjs/resources/content/ to let git natively resolve diffs and deletions.

tobiu added the enhancement label on Apr 10, 2026, 7:32 PM
tobiu added the ai label on Apr 10, 2026, 7:32 PM
tobiu added the build label on Apr 10, 2026, 7:32 PM
tobiu referenced in commit 07e8572 - "feat: Unified Data Sync Pipeline (#9869)" on Apr 10, 2026, 7:32 PM
tobiu cross-referenced by PR #9870 on Apr 10, 2026, 7:32 PM
tobiu referenced in commit a7e9b73 - "feat: Unified Data Sync Pipeline (#9869) (#9870)" on Apr 10, 2026, 7:35 PM
tobiu closed this issue on Apr 10, 2026, 7:35 PM