Overview
The DevIndex automation pipeline originally only scraped opt-in/opt-out statuses and generated the users.jsonl output. However, the neomjs/portal application relies on an array of generated indices (tickets.json, labels.json, releases.json) and raw Markdown ticket content (resources/content/issues and issue-archive) to natively render live tracking without needing a full framework version bump. Additionally, SEO files like sitemap.xml and llms.txt should represent the live state of the site.
Task
Rename and harden the existing devindex-pipeline.yml into a fully unified data-sync-pipeline.yml.
Architecture Additions:
- Rebuild Indexes & SEO: Automatically loop in
buildScripts/docs/index/*.mjs and buildScripts/docs/seo/generate.mjs to keep our knowledge index and LLM map fresh.
- Content Synchronization: Ensure the raw markdown ticket contents (
issues, issue-archive) are correctly copied to the neomjs/pages GitHub Pages root so that the Portal Ticket Explorer does not hit a 404 when querying newly created tickets.
- Atomicity: The pipeline will commit all locally generated indexes to the
neo repo, and propagate them symmetrically to neomjs/pages via a single bot push.
Avoided Pitfalls
Using cp -r for resources/content/* initially poses a problem since a ticket moving from issues to issue-archive would not be natively deleted in temp_pages. We avoid ghost collisions by running a clean rm -rf on the target directories inside the clone, copying fresh, and using git add -A node_modules/neo.mjs/resources/content/ to let git natively resolve diffs and deletions.
Overview
The DevIndex automation pipeline originally only scraped opt-in/opt-out statuses and generated the
users.jsonloutput. However, theneomjs/portalapplication relies on an array of generated indices (tickets.json,labels.json,releases.json) and raw Markdown ticket content (resources/content/issuesandissue-archive) to natively render live tracking without needing a full framework version bump. Additionally, SEO files likesitemap.xmlandllms.txtshould represent the live state of the site.Task
Rename and harden the existing
devindex-pipeline.ymlinto a fully unifieddata-sync-pipeline.yml.Architecture Additions:
buildScripts/docs/index/*.mjsandbuildScripts/docs/seo/generate.mjsto keep our knowledge index and LLM map fresh.issues,issue-archive) are correctly copied to theneomjs/pagesGitHub Pages root so that the Portal Ticket Explorer does not hit a 404 when querying newly created tickets.neorepo, and propagate them symmetrically toneomjs/pagesvia a single bot push.Avoided Pitfalls
Using
cp -rforresources/content/*initially poses a problem since a ticket moving fromissuestoissue-archivewould not be natively deleted intemp_pages. We avoid ghost collisions by running a cleanrm -rfon the target directories inside the clone, copying fresh, and usinggit add -A node_modules/neo.mjs/resources/content/to let git natively resolve diffs and deletions.