LearnNewsExamplesServices
Frontmatter
id13260
titleCouple the index+SEO derive to the content sync — extract a shared bundle reused by prepare.mjs, the data-sync pipeline, and a post-sync_all hook
stateClosed
labels
enhancementairefactoringbuild
assigneesneo-gpt
createdAtJun 14, 2026, 10:06 PM
updatedAtJun 21, 2026, 2:17 AM
githubUrlhttps://github.com/neomjs/neo/issues/13260
authorneo-opus-grace
commentsCount2
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 15, 2026, 12:27 AM

Couple the index+SEO derive to the content sync — extract a shared bundle reused by prepare.mjs, the data-sync pipeline, and a post-sync_all hook

Closed v13.1.0/archive-v13-1-0-chunk-2 enhancementairefactoringbuild
neo-opus-grace
neo-opus-grace commented on Jun 14, 2026, 10:06 PM

Context / Friction

GH_SyncService.runFullSync() — invoked by both ai:sync-github-workflow (CLI) and the MCP sync_all tool — emits resources/content/ (issues/PRs/discussions/release-notes) and stops. The sitemap + index JSONs are derived from that content (buildScripts/docs/index/*.mjs → index JSONs, buildScripts/docs/seo/generate.mjssitemap.xml / llms.txt), but nothing fires the derive. So the invariant "content and its derived index are coherent" silently breaks on every manual/MCP sync — observed live as a 12996-vs-13257 gap (content fresh, sitemap stale), which makes the sitemap-driven middleware SSG skip the newest tickets.

The data-sync-pipeline.yml only stays coherent because it happens to run both (DevIndex content + an inlined "Rebuild Indexes & SEO" step that duplicates the 7 node commands).

Grounded findings

  • The 4 content index builders + 2 SEO generators are local-only pure derives (no GH token/API; read resources/content via fast-glob) — verified. The label index builder is a separate remote GitHub/SDK derive and should stay explicit for release/CI callers rather than becoming part of the sync_all post-hook.
  • They are already importable functions (createTicketIndex, getSitemapXml, getLlmsTxt, …) — buildScripts/release/prepare.mjs lines 138-155 already calls them as a bundle (its "Category B"), alongside release-only version-bump work ("Category A").
  • Cost: index builders ~2.6s, SEO generator ~27s (CPU-bound) — cheap; generate.mjs is already shallow-clone-aware (hasReliableGitHistory() preserves existing lastmod to avoid hourly churn), so CI placement is fine.

The fix (option B — operator-confirmed)

  1. Extract prepare.mjs Category B (lines 138-155) into one named function — e.g. rebuildContentIndexesAndSeo() — a thin wrapper over the 7 already-imported functions.
  2. Reuse it in three callers:
    • prepare.mjs (release) — call it after the version bump.
    • data-sync-pipeline.yml — replace the 7 inlined node commands with one call (kills the duplication; drop the dead GH token on that step).
    • A post-sync_all hook in runFullSync()'s orchestration — so every content sync (CLI and MCP) regenerates the derived artifacts.
  3. (Defense-in-depth, optional) make the middleware build.mjs derive its route list from the content tree, so a stale sitemap can't drop content.

Design decision to settle

Placement of the post-sync hook: runFullSync is doc-scoped to agent-OS content emission, so the derive should not be buried inside it (concern-mixing). Prefer a thin orchestration wrapper / post-emit hook that both the CLI and the MCP sync_all handler flow through — covering the MCP path is a hard requirement, or an agent syncing via sync_all re-opens the gap.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
rebuildContentIndexesAndSeo() shared derive helper (new) This ticket + current buildScripts/release/prepare.mjs derive bundle Export one named async helper that runs the local content index builders and regenerates apps/portal/sitemap.xml + apps/portal/llms.txt with canonical base URL https://neomjs.com; release/CI callers may opt into the remote label index explicitly If any selected derive fails, reject/throw so the caller fails instead of publishing incoherent derived artifacts JSDoc on the helper naming generated artifacts, includeLabelIndex, and failure behavior Focused unit coverage for helper sequencing/outputs plus node --check
buildScripts/release/prepare.mjs release path Current release script order: version/package work first, then index+SEO derive Replace the inline Category B derive block with a call to the shared helper after release-only version work, preserving release-only behavior by passing includeLabelIndex: true Helper failure preserves current fail-fast release behavior Release script import/call remains self-explanatory; helper JSDoc owns detail Unit or static assertion that prepare.mjs imports/calls the shared helper; no duplicate inline seven-command bundle remains
.github/workflows/data-sync-pipeline.yml Rebuild Indexes & SEO step Current workflow step duplicates seven node commands after DevIndex update; labels.mjs uses GH_LabelService and therefore still needs GitHub credentials Replace the inline seven commands with one repo-local helper invocation/command using --include-labels; keep GITHUB_TOKEN for the label derive because source inspection proves it is needed If helper exits non-zero, the workflow fails before committing stale derived artifacts Workflow step name may stay; no additional docs required Static test/grep assertion that the workflow uses the shared command and no longer contains the duplicated seven derive commands
CLI GitHub Workflow sync path (ai/scripts/maintenance/syncGithubWorkflow.mjs) CLI currently invokes GH_SyncService.runFullSync() via the canonical SDK route A successful CLI content sync must run the shared local-only content derive after content emission and before reporting success to the operator If content sync fails, do not run derive; if derive fails after content sync, surface a non-zero/failed outcome rather than silent degradation JSDoc/comment at the orchestration boundary if a wrapper is introduced Focused unit coverage around the shared orchestration wrapper; CLI path delegates through it
MCP sync_all path (ai/mcp/server/github-workflow/toolService.mjs) sync_all currently delegates through the dev-branch guard to SyncService.runFullSync() A successful MCP sync_all leaves content and derived index/SEO artifacts coherent via the local-only helper default; it must not import or execute the remote label derive If derive fails, the MCP call rejects/fails rather than returning success with stale sitemap/index data OpenAPI description only if observable semantics are changed; otherwise helper/orchestration JSDoc Unit/integration assertion that the MCP delegate path reaches the shared post-sync derive wrapper

Acceptance Criteria

  • A single rebuildContentIndexesAndSeo() (or equivalent) is the one source of truth for the index+SEO derive.
  • prepare.mjs, the data-sync pipeline, and the content-sync path all call it — no inline duplication remains.
  • A content sync via the MCP sync_all tool leaves the sitemap coherent with the emitted content (the 12996-vs-13257 gap cannot recur).

Evidence: L2 (unit coverage of the extracted bundle + a coherence assertion that a sync leaves sitemap routes == content) → L2 required (deterministic local derive).

Refs #12964 (the middleware automation umbrella this serves).

Authored by Claude Opus 4.8 (Claude Code), @neo-opus-grace (Grace).