Post-release: automate middleware SSR rebuild + Cloud Run deploy (8h local)

Context

Operator directive (2026-06-12, v13 wrap night): the public site's SSR layer (private repo neomjs/middleware-v2) must follow neo's content cadence automatically. Today neo's content moves hourly (.github/workflows/data-sync-pipeline.yml regenerates the sitemap + portal data and pushes to dev), while SSR route regeneration + Cloud Run deployment happen manually in bursts (weeks apart). The v13 cut runbook (epic #12696 comments, steps 4-5) covers the one-time post-publish refresh; no ticket existed for the recurring automation — this is it.

Release classification: post-release — explicitly out of the v13 cut; the cut's one-time refresh remains runbook step 4.

The Problem

Constraint triangle, operator-stated and verified tonight:

• GitHub Actions cannot generate routes: the SSG boots one full Neo app per route (~12.5s avg/route, 120s cap); a backlog wave is thousands of routes (~2h at 14-way concurrency on an 18-core host) — far beyond hosted-runner economics.
• Cloud-side generation is the expensive path for the same reason — thousands of app boots per wave.
• ⇒ generation is local-only. What remains automatable: a local scheduled rebuild + a deploy path that doesn't depend on a human.

Tonight's deep-map (4-agent sweep over both repos) additionally found the current chain is not automatable as-is:

1. The Actions deploy is broken at current scale: the image build COPYs the full prerendered tree (~7.5 GB, ~12.3k pages); the last deploy run died on runner disk exhaustion — production currently serves a build several content-generations old, and the most recent route commit never shipped.
2. The raw-markdown layer is stale and old-shape: dist/raw predates the chunk-tree rename (legacy issue-archive/ layout, ~2 months old) — archived-ticket /raw requests 404 in prod today; pulls/discussions content (now in neo's tree) has zero /raw coverage.
3. The deploy script is unsafe unattended: blanket git add .; commit-failure detection that can never fire (inherited stdio makes the "nothing to commit" stdout check dead — real failures exit 0 silently); per-route render failures logged but exit 0; no lockfile; never pulls the sibling neo checkout (split freshness: routes/sitemap from the LOCAL tree, raw md from a fresh REMOTE clone).
4. Push-size ceiling: full-regen commits are multi-GB raw; the last one required manual 3-way splitting to push at all.

The Architectural Reality (what already works for us)

• The SSG is already incremental and short-circuits: a route→lastmod cache rebuilds only changed routes and exits immediately when the sitemap is unchanged — an 8h timer is effectively change-gated for free. Caveat: sitemap lastmods are bulk-stamped (few distinct values), so content waves rebuild in large groups, not trickles.
• neo-side lastmod generation is already cache-friendly (git-history-derived with stable fallback, explicitly so hourly data-sync doesn't churn site-wide lastmods).
• The container never reads git at runtime — committed-dist is load-bearing only because the image builds from an Actions checkout. All image inputs exist locally after a build ⇒ deploy-from-disk is architecturally clean (cloud image-assembly of a prebuilt tree is minutes of build time — cheap; it is route generation in the cloud that's expensive, and that stays local).
• Measured steady-state: ~67 routes/min at default concurrency; no-change cycles cost seconds.

The Fix (phased)

Phase 0 — one-time reconciliation (attended, post-v13-publish; extends runbook step 4):

• Re-shape the raw layer: clean re-fetch into the chunk-tree layout; add pulls + discussions to the fetch set and to the server's raw-index walk + dynamic id-resolution; delete legacy-layout trees (also removes the duplicate-basename last-write-wins hazard).
• Re-sync the theme-map snapshot from neo (stale vs v13 — new/renamed components silently lose CSS otherwise).
• Retire the dead pre-chunk fetch-mock branch in the SSR service.
• Dry-run a handful of never-rendered pulls/discussions routes against the readiness heuristics before bulk-building (a non-hydrating view costs the full settle-cap per route, thousands of times).
• First full rebuild + first deploy via the Phase-2 direct path (the Actions path cannot ship current size).

Phase 1 — harden the pipeline for unattended runs:

• Lockfile/overlap guard; explicit pathspec staging (prerendered tree + caches only); real commit-failure detection (git diff --cached --quiet semantics); failure-count threshold that aborts deploy (no more exit-0-with-holes); pre-run git pull of the sibling neo checkout gated on a clean tree; unattended concurrency cap flag (host-melt precedent exists); structured run-log + completion/failure notification.

Phase 2 — the 8h automation (local, macOS):

• LaunchAgent (not naive cron: survives sleep via scheduled wake or caffeinate-wrapped runs, ProcessType: Background), every 8h: pull neo → fetch content (from the local tree — unify the split freshness paths) → incremental build → threshold-gated deploy-from-disk to Cloud Run (cloud image-build of the prebuilt tree, or local docker build+push) — demoting the git push to optional archival and bypassing both the runner-disk ceiling and the pack-size ceiling.
• Post-deploy health check (healthz + one sampled bot route + one /raw route).

Phase 3 — hygiene:

• Image-registry cleanup policy (per-deploy images accrue with no pruning).
• Evaluate untracking the prerendered tree from git entirely (repo holds sources + caches; disk holds artifacts) once deploy-from-disk is proven.

Contract Ledger Matrix

Acceptance Criteria

• Phase 0: prod serves current-generation content on all three surfaces (bot HTML, /raw incl. archived + pulls/discussions, human proxy untouched); stale-layout trees removed.
• Phase 1: an injected route-failure aborts the deploy; an overlapping run is skipped; a dirty neo checkout halts with a clear log line.
• Phase 2: two consecutive unattended cycles observed — one with content changes (deploys) and one without (short-circuits, no deploy).
• Phase 3: registry holds a bounded image set.

Out of Scope

• neomjs/pages updater (runbook step 5 — npm-registry-keyed, separate cadence).
• Route generation in CI/cloud (the constraint this design exists to respect).
• The v13 cut itself (runbook owns it).

Related

• Epic #12696 (cut runbook steps 4-5 comments) · neo .github/workflows/data-sync-pipeline.yml (the upstream hourly cadence) · private repo neomjs/middleware-v2 (implementation surface — specifics live there)

Live latest-open sweep: checked latest 20 open issues at 2026-06-12T05:20Z; no equivalent (closest: #10757 periodic-audit cron, different domain). A2A in-flight sweep: clean — the active release-readiness audit lane (Mnemosyne) AUDITS the manual runbook; this ticket owns the recurring automation; findings handed to her separately.

Origin Session ID: c29b5ccc-d711-4cde-8401-32d1e4bbc4f1

Retrieval Hint: "middleware SSR rebuild automation 8h local LaunchAgent deploy-from-disk Cloud Run runner disk ceiling"