Reduce per-issue log volume during gh-workflow sync (downgrade per-item INFO to DEBUG + add batch-summary checkpoints)

Context

Post-#11470-merge operator-side npm run ai:sync-github-workflow run produced "very intense" per-item log output during the clean-slate sync. Operator's empirical observation 2026-05-16: the log volume "would crush your context window" — an agent re-running the sync from a Claude Code / Antigravity / Codex harness would burn its full context budget on per-issue Created/Moved/Updated lines before the sync completes.

This is an MX-friction issue: the canonical ai:sync-github-workflow CLI surface is operator-trusted (filed via PR #11470 specifically to escape MCP-request-timeout constraints), but the log surface is currently shaped for human terminal-watching rather than agent-runnable invocation.

Body revision 2026-05-16 (after @neo-gpt V-B-A correction): The initial prescription was wrong-shape. The real root cause is that ai/mcp/server/github-workflow/logger.mjs has no level filtering — it gates ALL levels (debug/info/warn/error) behind a single aiConfig.debug boolean. The CLI script buildScripts/ai/syncGithubWorkflow.mjs sets GH_Config.data.debug = true, so every logger.X call prints regardless of level. Simply moving per-item logs from info → debug does NOT reduce noise.

Worse: when aiConfig.debug = false, the logger is completely silent — even logger.error() calls are suppressed. That's a separate substrate bug (errors should always print) but worth naming here since the level-filtering fix touches the same module.

Problem

Two coupled substrate concerns:

1. logger.mjs lacks level filtering. The gate is binary on/off via aiConfig.debug. Level granularity (debug/info/warn/error) exists in the API but doesn't affect output — all-or-nothing semantics.

2. Per-item log volume on full clean-slate sync is operationally impractical for agent invocation. Theoretical ceiling:

   • ~8,500 issues × 3 possible log lines per item (Created / Moved / Updated)
   • ~2,800 PRs × 3 same shapes
   • ~165 discussions × 1-2
   • ~166 release notes × 1-2

   That's ~35k+ log lines in a single sync run.

Both must be fixed together — fixing the logger without rebalancing the per-item logs leaves the agent-runnable invocation unchanged. Fixing the per-item logs without level filtering leaves operators with no way to opt into per-item visibility.

Architectural Reality

ai/mcp/server/github-workflow/logger.mjs (full text):

const createLogMethod = (level) => {
    return (...args) => {
        if (aiConfig.debug) {
            console.error(`[${level.toUpperCase()}]`, ...args);
        }
    };
};

logger.debug = createLogMethod('debug');
logger.info  = createLogMethod('info');
logger.log   = createLogMethod('log');
logger.warn  = createLogMethod('warn');
logger.error = createLogMethod('error');

Levels are tagged at log-time but ignored at filter-time. Level filtering needs to be added.

The CLI script's GH_Config.data.debug = true is the explicit visibility opt-in for operator-side runs; the MCP server context uses aiConfig.debug = false to keep stdio clean.

The Fix (revised per V-B-A)

Two-part substrate change:

Part 1: Add level filtering to logger.mjs

const LEVEL_PRIORITY = {error: 0, warn: 1, info: 2, log: 2, debug: 3};

const createLogMethod = (level) => {
    return (...args) => {
        // Always print errors regardless of debug flag (fail-loud principle).
        if (level === 'error') {
            console.error(`[${level.toUpperCase()}]`, ...args);
            return;
        }
        // Other levels respect aiConfig.logLevel (or fall back to current binary behavior).
        const configured = aiConfig.logLevel || (aiConfig.debug ? 'debug' : 'warn');
        if (LEVEL_PRIORITY[level] <= LEVEL_PRIORITY[configured]) {
            console.error(`[${level.toUpperCase()}]`, ...args);
        }
    };
};

This:

• Always prints error (fixes the silent-error bug).
• Respects aiConfig.logLevel if set (new config field).
• Falls back to binary debug behavior if logLevel is absent — preserves existing semantics for callers that haven't migrated to per-level config.
• Per-syncer per-item logs at info are emitted by default if logLevel: 'info' (current behavior); operator can quiet via logLevel: 'warn' or amplify via logLevel: 'debug'.

Part 2: Add logLevel config field + sensible defaults

• ai/mcp/server/github-workflow/config.template.mjs: add logLevel: 'warn' as the substrate default (quiet by default; phase headers + warnings + errors only).
• buildScripts/ai/syncGithubWorkflow.mjs: remove GH_Config.data.debug = true; replace with GH_Config.data.logLevel = 'info' (phase headers + per-item visible during CLI sync). Optional CLI flag --verbose → set logLevel: 'debug' for full diagnostic output.
• Optionally environment variable NEO_LOG_LEVEL override.

Part 3: Move per-item events to info where they already aren't (or keep at info)

Per-item Created/Moved/Updated events stay at info (their current level) — they were already correctly tagged; the issue was logger.mjs not filtering. With logLevel: 'warn' substrate default, MCP-server mode emits only phase-headers + warns/errors. With logLevel: 'info' CLI default, operator sees per-item progress.

Acceptance Criteria

• ai/mcp/server/github-workflow/logger.mjs respects a logLevel config (priority-based filtering: error < warn < info/log < debug).
• logger.error() calls print regardless of logLevel setting (fail-loud invariant).
• ai/mcp/server/github-workflow/config.template.mjs adds logLevel: 'warn' as substrate default — preserves quiet MCP-server stdio behavior.
• buildScripts/ai/syncGithubWorkflow.mjs removes the GH_Config.data.debug = true line; replaces with GH_Config.data.logLevel = 'info' (or equivalent; operator can pass --verbose for debug-level).
• (Optional) NEO_LOG_LEVEL env var override for ad-hoc runs.
• Backward compatibility: callers that still set aiConfig.debug = true continue to get all levels (debug=true maps to logLevel='debug' if logLevel is not explicitly set).
• Manual verification: npm run ai:sync-github-workflow produces phase-header + per-item INFO lines without flooding (since logLevel: 'info' filters debug-only chatter).
• Unit test for logger.mjs priority filtering: verifies logLevel: 'warn' suppresses info + debug but emits error + warn; verifies error always prints.

Out of Scope

• Migrating callers to use logger.debug for per-item events instead of info — reverse-direction of original prescription; not needed once level filtering is real.
• Wider logger refactor (output sinks, file logging, structured JSON output) — narrow to the level-filtering gap.
• Same pattern in KB/Memory-Core/Neural-Link loggers — sibling concern; file separately if friction surfaces.

Avoided Traps

• Single hardcoded LOG_LEVEL thresholds in logger.mjs: keep it minimal — error<warn<info<debug ordering is standard; no need for finer granularity in v1.
• Removing all per-item logs entirely: rejected — operators occasionally need per-item visibility for debugging stuck syncs. Preserve via logLevel: 'info' or debug opt-in.
• Switching to a progress-bar UI: rejected — must work non-interactively.
• (Original prescription) Moving per-item logs from .info → .debug without first adding level filtering: rejected per @neo-gpt V-B-A 2026-05-16 17:31Z — the level renaming would be invisible to operators because logger.mjs doesn't filter by level.

Related

• Parent context: PR #11470 merged 2026-05-16 (ai:sync-github-workflow CLI enabler) — operator's first full-sync run surfaced this friction
• Adjacent: #11474 / PR #11476 — defensive null-safety in IssueSyncer.#formatTimelineEvent (the other surfaced friction from the same sync run)
• V-B-A correction credit: @neo-gpt 2026-05-16T17:31Z — falsified the original info → debug prescription by surfacing logger.mjs's missing level-filter, prevented wrong-shape implementation
• Authority anchor: operator-stated framing 2026-05-16: "very intense [logs] ... triggering the script on your own would crush your context window"

Origin Session ID: 0064efde-455e-4ecd-a26f-574381b3766a

Retrieval Hint: query_raw_memories(query="ai:sync-github-workflow log volume logger level filtering aiConfig.debug binary gate level filtering MX friction")