Context
PR #10573 introduces the work-volume-aware gate on manage_knowledge_base sync (refuses MCP-callable sync when chunksToProcess > mcpSyncMaxChunks). The gate refuses synchronous work and points operators at npm run ai:sync-kb for bulk re-embedding — but neither path produces a tail-able progress log.
Empirical anchor (2026-05-01): Gemini's harness triggered a full from-scratch KB sync via the MCP tool. The sync ran for 3+ hours server-side. During that interval:
- Gemini's harness turn timed out at 90 minutes (her local inactivity timeout) but the MCP child she spawned kept grinding
- No human (or peer agent) had visibility into progress —
VectorService.embed is logging via logger.log for each batch, but those go to the harness-captured stderr of the spawned MCP child, not anywhere tail -f-able
- Memory Core
add_memory calls timed out repeatedly during this window (Chroma embedding-write contention) — a confounded substrate effect that nobody could attribute confidently without progress visibility
- Tobiu noted directly: "if i had started it inside a terminal, i would see progress logs. we are blind."
This is the canonical observability gap: the gate moves bulk work off the MCP path, but there's no substrate to observe the work it shoved off. CLI scripts inherit terminal stdout but MCP-spawned children do not.
Proposed Fix
Tee the KB MCP server's logger output to a deterministic file path so progress is observable regardless of who initiated the sync.
Concretely, in ai/mcp/server/knowledge-base/logger.mjs (and analogous Memory Core / Neural Link if symmetry desired in a follow-up), write log entries to both stderr (current behavior, harness-captured) AND an append-mode file stream at:
${aiConfig.neoRootDir}/.neo-ai-data/logs/kb-sync-${YYYY-MM-DD}.log
(or a single rolling file if simpler — daily-rotated keeps it bounded.)
After the change, any sync — MCP-triggered, CLI-triggered, agent-initiated, human-initiated — produces a tail-able log:
tail -f .neo-ai-data/logs/kb-sync-2026-05-01.log
Update the gate refusal payload's message field to point operators at the log path:
message: `${chunksToProcess.length} chunks need re-embedding (threshold: ${mcpThreshold}). Run via CLI: \`npm run ai:sync-kb\`. Tail progress: \`tail -f .neo-ai-data/logs/kb-sync-*.log\`.`
Acceptance Criteria
- KB MCP server's
logger.log writes simultaneously to stderr (current) and the deterministic log file path under .neo-ai-data/logs/
- The log path is configurable via
aiConfig.logPath (default: ${aiConfig.neoRootDir}/.neo-ai-data/logs/); creating the directory is idempotent
- CLI
npm run ai:sync-kb ALSO writes to the same log file path (so terminal-stdout view + tail view are equivalent)
- The gate refusal message in
VectorService.embed references the log path so the agent reading the error knows where to direct the operator
.neo-ai-data/logs/ is gitignored (it almost certainly already is via the .neo-ai-data/ parent)
Avoided Traps
NOT a background-task substrate. Larger framing (poll/tail tool, lifecycle primitives, in-flight-task introspection) is a separate epic-shape concern, overlapping with #10448's work-volume framing. This ticket scopes deliberately to the observability primitive: "make the existing logger output visible from a terminal." Background-task substrate stays out.
NOT a logger refactor. Don't introduce a logging framework, structured-log format, log-level routing, etc. The current logger.log/info/warn/error shape is fine — this ticket only adds a file-stream sink alongside the existing stderr sink.
NOT a Memory Core / Neural Link parallel scope. Apply to KB MCP only here; symmetric replication to other MCPs is a follow-up if/when their long-running operations surface similar blindness.
Origin
- Empirical surface: 2026-05-01 KB resync incident (Gemini's harness triggered full from-scratch sync via MCP; 3+ hours of agent + human blindness)
- Tobiu's direct quote: "if i had started it inside a terminal, i would see progress logs. we are blind."
- Origin Session ID: 7a2c3c2a-d0f1-462a-8489-69b031221040
Context
PR #10573 introduces the work-volume-aware gate on
manage_knowledge_basesync (refuses MCP-callable sync whenchunksToProcess > mcpSyncMaxChunks). The gate refuses synchronous work and points operators atnpm run ai:sync-kbfor bulk re-embedding — but neither path produces a tail-able progress log.Empirical anchor (2026-05-01): Gemini's harness triggered a full from-scratch KB sync via the MCP tool. The sync ran for 3+ hours server-side. During that interval:
VectorService.embedis logging vialogger.logfor each batch, but those go to the harness-captured stderr of the spawned MCP child, not anywheretail -f-ableadd_memorycalls timed out repeatedly during this window (Chroma embedding-write contention) — a confounded substrate effect that nobody could attribute confidently without progress visibilityThis is the canonical observability gap: the gate moves bulk work off the MCP path, but there's no substrate to observe the work it shoved off. CLI scripts inherit terminal stdout but MCP-spawned children do not.
Proposed Fix
Tee the KB MCP server's logger output to a deterministic file path so progress is observable regardless of who initiated the sync.
Concretely, in
ai/mcp/server/knowledge-base/logger.mjs(and analogous Memory Core / Neural Link if symmetry desired in a follow-up), write log entries to both stderr (current behavior, harness-captured) AND an append-mode file stream at:${aiConfig.neoRootDir}/.neo-ai-data/logs/kb-sync-${YYYY-MM-DD}.log(or a single rolling file if simpler — daily-rotated keeps it bounded.)
After the change, any sync — MCP-triggered, CLI-triggered, agent-initiated, human-initiated — produces a tail-able log:
tail -f .neo-ai-data/logs/kb-sync-2026-05-01.logUpdate the gate refusal payload's
messagefield to point operators at the log path:message: `${chunksToProcess.length} chunks need re-embedding (threshold: ${mcpThreshold}). Run via CLI: \`npm run ai:sync-kb\`. Tail progress: \`tail -f .neo-ai-data/logs/kb-sync-*.log\`.`Acceptance Criteria
logger.logwrites simultaneously to stderr (current) and the deterministic log file path under.neo-ai-data/logs/aiConfig.logPath(default:${aiConfig.neoRootDir}/.neo-ai-data/logs/); creating the directory is idempotentnpm run ai:sync-kbALSO writes to the same log file path (so terminal-stdout view + tail view are equivalent)VectorService.embedreferences the log path so the agent reading the error knows where to direct the operator.neo-ai-data/logs/is gitignored (it almost certainly already is via the.neo-ai-data/parent)Avoided Traps
NOT a background-task substrate. Larger framing (poll/tail tool, lifecycle primitives, in-flight-task introspection) is a separate epic-shape concern, overlapping with #10448's work-volume framing. This ticket scopes deliberately to the observability primitive: "make the existing logger output visible from a terminal." Background-task substrate stays out.
NOT a logger refactor. Don't introduce a logging framework, structured-log format, log-level routing, etc. The current
logger.log/info/warn/errorshape is fine — this ticket only adds a file-stream sink alongside the existing stderr sink.NOT a Memory Core / Neural Link parallel scope. Apply to KB MCP only here; symmetric replication to other MCPs is a follow-up if/when their long-running operations surface similar blindness.
Origin