LearnNewsExamplesServices
Frontmatter
id10576
titleKB sync observability: tee MCP child logger output to .neo-ai-data/logs/ for tail-able progress
stateClosed
labels
enhancementaiarchitecture
assigneesneo-opus-4-7
createdAtMay 1, 2026, 3:03 PM
updatedAtMay 1, 2026, 3:46 PM
githubUrlhttps://github.com/neomjs/neo/issues/10576
authorneo-opus-4-7
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtMay 1, 2026, 3:46 PM

KB sync observability: tee MCP child logger output to .neo-ai-data/logs/ for tail-able progress

Closedenhancementaiarchitecture
neo-opus-4-7
neo-opus-4-7 commented on May 1, 2026, 3:03 PM

Context

PR #10573 introduces the work-volume-aware gate on manage_knowledge_base sync (refuses MCP-callable sync when chunksToProcess > mcpSyncMaxChunks). The gate refuses synchronous work and points operators at npm run ai:sync-kb for bulk re-embedding — but neither path produces a tail-able progress log.

Empirical anchor (2026-05-01): Gemini's harness triggered a full from-scratch KB sync via the MCP tool. The sync ran for 3+ hours server-side. During that interval:

  • Gemini's harness turn timed out at 90 minutes (her local inactivity timeout) but the MCP child she spawned kept grinding
  • No human (or peer agent) had visibility into progress — VectorService.embed is logging via logger.log for each batch, but those go to the harness-captured stderr of the spawned MCP child, not anywhere tail -f-able
  • Memory Core add_memory calls timed out repeatedly during this window (Chroma embedding-write contention) — a confounded substrate effect that nobody could attribute confidently without progress visibility
  • Tobiu noted directly: "if i had started it inside a terminal, i would see progress logs. we are blind."

This is the canonical observability gap: the gate moves bulk work off the MCP path, but there's no substrate to observe the work it shoved off. CLI scripts inherit terminal stdout but MCP-spawned children do not.

Proposed Fix

Tee the KB MCP server's logger output to a deterministic file path so progress is observable regardless of who initiated the sync.

Concretely, in ai/mcp/server/knowledge-base/logger.mjs (and analogous Memory Core / Neural Link if symmetry desired in a follow-up), write log entries to both stderr (current behavior, harness-captured) AND an append-mode file stream at:

${aiConfig.neoRootDir}/.neo-ai-data/logs/kb-sync-${YYYY-MM-DD}.log

(or a single rolling file if simpler — daily-rotated keeps it bounded.)

After the change, any sync — MCP-triggered, CLI-triggered, agent-initiated, human-initiated — produces a tail-able log:

tail -f .neo-ai-data/logs/kb-sync-2026-05-01.log

Update the gate refusal payload's message field to point operators at the log path:

message: `${chunksToProcess.length} chunks need re-embedding (threshold: ${mcpThreshold}). Run via CLI: \`npm run ai:sync-kb\`. Tail progress: \`tail -f .neo-ai-data/logs/kb-sync-*.log\`.`

Acceptance Criteria

  1. KB MCP server's logger.log writes simultaneously to stderr (current) and the deterministic log file path under .neo-ai-data/logs/
  2. The log path is configurable via aiConfig.logPath (default: ${aiConfig.neoRootDir}/.neo-ai-data/logs/); creating the directory is idempotent
  3. CLI npm run ai:sync-kb ALSO writes to the same log file path (so terminal-stdout view + tail view are equivalent)
  4. The gate refusal message in VectorService.embed references the log path so the agent reading the error knows where to direct the operator
  5. .neo-ai-data/logs/ is gitignored (it almost certainly already is via the .neo-ai-data/ parent)

Avoided Traps

  • NOT a background-task substrate. Larger framing (poll/tail tool, lifecycle primitives, in-flight-task introspection) is a separate epic-shape concern, overlapping with #10448's work-volume framing. This ticket scopes deliberately to the observability primitive: "make the existing logger output visible from a terminal." Background-task substrate stays out.

  • NOT a logger refactor. Don't introduce a logging framework, structured-log format, log-level routing, etc. The current logger.log/info/warn/error shape is fine — this ticket only adds a file-stream sink alongside the existing stderr sink.

  • NOT a Memory Core / Neural Link parallel scope. Apply to KB MCP only here; symmetric replication to other MCPs is a follow-up if/when their long-running operations surface similar blindness.

Origin

  • Empirical surface: 2026-05-01 KB resync incident (Gemini's harness triggered full from-scratch sync via MCP; 3+ hours of agent + human blindness)
  • Tobiu's direct quote: "if i had started it inside a terminal, i would see progress logs. we are blind."
  • Origin Session ID: 7a2c3c2a-d0f1-462a-8489-69b031221040
tobiu referenced in commit b28e3a8 - "feat(kb): always-on file sink for KB MCP server logger (#10576) (#10580) on May 1, 2026, 3:46 PM
tobiu closed this issue on May 1, 2026, 3:46 PM