LearnNewsExamplesServices
Frontmatter
id12774
titleAdd Knowledge Base runtime freshness to healthcheck
stateClosed
labels
enhancementaiarchitecturemodel-experience
assigneesneo-gpt
createdAtJun 8, 2026, 5:40 PM
updatedAtJun 8, 2026, 8:13 PM
githubUrlhttps://github.com/neomjs/neo/issues/12774
authorneo-gpt
commentsCount0
parentIssue12740
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 8, 2026, 8:13 PM

Add Knowledge Base runtime freshness to healthcheck

Closed v13.0.0/archive-v13-0-0-chunk-17 enhancementaiarchitecturemodel-experience
neo-gpt
neo-gpt commented on Jun 8, 2026, 5:40 PM

Context

During the v13 cost-safety finish pass on 2026-06-08, two already-merged MCP contract fixes still appeared absent from the live tool surfaces:

  • GitHub Workflow list_issues source already supports lean projections from #12706, but the active MCP tool schema exposed in this harness still lacked the projection argument.
  • Knowledge Base SearchService.ask() source already provider-routes synthesis via buildChatModel() from #12741, but a live ask_knowledge_base call in this session still attempted gemini-3.5-flash through the deleted Gemini key and failed with API_KEY_INVALID.

V-B-A evidence from this creation pass:

  • ai/services/github-workflow/HealthService.mjs includes runtimeFreshness support from #12765.
  • Live github-workflow.healthcheck() returned healthy GitHub CLI status but no runtimeFreshness block, proving the active GitHub Workflow MCP runtime is stale relative to the checked-out source.
  • ai/services/knowledge-base/SearchService.mjs imports buildChatModel and reads aiConfig.modelProvider, confirming the source-side provider-routing fix exists.
  • Live knowledge-base.healthcheck() returned healthy Chroma/embedding status but no runtime freshness metadata.
  • ai/services/knowledge-base/HealthService.mjs does not contain runtimeFreshness, gitHead, openApiDigest, or configDigest diagnostics.
  • Live duplicate sweep: checked current open issues on 2026-06-08 and found adjacent but non-equivalent #12765 (GitHub Workflow runtime freshness) and #12772 (Memory Core runtime freshness). No KB-specific healthcheck runtime-freshness leaf exists.
  • Semantic ticket sweep with query_documents(type='ticket') did not surface an equivalent KB healthcheck freshness ticket.

The Problem

Knowledge Base can be infrastructure-healthy while still running stale source/config/tool-schema code. That distinction matters during v13 because KB is the default Neo knowledge tool and it directly intersects the cost-safety incident class:

  1. A stale KB process can keep using the old Gemini-only synthesis path even after provider-routing has merged.
  2. A stale KB OpenAPI/tool schema can hide newly merged tool contract changes from agents.
  3. The current KB healthcheck reports Chroma and embedding readiness only, so agents cannot falsify stale-runtime drift through the standard healthcheck path.

The result is duplicate diagnosis work: agents see a live KB behavior, inspect current source, find the source already fixed, and have no structured healthcheck field proving the process simply needs restart/reconnect.

The Architectural Reality

  • Owning healthcheck service: ai/services/knowledge-base/HealthService.mjs.
  • Owning MCP schema: ai/mcp/server/knowledge-base/openapi.yaml.
  • Runtime config surface: ai/mcp/server/knowledge-base/config.mjs and config.template.mjs.
  • Existing sibling precedent: ai/services/github-workflow/HealthService.mjs exposes boot-vs-current gitHead and openApiDigest; ai/services/memory-core/HealthService.mjs extends that shape with configDigest for ignored local config drift.
  • KB stale-runtime risk needs the Memory Core shape, not only the GitHub Workflow shape, because KB provider defaults can live in the gitignored migrated config overlay.

The Fix

Add a compact runtimeFreshness diagnostic to Knowledge Base healthcheck.

Expected implementation shape:

  • Capture boot-time runtime identity when HealthService.mjs is loaded: gitHead, configDigest, and openApiDigest.
  • On every direct healthcheck response, compare boot identity with the current checkout/config/OpenAPI identity.
  • Return runtimeFreshness.status as current, stale, or unknown plus field-level stale booleans, details, and a restart hint.
  • Treat stale runtime as observability metadata, not an outage. Do not flip top-level healthcheck status solely because runtime freshness is stale.
  • Keep the payload lean and unit-covered; do not add Chroma query canaries, mailbox previews, or broad diagnostics in this ticket.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
knowledge-base.healthcheck() payload HealthService.mjs + sibling runtime-freshness precedents Include runtimeFreshness with boot/current source identity comparison If reads fail, return status:'unknown' with bounded details OpenAPI schema + JSDoc Unit tests for current, stale, and unknown identity states
KB config freshness ai/mcp/server/knowledge-base/config.mjs / template Include configDigest so ignored local provider/config overlays can be detected If config file is unreadable, mark only that field unknown Healthcheck schema Unit test stubs config digest mismatch
KB OpenAPI/tool-schema freshness openapi.yaml Include openApiDigest so stale tool definitions are visible If OpenAPI file is unreadable, mark only that field unknown Healthcheck schema Unit test stubs OpenAPI digest mismatch
Agent restart decision v13 cost-safety workflow Agents can distinguish source-fixed-but-runtime-stale from a real source regression If runtime freshness is unavailable, agents still inspect source and restart manually PR body post-merge validation Manual post-merge healthcheck after KB MCP restart

Decision Record impact

none. This extends an existing MCP healthcheck observability pattern to the KB server and aligns with the local-first/cost-safety direction in #12740.

Acceptance Criteria

  • knowledge-base.healthcheck() returns a runtimeFreshness block with status, startedAt, boot, current, stale, details, and hint fields.
  • runtimeFreshness compares gitHead, configDigest, and openApiDigest.
  • runtimeFreshness.status:'stale' does not by itself change top-level healthcheck status from healthy.
  • Cached healthy healthchecks still return request-fresh runtime freshness, or the implementation documents why KB healthcheck cache semantics differ from the sibling services.
  • Unit coverage proves current, stale, and unknown runtime identity states without requiring live Chroma or GitHub network.
  • OpenAPI schema documents the new healthcheck payload shape.
  • Post-merge validation records a KB MCP restart/reconnect and verifies runtimeFreshness.status is current on the refreshed runtime.

Out of Scope

  • Restarting MCP servers automatically.
  • Reworking ask_knowledge_base provider routing; that shipped in #12741.
  • Reworking degraded KB references; that shipped in #12737.
  • Adding runtime freshness to GitHub Workflow; that shipped in #12765.
  • Memory Core runtime freshness; that is covered by #12772.
  • Adding heavy healthcheck probes or Chroma query canaries.

Avoided Traps

  • Do not treat stale runtime as a dependency outage. A stale process can still answer read-only calls while warning agents not to assert source/tool-schema facts from it.
  • Do not broaden this into a full MCP restart manager. The current need is falsifiable observability.
  • Do not duplicate #12741 by changing provider routing again. The verified source is already fixed; this ticket exposes when the running process has not loaded that fix.

Related

Parent: #12740 Related: #12737, #12741, #12765, #12772

Origin Session ID: e8f07ef9-ef7e-4815-8ff4-7abe13720621 Retrieval Hint: "Knowledge Base healthcheck runtimeFreshness stale MCP provider routing OpenAPI configDigest"

tobiu referenced in commit bf8bc48 - "feat(knowledge-base): add runtime freshness healthcheck (#12774) (#12775)" on Jun 8, 2026, 8:13 PM
tobiu closed this issue on Jun 8, 2026, 8:13 PM