LearnNewsExamplesServices
Frontmatter
id16895
titleThe Knowledge Base embedding probe deadline is a frozen literal, so a slow plane cannot raise the one deadline that marks it degraded
stateClosed
labels
bugai
assignees[]
createdAtAug 10, 2026, 5:15 PM
updatedAtAug 10, 2026, 8:52 PM
githubUrlhttps://github.com/neomjs/neo/issues/16895
authorneo-opus-grace
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 10, 2026, 8:52 PM

The Knowledge Base embedding probe deadline is a frozen literal, so a slow plane cannot raise the one deadline that marks it degraded

neo-opus-grace
neo-opus-grace commented on Aug 10, 2026, 5:15 PM

Env names corrected 2026-08-10. This body named NEO_KB_EMBEDDING_PROBE_*; the implementation declares NEO_KB_HEALTHCHECK_EMBEDDING_PROBE_*, matching the healthcheck config block it lives in and Memory Core's sibling naming. Falsified by @neo-gpt-emmy with a child-process probe: the ticket-named variable left the value at 30000 while the implemented name resolved 180000. Closing on the old names would have shipped operator instructions for variables that do not exist.

Context

Found 2026-08-10 on a live CPU-only deployment running 3f9f8343a8. Its Knowledge Base reports:

Knowledge Base embedding probe failed: consumer-probe-timeout:EMBEDDING_PROBE_TIMEOUT
  — backing off 240000ms (streak 4, deadline 30000ms)

Thirty seconds is not enough for a cold embed on CPU-only hardware, and there is no way to change it.

Plan-Authority: INDEPENDENT — runtime defect. Related: #16706, #16860.

The Problem

ai/services/knowledge-base/HealthService.mjs:14-21:

const embeddingProbePolicy = Object.freeze({
    cadenceMs      : 60 * 1000,
    timeoutMs      : 30 * 1000,
    healthyTtlMs   : 60 * 1000,
    failureTtlMs   : 30 * 1000,
    failureTtlMaxMs: 10 * 60 * 1000
});

Five frozen literals with no config leaf and no environment variable. Six consumers read them (:66, :403, :474-478). A deployment whose hardware cannot meet a 30s embed deadline has no lever at all — not in .env, not in compose, not in config.mjs.

The asymmetry makes this a defect rather than a design choice. Memory Core's equivalent probe is configurable — ai/mcp/server/memory-core/configBase.mjs:387:

embeddingWriteCanaryTimeoutMs: leaf(30000, 'NEO_MEMORY_HEALTHCHECK_EMBEDDING_WRITE_CANARY_TIMEOUT_MS', 'number')

Same probe shape, same default, same provider, same failure mode — one is tunable and one is not. The KB side simply never got a leaf.

What it costs, concretely. On that deployment the operator raised every reachable embedding deadline to 180s. The KB probe kept firing at 30s and kept reporting degraded, because it was the one deadline no configuration could reach. Time was spent verifying compose wiring that could never have worked.

The Architectural Reality

  • The leaf shape already exists in the same config file: NEO_KB_ASK_SYNTHESIS_TIMEOUT_MS (:241) and NEO_KB_ASK_SYNTHESIS_TIMEOUT_MS_REMOTE (:252). Nothing new is being invented.
  • timeoutMs is the only one with a live incident behind it. The other four are exposed for symmetry, not speculation — a deployment that needs a longer deadline needs a matching cadence, or it simply queues probes.
  • Consumers already destructure with defaults, so threading the leaf is a change to the default's source and not to call sites.

The Fix

Convert embeddingProbePolicy to config leaves under the knowledge-base config, mirroring the Memory Core naming exactly so the two are greppable as a pair.

leaf env default
embeddingProbe.timeoutMs NEO_KB_HEALTHCHECK_EMBEDDING_PROBE_TIMEOUT_MS 30000
embeddingProbe.cadenceMs NEO_KB_HEALTHCHECK_EMBEDDING_PROBE_CADENCE_MS 60000
embeddingProbe.healthyTtlMs NEO_KB_HEALTHCHECK_EMBEDDING_PROBE_HEALTHY_TTL_MS 60000
embeddingProbe.failureTtlMs NEO_KB_HEALTHCHECK_EMBEDDING_PROBE_FAILURE_TTL_MS 30000
embeddingProbe.failureTtlMaxMs NEO_KB_HEALTHCHECK_EMBEDDING_PROBE_FAILURE_TTL_MAX_MS 600000

Defaults unchanged, so no existing deployment moves.

Contract Ledger

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
KB embedding probe deadline this ticket env-settable leaf shipped 30000 default when unset JSDoc naming the MC counterpart spec: a set env value reaches the probe
the four sibling policy values symmetry with the above env-settable leaves shipped defaults same same
probe failure detail existing keeps printing the deadline in force unchanged the printed value tracks the leaf

Acceptance Criteria

  • Setting NEO_KB_HEALTHCHECK_EMBEDDING_PROBE_TIMEOUT_MS changes the deadline the probe actually applies — proven by a spec that fails against the frozen literal, not by asserting the leaf resolves.
  • Unset leaves the shipped behaviour byte-identical. This is the non-vacuity arm: a change that made every deployment adopt a new default would pass the first AC and silently alter every plane.
  • The failure detail string continues to print the deadline in force, and that value tracks the leaf. This is what makes the setting verifiable from outside the container — the reason this defect was diagnosable at all.
  • ADR-0019 §3 self-audit recorded in the PR for the five new leaves.
  • Object.freeze on a resolved-config object is not reintroduced anywhere in this path.

Out of Scope

  • Changing any default. This makes the values reachable; choosing new ones is a deployment decision.
  • The ingestion-path embed deadline (NEO_OLLAMA_EMBEDDING_TIMEOUT_MS), which is already a leaf and already reaches containers. The probe and the ingestion path are different deadlines and conflating them cost real time today.
tobiu referenced in commit 9dc2a09 - "fix(knowledge-base): the embedding-probe policy is a frozen literal no deployment can reach (#16895) (#16899) on Aug 10, 2026, 8:52 PM
tobiu closed this issue on Aug 10, 2026, 8:52 PM