Measured friction from a live incident on an adopter deployment (2026-08-18). An embedding lane was at 96.7% of its memory ceiling and four tenant repos had been failing for four hours. get_deployment_state_snapshot answered every health question I had — usage bytes, limit bytes, pressure disposition, saturation facts, restart churn, provider lane shape, a log tail.
It could not answer "what is this service's NEO_KB_EMBEDDING_BATCH_SIZE right now?"
That mattered because an earlier deploy letter of ours had told the operator to set it to 1, and I needed to know whether that instruction had been applied and never reverted.
"Unknown" overstates it, and the precise version is the better argument for this ticket. The declared value is readable — our compose declares ${NEO_KB_EMBEDDING_BATCH_SIZE:-5}, and I read it. What is unreadable is whether that is what runs, across three divergence points we ourselves created: the client host is not a checkout, so its copy of compose.yml can lag the revision we are reading; the host .env is an override slot we published; and our own deploy letter instructed a value into it.
So the gap is not we know nothing. It is we can read what we intended and cannot read what is running — and reporting the declared value as the effective one is exactly the confident-wrong-answer this ticket exists to remove. It is also the answer I nearly gave.
So the one question I had to escalate to a human was a question about our own configuration, on a plane we author end to end.
This body was rewritten 2026-08-18 after the original Fix section proved wrong. It prescribed a bridge-side AiConfig read for values owned by a different process. The reasoning and the corrected shape are preserved below under The Architectural Reality; the original is in issuecomment-5330285332.
The Problem
snapshot.services[] carries stats, memoryPressure, diagnosis, restartChurn, providerActivity, providerResidency, providerLaneShape and logs — and no view of resolved configuration. inspect carries container identity and state but not Config.Env.
The gap is asymmetric in the way that matters: we can see what a service is doing and never what we told it to do. Diagnosis then proceeds against assumed inputs. In this incident the assumed input and the possible real input differed by 5×, on the exact knob that governs ingestion throughput.
There is a second-order cost. Because the value is unknowable remotely, the only way to confirm a config instruction landed is to ask the operator to run a grep — the pattern we have explicitly ruled out for anything we can self-serve. Config was the one carve-out, and a carve-out by omission rather than by decision.
The Architectural Reality
The process boundary is the load-bearing fact, and the first version of this ticket got it wrong.
batchSize / batchDelay / maxRetries are knowledge-base server leaves (ai/mcp/server/knowledge-base/configBase.mjs:693/:715/:735). DeploymentStateBridgeService runs in the orchestrator. A bridge-side read for those paths resolves the orchestrator's config tree and publishes it under the KB server's name. On a deployment whose per-service .env diverges from the compose default — the only deployment anyone would consult this field for — that publishes a confidently wrong number.
That is strictly worse than the absence it replaces, and it is the same failure this ticket complains about: an absent field says cannot answer and gets checked; a wrong-process field says answered and does not.
The channel already existed.heapObservation is provenance: 'self-reported' — the owning process writes a file, the bridge reads and bounds it, and unavailableReason carries the reason when the process cannot answer (which is why not-node reads correctly on a non-Node container instead of looking like a broken probe). Resolved config is the second fact of that kind.
providerLaneShape is the shape precedent for a narrow declared-and-observed projection rather than an environment dump.
The Fix (as shipped)
Four pieces, in dependency order:
ai/mcp/server/shared/helpers/resolvedConfigDisclosure.mjs — the disclosure boundary. A pure function over an already-resolved config object; reads no environment and imports no config singleton.
ai/mcp/server/shared/services/ResolvedConfigReporterService.mjs — the owning process projects its allowlisted subset and publishes it once, atomically, after boot().
DeploymentStateBridgeService.readResolvedConfig — relays it as services[].resolvedConfig, beside providerLaneShape. Resolves nothing itself.
BaseServer.getResolvedConfigDisclosure — one hook returning {config, allowlist}; kb-server declares the three embedding knobs.
Where the allowlist is enforced, and why it is not the relay. The writer applies it before anything reaches disk, so an unallowlisted value never leaves the owning process. No downstream relay, snapshot copy, log or future consumer can surface what was never emitted, and there is no second place a filter has to be re-applied correctly.
Validity is bounded by incarnation, not by elapsed time. A heap number is resampled because it moves. Config is fixed at boot and runtime mutation of the shared tree is forbidden, so an old record is not degraded and refusing it on age would hide a correct answer. A restart does invalidate it — the container may have come back with different env — so a record stamped before the current incarnation start is stale-incarnation. An unparseable incarnation start does not discard the record: that is an instrument gap, and refusing on it would convert "cannot tell which incarnation" into "configuration unknown".
Published after boot(), which is load-bearing rather than tidy.loadCustomConfig() runs inside boot(), so publishing earlier would disclose the pre-overlay values — naming the defaults as this deployment's effective configuration, the exact false answer the channel replaces.
The golden rule, and why it holds by construction
These tools must never be able to read secrets. Satisfied without relying on a filter:
No environment access. Nothing in the boundary reads process.env; it is a pure function over a config object handed in. There is no path from disclosure to the environment, so there is nothing for a filter to miss.
Secrets are not config leaves. Credentials arrive as credentialRef: env:NAME resolved at their point of use, deliberately outside the tree this projection can see.
Allowlist, never denylist. A denylist fails open on every future key. An allowlist's failure mode is a missing value — visible, harmless, fixed by a reviewed addition.
No wildcards or prefix matching.embedding.* would silently admit a future embedding.apiKey. Refused at load rather than left to review.
Disclosure kinds as a second floor. Every entry declares the primitive it may reveal, and there is no free string kind, because a free string is the shape a credential has. A path declared number cannot carry a token even after a refactor moves something unexpected behind it.
The kind is deliberately not the config leaf's type. The leaf already owns the value domain; restating positiveInt here would define one thing twice.
Decision Record impact
aligned-with ADR-0019 — every config read happens at its use site in the owning process; no parallel resolution path, no new leaf, no formula, no runtime mutation.
Acceptance Criteria
AC-1:snapshot.services[].resolvedConfig reports the allowlisted subset, self-reported by the owning service and relayed, with no process.env read anywhere in the path and no bridge-side resolution of another process's config.
AC-2 (security, fails first): a spec asserts a non-allowlisted path — including one holding a credential-shaped string — is absent from the projection, and it is mutation-proven: replacing the projection with a full-subtree dump turns it red.
AC-3: the allowlist refuses wildcard and prefix entries at load, so embedding.*-style widening cannot be introduced without failing a test.
AC-4: an allowlisted path whose value violates its declared kind is omitted with a reason, not coerced and not emitted.
AC-5: the allowlist is frozen and is not derived from configuration or environment; a deployment cannot extend it.
AC-6 (over-claim, twin of AC-2): an entry the owning service has not reported is absent with a reason, never defaulted and never filled from the bridge's own tree. disclosed is null on every unavailable arm, never {}.
AC-7: ADR-0019 conformance over the diff, naming A1, A5, A6, B1, B3, B4 as checked.
AC-8: the incident question is answerable from the snapshot alone — the effective embedding batch size for a running service, with no operator involvement.
AC-9: the field is exercised through collectServiceSnapshot, not only through the reader in isolation — an isolated corpus cannot catch an unreachable call site.
AC-10: paths resolve against the real config proxy, with a control proving the proxy hides them from in while the values read fine.
Out of Scope
Exposing inspect.Config.Env, in whole or in filtered form. It starts from the surface containing every secret and tries to subtract.
Writing configuration through the snapshot. Read-only; a remote config write is a different blast radius and needs its own decision.
Provider slot geometry — providerLaneShape already covers it.
Renaming the self-report directory's config key. It is heapObservation.dir for historical reasons while its meaning is broader; the rename has its own callers.
mc-server's declaration. The hook exists and defaults to publishing nothing; adding its seed set wants its own incident-driven test of which values have been wanted.
Avoided Traps
Filtering an environment dump. Redaction inverts the safety default: every future secret is exposed until someone remembers to add a pattern. The industry-standard SECRET_PATTERNS regex list is precisely the denylist this refuses.
Making the allowlist configurable. It reads as good practice and defeats the mechanism: a deployment that can extend the list can name a credential path.
Asserting the feature works without asserting what it refuses. A test that only checks the allowlisted values appear passes equally against a full-subtree dump.
Trusting a plain-object fixture for a proxy-backed subject. The walker decided presence with in, which reads false on a config proxy that has a get trap and no has trap — every fixture green, production silently disclosing nothing. Caught only by wiring it to the real config.
A second identity hook. Reusing the heap channel's service key avoids two service keys for one process, which is the mis-attribution hazard that channel already documents.
Related
#17357 — startup facts aging out of the log tail; the sibling observability gap from the same incident.
#17344 — one global transport leaf serving four servers whose transports diverged; same "config we authored is not observable" family.
#17349 / #17345 — tenant-sync behaviours whose diagnosis this shortens.
tobiu referenced in commit 2050825 - "feat(ai): a service reports its own resolved config, so nobody has to ask the operator (#17356) (#17362) on Aug 19, 2026, 7:53 AM
Context
Measured friction from a live incident on an adopter deployment (2026-08-18). An embedding lane was at 96.7% of its memory ceiling and four tenant repos had been failing for four hours.
get_deployment_state_snapshotanswered every health question I had — usage bytes, limit bytes, pressure disposition, saturation facts, restart churn, provider lane shape, a log tail.It could not answer "what is this service's
NEO_KB_EMBEDDING_BATCH_SIZEright now?"That mattered because an earlier deploy letter of ours had told the operator to set it to
1, and I needed to know whether that instruction had been applied and never reverted."Unknown" overstates it, and the precise version is the better argument for this ticket. The declared value is readable — our compose declares
${NEO_KB_EMBEDDING_BATCH_SIZE:-5}, and I read it. What is unreadable is whether that is what runs, across three divergence points we ourselves created: the client host is not a checkout, so its copy ofcompose.ymlcan lag the revision we are reading; the host.envis an override slot we published; and our own deploy letter instructed a value into it.So the gap is not we know nothing. It is we can read what we intended and cannot read what is running — and reporting the declared value as the effective one is exactly the confident-wrong-answer this ticket exists to remove. It is also the answer I nearly gave.
So the one question I had to escalate to a human was a question about our own configuration, on a plane we author end to end.
The Problem
snapshot.services[]carriesstats,memoryPressure,diagnosis,restartChurn,providerActivity,providerResidency,providerLaneShapeandlogs— and no view of resolved configuration.inspectcarries container identity and state but notConfig.Env.The gap is asymmetric in the way that matters: we can see what a service is doing and never what we told it to do. Diagnosis then proceeds against assumed inputs. In this incident the assumed input and the possible real input differed by 5×, on the exact knob that governs ingestion throughput.
There is a second-order cost. Because the value is unknowable remotely, the only way to confirm a config instruction landed is to ask the operator to run a
grep— the pattern we have explicitly ruled out for anything we can self-serve. Config was the one carve-out, and a carve-out by omission rather than by decision.The Architectural Reality
The process boundary is the load-bearing fact, and the first version of this ticket got it wrong.
batchSize/batchDelay/maxRetriesare knowledge-base server leaves (ai/mcp/server/knowledge-base/configBase.mjs:693/:715/:735).DeploymentStateBridgeServiceruns in the orchestrator. A bridge-side read for those paths resolves the orchestrator's config tree and publishes it under the KB server's name. On a deployment whose per-service.envdiverges from the compose default — the only deployment anyone would consult this field for — that publishes a confidently wrong number.That is strictly worse than the absence it replaces, and it is the same failure this ticket complains about: an absent field says cannot answer and gets checked; a wrong-process field says answered and does not.
The channel already existed.
heapObservationisprovenance: 'self-reported'— the owning process writes a file, the bridge reads and bounds it, andunavailableReasoncarries the reason when the process cannot answer (which is whynot-nodereads correctly on a non-Node container instead of looking like a broken probe). Resolved config is the second fact of that kind.providerLaneShapeis the shape precedent for a narrow declared-and-observed projection rather than an environment dump.The Fix (as shipped)
Four pieces, in dependency order:
ai/mcp/server/shared/helpers/resolvedConfigDisclosure.mjs— the disclosure boundary. A pure function over an already-resolved config object; reads no environment and imports no config singleton.ai/mcp/server/shared/services/ResolvedConfigReporterService.mjs— the owning process projects its allowlisted subset and publishes it once, atomically, afterboot().DeploymentStateBridgeService.readResolvedConfig— relays it asservices[].resolvedConfig, besideproviderLaneShape. Resolves nothing itself.BaseServer.getResolvedConfigDisclosure— one hook returning{config, allowlist};kb-serverdeclares the three embedding knobs.Where the allowlist is enforced, and why it is not the relay. The writer applies it before anything reaches disk, so an unallowlisted value never leaves the owning process. No downstream relay, snapshot copy, log or future consumer can surface what was never emitted, and there is no second place a filter has to be re-applied correctly.
Validity is bounded by incarnation, not by elapsed time. A heap number is resampled because it moves. Config is fixed at boot and runtime mutation of the shared tree is forbidden, so an old record is not degraded and refusing it on age would hide a correct answer. A restart does invalidate it — the container may have come back with different env — so a record stamped before the current incarnation start is
stale-incarnation. An unparseable incarnation start does not discard the record: that is an instrument gap, and refusing on it would convert "cannot tell which incarnation" into "configuration unknown".Published after
boot(), which is load-bearing rather than tidy.loadCustomConfig()runs insideboot(), so publishing earlier would disclose the pre-overlay values — naming the defaults as this deployment's effective configuration, the exact false answer the channel replaces.The golden rule, and why it holds by construction
These tools must never be able to read secrets. Satisfied without relying on a filter:
process.env; it is a pure function over a config object handed in. There is no path from disclosure to the environment, so there is nothing for a filter to miss.credentialRef: env:NAMEresolved at their point of use, deliberately outside the tree this projection can see.embedding.*would silently admit a futureembedding.apiKey. Refused at load rather than left to review.stringkind, because a free string is the shape a credential has. A path declarednumbercannot carry a token even after a refactor moves something unexpected behind it.The
kindis deliberately not the config leaf's type. The leaf already owns the value domain; restatingpositiveInthere would define one thing twice.Decision Record impact
aligned-with ADR-0019— every config read happens at its use site in the owning process; no parallel resolution path, no new leaf, no formula, no runtime mutation.Acceptance Criteria
snapshot.services[].resolvedConfigreports the allowlisted subset, self-reported by the owning service and relayed, with noprocess.envread anywhere in the path and no bridge-side resolution of another process's config.embedding.*-style widening cannot be introduced without failing a test.disclosedisnullon every unavailable arm, never{}.collectServiceSnapshot, not only through the reader in isolation — an isolated corpus cannot catch an unreachable call site.inwhile the values read fine.Out of Scope
inspect.Config.Env, in whole or in filtered form. It starts from the surface containing every secret and tries to subtract.providerLaneShapealready covers it.heapObservation.dirfor historical reasons while its meaning is broader; the rename has its own callers.mc-server's declaration. The hook exists and defaults to publishing nothing; adding its seed set wants its own incident-driven test of which values have been wanted.Avoided Traps
SECRET_PATTERNSregex list is precisely the denylist this refuses.in, which readsfalseon a config proxy that has agettrap and nohastrap — every fixture green, production silently disclosing nothing. Caught only by wiring it to the real config.Related
learn/agentos/decisions/0019-aiconfig-reactive-provider-ssot.md.Handoff Retrieval Hints
"resolved config self-reported allowlist leaf paths not env names""snapshot cannot answer effective batch size, asked the operator instead""config proxy has no has trap, in operator reads false while value resolves"heapObservationfor the channel,providerLaneShapefor the projection shape.Origin Session ID: 9ccc2fa1-8843-4796-8e85-5e151c0392d2
— Vega (Claude Opus 5, Claude Code) 🌿