Context
#16780 AC-7 asks that an operator be able to answer "is this provider load explained by pending work?" from the public surface alone — no shell, no log parsing. Tracing it produced a smaller and more actionable finding than the AC's wording implies: both halves already exist and are already computed. Nothing connects them.
Duplicate sweep: this is a delivery leaf of #16780, which stays open for AC-1/2/3/5/6/8 and the KB half. No other open ticket claims the drain receipt's exposure — #16770 delivered provider-stage attribution (the numerator) and explicitly does not bound anything.
The Problem
The denominator exists, per cycle. drainWalOnce (ai/daemons/embed/drainCycle.mjs) computes {pending, embedded, compensated, failed, metadataOnly, unverifiable, cooling, prunedSegments} — with summary.pending read from the complete, deliberately un-limited pending scan — and disposition.recordCycle(summary) retains it in a createDrainDispositionTracker receipt ({state, drainedClean, reason, counts, at}), returned on the loop handle as getDisposition.
The numerator is attributable in-process. addWalRecords wraps the write in runWithProviderActivityContext({operationStage: 'mc-wal-drain-embedding', service: 'memory-core'}). The payload is {ids, metadatas, documents} with no embeddings, but the collection carries a Neo-side embeddingFunction (createDynamicTextEmbeddingFunction), so the provider call happens in this process, inside that context, and lands in the ledger get_memory_core_tool_metrics already reports.
The receipt reaches nothing. Verified three ways:
| check |
result |
getDisposition production consumers |
zero — only its own two definition sites |
drainDisposition.mjs importers |
4 files: 2 specs + the 2 loops that define the tracker |
drainedClean in any openapi.yaml |
zero |
And the logging is inverted, which is why this went unnoticed for ~24h on a live plane:
if (summary.pending > 0 || summary.prunedSegments > 0) {
log('INFO', `WAL drain cycle: ${JSON.stringify(summary)}`);
}A cycle with zero pending logs nothing. The precise state the parent ticket exists for — provider busy, pending zero — emits no line at all. Silence is indistinguishable from idle-and-healthy, so even the log-parsing escape hatch AC-7 wants to remove does not cover the anomaly case.
The Fix
Publish the existing receipt on the surface that already carries the attribution. No new tool (the parent's constraint), no new measurement, no new daemon.
Two properties are the substance, not the plumbing:
- Absence must never render as zero. A process that does not host the drain has an unknown backlog, not an empty one. Zero pending against live provider load is the alarm condition, so a defaulted zero would synthesise the exact alarm the projection exists to detect — on every non-hosting process.
- Comparability must be stated, not assumed. The receipt describes the last cycle;
providerActivity describes a lookback window. They are a ratio only when the cycle landed inside the window. A single precomputed ratio would be more convenient and less true.
Deliberately not a composed global number: the denominator is genuinely distributed across the MC WAL, KB resume state, and separate processes. Each process reports its own; the operator composes.
Acceptance Criteria
Out of Scope
- The KB ingestion path. It has its own recorder (
KBRecorderService) and its own pending set; it is verified before it is claimed, not assumed from this one.
- AC-3 (re-embed ratio), AC-5 (declared concurrency), AC-1/AC-2/AC-6 — all stay on
#16780, all feed this same surface.
- Alarming on the ratio. This makes the disproportion readable; deciding a threshold is a separate act with its own false-positive burden.
Avoided Traps
- Defaulting an unknown denominator to zero. The most natural-looking default is the alarm value.
- Precomputing one ratio across two windows. Convenient, and wrong whenever the drain has been quiet.
- Building a new metric. The measurement was never missing — only its wiring.
- Assuming the KB path mirrors this one because both embed. Different recorder, different pending set.
Related
#16780 (parent — the reporting half continues there) · #16770 (provider-stage attribution — the numerator) · #16822 / #16826 (the bound and its durability) · #16566 · #16830 (a stuck runner would make this projection's pending the deciding evidence) · #16706
Origin Session ID: 87f453f9-aa80-4487-9ed1-b5d91e052c43
Retrieval Hint: wal drain disposition receipt orphaned unexposed provider activity denominator pending zero unavailable truncated coverageStartedAt window aggregate
Context
#16780AC-7 asks that an operator be able to answer "is this provider load explained by pending work?" from the public surface alone — no shell, no log parsing. Tracing it produced a smaller and more actionable finding than the AC's wording implies: both halves already exist and are already computed. Nothing connects them.Duplicate sweep: this is a delivery leaf of
#16780, which stays open for AC-1/2/3/5/6/8 and the KB half. No other open ticket claims the drain receipt's exposure —#16770delivered provider-stage attribution (the numerator) and explicitly does not bound anything.The Problem
The denominator exists, per cycle.
drainWalOnce(ai/daemons/embed/drainCycle.mjs) computes{pending, embedded, compensated, failed, metadataOnly, unverifiable, cooling, prunedSegments}— withsummary.pendingread from the complete, deliberately un-limited pending scan — anddisposition.recordCycle(summary)retains it in acreateDrainDispositionTrackerreceipt ({state, drainedClean, reason, counts, at}), returned on the loop handle asgetDisposition.The numerator is attributable in-process.
addWalRecordswraps the write inrunWithProviderActivityContext({operationStage: 'mc-wal-drain-embedding', service: 'memory-core'}). The payload is{ids, metadatas, documents}with noembeddings, but the collection carries a Neo-sideembeddingFunction(createDynamicTextEmbeddingFunction), so the provider call happens in this process, inside that context, and lands in the ledgerget_memory_core_tool_metricsalready reports.The receipt reaches nothing. Verified three ways:
getDispositionproduction consumersdrainDisposition.mjsimportersdrainedCleanin anyopenapi.yamlAnd the logging is inverted, which is why this went unnoticed for ~24h on a live plane:
if (summary.pending > 0 || summary.prunedSegments > 0) { log('INFO', `WAL drain cycle: ${JSON.stringify(summary)}`); }A cycle with zero pending logs nothing. The precise state the parent ticket exists for — provider busy, pending zero — emits no line at all. Silence is indistinguishable from idle-and-healthy, so even the log-parsing escape hatch AC-7 wants to remove does not cover the anomaly case.
The Fix
Publish the existing receipt on the surface that already carries the attribution. No new tool (the parent's constraint), no new measurement, no new daemon.
Two properties are the substance, not the plumbing:
providerActivitydescribes a lookback window. They are a ratio only when the cycle landed inside the window. A single precomputed ratio would be more convenient and less true.Deliberately not a composed global number: the denominator is genuinely distributed across the MC WAL, KB resume state, and separate processes. Each process reports its own; the operator composes.
Acceptance Criteria
get_memory_core_tool_metricscarries the drain receipt besideproviderActivity, so one reading holds both halves. —MemoryCoreRecorderService.getWalDrainProjection, returned aswalDrain;e8b852d122.A process that does not host the drain reports
status: 'unavailable'withcounts: null— a fixture asserts the counts are null and fails if they are zeroed. — an absent drain host reports unavailable with NULL counts; red-proved by zeroing that branch: fails with "an unknown backlog must never read as an empty one".A hosted drain reports its per-cycle counts and
drainedClean. — a hosted drain publishes its own per-cycle counts beside provider activity.A lookback this process cannot fully attest is reported as partial, never as a total, and the covered interval is on the surface so the partiality is bounded rather than merely flagged. —
window.truncated+window.coverageStartedAt. Proven in the producer's own spec (ai/daemons/shared/drainDisposition.spec.mjs), with an eviction control and a fully-covered negative control so "always partial" cannot satisfy it.A throwing receipt degrades to
partialwith null counts and does not fail the metrics call. — a throwing receipt degrades to partial with null counts, and never breaks the metrics call.The response's declared key set records the addition rather than being loosened, and the openapi schema declares the field. — the existing
Object.keys(metrics)pin was updated, not relaxed;WalDrainProjectionResponsedeclared + added to the required set;ai:lint-openapi-service-parityOK.Coverage fails against today's code and passes against the repair. — 1663 passed; per-test mutation run reddens the absent-host case while both controls stay green.
Out of Scope
KBRecorderService) and its own pending set; it is verified before it is claimed, not assumed from this one.#16780, all feed this same surface.Avoided Traps
Related
#16780(parent — the reporting half continues there) ·#16770(provider-stage attribution — the numerator) ·#16822/#16826(the bound and its durability) ·#16566·#16830(a stuck runner would make this projection'spendingthe deciding evidence) ·#16706Origin Session ID: 87f453f9-aa80-4487-9ed1-b5d91e052c43
Retrieval Hint:
wal drain disposition receipt orphaned unexposed provider activity denominator pending zero unavailable truncated coverageStartedAt window aggregate