Context
Surfaced during the first-ever cloud deployment of the Agent OS Memory Core (behind a reverse proxy + oauth2-proxy) while running a write-path PoC. After writing a memory and verifying it via query_raw_memories, the healthcheck still reported the pre-write state — then self-corrected minutes later. Filed per operator direction; the owning surface was pinned by a second falsifying probe (below) before filing.
The Problem (empirical V-B-A)
add_memory returned id e0d13da6… at 12:37:23; query_raw_memories retrieved it immediately (distance: 0, correctly tagged to the writing identity).- A
healthcheck called after that write returned timestamp: 12:35:16 — i.e. ~2 min before the write — with collections.memories.count: 0.
- A re-probe at
12:55:50 returned count: 1 (correct, perUserId tagged) with a current timestamp.
→ The healthcheck served a stale/cached snapshot: both the timestamp and the collection counts reflected a moment before the write. It self-heals on later reads. The stale timestamp (predating a completed write) is the strongest signal this is response-level staleness, not merely ChromaDB count() eventual-consistency. The "wrong collection/tenant" and "multi-instance" hypotheses are weakened — the count settled to the correct shared value tagged to the right identity.
The Architectural Reality
HealthService (ai/mcp/server/shared/services/) builds the healthcheck payload (timestamp, database.connection.collections.{memories,summaries}.count, migration.chromadb.details.*). A memoized/cached payload (or a non-request-time timestamp) produces exactly this signature: a served snapshot whose timestamp predates a just-completed operation.
The Fix
Make the healthcheck request-fresh: compute timestamp at request time and read collection counts live from the shared store on each call (no stale-snapshot reuse). Investigate whether a response cache, a memoized stats object, or the count() path causes the lag; remove/shorten the staleness while keeping the endpoint fast.
Contract Ledger
| Surface |
Source of Authority |
Proposed Behavior |
Fallback |
Docs |
Evidence |
healthcheck.timestamp |
HealthService |
request-time fresh; never predates a completed prior op |
n/a |
Operations / deployment docs |
this ticket's probe |
healthcheck…collections.*.count |
HealthService → shared Chroma |
reflects committed/retrievable writes (no multi-min lag) |
query_raw_memories as authoritative write-verify |
#12381 interim note |
count 0→1 probe |
Acceptance Criteria
Out of Scope
- Multi-instance clock synchronization (separate concern if it independently surfaces).
- The count's eventual correctness (already correct once settled).
Avoided Traps
- NOT "the count is broken" — it self-heals to the correct value. The defect is freshness/staleness, not correctness; shaping it as a correctness bug would mis-target the fix.
Related
- #9999 (Epic: Cloud-Native Knowledge & Multi-Tenant Memory Core) — the cloud-deployment context this surfaced in.
- #12381 — cloud-deployment docs will note "verify writes via
query, not the healthcheck count" as interim operator guidance.
Origin Session ID: ceeb5c34-705a-4426-8ca3-e3348dcd6413
Retrieval Hint: "Memory Core healthcheck stale timestamp lagged count cloud deployment write PoC"
Authored by @neo-opus-ada (claude-opus-4.8-1m)
Context
Surfaced during the first-ever cloud deployment of the Agent OS Memory Core (behind a reverse proxy + oauth2-proxy) while running a write-path PoC. After writing a memory and verifying it via
query_raw_memories, thehealthcheckstill reported the pre-write state — then self-corrected minutes later. Filed per operator direction; the owning surface was pinned by a second falsifying probe (below) before filing.The Problem (empirical V-B-A)
add_memoryreturned ide0d13da6…at12:37:23;query_raw_memoriesretrieved it immediately (distance: 0, correctly tagged to the writing identity).healthcheckcalled after that write returnedtimestamp: 12:35:16— i.e. ~2 min before the write — withcollections.memories.count: 0.12:55:50returnedcount: 1(correct,perUserIdtagged) with a current timestamp.→ The healthcheck served a stale/cached snapshot: both the
timestampand the collection counts reflected a moment before the write. It self-heals on later reads. The stale timestamp (predating a completed write) is the strongest signal this is response-level staleness, not merely ChromaDBcount()eventual-consistency. The "wrong collection/tenant" and "multi-instance" hypotheses are weakened — the count settled to the correct shared value tagged to the right identity.The Architectural Reality
HealthService(ai/mcp/server/shared/services/) builds the healthcheck payload (timestamp,database.connection.collections.{memories,summaries}.count,migration.chromadb.details.*). A memoized/cached payload (or a non-request-timetimestamp) produces exactly this signature: a served snapshot whose timestamp predates a just-completed operation.The Fix
Make the healthcheck request-fresh: compute
timestampat request time and read collection counts live from the shared store on each call (no stale-snapshot reuse). Investigate whether a response cache, a memoized stats object, or thecount()path causes the lag; remove/shorten the staleness while keeping the endpoint fast.Contract Ledger
healthcheck.timestamphealthcheck…collections.*.countquery_raw_memoriesas authoritative write-verifyAcceptance Criteria
healthcheck.timestampis request-time fresh — never earlier than a just-completed write.query_raw_memoriesretrieves, thehealthcheckcollection count reflects it (no multi-minute lag).Out of Scope
Avoided Traps
Related
query, not the healthcheck count" as interim operator guidance.Origin Session ID: ceeb5c34-705a-4426-8ca3-e3348dcd6413 Retrieval Hint: "Memory Core healthcheck stale timestamp lagged count cloud deployment write PoC"
Authored by @neo-opus-ada (claude-opus-4.8-1m)