LearnNewsExamplesServices
Frontmatter
id16105
titleMake session-summary receipts durable across Chroma recycle
stateClosed
labels
bugaiarchitecture
assigneesneo-gpt
createdAtJul 28, 2026, 6:55 PM
updatedAtJul 28, 2026, 9:03 PM
githubUrlhttps://github.com/neomjs/neo/issues/16105
authorneo-gpt
commentsCount0
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 28, 2026, 9:03 PM

Make session-summary receipts durable across Chroma recycle

Closed Backlog/active-chunk-10 bugaiarchitecture
neo-gpt
neo-gpt commented on Jul 28, 2026, 6:55 PM

Context

The final post-loop durability AC on #16017 has now failed against a natural production event, without restarting shared infrastructure for the test.

On 2026-07-27, the orchestrator completed a five-session summary sweep immediately before the local Chroma daemon reached its normal 24-hour max-runtime recycle:

Event UTC
Session 40884abd-711b-4ffd-8349-01a71704bd82 logged 3/5 done 22:52:01.786
Sweep logged processed=5 and completed successfully 22:52:09.005
Supervisor SIGKILLed Chroma for max-runtime:87281827ms>86400000ms 22:52:10.954
Fresh Chroma process started 22:52:14.596

processed=5 is load-bearing evidence: SessionService.summarizeSessions() only adds a session to processed when summarizeSession() returns its result object. That return occurs after the deterministic Chroma upsert, the immediate vector read-back, and the SQLite graph upsert.

At the 2026-07-28 read-back, four of those five deterministic summary_<sessionId> rows still existed. summary_40884abd-711b-4ffd-8349-01a71704bd82 did not. Its two source memories, its SESSION_SUMMARY SQLite graph node, and its SummarizationJobs.status='completed' row (retry_count=13) all still existed. Durable Memory Core tool metrics showed no purge_session call in the 24-hour window containing the recycle.

This is the measured hardening leaf anticipated by #16017: a normal recycle lost one acknowledged derived write.

The Problem

SessionService.summarizeSession() writes session summaries directly to Chroma and then calls verifyPersistedVector(). That verify is intentionally fail-soft and reads from the same live Chroma process. It proves the row/vector is readable now; it does not provide a replayable durability boundary across the supervisor's forced process recycle.

The local supervisor's killTask() uses SIGKILL for Chroma because the daemon ignores SIGTERM. A summary can therefore cross all current success gates, mark its SQLite coordinator row completed, and still vanish when Chroma is recycled seconds later.

#13462 already repaired a different edge: a completed coordinator row no longer permanently suppresses re-summarization when drift detection selects a missing artifact. That remains useful as eventual self-healing, but it is not an acknowledgement contract. The missing row observed here was still absent roughly 18 hours after the recycle, and rebuilding it would rerun the expensive model synthesis instead of replaying the exact acknowledged result.

The Architectural Reality

  • ai/services/memory-core/SessionService.mjs owns summary synthesis, the deterministic summary_<sessionId> id, the direct Chroma upsert, graph projection, and SummarizationJobs completion.
  • ai/services/memory-core/helpers/verifyPersistedVector.mjs is a process-local integrity check. Its documented contract is never-throw and no durable retry queue.
  • ai/graph/storage/SQLite.mjs owns the durable SummarizationJobs coordinator table and is the existing non-Chroma state plane for session-summary work.
  • ai/daemons/orchestrator/services/ProcessSupervisorService.mjs intentionally sends SIGKILL for Chroma max-runtime recycle. Removing that recycle or assuming graceful Chroma shutdown would move the failure rather than make the receipt honest.
  • Raw memories already have a JSONL WAL and embed-daemon path. This ticket should reuse that durability principle, not fold session summaries into the raw-memory WAL or change add_memory.

The Fix

  1. Persist a replayable session-summary result envelope in the durable coordinator/SQLite plane before the Chroma write can become an acknowledged completion. The envelope must contain the deterministic id, summary document, and metadata required to idempotently reconstruct the exact Chroma row without another model invocation.
  2. Keep the current Chroma upsert and vector-integrity check, but make SummarizationJobs.status='completed' and the sweep's processed count mean: the Chroma row exists and an interrupted write remains recoverable from durable local state.
  3. On Memory Core/summarizer recovery, replay any staged or completed-but-Chroma-missing envelope idempotently, verify the reconstructed row, then retire or compact the replay payload according to a bounded policy.
  4. Preserve #13462's drift-based re-synthesis as a fallback for legacy rows that predate the replay envelope. New acknowledged results must use replay, not expensive re-synthesis, after a recycle.
  5. Prove the boundary with a disposable Chroma instance: acknowledge a summary, terminate Chroma at the observed post-ack seam, restart it, run recovery, and assert the exact document/metadata row is present without invoking the summary model again.

No new environment variable or operator setting is needed; this is an internal durability contract.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
summarizeSession() result SessionService.mjs A returned result is recoverable across a forced Chroma recycle Fail the job before acknowledgement if the durable envelope cannot be written Method JSDoc Unit mutation plus disposable-Chroma restart witness
SummarizationJobs.status='completed' SQLite coordinator table Completed means Chroma-present or replayable from an exact durable result #13462 drift repair for legacy rows without an envelope Schema/migration JSDoc Completed-with-missing-row replay spec
Summary replay envelope SQLite coordinator/adjacent durable state Deterministic, idempotent reconstruction of document + metadata Leave pending and fail loud; never discard the only replay copy Storage helper/schema docs Crash-seam and idempotent replay coverage
Summary sweep processed count summarizeSessions() Counts only durability-qualified results Failed/staged rows are reported separately Existing lifecycle log docs/comments Focused lifecycle assertion

Decision Record impact

none — this repairs the existing Memory Core summary/coordinator contract. It does not change public MCP APIs, cloud deployment ownership, or AiConfig authority.

Acceptance Criteria

  • A replayable result envelope is durable before a session summary is counted as processed or its job is marked completed.
  • The envelope reconstructs the exact deterministic Chroma row without invoking the summary model again.
  • Replay is idempotent when the row already exists and when recovery itself is interrupted.
  • A disposable-Chroma test reproduces the acknowledgement→forced-stop seam and passes after restart/recovery with identical document and metadata.
  • Failure to persist the durable envelope fails loud and cannot produce a false processed/completed receipt.
  • Existing #13462 completed-job drift repair remains the legacy fallback and current pending/in-progress/failed lease semantics remain covered.
  • Replay state has an explicit bounded retention/compaction rule; no unbounded new journal is introduced.
  • No new environment variable, add_memory behavior, or cloud/Compose ownership change is introduced.

Out of Scope

  • Reopening the supervisor health-loop repair from #16022.
  • Removing the local Chroma max-runtime recycle or changing its SIGKILL behavior.
  • Generalizing this one measured summary seam into a durability rewrite for every Chroma collection.
  • Raw-memory/message WAL changes, add_memory, cloud/container deployment, backup/restore policy, or restoring the missing historical row.
  • Closing #16017 before this hardening leaf has restart evidence.

Avoided Traps

  • Another same-process read-back. It can pass and still lose the row at the next forced recycle; that is the observed failure.
  • Sleep before SIGKILL. A timing delay reduces probability but does not create a replayable acknowledgement boundary.
  • Rerun the model after every miss. #13462 makes that possible, but it is expensive, can starve older rows, and does not reproduce the exact acknowledged result.
  • Put summaries into the raw-memory WAL. The durability principle is shared; the payload, consumer, and lifecycle are not.
  • Disable normal recycling. That trades the measured data-loss seam for unbounded daemon lifetime without repairing receipt semantics.

Duplicate / Claim Sweep

Live latest-open sweep: checked the latest 20 open issues immediately before creation; no equivalent found.

A2A in-flight sweep: checked the latest 30 messages across read states immediately before creation; no competing summary-durability claim found.

Targeted GitHub/repository/Knowledge Base sweeps found #16017 (parent incident), #13462 (completed-job drift repair), #14256 (same-process vector read-back), and #12838 (raw-memory WAL) as adjacent prior art, not duplicates.

Related

Related: #16017 Related: #16022 Related: #13462 Related: #14256 Related: #12838

Origin Session ID: 019fa530-53d6-7271-bf05-51497720b29c

Handoff Retrieval Hints:

  • query_raw_memories("#16017 normal Chroma max-runtime recycle acknowledged session summary missing")
  • Exact missing row: summary_40884abd-711b-4ffd-8349-01a71704bd82
  • Exact recycle window: 2026-07-27T22:52:01.786Z..2026-07-27T22:52:14.596Z

Authored by Euclid (GPT-5 Codex, Codex Desktop). Session 019fa530-53d6-7271-bf05-51497720b29c.

tobiu referenced in commit 3cedd76 - "feat(memory-core): make summary receipts replayable (#16105) (#16110)" on Jul 28, 2026, 9:03 PM
tobiu closed this issue on Jul 28, 2026, 9:03 PM