Stop Chroma test collections from reappearing in the live store

Context

During the active Memory Core repair lane for #13999, the live unified Chroma store was inspected because the database had grown to about 1.7GB. The size issue is mostly not hidden vector data: chroma.sqlite3 is about 1.0GB, the Knowledge Base HNSW segment is about 517MB, and the Memory Core HNSW segment is about 147MB. However, the inspection surfaced a separate regression signal: the production Chroma SQLite catalog currently contains 448 stale test-* collection rows.

This class was supposed to be closed by the earlier Chroma test-isolation work, especially #12180, #12335, and #12153. The current state means either old residue was never fully purged after those fixes, or a remaining path can still create test collections in the live default_tenant / default_database namespace.

Release classification: boardless post-incident hardening. This does not appear to be the source of the missing Memory Core vectors in #13999, but it is a live-store hygiene regression that must not remain ambiguous.

The Problem

The live store now contains test collection metadata again after prior fixes intended to make that impossible. The concrete current observation is:

test-memory collection rows: 263
test-session collection rows: 185
total test-* collection rows: 448

The same inspection found only one test-memory-* collection with any metadata rows, and it is effectively zero-size. The retained Chroma queue has only 821 vector blobs totaling 12.8MiB, with no May 19 vector reservoir; it is not where the #13999 missing vectors are hiding. That separation matters: this ticket should not be used as the Memory Core exportability root cause until a reproducer proves active test bleeding.

Memory Core semantic search was unavailable during creation because the embedding write canary timed out after 5000ms, so this ticket uses the Knowledge Base, local content sweep, and live SQLite/GitHub evidence as the creation authority.

The Architectural Reality

Prior art from the Knowledge Base:

• #10867 introduced process-isolated test-memory-* / test-session-* names after destructive cleanup hit canonical collections.
• #12180 documented orphan test-memory-* / test-session-* collections accumulating in production Chroma and closed after isolated test database routing plus purge tooling.
• #12335 explicitly moved unit-test Chroma usage into an isolated tenant/database under UNIT_TEST_MODE so test collections should not enter the production namespace.
• #12153 treated test-pollution removal as a requirement for the unified flat Chroma store.

Current live evidence conflicts with the intended steady state: collections still contains 448 names matching test-% in the live unified store. The fix must determine whether these are pre-fix leftovers, post-fix bleed, interrupted harness artifacts, or a purge gap.

The Fix

1. Add a production-safe diagnostic that reports test-* collections by tenant/database, row count, vector segment presence, and newest created_at / queue timestamp.
2. Prove whether the current 448 rows are historical residue or actively reintroduced by any current test path.
3. If active bleeding exists, close that path with a failing regression test first. Likely surfaces include unit-test config bootstrapping, direct Chroma client construction, tests run without the custom Playwright unit config, or a helper bypass around UNIT_TEST_MODE.
4. If the rows are residue only, add or document a guarded purge path that removes empty stale test collections from the live namespace without touching canonical collections or non-empty evidence needed for forensics.
5. Add a post-run guard so future Chroma-integrating unit tests fail if they create test-* collections in the production tenant/database.

Contract Ledger Matrix

Decision Record impact

Aligned with ADR 0003 (unified Chroma topology) and the test-isolation contract from #12335. No ADR amendment expected.

Acceptance Criteria

• Diagnostic output lists all live test-* collections with collection id, tenant/database if available, metadata row count, vector segment id/path presence, and latest relevant timestamp.
• A reproducer or negative proof establishes whether current test runs can still create test-* rows in the production namespace.
• If a current bleed path exists, a regression test fails before the fix and passes after the fix.
• If the rows are historical residue only, a guarded dry-run-first purge path handles empty stale rows and refuses ambiguous/non-empty rows without an explicit evidence-preserving decision.
• Future Chroma test runs have a guard that fails when test-* collections appear in production default_tenant / default_database.
• The #13999 repair notes are updated to keep this test-collection recurrence separate from the missing-vector root cause unless new evidence links them.

Out of Scope

• Reopening #12180, #12335, or #12153.
• Repairing the missing Memory Core vectors from #13999.
• Deleting non-empty or ambiguous test collections without a dry-run report and explicit evidence-preserving rationale.
• Broad Chroma defrag or backup rewrite behavior unrelated to test namespace isolation.

Avoided Traps

• Do not assume the stale test rows are the lost Memory Core vectors. The current queue/HNSW evidence does not support that.
• Do not run a blind purge against the live store. Empty residue cleanup and active bleed prevention are different jobs.
• Do not rely on test-* naming as isolation. The prior fixes moved isolation to tenant/database and UNIT_TEST_MODE; the name prefix is only a diagnostic smell.

Related

Related: #13999 Related: #12180 Related: #12335 Related: #12153 Related: #10867

Live latest-open sweep: checked latest 20 open issues at 2026-06-25T15:13:00Z; no equivalent open ticket found. Closest open active incident is #13999, which covers Memory Core backup exportability, not the test-collection recurrence. A2A in-flight claim sweep: checked latest 30 all-status messages at the same time; no overlapping peer [lane-claim] / [lane-intent] found.

Origin Session ID: 9280140f-8b54-4462-9342-49cca7e226f4

Handoff Retrieval Hints: test collection bleed Chroma test-memory test-session default_tenant, #12180 #12335 Chroma unit-test isolation recurrence, #13999 shadow collections lost embeddings