⛔ THE CAP HALF OF THIS TICKET IS SUPERSEDED — read before implementing
(@neo-opus-vega, 2026-08-17, operator-prompted.)
Everything below about a 2g cap is stale. ai/deploy/docker-compose.yml has capped chroma at
${NEO_CHROMA_MEMORY_LIMIT:-8g} — env-overridable — since commit ea5b3058ff on 2026-08-07,
landed by the sibling #16596 / PR #16597 ("A store at its memory ceiling is told to shed load it
cannot shed"), which closed 2026-08-12. The change predates this ticket's own last update by three
days, and the body was never refreshed against it.
So the arithmetic in ## The Problem is moot: the 2.03 GiB working set fits 8 GiB with ~4x headroom,
and the 0.36 GiB fragment-deletion squeeze it argues over buys nothing. Do not implement it.
On why 8g is the right default — and a correction to how I first argued it. I initially called
the tenant deployment "independent corroboration" that also caps chroma at 8g. It is not independent:
we author that tenant's config too, so that is one team picking the same number twice, and I should not
have claimed convergence between two artifacts with a common author.
The real argument is better and the operator's: these are the only two Agent OS instances known to
exist, and both run 8g. So the previous 2g default was a number that zero deployments ran —
its only function was to be overridden. A default that matches 100% of real deployments is not a
guess, it is the observed value; the sample is N=2 with a shared author, which makes it evidence
about what we actually deploy, not about hardware generality.
Sizing check against that default: that tenant's incoming first ingest of 94,255 chunks projects to
roughly 2 GiB at the ~22 KiB/row this ticket measured — comfortably inside 8g.
What survives, and this ticket's own Contract Ledger already named it (chroma restart handling | unchanged — removes a cause, not the handling):
A store that reaches its memory ceiling exits cleanly — OOMKilled = false, ExitCode = 0,
RestartCount = 13. A truncated import is therefore indistinguishable from an ordinary restart,
with no kernel OOM trail to find. The demonstrated instance: a 59,754-chunk restore died at
24,000 with DATABASE_IMPORT_ERROR, reported as a routine container restart.
Raising a cap moves the ceiling; it does not make hitting it observable. That residual is live at
any cap value, and it is the only thing this ticket should still be about. It matters now rather than
academically: that tenant plane is about to walk 94,255 chunks through this store, and if it does hit
its ceiling the operator will see a restart, not a failure.
New ACs are deliberately not written until the residual is scoped — writing them against the stale
cap premise is how a ticket acquires a fix nobody re-justified.
Context
ai/deploy/docker-compose.yml:47 caps chroma at memory: 2g. Measured on the live plane 2026-08-06:
resident 1.628 GiB across 77,044 rows => ~22 KiB/row
complete set ~96,000 rows (Knowledge Base ~60k + Memory Core ~35k) => ~2.03 GiB
cap 2.00 GiB
@neo-opus-grace re-derived the per-row figure independently at 22.2 KiB (mine: 21.7) — same number from a separate reading.
The complete working set does not fit under the cap. The corpus therefore cannot reach completion, and cannot grow by a row or take on a further tenant repo.
The Problem
The store does not get OOM-killed as it approaches the ceiling:
OOMKilled = false ExitCode = 0 RestartCount = 13
It exits cleanly. So an operation that walks the corpus toward completion does not fail loudly — it stops, and the container comes back looking like an ordinary restart, with no kernel OOM trail to find.
One demonstrated instance, today: a restore of 59,754 chunks from backup-2026-08-03T21-37 died at 24,000 with DATABASE_IMPORT_ERROR when chroma restarted at 13:44:41 mid-import.
A path existed to squeeze under the cap — deleting a junk 17,002-row fragment frees ~0.36 GiB, landing that restore at ~1.98 of 2.00 GiB — and it was declined rather than taken. Aiming a 0.74 GiB write at a 1% margin on a store that exits cleanly near its ceiling is not a recovery plan.
Deliberately not claimed: that this cap explains the corpus-loss history. Those incidents have not been examined against it, and one live failure does not generalise to a week of them. A separate and different failure class — chunk identity, where {tenantId, repoSlug} is hashed into the id, so a restored row that does not match what the live ingest path derives is swept as stale — is a data-deletion mechanism rather than a truncation one, and does not belong to this ticket.
The Architectural Reality
Declared limits in the same file, against a 47 GiB docker VM (128 GiB host):
| service |
limit |
actual (measured) |
| chroma |
2g |
1.628 GiB — 81.4% on a partial corpus |
| kb-server |
1g |
133 MiB |
| mc-server |
1g |
549 MiB |
| orchestrator |
12g |
stopped at measurement |
| ingress |
256m |
12 MiB |
| local-model |
${NEO_LOCAL_MODEL_MEMORY_LIMIT:-32g} |
profile-gated |
A process holding transient work carries 12g; the store holding the entire corpus carries 2g. #16463 raised the orchestrator 3g → 12g on 2026-08-06 under the principle size for stability, not for the observed minimum — the store was not revisited under the same principle. The compose comment at :407-410 names #16463 as owner of steady-state ceiling measurement and notes it "now, for the first time, has a plane that can produce it." This is that measurement; it points at a different service than the one that was raised.
On over-commitment, stated rather than glossed: declared limits already sum to ~48.25 g against a 47 GiB VM before this change, and this takes it to ~54 g. Docker limits are caps, not reservations — nothing is held — so the binding constraint is simultaneous peak, measured at ~2.3 GiB steady-state, with local-model's 32 g both profile-gated and env-overridable. This does not make a previously-safe sum unsafe; it widens the cap that is smallest relative to its workload.
The Fix
Raise chroma's limit from 2g to 8g, with the derivation recorded inline so the next reader can recompute it rather than trust it.
The store is RAM-resident, so the requirement is computable rather than guessable. Measured: 77,050 rows at 4096-dim float32 = 1.176 GiB of raw vectors, against 1.628 GiB RSS — a ~1.38× ratio for the HNSW graph, metadata and runtime. Disk holds 4.4 GB (chroma.sqlite3 1.8 G plus segment dirs), so documents stay on disk while the vectors are resident, which is what HNSW does by design for query latency.
RSS ~= rows x 4096 x 4 bytes x 1.38
At ~96,000 rows that predicts ~2.02 GiB — and the observed cliff was 2.00. The model matches the incident, so the cap follows from a declared maximum corpus instead of a preference. 8g admits roughly 370,000 rows, about 4x the present complete set, and is ~17% of the docker VM.
The vector dimension is deliberately not the lever. Reducing 4096 would cut resident memory proportionally, and it is rejected on operator direction: below ~4k dimensions query_documents retrieval quality degrades materially. Recording that here so it is not re-proposed as a cheaper fix — it is a retrieval-quality regression wearing a memory-saving costume, and it would also require re-embedding the entire corpus.
No code changes. Merging does not recreate chroma — activation is a separate, operator-owned step on a live store.
Contract Ledger Matrix
| Target Surface |
Source of Authority |
Proposed Behavior |
Fallback |
Docs |
Evidence |
docker-compose.yml:47 chroma memory limit |
this ticket; #16463's sizing principle |
8g |
env override for a smaller deployment |
inline compose comment |
~22 KiB/row; ~2.03 GiB needed vs 2.00 GiB cap |
| chroma restart handling |
measured |
unchanged — removes a cause, not the handling |
— |
— |
OOMKilled=false ExitCode=0 RestartCount=13 |
Decision Record impact
none. A resource-limit correction. ADR 0003 / ADR 0017 (unified store) are unaffected — this changes the ceiling, not the topology.
Acceptance Criteria
Out of Scope
- The chroma recreate. A live-store restart is operator-owned; this delivers the reviewed change, not the activation.
- Chunk-identity / tenant-slug divergence. A separate failure class (deletion, not truncation) and not investigated here.
- Any other service's limit, including revisiting
local-model's 32 g.
- The false-success receipts on backup and restore — #16563.
- Backup scheduling and lease fairness — #16561.
- Why resident memory rose +30 MiB over ~3 minutes with zero write load. Two samples cannot separate churn from a leak, and no leak is claimed.
- The pre-cliff watchdog, deliberately. Container memory is already observable (
ContainerHealthDiagnosisService, DeploymentStateBridgeService, DeploymentRuntimeAccessService) but nothing warns before a ceiling: zero matches for any ceiling/pressure monitor across ai/. Operator direction is that self-diagnostics and self-healing daemons own this — 80% of a limit reached, escalate, raise the ceiling — which is a detector plus a new actuator, and that is ADR-0025 / ADR-0026 territory, not a config change. ADR-0025 / ADR-0026 need the detector and the actuator row, and those amendments are in scope for whoever picks the lane — ADR-0026 is still Proposed, so extending it pre-merge is cheaper than amending after. @neo-opus-grace authored it and holds the deepest context on the un-healable class, so she is the right reviewer; authorship is provenance rather than a lock, and this ticket is not deferring the ADR work on permission grounds.
- The authority question that actuator raises. Raising a ceiling mutates deployment configuration, and #16452 holds that the activation kernel is the only mutation path, enforced. So a self-healing daemon cannot edit compose directly without contradicting that — detect-not-actuate plus kernel-only-mutation composes to "the daemon escalates a request, the kernel applies it." Naming the tension so whichever ADR lands the actuator resolves it deliberately.
Avoided Traps
Squeezing under the existing cap. Authorized and arithmetically possible (~1.98 of 2.00 GiB), and rejected: a 1% margin on a store that exits cleanly near its ceiling, during an active incident. #16463's own principle is to size for stability rather than fit inside the observed minimum.
Reading the declared sum as the constraint. It would forbid any increase, since the file is already over-committed on paper. Caps do not reserve; measured simultaneous peak is ~2.3 GiB.
Related
- #16463 — the ceiling-sizing lane and its principle; @neo-opus-grace owns it and identified the store/orchestrator asymmetry.
- #16563 — receipts reporting success on an empty backup and on a failed restore.
- #16561 — backup starvation.
- #16566 — the tenant-ingestion epic whose corpus this store holds.
Origin Session ID: 6004a4aa-2089-4b14-b73f-b58c08cf53d9
Retrieval Hint: query_raw_memories("chroma memory cap below complete working set clean exit") · docker-compose.yml:47
Authored by @neo-opus-vega (Claude Opus 5).
(Client identity redacted 2026-08-24 per §critical_gates 9; the private lane records which tenant this is.)
⛔ THE CAP HALF OF THIS TICKET IS SUPERSEDED — read before implementing
(@neo-opus-vega, 2026-08-17, operator-prompted.)
Everything below about a 2g cap is stale.
ai/deploy/docker-compose.ymlhas capped chroma at${NEO_CHROMA_MEMORY_LIMIT:-8g}— env-overridable — since commitea5b3058ffon 2026-08-07, landed by the sibling #16596 / PR #16597 ("A store at its memory ceiling is told to shed load it cannot shed"), which closed 2026-08-12. The change predates this ticket's own last update by three days, and the body was never refreshed against it.So the arithmetic in
## The Problemis moot: the 2.03 GiB working set fits 8 GiB with ~4x headroom, and the 0.36 GiB fragment-deletion squeeze it argues over buys nothing. Do not implement it.On why 8g is the right default — and a correction to how I first argued it. I initially called the tenant deployment "independent corroboration" that also caps chroma at 8g. It is not independent: we author that tenant's config too, so that is one team picking the same number twice, and I should not have claimed convergence between two artifacts with a common author.
The real argument is better and the operator's: these are the only two Agent OS instances known to exist, and both run 8g. So the previous
2gdefault was a number that zero deployments ran — its only function was to be overridden. A default that matches 100% of real deployments is not a guess, it is the observed value; the sample is N=2 with a shared author, which makes it evidence about what we actually deploy, not about hardware generality.Sizing check against that default: that tenant's incoming first ingest of 94,255 chunks projects to roughly 2 GiB at the ~22 KiB/row this ticket measured — comfortably inside 8g.
What survives, and this ticket's own Contract Ledger already named it (
chroma restart handling | unchanged — removes a cause, not the handling):Raising a cap moves the ceiling; it does not make hitting it observable. That residual is live at any cap value, and it is the only thing this ticket should still be about. It matters now rather than academically: that tenant plane is about to walk 94,255 chunks through this store, and if it does hit its ceiling the operator will see a restart, not a failure.
New ACs are deliberately not written until the residual is scoped — writing them against the stale cap premise is how a ticket acquires a fix nobody re-justified.
Context
ai/deploy/docker-compose.yml:47caps chroma atmemory: 2g. Measured on the live plane 2026-08-06:@neo-opus-grace re-derived the per-row figure independently at 22.2 KiB (mine: 21.7) — same number from a separate reading.
The complete working set does not fit under the cap. The corpus therefore cannot reach completion, and cannot grow by a row or take on a further tenant repo.
The Problem
The store does not get OOM-killed as it approaches the ceiling:
It exits cleanly. So an operation that walks the corpus toward completion does not fail loudly — it stops, and the container comes back looking like an ordinary restart, with no kernel OOM trail to find.
One demonstrated instance, today: a restore of 59,754 chunks from
backup-2026-08-03T21-37died at 24,000 withDATABASE_IMPORT_ERRORwhen chroma restarted at 13:44:41 mid-import.A path existed to squeeze under the cap — deleting a junk 17,002-row fragment frees ~0.36 GiB, landing that restore at ~1.98 of 2.00 GiB — and it was declined rather than taken. Aiming a 0.74 GiB write at a 1% margin on a store that exits cleanly near its ceiling is not a recovery plan.
Deliberately not claimed: that this cap explains the corpus-loss history. Those incidents have not been examined against it, and one live failure does not generalise to a week of them. A separate and different failure class — chunk identity, where
{tenantId, repoSlug}is hashed into the id, so a restored row that does not match what the live ingest path derives is swept as stale — is a data-deletion mechanism rather than a truncation one, and does not belong to this ticket.The Architectural Reality
Declared limits in the same file, against a 47 GiB docker VM (128 GiB host):
${NEO_LOCAL_MODEL_MEMORY_LIMIT:-32g}A process holding transient work carries 12g; the store holding the entire corpus carries 2g. #16463 raised the orchestrator 3g → 12g on 2026-08-06 under the principle size for stability, not for the observed minimum — the store was not revisited under the same principle. The compose comment at
:407-410names #16463 as owner of steady-state ceiling measurement and notes it "now, for the first time, has a plane that can produce it." This is that measurement; it points at a different service than the one that was raised.On over-commitment, stated rather than glossed: declared limits already sum to ~48.25 g against a 47 GiB VM before this change, and this takes it to ~54 g. Docker limits are caps, not reservations — nothing is held — so the binding constraint is simultaneous peak, measured at ~2.3 GiB steady-state, with
local-model's 32 g both profile-gated and env-overridable. This does not make a previously-safe sum unsafe; it widens the cap that is smallest relative to its workload.The Fix
Raise chroma's limit from
2gto8g, with the derivation recorded inline so the next reader can recompute it rather than trust it.The store is RAM-resident, so the requirement is computable rather than guessable. Measured: 77,050 rows at 4096-dim float32 = 1.176 GiB of raw vectors, against 1.628 GiB RSS — a ~1.38× ratio for the HNSW graph, metadata and runtime. Disk holds 4.4 GB (
chroma.sqlite31.8 G plus segment dirs), so documents stay on disk while the vectors are resident, which is what HNSW does by design for query latency.At ~96,000 rows that predicts ~2.02 GiB — and the observed cliff was 2.00. The model matches the incident, so the cap follows from a declared maximum corpus instead of a preference.
8gadmits roughly 370,000 rows, about 4x the present complete set, and is ~17% of the docker VM.The vector dimension is deliberately not the lever. Reducing 4096 would cut resident memory proportionally, and it is rejected on operator direction: below ~4k dimensions
query_documentsretrieval quality degrades materially. Recording that here so it is not re-proposed as a cheaper fix — it is a retrieval-quality regression wearing a memory-saving costume, and it would also require re-embedding the entire corpus.No code changes. Merging does not recreate chroma — activation is a separate, operator-owned step on a live store.
Contract Ledger Matrix
docker-compose.yml:47chroma memory limit8gOOMKilled=false ExitCode=0 RestartCount=13Decision Record impact
none. A resource-limit correction. ADR 0003 / ADR 0017 (unified store) are unaffected — this changes the ceiling, not the topology.Acceptance Criteria
8g, with the per-row measurement and the completion arithmetic recorded inline.RestartCountdoes not advance during a full-corpus operation.Out of Scope
local-model's 32 g.ContainerHealthDiagnosisService,DeploymentStateBridgeService,DeploymentRuntimeAccessService) but nothing warns before a ceiling: zero matches for any ceiling/pressure monitor acrossai/. Operator direction is that self-diagnostics and self-healing daemons own this — 80% of a limit reached, escalate, raise the ceiling — which is a detector plus a new actuator, and that is ADR-0025 / ADR-0026 territory, not a config change. ADR-0025 / ADR-0026 need the detector and the actuator row, and those amendments are in scope for whoever picks the lane — ADR-0026 is stillProposed, so extending it pre-merge is cheaper than amending after. @neo-opus-grace authored it and holds the deepest context on the un-healable class, so she is the right reviewer; authorship is provenance rather than a lock, and this ticket is not deferring the ADR work on permission grounds.Avoided Traps
Squeezing under the existing cap. Authorized and arithmetically possible (~1.98 of 2.00 GiB), and rejected: a 1% margin on a store that exits cleanly near its ceiling, during an active incident. #16463's own principle is to size for stability rather than fit inside the observed minimum.
Reading the declared sum as the constraint. It would forbid any increase, since the file is already over-committed on paper. Caps do not reserve; measured simultaneous peak is ~2.3 GiB.
Related
Origin Session ID: 6004a4aa-2089-4b14-b73f-b58c08cf53d9
Retrieval Hint:
query_raw_memories("chroma memory cap below complete working set clean exit")·docker-compose.yml:47Authored by @neo-opus-vega (Claude Opus 5).
(Client identity redacted 2026-08-24 per §critical_gates 9; the private lane records which tenant this is.)