Context
The cloud-scale bottleneck in ai:restore is now measured. The Knowledge Base importer reads every JSONL row into records[] before its first 500-row write; Memory Core does the same before 250-row writes and, in merge mode, also retains file-wide existing/missing sets.
The disposable matrix in #15695 measured 20,000 preserved 4,096-dimensional vectors at 1,232.9 MiB Node RSS for MC and 1,332.2 MiB for KB, before counting the separate Chroma process. Neither importer made its first store request before EOF. Provider healthy/cold/unreachable produced zero embedding calls, so the pressure is parse retention plus store serialization, not re-embedding.
The Problem
Downstream Chroma writes are chunked, but upstream parsing is not. A constrained Agent OS deployment can therefore exhaust its Node budget before making useful restore progress. MC merge also defeats the intended bound by retaining whole-file ID sets after parsing.
The Architectural Reality
KB_DatabaseService.importDatabase({file, mode, confirmation}) owns KB JSONL ingestion and currently returns {message, imported, mode}.Memory_DatabaseService.importDatabase({file, mode, reEmbed=false, confirmation, preserveDeliveryReadState=false}) owns MC JSONL ingestion and currently returns {message, imported, mode, counts}.- Existing write bounds remain 500 rows for KB and 250 rows for MC; this ticket changes when those batches flush, not their measured sizes.
- #15691 owns full-file parse/vector/bundle compatibility validation before any restore mutation.
- #15693 owns the autonomous ADR-0027 envelope: orchestrator selection/admission, durable restore-run state, retry policy, heavy-maintenance lease, writer fencing, resumable shadow identity, validation-clean promotion, rollback, and durable receipts.
- Explicit vectors stay explicit. This ticket must not call the embedding provider or acquire re-embedding authority.
The Fix
- In both DatabaseService importers, parse JSONL asynchronously and flush as soon as the owning batch reaches its existing bound. Do not materialize a whole-file
records[].
- In MC merge mode, run
collection.get({ids, include: []}) for the current batch only, add only that batch's missing records, update aggregate counts, and release all batch-local IDs/rows before reading the next batch.
- Preserve current public method arguments and return envelopes. This ticket introduces no resume cursor, checkpoint store, scheduling hook, or durable receipt API.
- Preserve current mode semantics and truthful aggregate counts across multiple files. A failed batch must not be reported as imported.
- Add deterministic instrumentation/specs proving that live buffered rows and merge-ID state never exceed the owning batch plus constant parser look-ahead.
- Re-run the 5,000/20,000 preserved-vector matrix with provider tracing and a disposable collection seam/private Chroma daemon.
Contract Ledger Matrix
| Target Surface |
Current Authority |
Behavior after this ticket |
Fallback / failure |
Docs |
Evidence |
KB_DatabaseService.importDatabase({file, mode='merge', confirmation}) |
ai/services/knowledge-base/DatabaseService.mjs |
Same arguments and {message, imported, mode} result; parse/validate/write one ≤500-row batch at a time |
Current batch fails; no later rows are read or counted |
Existing method JSDoc, updated with the bounded-buffer invariant |
Unit seam + 5k/20k fixture |
Memory_DatabaseService.importDatabase({file, mode, reEmbed=false, confirmation, preserveDeliveryReadState=false}) |
ai/services/memory-core/DatabaseService.mjs |
Same arguments and {message, imported, mode, counts} result; parse/validate/write one ≤250-row batch at a time |
Merge records a failed batch consistently with the current contract; replace remains fail-fast |
Existing method JSDoc, updated with the bounded-buffer invariant |
Unit seam + 5k/20k fixture |
| MC merge existence filter |
Current per-ID preserve-live merge semantics |
get(ids) and missing-row selection are batch-local; no file-wide Set or missing array |
A failed lookup/write affects only the active batch and is reflected in counts/error behavior |
MC importer JSDoc |
Peak batch/ID instrumentation |
| Restore-run checkpoint / shadow / receipt |
#15693 + ADR 0027 |
Unchanged and not implemented here; consumes this bounded primitive later |
#15693 fails closed without its own validated descriptor/state |
#15693 / ADR 0027 |
#15693 crash-point matrix |
Decision Record impact
No ADR amendment. This ticket is a bounded data-plane prerequisite for ADR 0027 / #15693; it does not implement an autonomous mutation action or move authority out of the orchestrator.
Acceptance Criteria
Out of Scope
- Full-file compatibility validation (#15691)
- Durable checkpoints, resume cursors, transient retry policy, or progress receipts (#15693)
- Autonomous bundle selection or action selection (#15639 / #15693)
- Re-embedding missing or incompatible vectors
- Changing the existing 500/250 write sizes without separate measurements
Avoided Traps
- Chunked upsert over a whole-file array — bounds transport payload, not process memory.
- File-wide merge sets — merely move the O(n) retention from rows to IDs.
- Importer-owned durable checkpoint — would create a second recovery authority and overlap #15693.
- Implicit re-embed during restore — makes provider availability a restore gate again.
Related
Origin Session ID: cb60301d-74a4-4024-b80d-2f7efdbf9cd1
Retrieval Hint: "bounded streaming vector restore importer 500 250 no records array no file-wide merge set"
Context
The cloud-scale bottleneck in
ai:restoreis now measured. The Knowledge Base importer reads every JSONL row intorecords[]before its first 500-row write; Memory Core does the same before 250-row writes and, in merge mode, also retains file-wide existing/missing sets.The disposable matrix in #15695 measured 20,000 preserved 4,096-dimensional vectors at 1,232.9 MiB Node RSS for MC and 1,332.2 MiB for KB, before counting the separate Chroma process. Neither importer made its first store request before EOF. Provider healthy/cold/unreachable produced zero embedding calls, so the pressure is parse retention plus store serialization, not re-embedding.
The Problem
Downstream Chroma writes are chunked, but upstream parsing is not. A constrained Agent OS deployment can therefore exhaust its Node budget before making useful restore progress. MC merge also defeats the intended bound by retaining whole-file ID sets after parsing.
The Architectural Reality
KB_DatabaseService.importDatabase({file, mode, confirmation})owns KB JSONL ingestion and currently returns{message, imported, mode}.Memory_DatabaseService.importDatabase({file, mode, reEmbed=false, confirmation, preserveDeliveryReadState=false})owns MC JSONL ingestion and currently returns{message, imported, mode, counts}.The Fix
records[].collection.get({ids, include: []})for the current batch only, add only that batch's missing records, update aggregate counts, and release all batch-local IDs/rows before reading the next batch.Contract Ledger Matrix
KB_DatabaseService.importDatabase({file, mode='merge', confirmation})ai/services/knowledge-base/DatabaseService.mjs{message, imported, mode}result; parse/validate/write one ≤500-row batch at a timeMemory_DatabaseService.importDatabase({file, mode, reEmbed=false, confirmation, preserveDeliveryReadState=false})ai/services/memory-core/DatabaseService.mjs{message, imported, mode, counts}result; parse/validate/write one ≤250-row batch at a timeget(ids)and missing-row selection are batch-local; no file-wideSetor missing arrayDecision Record impact
No ADR amendment. This ticket is a bounded data-plane prerequisite for ADR 0027 / #15693; it does not implement an autonomous mutation action or move authority out of the orchestrator.
Acceptance Criteria
Out of Scope
Avoided Traps
Related
Origin Session ID:
cb60301d-74a4-4024-b80d-2f7efdbf9cd1Retrieval Hint: "bounded streaming vector restore importer 500 250 no records array no file-wide merge set"