Backup verification crashes on >512MB JSONL files (ERR_STRING_TOO_LONG)

Context

Discovered during the operator-directed 2026-06-26 #13999 Memory Core recovery. After the partial-promotion repair restored canonical exportability, npm run ai:backup ran clean through all 7 export steps — the vector-loss blocker is fixed — then crashed in the post-export integrity verification:

Verifying bundle integrity (row-count parity)...
❌ Backup failed: Error: Cannot create a string longer than 0x1fffffe8 characters
    at Buffer.toString (node:buffer:925:14)
  code: 'ERR_STRING_TOO_LONG'

This is a latent bug exposed by two things at once: (1) the export only now started completing (it previously died earlier at the Memory Core export step, so verification was never reached), and (2) the corpus grew — the just-written exports are 1.2 GB (mc/memory-backup-*.jsonl) and 1.0 GB (kb/knowledge-base-backup-*.jsonl).

The Problem

verifyBundleIntegrity counts each exported JSONL's rows (for source/bundle parity) by reading the whole file into a single string and splitting on newlines:

const content = await fs.readFile(path.join(dir, file), 'utf8');   // backup.mjs:300
bundleCount += content.split('\n').filter(line => line.trim()).length;

V8's maximum string length is 0x1fffffe8 (~512 MB). The 1.2 GB Memory Core export and 1.0 GB Knowledge Base export exceed it, so the internal Buffer.toString() throws ERR_STRING_TOO_LONG. The bundle data is fully written and complete; only the self-verification crashes — which means the canonical backup can never certify as a usable recovery source for a normally-sized corpus. That directly undermines the parent ticket's "verify restorability" goal and the v13.1 "verified-restorable backup" release-gate clause.

The Architectural Reality

• ai/scripts/maintenance/backup.mjs:299-302 — the verifyBundleIntegrity per-file row-count loop (the fs.readFile(..., 'utf8') + content.split('\n')).
• The same function gained its empty status in #14048 (sibling slice); this is the second latent defect in the same parity counter.
• node:readline streaming is already the established pattern in this exact directory: ai/scripts/maintenance/restore.mjs, ingestTenant.mjs, plus ai/services/knowledge-base/VectorService.mjs / DatabaseService.mjs. The fix lifts that precedent.

The Fix

Replace the whole-file read with a streaming line count — readline.createInterface({input: fs.createReadStream(file), crlfDelay: Infinity}), incrementing for each non-empty trimmed line. This is the exact semantic equivalent of the current split('\n').filter(line => line.trim()), with bounded memory and no string-size ceiling. All four verdict statuses (pass / empty / fail / skipped) are preserved unchanged.

Contract Ledger Matrix

Decision Record impact

aligned-with #14030 (backup-substrate trustworthiness) and ADR-0025/ADR-0026 (the backup feeds the immune system's recovery-source-of-last-resort). No ADR amended — pure mechanism fix.

Acceptance Criteria

• verifyBundleIntegrity counts bundle JSONL rows via a streaming reader (no whole-file .toString()), so files >512 MB verify without ERR_STRING_TOO_LONG.
• Row-count semantics are unchanged: non-empty trimmed lines are counted; pass / empty / fail / skipped verdicts are identical to before for small files.
• npm run ai:backup completes the verification step on the current ~1.2 GB MC / ~1.0 GB KB exports (manual evidence in the PR).
• Unit coverage: a large (>512 MB-class, or stubbed large-stream) file is counted correctly without throwing; an existing small-file parity case still passes.

Out of Scope

• Reducing export size / not serializing full embeddings into the JSONL (#14079 — the reason the files are 1+ GB).
• The empty-parity semantics (#14048, shipped) and the alert-on-failure / restorability-import / retention-SLA slices (#14030 AC1-3, shipped).

Avoided Traps

• "Just raise the string limit." 0x1fffffe8 is a hard V8 cap, not configurable — streaming is the only correct fix.
• Naive \n-byte counting. A raw byte scan would mishandle the existing empty-line filter and the trailing-newline case; readline preserves the exact non-empty-line semantics the parity check relies on.

Related

• #14030 (parent — backup reliability / verify restorability).
• #14048 (sibling — the empty-parity fix in the same verifyBundleIntegrity function).
• #13999 (the Memory Core recovery whose now-completing export exposed this).
• #14079 (Memory Core store bloat — why the exported files exceed 512 MB).

Handoff Retrieval Hints

• query_raw_memories("backup verifyBundleIntegrity ERR_STRING_TOO_LONG 512MB streaming line count")
• Exact anchors: verifyBundleIntegrity (ai/scripts/maintenance/backup.mjs:299-302), node:readline precedent in restore.mjs.

Authored by Vega (Claude Opus 4.8) — surfaced by @tobiu during the 2026-06-26 live #13999 recovery; premise reproduced via npm run ai:backup on dev.