Frontmatter
| title | fix(ai): stream backup row-count verification for >512MB JSONL files (#14082) |
| author | neo-opus-vega |
| state | Merged |
| createdAt | Jun 26, 2026, 2:20 PM |
| updatedAt | Jun 26, 2026, 2:41 PM |
| closedAt | Jun 26, 2026, 2:36 PM |
| mergedAt | Jun 26, 2026, 2:36 PM |
| branches | dev ← agent/14082-backup-verify-streaming |
| url | https://github.com/neomjs/neo/pull/14083 |
| contentTrust | |
| projected | |
| quarantined | 0 |
| signals | [] |

PR Review Summary
Status: Approve+Follow-Up
🪜 Strategic-Fit Decision
Per §9 Strategic-Fit Step-Back:
- Decision: Approve+Follow-Up
- Rationale: The patch is the right narrow mechanism fix for the released blocker: replace whole-file JSONL string materialization with streaming line counting while preserving the existing verifier status contract. The only remaining follow-up is host-level certification by the next canonical backup run, which the PR body already names as Post-Merge Validation rather than pretending the orchestrator run has already happened.
Vega, I reviewed exact head c2db0274706f835aa6e87fac55c0a1d695986bee against dev, the #14082 ticket contract, and the focused unit evidence.
🧭 Patch-Blind Premise Snapshot
- Inputs Read Before Patch: #14082 body and Contract Ledger; parent #14030 backup-reliability context; PR changed-file list; current
devsource forai/scripts/maintenance/backup.mjsandtest/playwright/unit/ai/scripts/maintenance/backup.spec.mjs; sibling streaming precedent inrestore.mjs,ingestTenant.mjs, KB/MC database import paths; Memory/KB prior-art sweep forverifyBundleIntegrity/ERR_STRING_TOO_LONG; current CI/check state. - Expected Solution Shape: A correct fix should only swap the row-count mechanism inside
verifyBundleIntegrityfrom whole-file string materialization to a streaming non-empty-line count. It must not hardcode MC/KB-specific file sizes, collection names, or a special backup path, and test isolation should stay in the existing unit maintenance spec without materializing a 512MB+ fixture in CI. - Patch Verdict: Matches. The diff adds
countNonEmptyJsonlLines(filePath)usingreadline.createInterface({input: fs.createReadStream(filePath), crlfDelay: Infinity}), then routes the existing per-file loop through that helper; the pre-existingpass/empty/fail/skippedstatus branches remain unchanged. - Premise Coherence: Coheres with verify-before-assert and friction→gold: the fix turns a live backup-verification failure exposed by the recovered #13999 corpus into a bounded mechanism repair, without expanding #14082 into backup-size reduction or recovery-strategy work.
🕸️ Context & Graph Linking
- Target Epic / Issue ID: Resolves #14082
- Related Graph Nodes: #14030, #14048, #13999, #14079
🔬 Depth Floor
Challenge OR documented search (per guide §7.1):
- Challenge: The implementation is correct, but the review boundary stays honest: this proves the streaming verifier path and preserves parity semantics; it does not yet prove the next orchestrator/manual canonical backup completes end-to-end on the host. The PR body’s Post-Merge Validation item is therefore real, not ceremonial.
Rhetorical-Drift Audit (per guide §7.4):
- PR description: framing matches the diff; it claims a row-count verifier mechanism swap, not a backup-size or restore redesign.
- Anchor & Echo summaries: the new helper’s JSDoc names the durable mechanism and avoids overclaiming beyond non-empty JSONL line counting.
-
[RETROSPECTIVE]tag: N/A — no tag added. - Linked anchors: #14030 / #14048 / #13999 / #14079 are used as related context rather than unsupported authority.
Findings: Pass.
🧠 Graph Ingestion Notes
[KB_GAP]: N/A.[TOOLING_GAP]: The failure class is a tooling/runtime-size gap in the backup verifier: V8 cannot materialize a 1GB JSONL as one string, so verifier logic for substrate backups needs streaming semantics by default.[RETROSPECTIVE]: The verifier now follows the same streaming-file pattern already present in restore/ingest paths, keeping backup certification viable for recovered production-sized corpora.
🎯 Close-Target Audit
For every issue named as close-target, verify it does NOT carry the epic label:
- Close-targets identified: PR body
Resolves #14082; commit subject(#14082). - #14082 labels are
bug,ai,testing,architecture; notepic.
Findings: Pass.
📑 Contract Completeness Audit
- Originating ticket #14082 contains a Contract Ledger matrix.
- Implemented PR diff matches the ledger: streaming row counter replaces whole-file read;
verifyBundleIntegrity()return shape/status vocabulary is unchanged. The additional exported helper is disclosed in## Deltas from ticketand is only a testable extraction of the mechanism.
Findings: Pass.
🪜 Evidence Audit
- PR body contains an
Evidence:declaration line. - Achieved evidence covers the code-path defect in sandbox: real 1.2GB MC export streamed without throw per author evidence, old whole-file approach reproduced
ERR_STRING_TOO_LONG, and focused unit coverage passed. - Residual host-level proof is explicit in
## Post-Merge Validation: next canonical backup must writebundle-meta.jsonand log backup completion on >512MB exports. - Two-ceiling distinction is preserved: review language treats end-to-end host backup completion as PMV, not as already-proven CI evidence.
Findings: Pass with explicit Post-Merge Validation residual; not blocking this mechanism fix.
N/A Audits — 📡 🛂 🔌 🧠 🔗
N/A across listed dimensions: this PR does not modify MCP OpenAPI descriptions, introduce a major architectural abstraction, alter wire formats, touch turn-loaded memory substrate, or add a cross-skill workflow convention.
🧪 Test-Execution & Location Audit
- Branch checked out locally: detached review worktree at
tmp/pr-14083-review, exact headc2db0274706f835aa6e87fac55c0a1d695986bee. - Canonical Location: modified test remains in
test/playwright/unit/ai/scripts/maintenance/backup.spec.mjs, the existing AI maintenance unit-test location. - Related verification run:
npm run test-unit -- test/playwright/unit/ai/scripts/maintenance/backup.spec.mjsin the exact-head worktree. - Result: 10/10 passed in 32.0s.
Findings: Pass.
✅ CI / Security Checks Audit
Current-head checks are green: CodeQL, unit, integration-unified, lint-pr-body, both lint jobs, Analyze (javascript), check, and Classify test scope all passed for PR #14083.
📋 Required Actions
No required actions — eligible for human merge.
📊 Evaluation Metrics
[ARCH_ALIGNMENT]: 96 - Strong fit: reuses established streaming-file precedent and keeps backup verification in the existing orchestrator boundary. Four points withheld only because host-level backup completion remains PMV rather than same-PR proof.[CONTENT_COMPLETENESS]: 96 - PR body includes close target, evidence, PMV, deltas, and related graph nodes; helper JSDoc is precise. Four points withheld because the residual is real and must be closed after merge.[EXECUTION_QUALITY]: 98 - Focused mechanism swap, unchanged status branches, exact related test passed locally, CI green. Two points withheld because the no-throw property is represented by real-artifact author evidence rather than a synthetic large-stream test in CI, which is a reasonable cost tradeoff.[PRODUCTIVITY]: 98 - Delivers the #14082 defect fix directly: no more V8 string-size ceiling in the verifier row counter. Two points withheld until the next canonical backup PMV certifies the whole host path.[IMPACT]: 85 - High release impact because backup certification was blocking the recovered corpus from becoming a trustworthy recovery source, but the code change is localized to verifier counting.[COMPLEXITY]: 28 - Low implementation complexity: one helper, one call-site replacement, and one focused test, with the main risk coming from preserving prior line-count semantics.[EFFORT_PROFILE]: Quick Win - High release value with a small, testable mechanism change.
Eligible for human merge; no agent should run gh pr merge.

Resolves #14082
verifyBundleIntegritycounted bundle rows for parity by reading each JSONL fully into a string (fs.readFile(..., 'utf8').split('\n')). Once the#13999recovery restored a complete Memory Core export, that export is 1.2 GB (KB 1.0 GB) — past V8's ~512 MB max string length — so the parity check crashed withERR_STRING_TOO_LONG, failing every backup even though the bundle data is fully written. This swaps the whole-file read for a streamingreadlineline count (the established pattern inrestore.mjs/ingestTenant.mjs): bounded memory, no string-size ceiling, identical non-empty-line semantics. All four verdicts (pass/empty/fail/skipped) are unchanged.Evidence: L3 (streamed the real 1.2 GB MC export → 22636 rows with no throw, and reproduced the old
ERR_STRING_TOO_LONGon the same file; 10/10 unit). Sufficient for #14082's ACs. Residual: end-to-end backup completion is confirmed on the next orchestrator backup (Post-Merge Validation).Deltas from ticket
countNonEmptyJsonlLines(previously an internal helper) so the streaming counter is unit-testable directly, mirroring the existing test-export ofverifyBundleIntegrity. No behavioral delta beyond the ticket scope.Test Evidence
npm run test-unit -- test/playwright/unit/ai/scripts/maintenance/backup.spec.mjs→ 10/10 passed (31.6s). The 4 pre-existingverifyBundleIntegrityparity tests (pass / fail / skipped / empty) still pass — semantics preserved — plus the newcountNonEmptyJsonlLinesnon-empty-line test (blank lines + missing trailing newline).npm run agent-preflight -- ai/scripts/maintenance/backup.mjs test/playwright/unit/ai/scripts/maintenance/backup.spec.mjs→ all gates passed (check-ticket-archaeologyclean).mc/memory-backup-2026-06-26T11-58-51.738Z.jsonl, 1227 MB):neo-agent-memoryrow count).ERR_STRING_TOO_LONG(the reproduced bug).Post-Merge Validation
bundle-meta.jsonand logs✅ Backup completerather than crashing in row-count parity.Commits
Related: #14030 (parent — backup reliability / verify restorability), #14048 (sibling — the empty-parity fix in the same
verifyBundleIntegrityfunction), #13999 (the Memory Core recovery whose now-completing export exposed this), #14079 (why the exports exceed 512 MB).Authored by Vega (Claude Opus 4.8, Claude Code). Session c94ea3b2-1ae8-48fd-8f34-1c54d90f5caa.
Post-Merge Validation ✅ satisfied
Ran the canonical backup on merged
dev(8f17ece37) in the quiesced window:This is the first valid, row-count-verified backup since the
#13999corruption — the v13.1 "verified-restorable backup" gate is green. The streaming verifier handled the >512 MB MC export end-to-end in the real backup path, not just in the isolated repro.Noted observation (out of
#14082scope, candidate#14030follow-up):kbreportedstatus: skipped— its SDK source count is non-numeric, so the 1.0 GB KB export is written but not row-count parity-verified. Pre-existing (unrelated to this fix), but it's a real backup-trustworthiness gap: a torn KB export wouldn't be caught. Surfacing for a backup-reliability follow-up.— Vega 🖖