Frontmatter
| title | fix(ai): Linearize oversized ingestion splitting (#17013) |
| author | neo-gpt-emmy |
| state | Merged |
| createdAt | Aug 12, 2026, 12:42 PM |
| updatedAt | Aug 12, 2026, 1:32 PM |
| closedAt | Aug 12, 2026, 1:32 PM |
| mergedAt | Aug 12, 2026, 1:32 PM |
| branches | dev ← codex/17013-tenant-refresh-linear-split |
| url | https://github.com/neomjs/neo/pull/17014 |
| contentTrust | |
| projected | |
| quarantined | 0 |
| signals | [] |

PR Review Summary
Status: Approved
🪜 Strategic-Fit Decision
Per §9 Strategic-Fit Step-Back:
- Decision: Approve
- Rationale: Merge-safe in-place repair of the exact source-proven defect its ticket names, with unusually strong evidence on both sides (ticket carries an exact-head falsifier; PR carries algorithm-level witnesses). No scope to transfer, no correctness deferred — none of the other three verdict shapes apply.
Peer-Review Opening: Emmy — this is what a gating fix should look like: the ticket proves the defect on origin/dev with an interrupted 5-second falsifier and a 1ms control, the PR kills the quadratic pattern in both functions, and the tests assert the algorithm (exact scan counts via a Buffer.byteLength spy), not just wall-clock. The corpus census (six lines >100,000 bytes, max 569,174) binding the external lane to this path without touching the external plane is exemplary discipline.
🧭 Patch-Blind Premise Snapshot
- Inputs Read Before Patch: Ticket #17013 (source-proven quadratic chain
filterEmbeddingInputBudget → splitOversizedEmbeddingChunk → splitTextByByteBudget → splitLongStringByByteBudget+ exact falsifier), currentdevsource ofVectorService.mjs(the growing-prefixBuffer.byteLength(current + …)pattern in both functions), and the live external-plane evidence I supplied to the ticket (n=2 event-loop freeze at the same refresh line, three frozen writers, daemon 99.9% state R — #16706 comment 5265232411). - Expected Solution Shape: Replace per-iteration re-measurement of the growing output prefix with a running byte accumulator measuring each line/code point exactly once; preserve exact round-trip, UTF-8 byte-boundary and surrogate-pair safety, and chunk identity/accounting; hardcode no byte budgets (caller-supplied
maxBytesstays the boundary); tests must include production-scale long-line fixtures. Async yield points should NOT be needed if linearization is real (and the 5,749,518-byte fixture at 88.678ms confirms they are not). - Patch Verdict: Matches the expected shape exactly. Both functions converted to accumulator form;
flushCurrent()preserves the never-flush-empty guard; the oversized-line fallback path unchanged in structure. The one extra hunk (collectionkey alignment inembedChunkscall) is a cosmetic in-file style fix. - Premise Coherence: Coheres — verify-before-assert end to end: the ticket refuses to use the external deployment as a test substrate and reproduces the defect from repository fixtures instead; friction→gold: a production incident became a permanent algorithmic guarantee with spy-asserted linearity.
🕸️ Context & Graph Linking
- Target Epic / Issue ID: Resolves #17013
- Related Graph Nodes: #16706 (epic, non-closing Related ✓), #16853 (sibling provider-recovery concern, correctly out of scope here)
- Origin Session ID: 3b6a56b3-6197-42d7-9bca-c5e9b1bb1496
🔬 Depth Floor
Challenge (per guide §7.1):
- Non-blocking follow-up concern — residual hotspots in the same lane: this PR removes the proven blocker, and the census strongly binds the blocked lane to this path. If the same repo lane still freezes the loop post-deploy, the next synchronous hotspot (parse/manifest side) gets its own ticket — do not fold it here. Named so the next observer doesn't read a recurrence as this fix failing.
- Unverified assumption, checked and cleared: byte-length additivity across accumulator items. For
splitTextByByteBudget, interior items always end in\nso no cross-item surrogate pairing can occur; forsplitLongStringByByteBudget,for…ofiterates code points, and adjacent valid surrogate pairs arrive as single items — additivity holds. Pathological lone-surrogate inputs could drift a part a few bytes pastmaxBytes(replacement-encoding asymmetry); with production budgets near 100KB this is cosmetic, and the pre-repair code had the same property. Not a regression; no action. - Timing assertions on shared CI:
<1000msbounds carry ~50× margin against the pre-repair >5s behavior, and the deterministicscannedCharsassertions are the real teeth — flake risk acceptable.
Rhetorical-Drift Audit (per guide §7.4):
- PR description: framing matches the diff (quadratic→linear claim is spy-verified, not asserted)
- Anchor & Echo summaries: the added JSDoc states the load-bearing constraint (event-loop starvation) without overshoot
-
[RETROSPECTIVE]tag: n/a in body — no inflation present - Linked anchors: #16706 comment cited actually establishes the external observation claimed
Findings: Pass.
🧠 Graph Ingestion Notes
[RETROSPECTIVE]: The review-worthy pattern: assert the algorithm, not the stopwatch. SpyingBuffer.byteLengthto prove each unit is measured exactly once (scannedChars === source.length,=== source.length * 2including the whole-text pre-check) turns a performance fix into a deterministic contract that cannot silently regress. This belongs in the unit-test idiom vocabulary.
N/A Audits — 📑 🪜 📡 🔗
N/A across listed dimensions: no public/consumed contract surface changed (internal split helpers, same signatures), close-target ACs fully covered by repository-local evidence (PR declares Evidence: L2 → L2 required, no residuals), no OpenAPI surfaces touched, no skills/conventions/MCP tools introduced.
🎯 Close-Target Audit
- Close-targets identified: #17013 (newline-isolated
Resolves✓) - #17013 confirmed not
epic-labeled; #16706 correctly non-closing underRelated:
Findings: Pass.
🧪 Test-Evidence & Location Audit
- Execution evidence: exact-head required CI green at
98643299a7; author non-CI receipts present and current-head (94/94 targeted unit run, 100,000-byte Unicode probe 3.748ms vs pre-repair >5s interrupted, 5,749,518-byte production-shaped fixture 88.678ms → 81 chunks exact round-trip, agent-preflight pass) - Reviewer falsifier: N/A — no named behavioral concern survived the additivity analysis above; the spy-based linearity assertions cover the algorithmic claim at exact head
- Test location: pass — both specs extend the canonical existing suites (
VectorService.WorkVolumeBranching.spec.mjs,IngestionService.spec.mjs) in place
Findings: Pass.
📋 Required Actions
No required actions — eligible for human merge.
📊 Evaluation Metrics
[ARCH_ALIGNMENT]: 100 - Repair lands inside the primitive that already owns both split helpers; no boundary moves, no provider specifics leaked, caller-suppliedmaxBytesremains the only budget authority. Checked for placement alternatives and hardcoded limits; none found.[CONTENT_COMPLETENESS]: 100 - Both modified methods carry JSDoc stating the constraint the code cannot show (quadratic re-measurement starves the shared event loop); PR body is a complete fat-ticket with census receipts, falsifiers, and an explicit deltas-from-ticket section.[EXECUTION_QUALITY]: 100 - Accumulator conversion verified equivalent including the surrogate/newline additivity analysis; never-flush-empty guard preserved; spy-asserted exact scan counts make the linearity claim deterministic; production ingestion path exercised end-to-end with lossless round-trip and unchanged accounting.[PRODUCTIVITY]: 100 - Every ticket AC delivered: linear work, exact content retention, byte boundaries, surrogate safety, chunk identity, bounded completion on production-scale fixtures.[IMPACT]: 90 - Removes the reproducible (n=2) event-loop blocker gating the external plane's multi-tenant ingestion convergence — the S2-critical defect of epic #16706; core ingestion primitive, not framework-wide.[COMPLEXITY]: 40 - Two focused function rewrites plus three tests; the care lives in Unicode boundary reasoning, not in breadth.[EFFORT_PROFILE]: Quick Win - Contained diff with outsized operational payoff on the constrained plane.
The next plane cycle after merge+deploy is the live confirmation: the previously blocking repo lane should refresh with the bridge writing throughout. 🖖
Resolves #17013
Related: #16706
Oversized raw-source splitting now measures each line or Unicode code point once instead of repeatedly measuring a growing prefix. Tenant refresh retains exact content, byte boundaries, surrogate-pair safety, and chunk identity while eliminating the synchronous quadratic event-loop block.
Evidence: L2 (real production methods under the repository-local unit harness plus a deterministic byte-work witness) → L2 required (all close-target ACs are repository-local). No residuals.
Deltas from ticket
None substantive. The repair stays inside the existing
VectorServiceprimitive. Coverage adds both direct linear-work witnesses and the productionIngestionService.ingestSourceFiles()raw-source path.After the PR opened, a second external observation reproduced the same discriminator: the same repository lane blocked all independent event-loop writers while a larger-file-count control lane kept yielding. A read-only corpus census then found six individual lines above 100,000 bytes (maximum 569,174), binding that lane to the repaired long-line path without using the external deployment as a test environment.
Test Evidence
npm run test-unit -- test/playwright/unit/ai/services/knowledge-base/VectorService.WorkVolumeBranching.spec.mjs test/playwright/unit/ai/services/knowledge-base/IngestionService.spec.mjs— 94/94 passed.splitOversizedEmbeddingChunk()at98643299a7processed the largest production-shaped 5,749,518-byte source fixture in 88.678 ms: 81 non-empty chunks, exact round-trip, maximum final provider input 85,901 bytes.npm run agent-preflight -- --change-class restoration --commit-subject 'fix(ai): linearize oversized ingestion splitting (#17013)' ...— passed; only unrelated stale-overlay warnings reported.git diff --cached --check— clean before commit.VectorService.WorkVolumeBranching.spec.mjscovers production-sized long lines, Unicode boundaries, many-line packing, byte-work linearity, and existing oversized-chunk identity behavior.IngestionService.spec.mjscovers a 105,000-character raw generated source through the realingestSourceFiles()split path, bounded completion, lossless materialization, and unchanged ingestion accounting.Post-Merge Validation
Authored by Emmy (GPT-5.6 Sol Ultra, Codex). Session 3b6a56b3-6197-42d7-9bca-c5e9b1bb1496.