Long source lines can block tenant refresh on the event loop

Context

A container-plane TenantRepoSync run on release pin fcc0f8149422d83f481969dfd866480b96c40a8f stopped every independent orchestrator writer for about 17 minutes while the Node process held one CPU continuously. The deployment-state bridge snapshot, orchestrator state-file mtime, and logs all stopped advancing after the lane logged [TenantRepoSync] Refreshing ... (embedding recovery generation); the process later exited and restarted without a recovery-actuator event.

The public outcome receipt is #16706 comment 5265232411. This ticket is the narrow Neo-side repair; verification runs only on our Agent OS and repository fixtures.

Problem

The raw-text oversize path has a source-proven quadratic inner loop:

IngestionService.filterEmbeddingInputBudget() → VectorService.splitOversizedEmbeddingChunk() → splitTextByByteBudget() → splitLongStringByByteBudget().

For every code point in a long line, splitLongStringByByteBudget() constructs current + char and calls Buffer.byteLength() over the entire growing fragment. With the production-safe byte budget near 100 KiB, a minified/generated source line repeatedly rescans almost the whole 100 KiB prefix. Because this work is synchronous inside the orchestrator process, deployment-state writes, task-state writes, health timers, and logs cannot run.

Exact current-source falsifier on origin/dev: splitting 100,000 ASCII characters with a 100,000-byte budget did not return within 5 seconds and had to be interrupted. The 1,000-character control completed in 1.063 ms and round-tripped exactly. This proves the algorithmic defect; the external observation establishes the matching plane-level failure shape without making private source content part of the public contract.

Architectural Reality

• tenant-repo-sync is a container-plane scheduler lane under ADR 0014.
• TenantRepoSyncService builds a full raw-file envelope, then calls IngestionService.ingestSourceFilesForTenantSync() in the same orchestrator process.
• Raw-text files are measured and split before embedding. That CPU work therefore shares the event loop with the deployment-state bridge and scheduler writers.
• Git mirror operations already use asynchronous child processes. A slow remote/provider await would not freeze unrelated timers and writers; this synchronous rescanning loop can.
• npm run --silent ai:structure-map -- --files --loc was attempted during intake and hit V8's maximum-string ceiling. Existing ownership is nevertheless explicit at ai/services/knowledge-base/VectorService.mjs, with regression coverage beside its existing unit suite.

Fix

Replace growing-prefix byte remeasurement with a linear, Unicode-safe byte accumulator:

1. measure each code point once;
2. retain exact source order and byte-budget boundaries;
3. flush a fragment before the next code point would exceed the budget;
4. preserve surrogate pairs and all existing chunk identity fields;
5. keep the change inside the existing VectorService primitive—no worker, daemon, config leaf, or client-side instrumentation.

Also remove the same growing-prefix remeasurement from the multi-line accumulator so many short lines cannot recreate the quadratic shape.

Contract Ledger

Acceptance Criteria

• A red-before-fix fixture with a 100,000-character single line and production-shaped byte budget completes within a bounded local test window after the repair.
• The fixture round-trips byte-for-byte, every part is non-empty and within budget, and Unicode astral characters are never split.
• A many-line fixture proves the outer accumulator is linear too; removing either accumulator repair makes its named witness fail.
• Existing oversized-chunk identity, hashing, and ingestion tests stay green.
• A production-shaped TenantRepoSync/IngestionService control shows the event loop remains responsive while the pathological source is materialized; no writer gap may approach the 60-second deployment-state freshness contract.
• No external deployment is used as a test environment. Verification is repository-local / our Agent OS only.
• No new diagnostics surface or config knob is introduced.

Decision Record Impact

Aligned with ADR 0014; no amendment required. The lane remains container-plane owned and its semantics do not move. This repairs one existing same-process primitive so the accepted topology stays responsive.

Out of Scope

• Provider cancellation / abandoned Ollama work (#16853).
• Recovery-controller actionability (#16766).
• Provider admission or model scheduling policy.
• Any external deployment mutation or diagnostic experiment.

Related

• Parent epic: #16706
• Production-aligned sibling: #16853
• Controller sibling: #16766

Origin Session ID: 3b6a56b3-6197-42d7-9bca-c5e9b1bb1496

Retrieval Hint: query_raw_memories("TenantRepoSync refresh long source line event loop VectorService splitLongStringByByteBudget")