Frontmatter
| title | fix(ai): chunk oversized kb sources before embedding (#14000) |
| author | neo-gpt |
| state | Merged |
| createdAt | Jun 25, 2026, 1:22 PM |
| updatedAt | Jun 25, 2026, 3:44 PM |
| closedAt | Jun 25, 2026, 3:20 PM |
| mergedAt | Jun 25, 2026, 3:20 PM |
| branches | dev ← codex/14000-kb-oversized-source-chunking |
| url | https://github.com/neomjs/neo/pull/14003 |
| contentTrust | |
| projected | |
| quarantined | 0 |
| signals | [] |

Author-side ADR-0019 correction: the challenged test setup was a real B4 violation shape. The ingestion spec was mutating the shared Provider singleton via memoryConfig/aiConfig to force local embedding caps. I pushed 6f306053b7 to replace those writes with a test-local resolveEmbeddingInputGuardrail override restored through the existing singleton seam.\n\nValidation after the fix:\n- npm run test-unit -- test/playwright/unit/ai/services/knowledge-base/KnowledgeBaseIngestionService.spec.mjs passed, 37/37.\n- npm run agent-preflight -- test/playwright/unit/ai/services/knowledge-base/KnowledgeBaseIngestionService.spec.mjs passed.\n- git diff --check passed.\n- rg found no remaining memoryConfig/localModels.embedding mutation in the touched ingestion spec slice.

No review body provided.
Resolves #14000
Adds deterministic oversized-source chunking inside
KnowledgeBaseIngestionServicebefore the final embedding guardrail. Recoverable raw fallback and parser-produced text chunks are split into embedding-safe sub-chunks, keep trace metadata, and receive stable rehashed IDs; unsplittable chunks still use the existing boundedKB_INGEST_INPUT_SIZE_EXCEEDEDdiagnostic.VectorServiceremains the final refusal guardrail, and no provider limits are raised.Evidence: L2 (unit-level ingestion path with mocked
VectorServiceboundary and deterministic ID assertions) -> L2 required (pre-provider KB ingestion behavior). No residuals.Related: #13999 Related: #13930 Related: #13929
Deltas from ticket
No response schema change was needed: the existing summary fields already distinguish embedded chunks from final skipped chunks. Split metadata is stored on generated chunk records as
oversizedSplit,oversizedSplitIndex,oversizedSplitTotal,oversizedSplitCharStart, andoversizedSplitCharEnd.Test Evidence
npm run agent-preflight -- ai/services/knowledge-base/KnowledgeBaseIngestionService.mjs test/playwright/unit/ai/services/knowledge-base/KnowledgeBaseIngestionService.spec.mjspassed.npm run test-unit -- test/playwright/unit/ai/services/knowledge-base/KnowledgeBaseIngestionService.spec.mjspassed: 37/37.git diff --checkpassed.merge-base HEAD origin/dev == origin/devatf97d1562a63272c86fb44e71e055111b19794ad2.Post-Merge Validation
resources/content/issues/chunk-2/issue-12065.mdand verify rows exist for thatsourcePathwithoversizedSplitmetadata instead of a zero-vector skip.Commits
2603f918cc-fix(ai): chunk oversized kb sources before embedding (#14000)Evolution
This spun out of the
#13999backup investigation after the operator's log sample proved a separate KB completeness gap: the KB store can remain exportable while an oversized source is missing entirely because the prior safety behavior skipped instead of splitting. This PR keeps the safety invariant and recovers vectors for split-safe sources.Authored by Euclid (GPT-5, Codex Desktop). Session f4d00667-a65a-4285-83f5-6761f3aea394.