LearnNewsExamplesServices
Frontmatter
titlefix(ai): chunk oversized kb sources before embedding (#14000)
authorneo-gpt
stateMerged
createdAtJun 25, 2026, 1:22 PM
updatedAtJun 25, 2026, 3:44 PM
closedAtJun 25, 2026, 3:20 PM
mergedAtJun 25, 2026, 3:20 PM
branchesdevcodex/14000-kb-oversized-source-chunking
urlhttps://github.com/neomjs/neo/pull/14003
contentTrust
projected
quarantined0
signals[]
Merged
neo-gpt
neo-gpt commented on Jun 25, 2026, 1:22 PM

Resolves #14000

Adds deterministic oversized-source chunking inside KnowledgeBaseIngestionService before the final embedding guardrail. Recoverable raw fallback and parser-produced text chunks are split into embedding-safe sub-chunks, keep trace metadata, and receive stable rehashed IDs; unsplittable chunks still use the existing bounded KB_INGEST_INPUT_SIZE_EXCEEDED diagnostic. VectorService remains the final refusal guardrail, and no provider limits are raised.

Evidence: L2 (unit-level ingestion path with mocked VectorService boundary and deterministic ID assertions) -> L2 required (pre-provider KB ingestion behavior). No residuals.

Related: #13999 Related: #13930 Related: #13929

Deltas from ticket

No response schema change was needed: the existing summary fields already distinguish embedded chunks from final skipped chunks. Split metadata is stored on generated chunk records as oversizedSplit, oversizedSplitIndex, oversizedSplitTotal, oversizedSplitCharStart, and oversizedSplitCharEnd.

Test Evidence

  • npm run agent-preflight -- ai/services/knowledge-base/KnowledgeBaseIngestionService.mjs test/playwright/unit/ai/services/knowledge-base/KnowledgeBaseIngestionService.spec.mjs passed.
  • npm run test-unit -- test/playwright/unit/ai/services/knowledge-base/KnowledgeBaseIngestionService.spec.mjs passed: 37/37.
  • git diff --check passed.
  • Freshness check passed after rebase: merge-base HEAD origin/dev == origin/dev at f97d1562a63272c86fb44e71e055111b19794ad2.

Post-Merge Validation

  • Run a KB sync including resources/content/issues/chunk-2/issue-12065.md and verify rows exist for that sourcePath with oversizedSplit metadata instead of a zero-vector skip.

Commits

  • 2603f918cc - fix(ai): chunk oversized kb sources before embedding (#14000)

Evolution

This spun out of the #13999 backup investigation after the operator's log sample proved a separate KB completeness gap: the KB store can remain exportable while an oversized source is missing entirely because the prior safety behavior skipped instead of splitting. This PR keeps the safety invariant and recovers vectors for split-safe sources.

Authored by Euclid (GPT-5, Codex Desktop). Session f4d00667-a65a-4285-83f5-6761f3aea394.

neo-gpt
neo-gpt commented on Jun 25, 2026, 3:02 PM

Author-side ADR-0019 correction: the challenged test setup was a real B4 violation shape. The ingestion spec was mutating the shared Provider singleton via memoryConfig/aiConfig to force local embedding caps. I pushed 6f306053b7 to replace those writes with a test-local resolveEmbeddingInputGuardrail override restored through the existing singleton seam.\n\nValidation after the fix:\n- npm run test-unit -- test/playwright/unit/ai/services/knowledge-base/KnowledgeBaseIngestionService.spec.mjs passed, 37/37.\n- npm run agent-preflight -- test/playwright/unit/ai/services/knowledge-base/KnowledgeBaseIngestionService.spec.mjs passed.\n- git diff --check passed.\n- rg found no remaining memoryConfig/localModels.embedding mutation in the touched ingestion spec slice.


tobiu
tobiu APPROVED reviewed on Jun 25, 2026, 3:19 PM

No review body provided.