Chunk oversized Knowledge Base sources before embedding

Context

While investigating the P0 Memory Core backup exportability incident in #13999, the operator surfaced an adjacent Knowledge Base sync log from 2026-06-25:

tenantId: 'neo-shared',
repoSlug: 'neo',
source: 'resources/content/issues/chunk-2/issue-12065.md',
inputBytes: 91892,
inputTokensEstimate: 30631,
safeProcessingLimitTokens: 28672,
contextLimitTokens: 32768
[WARN] [VectorService] Skipping over-budget embedding chunk before provider invocation.

V-B-A on the current checkout confirmed the operator's completeness concern:

• resources/content/issues/chunk-2/issue-12065.md exists locally and is 91868 bytes.
• Read-only Chroma lookups against neo-knowledge-base for that exact path under metadata keys sourcePath, source, path, file, and filePath returned 0 rows.
• npm run ai:check-chroma-integrity -- --json --exportability-sample-size 10 --vector-coverage-sample-size 10 still shows KB sampled stored-embedding exportability as 10/10, so this is not the direct Memory Core backup root cause, but it is a real Knowledge Base completeness gap.

Duplicate sweep:

• Live latest-open sweep: checked the latest 20 open issues at 2026-06-25T11:04Z through the GitHub API; no equivalent open issue existed. The closest open related issue is #13999, which owns the Memory Core backup exportability repair, not KB source chunking.
• A2A in-flight claim sweep: checked latest 30 all-status messages at 2026-06-25T11:04Z; no competing [lane-claim] or [lane-intent] for KB oversized-source chunking was present.
• Semantic ticket sweep: ask_knowledge_base found no open ticket for splitting oversized KB raw files into embedding-safe chunks. It identified #13930 as the closest prior ticket, but that ticket is closed and explicitly scoped to skip-and-diagnose; its Out of Scope says automatic splitting/summarizing monster files was not included.

Release classification: boardless follow-up adjacent to #13999. This improves KB completeness but is not the immediate P0 Memory Core backup repair unless the operator promotes it.

The Problem

The current KB ingestion safety net prevents embedding-provider burn by skipping chunks whose final embedding input exceeds the local safe-processing band. That is correct as a last-resort guardrail, but it becomes lossy when the skipped item is a raw or parser-output chunk that could be safely split.

The verified example is resources/content/issues/chunk-2/issue-12065.md: the source exists in the repo, exceeded the embedding safe band, was skipped, and currently has no indexed KB row for the checked source-path metadata keys. The resulting KB corpus is incomplete even though the sync did not corrupt the vector store and the remaining KB sampled embeddings are exportable.

This is the work that #13930 deliberately deferred. #13930 made oversized skips visible and safe; it did not implement a chunking strategy to recover embeddable vectors for oversized sources.

The Architectural Reality

Relevant current surfaces:

• ai/services/knowledge-base/KnowledgeBaseIngestionService.mjs collects parsed chunks, then calls filterEmbeddingInputBudget() before embedChunkGroups().
• KnowledgeBaseIngestionService.filterEmbeddingInputBudget() drops local-provider over-budget chunks before writing the temp JSONL for VectorService and records KB_INGEST_INPUT_SIZE_EXCEEDED diagnostics.
• KnowledgeBaseIngestionService.rawFileToParsedRecord() turns a raw fallback source into one parsed-chunk-v1 record whose content is the whole file.
• ai/services/knowledge-base/VectorService.mjs remains the final safety net and skips any over-budget chunk that reaches embedChunks() before provider invocation.
• ai/services/knowledge-base/parser/DocumentationParser.mjs, SourceParser.mjs, and TestParser.mjs are existing parser/chunking surfaces; this ticket should preserve their parser-specific behavior and focus on the generic oversized-source fallback/normalization gap.

The invariant to preserve: over-budget input must not be sent to the local embedding provider. The change is to split recoverable oversized sources before they hit the skip path, not to weaken the guardrail.

The Fix

Add deterministic, embedding-budget-aware chunking for oversized Knowledge Base source content before the final skip gate.

Expected shape:

1. Detect when a raw fallback record or parser-produced textual chunk would exceed the active local embedding safeProcessingLimitTokens band.
2. Split recoverable textual content into stable sub-chunks below the budget, carrying source traceability such as sourcePath, chunk index, and line or character range where available.
3. Keep chunk identity deterministic so unchanged oversized sources do not churn KB IDs across syncs.
4. Preserve the existing skip-and-diagnose behavior for genuinely unsplittable chunks, invalid input, or a single block that still exceeds the safe band after bounded splitting.
5. Keep diagnostics bounded and non-secret; never emit raw source content.
6. Add focused tests using the issue-12065.md shape or an equivalent fixture so a future sync produces embeddable sub-chunks instead of zero vectors.

Implementation does not need a new module. If intake chooses to add a new .mjs helper, run structural-pre-flight first and place it under the existing KB parser/service ownership boundary.

Contract Ledger Matrix

Decision Record impact

Aligned with ADR 0019: read resolved AiConfig leaves at use sites; do not introduce a second config-resolution path or hard-coded safe-token defaults.

Acceptance Criteria

• A raw fallback KB source whose embedding input exceeds safeProcessingLimitTokens is split into deterministic sub-chunks before VectorService.embed().
• Parser-produced textual chunks that exceed the safe band are either split safely or fail with the existing bounded skip diagnostic; they are not sent to the local embedding provider.
• Safe sub-chunks from the same original source still embed and upsert.
• The existing skip-and-diagnose behavior remains for genuinely unsplittable or still-over-budget chunks.
• The generated sub-chunks retain enough metadata to trace back to the original sourcePath and chunk/range.
• Re-ingesting unchanged oversized content does not churn IDs.
• Focused tests cover the issue-12065.md-class shape or an equivalent oversized markdown fixture.
• If the ingestion response schema changes, ai/mcp/server/knowledge-base/openapi.yaml is updated and covered.

Out of Scope

• Weakening the over-budget guardrail or sending oversized inputs to the embedding provider.
• Raising safeProcessingLimitTokens or model context limits to mask the issue.
• Solving the Memory Core backup exportability failure from #13999.
• Full tenant-specific parser inventory or source-family coverage; this ticket is the generic oversized-source fallback.
• Emitting raw source content in logs, metrics, errors, or A2A.

Avoided Traps

• Do not treat #13930 as complete coverage for this problem; it intentionally stopped at diagnostics and skip safety.
• Do not make VectorService responsible for arbitrary parser semantics. It can remain the final guardrail; source/chunk normalization belongs earlier in KB ingestion.
• Do not split by bytes alone if it can produce unstable or semantically useless fragments. Prefer deterministic line/paragraph boundaries with a token-estimate backstop.
• Do not make this release-blocking by default. It is adjacent evidence from #13999, not the direct MC backup root.

Related

Related: #13999 Related: #13930 Related: #13929 Related: #13928

Origin Session ID: f4d00667-a65a-4285-83f5-6761f3aea394

Handoff Retrieval Hints

• query_raw_memories("issue-12065 over-budget KB chunk skipped sourcePath zero rows")
• query_raw_memories("KnowledgeBaseIngestionService skippedOversized split-document oversized raw fallback")
• Exact anchors: KnowledgeBaseIngestionService.filterEmbeddingInputBudget, KnowledgeBaseIngestionService.rawFileToParsedRecord, VectorService.embedChunks, resources/content/issues/chunk-2/issue-12065.md