KB chunking: separate ticket/PR body + each comment + each review cycle

Context

Tickets and PRs accumulate content across cycles (body + N comments; PR body + multiple review rounds). Measured this session (sizes only, bodies never read): no single body/review is over the ~30k-token qwen3 embedding cap today (largest open-issue body ≈ 6.7k tok #10074; largest first-round PR review ≈ 3.8k tok), but the whole-artifact blob trends toward it — #13999 body + 10 comments = 27,703 chars ≈ 7–8k tok and growing; a multi-round PR (body + 3-4 ~3k-tok reviews) reaches ~15k+.

The Architectural Reality (VERIFIED — read, not assumed)

The KB chunks tickets/PRs at whole-artifact granularity — one chunk per file, no element separation:

• ai/services/knowledge-base/source/TicketSource.mjs:62-106 — extract() reads each issue-*.md whole and writes one {type:'ticket', content} chunk per file.
• ai/services/knowledge-base/source/PullRequestSource.mjs (~66-97) — same shape: one chunk per PR .md (body + review conversations combined).
• Over-cap handling is structure-blind: KnowledgeBaseIngestionService.filterEmbeddingInputBudget drops over-budget chunks (→ artifact unindexed, the #14000 class), and VectorService.splitOversizedEmbeddingChunk → splitTextByByteBudget(content, maxContentBytes) splits by byte budget mid-content, blind to body/comment/review boundaries (→ semantic incoherence; a chunk can split across a comment boundary).

So a growing multi-cycle artifact is one growing chunk → eventually over-cap → dropped or blind-byte-split.

The Problem

A whole-artifact chunk that crosses the ~30k-token cap is either dropped (unindexed) or blind-split (incoherent). Both degrade retrieval; on a path without a hard skip (Memory Core) an over-cap input fails to embed → metadata-without-vector (the #13999 class). Multi-cycle tickets/PRs are a latent ingestion hazard, and the existing byte-split is a poor fallback (structure-blind).

The Fix

Enhance the per-entity Sources (TicketSource / PullRequestSource / DiscussionSource) to emit per-element chunks at extract time — body, each comment, each PR review cycle as separate chunks — instead of one whole-file chunk. Each element is small (measured ≤ ~7k tok body, ≤ ~0.9k tok comment), so coverage stays complete and no element approaches the cap regardless of cycle count, and the blind byte-split stops being the fallback for these entity types. Preserve per-element provenance (source artifact + element type/index) for retrieval.

Acceptance Criteria

• TicketSource / PullRequestSource / DiscussionSource emit separate chunks per structural element (body, each comment, each PR review cycle).
• Each emitted chunk carries element provenance (source artifact + element type/index).
• A synthetic large multi-comment/multi-review artifact produces only sub-cap, structurally-coherent chunks (no drop, no mid-element byte-split) — covered by a test.
• The structure-blind splitTextByByteBudget path is no longer reached for these entity types under normal cycle accumulation (it may remain a last-resort guard for a single oversized element).
• No retrieval regression for single-body artifacts.

Out of Scope

• Oversized single source file splitting (#14000 — distinct: splits one big file; this splits a multi-element artifact).
• The Memory Core write-path atomic-vector-write invariant (#14029 — the prevention complement).

Related

• #14000 — oversized-source chunking (sibling chunking concern, different granularity).
• #14029 — atomic vector-write invariant (over-cap → metadata-without-vector prevention).
• #14026 — detect-signal (catches an unindexed/uncovered artifact).
• #13999 — the incident; its growing comment thread is the live example.

Origin Session ID: 58acd7a6-778e-4fe6-b85a-4569802ac57b

Handoff Retrieval Hints: query_raw_memories("KB chunking per-element ticket PR body comment review qwen3 cap TicketSource splitTextByByteBudget"); anchors: ai/services/knowledge-base/source/TicketSource.mjs:62, source/PullRequestSource.mjs, VectorService.splitOversizedEmbeddingChunk.

Authored-by: @neo-opus-vega (Vega, Claude Opus 4.8)

Edit note: body grounded on a verified read of the current chunker (TicketSource/PullRequestSource/splitOversizedEmbeddingChunk) after an operator-flagged V-B-A miss — the original premise was asserted before the parsers were read. Premise confirmed; evidence now inline.