Chunking parsers for pull-mode tenant-repo ingestion

Context

Neo's pull-mode tenant ingestion (the orchestrator tenant-repo-sync lane) materializes tenant repos through RawRepoSource — one envelope per file, embedded as one vector. On the first production multi-tenant plane this collided with embedding geometry twice in one week:

• Individual repo files run past 25k tokens. A whole-file input at that length is both semantically mushy as a single vector and disproportionately expensive to embed — the provider-infeasibility RCA and the geometry mechanics live in the #17062–#17070 batch (especially #17070, which bound the resolved safe band through composition and typed overflow handling).
• The deployment layer now deliberately caps the admitted embedding band (16,384-token slots / 14,336-token safe band). Oversized whole files refuse loudly pre-provider, and #17133 grades a twice-expired call ceiling to undeliverable-at-geometry — a named terminal state. The refusal list enumerates exactly the files a chunking parser must cover.

The operating model is settled: the deployment author configures parsing for the tenant — the tenant configures nothing. That keeps this epic's consumer a deployment artifact (config + docs), never a self-serve parsing UI.

The Problem

The dispatch mechanism for tenant parsing already exists end-to-end — what is missing is anything production-grade to dispatch to:

• Per-repo parser selection is plumbed through the pull lane (ai/daemons/orchestrator/services/TenantRepoSyncService.mjs:2230 forwards repo.parserId), and ai/services/knowledge-base/IngestionService.mjs:1577-1580 resolves file.parserId || 'raw-text' against the registry, failing loudly on an unknown id.
• The tenant config model already resolves useDefaultSources / useDefaultParsers / customSources / customParsers / sourcePaths through the 3-tier chain (graph node → bootstrap yaml → aiConfig; IngestionService.mjs:1740-1797).
• The registration pattern is proven by the cloud-deployment exemplar (ai/examples/cloud-deployment/minimal-external-workspace — ProtoParser, registered by parserId via customParsers).
• But the only universally applicable parser is raw-text (whole-file). Neo's default Source/Parser set is neo-repo-specific (learn content, framework src, tickets, releases). An arbitrary tenant code repo has no chunking story: no generic code chunker, no markdown/docs chunker, no size-bounded fallback, no geometry guarantee, and no documented recipe a deployment author can follow.

Net effect on the production plane: tenant repos either ingest whole files (mushy retrieval) or refuse at geometry (no ingestion progress) — both hostage to file size.

The Architectural Reality

• Pull lane: TenantRepoSyncService (in the 43-file ai/daemons/orchestrator/services/, per the structure map) → diff-to-ingest envelopes → IngestionService.ingestSourceFiles → parser dispatch → VectorService.embed.
• Registry: ai/services/knowledge-base/source/_export.mjs (SourceRegistry) with RawRepoSource.mjs as sibling; default-source auto-registration honors useDefaultSources !== false (ai/services/knowledge-base/DatabaseService.mjs:11,825).
• Config: ai/mcp/server/knowledge-base/configBase.mjs:529-566 declares the leaves; ADR 0019 governs consumption (read at the resolved leaf, never re-derive).
• Geometry: the admitted band is deployment-declared (NEO_LOCAL_MODELS_EMBEDDING_CONTEXT_LIMIT_TOKENS, plus the providerLaneDeclaration provenance namespace from #17069); #17070 bound the safe band; #17133 named the terminal state.
• Phase 0/1 contracts from the v13.0.0 tenant cluster (archived): path-identity tuple {tenantId, repoSlug, rootKind, sourcePath}, parsed-chunk-v1 schema, server-stamped write isolation, $in-filtered read isolation.

Intended solution shape

Tenant repo files route through registered chunking parsers with a hard geometry contract, configured per repo by the deployment author:

1. Parsers and Sources that emit SEMANTICALLY MEANINGFUL units — a function, a method, a class, a heading section, a config block. That is what a parser is for. Reasonable chunk size is the consequence of cutting on real boundaries, never the objective: a parser that splits by size produces chunks that retrieve badly, because a half-function is not a unit of meaning. Per-language fidelity is incremental; where structure is not cheaply available the honest answer is that the family has no parser yet, not a byte-splitter wearing a parser's name.
2. Oversize handling is NOT this epic's problem — the logic already exists. IngestionService.filterEmbeddingInputBudget drops oversized chunks before the VectorService write "for EVERY provider, recognized or not", with splitOversizedEmbeddingChunk ahead of it and VectorService as the final safety net. A chunk that is still too big after semantic parsing is hard-cut there, today. Building a geometry contract into the parser layer would duplicate that machinery one layer too high.
3. Deployment-tier configuration: per-repo parser selection in the existing bootstrap tier (repo entries in the kb-config yaml + the graph-node tier) — config-only on the consuming plane, ADR-0019-clean. Today's plumbing is one parserId per repo; the shape must allow extension-dispatch beneath a single repo-level id, because real repos mix code, docs, and data files.
4. Parser identity participates in checkpoint/contract keying, so a parser change forces a clean re-ingest of affected repos instead of a mixed-generation corpus — the ingestContractVersion mechanism, sibling to how vectorDimension already keys the sync (TenantRepoSyncService.mjs:1101).
5. Deployment-author docs: the consumption recipe, plus the retrieval-visible metadata parsed chunks carry (path-identity tuple + symbol/heading anchors).

Sub-tickets are linked incrementally via native relationships as decomposition clarifies; ACs live in the subs. This needs an epic rather than a single ticket because the deliverables span distinct substrates — the KB parser registry, the orchestrator pull lane's keying, the config tiers, and docs — converging on one outcome: tenant repos ingest as parsed, geometry-safe chunks.

Out of Scope

• Tenant self-serve parser authoring or configuration UI — the deployment author configures.
• Per-repo visibility/membership ACLs (tracked separately in the tenant lineage).
• Retrieval-side ask/query work (the #16566 sibling lineage owns sync-scheduling and retrieval-horizon concerns).
• Any specific deployment's config artifacts — private planes consume this epic; they are not its deliverable.
• The tenant source-family inventory, and volume-gate-aware bulk/backfill ingestion — owned by sub #11735, not restated here. This epic builds layout-agnostic chunkers; #11735 enumerates which families a given tenant actually has and verifies coverage against that enumeration, and owns the bulk/backfill path across families. Filed 2026-05-21 by @neo-opus-ada and re-parented here on 2026-08-17 after its own gating condition fired on a deployed tenant (raw-text fallback where structured parsing was expected) and its original parent #11730 closed.

Avoided Traps

• A per-language AST parser zoo up front — rejected. Language fidelity arrives incrementally behind the same registry ids.
• Any logic built AROUND what parsers are for — STRICT DECLINE (operator ruling, 2026-08-17). Size-bounded splitters, chunker-enforced geometry contracts, undeliverable-at-geometry-by-construction: all of it re-implements filterEmbeddingInputBudget / splitOversizedEmbeddingChunk at the wrong layer. Parsers emit meaningful units; the existing hard cut-off handles whatever is still oversize.
• Treating this as a SIZE problem at all — rejected. Whole-file chunks are bad because a whole file is not a unit of meaning, not because it is long. A 200-line file and a 20,000-line file both retrieve better as functions. Sizing evidence justifies priority, never the design.
• Raising the geometry ceiling instead of chunking — rejected, measured: large-input embedding is infeasible on CPU planes (#17062–#17070 RCA) and a >25k-token single vector retrieves poorly everywhere.
• Client-side parsing (the parsed-chunk-v1 push path) as this program's delivery vehicle — rejected here: the settled operating model is server-side parsing configured by the deployment author. The push contract remains valid for other integrations.
• Treating raw-text as deletable — it stays the explicit fallback for unmatched file types; parsed repos simply stop defaulting to it.

Related

• Tenant lineage (archived, Phase 0/1/2): #11625, #11626, #11635, #11637, #11731, #11791 · Discussion #12034 (tenant-state control plane) · #15667, #15748 (closed hardening)
• Geometry/provider: #17062, #17066, #17070, #17133, #17069
• Sibling epic: #16566 (tenant ingestion failure stages; scheduling + retrieval horizon)
• Sub: #11735 (tenant source-family inventory + parser coverage verification + bulk/backfill) — re-parented from the closed #11730

Scope narrowing (2026-08-17)

This epic is narrower than originally filed. Two things it once claimed as deliverables already exist and are not built here:

• Parser dispatch. IngestionService.resolveParser (:1610) reads the registry via getParserIds?.() / getParsers?.(); dispatch runs at :1577-1596 on file.parserId || 'raw-text', failing loudly with KB_PARSER_NOT_REGISTERED. Both customSources and customParsers are live extension points, resolved per tenant through IngestionService.getTenantConfig's 3-tier chain. Nothing to wire. (Residual doc defect, not this epic's: SourceRegistry's JSDoc still defers execution wiring to "Phase 2 / Phase 3 (#11626 / #11627)" — both closed, wiring landed.)
• Oversize cut-off. IngestionService.filterEmbeddingInputBudget drops oversized chunks before the VectorService write for every provider, with splitOversizedEmbeddingChunk ahead of it and VectorService as final safety net.

What remains: parsers that cut on real semantic boundaries, per-repo configuration recipes, and parser identity in checkpoint keying.

Sub relationship (2026-08-17)

#11735 (@neo-opus-ada, 2026-05-21) predates this epic and covers overlapping ground; it was re-parented here after its own gating condition fired and its original parent #11730 closed. Of its three scope items only parser-dispatch coverage collides with this epic; its inventory and bulk/backfill halves are unique and are owned there.

Decision Record impact

aligned-with ADR 0014 (cloud deployment topology / scheduler task taxonomy) and ADR 0019 (AiConfig SSOT consumption). No new ADR at epic level; a sub introducing a new contract surface declares its own.

Live latest-open sweep: checked latest 20 open issues (created-desc) at 2026-08-16T21:05Z — no equivalent; nearest neighbors #17238 (data-sync epic) and #16566 (sibling lineage). A2A in-flight claim sweep: latest 30 messages scanned — no overlapping claim. Structure map gate: run (--files --loc; owning folders cited above).

Origin Session ID: f4ac243f-ea07-4c54-828e-a7f5c32f98b1 Retrieval Hint: query_raw_memories("tenant repo chunking parsers pull-mode geometry admitted band 16k ceiling")