Surface oversized KB raw-file skips in diagnostics

Context

While reviewing #13928 / PR #13929, the operator raised a separate tenant-repository ingestion risk: custom parsers are not yet deployed for every tenant source family, so a large raw source file can enter ingest_source_files as one monolithic file/chunk. PR #13929 stops local embedding providers from grinding on an over-budget final embedding input, but that protection is intentionally scoped to VectorService and keeps skip details internal/logged so the current PR does not widen MCP response contracts.

Live latest-open sweep: checked latest 20 open issues at 2026-06-23T14:14:05Z; no equivalent active issue found. Related broad parser inventory ticket #11735 exists, but it is parked/re-triage scoped to parser coverage deepening, not this immediate burn-prevention diagnostic gap. A2A in-flight sweep: checked latest 30 messages; no competing lane-claim or ticket-creation claim for this scope found.

Release classification: post-#13929 hardening follow-up, not a #13929 merge blocker unless the operator promotes it.

The Problem

ingest_source_files has a coarse up-front volume gate, but raw files count as one item before server-side parsing. A 5k-line source file can therefore pass the MCP facade, become one raw-text parsed chunk, and only be detected later at the embedding boundary.

After #13929, that late detection should prevent model CPU burn. The remaining problem is observability and operator actionability:

• the ingestion summary does not expose an oversized-skipped count/detail;
• daemon diagnostics cannot reliably distinguish "legitimate zero embeddings" from "monster file skipped because parser coverage/chunking is missing";
• relying only on graph rows is insufficient when embedding/model threads are wedged or skipped before graph write;
• broad custom parser coverage belongs to #11735, but a fail-safe diagnostic must exist before all tenant-specific parsers are ready.

The Architectural Reality

• ai/mcp/server/knowledge-base/ingestSourceFilesTool.mjs:53 documents that raw files count as 1 and that the gate is only a coarse pre-parse guard.
• ai/services/knowledge-base/KnowledgeBaseIngestionService.mjs:650 resolves a raw string payload into parser output or falls back to raw text.
• ai/services/knowledge-base/KnowledgeBaseIngestionService.mjs:1054 turns fallback raw text into one parsed-chunk-v1 record whose content is the whole file.
• ai/services/knowledge-base/KnowledgeBaseIngestionService.mjs:283 increments summary.embeddingsGenerated from VectorService.embed() but has no skipped-oversized summary field.
• ai/services/knowledge-base/VectorService.mjs:315 is now the safety net that emits ConsumerFriction and logger warning before local provider invocation.
• ai/mcp/server/knowledge-base/openapi.yaml:971 defines IngestSourceFilesResponse; any new ingestion-summary field must update this schema.

The Fix

Add an ingestion-layer oversized-input diagnostic gate after raw/client parser output is normalized but before embedChunkGroups() writes the temp JSONL for VectorService.

The gate should:

• evaluate the same local embedding safe-processing band used by #13929 for local providers;
• drop over-budget chunks from the embed queue before VectorService.embed() is called;
• emit a bounded, non-secret diagnostic with tenantId, repoSlug, sourcePath, parser id/version, byte count, token estimate, and safe threshold, never raw content;
• expose the skip to callers via the ingestion summary and IngestSourceFilesResponse OpenAPI schema;
• record/log the event through existing KB logging/recording surfaces so #13914 / #13924 diagnostic tooling can consume it without any new public surface outside Knowledge Base / Memory Core.

Contract Ledger Matrix

Decision Record impact

Aligned with ADR 0019: read AiConfig resolved leaves at use sites; do not add fallback defaults, alias config, or create a second config-resolution path.

Acceptance Criteria

• Oversized raw fallback chunks are skipped before VectorService.embed() / local provider invocation.
• Oversized client-provided parsedChunks are also skipped before embedding.
• Safe chunks in the same payload still embed.
• The ingestion summary exposes a bounded skipped-oversized signal and stable error code.
• ai/mcp/server/knowledge-base/openapi.yaml is updated if the response shape gains a field.
• KB logging/recording emits non-secret diagnostic metadata that #13914 / #13924 tooling can consume.
• Focused unit coverage proves raw fallback, client parsed-chunk, safe-remainder, and schema behavior.

Out of Scope

• Implementing tenant-specific custom parsers or full source-family inventory; #11735 owns that broader work.
• Automatically splitting or summarizing monster files. This ticket is skip-and-diagnose, not chunking strategy.
• Adding any public surface outside Knowledge Base / Memory Core.
• Private deployment config changes.

Avoided Traps

• Do not solve this by raising context limits or mcpSyncMaxChunks; that only postpones the failure and can reintroduce burn.
• Do not expose raw source content in diagnostics.
• Do not make the daemon depend on graph rows for a pre-embedding skip; the whole point is diagnosing cases where nothing reaches the graph.
• Do not broaden #13929; its PR remains the model-burn guardrail and this ticket handles follow-up diagnostics.

Related

• #13928
• #13929
• #13914
• #13924
• #11735

Origin Session ID: 1be84a2d-a911-424a-bfb0-10a51fff6303

Retrieval Hint: query_raw_memories("qwen embedding burn oversized raw tenant repo file skip diagnostics #13928 #13929")