Add default raw-ingest Source for tenants with unknown repo shape

Context

Tenth friction signal from the first-real-world cloud-deployment exercise (siblings #12014 through #12028). Surfaced while planning R3 tenant ingestion (the polling path for an external mirror repo as a first tenant smoke). To ingest an arbitrary tenant repo today, the operator must author a CustomSource per CustomSources.md. For a "drop-in, just-let-me-see-what-happens" smoke against any unknown-shape repo, that is meaningful friction.

The Problem

Neo's ai/services/knowledge-base/source/ ships 10 typed Source classes:

AdrSource          DiscussionSource   PullRequestSource
ApiSource          LearningSource     ReleaseNotesSource
ConceptSource      TestSource         TicketSource
SourceRegistry

Each is Neo-shape-specific:

• LearningSource enumerates learn/
• AdrSource enumerates learn/agentos/decisions/
• TicketSource enumerates resources/content/issues/
• ApiSource enumerates JSDoc output
• etc.

For an arbitrary tenant repo (a client tenant's product repo, a partner's checkout, an internal tooling repo), none of these enumerators fit. The operator's options today:

1. Author a CustomSource per CustomSources.md — meaningful upfront effort; requires understanding parsed-chunk-v1 + the Source authoring contract; needs deployment-side wiring through aiConfig.customSources or kb-config.yaml. Right answer when the tenant's repo shape is known and ingestion ergonomics matter. Wrong answer for "let me ingest this and see what happens."

2. Set sourcePaths overrides on Neo's existing classes — only works if the tenant repo's shape happens to align with Neo's expectations (it almost never does for non-Neo repos).

3. Manual per-file ingest_source_files MCP calls — the Day-0 Milestone 2/4 path. Works file-by-file; doesn't scale beyond a smoke.

The parser layer already has a raw-text fallback per Overview.md: "with no parser match, the raw-text fallback ingests the whole file as one chunk". But the Source layer has no equivalent — there's no Source class that just enumerates every file in a repo and lets the parser-layer fallback handle the content.

The Architectural Reality

This is a Source-class gap, not a Parser-class gap. Distinct from #11735 (which is about deepening parser coverage across more file formats — the per-format depth of parsing). This ticket is about the enumeration layer one level up.

Affected:

• ai/services/knowledge-base/source/ — needs a new class (e.g., RawRepoSource).
• SourceRegistry — needs to register it as a built-in.
• aiConfig default-source defaults — needs a decision: does the new Source become a default for useDefaultSources: true, OR is it specifically the fallback for tenant configs that don't register any Source?
• kb-config.yaml schema — may need a defaultRawIngest: true flag (or implicit-when-no-customSources behavior).

The Fix — proposed shape

Add RawRepoSource (working name) under ai/services/knowledge-base/source/:

• Enumeration: recursively walks the configured repo root, emits each file as a chunk-source.
• Filter: respects a configurable allow/deny list (e.g., default-deny on .git, node_modules, dist, binary file extensions). Sensible defaults so a first-time operator gets reasonable behavior.
• Parser dispatch: hands each file to the existing parser registry. With no registered parser match, falls through to the raw-text parser (the existing parser-layer fallback). One chunk per file.
• Chunk metadata: stamps parsedChunks[].sourcePath with repo-relative path; kind defaults to 'raw' or matches the file extension when known (e.g., .md → 'doc-section'); name is the basename.

Defaulting behavior — implementer's call between two shapes:

| Option A | RawRepoSource becomes a built-in default that fires when a tenant has useDefaultSources: false AND no customSources registered. Implicit fallback. | | Option B | RawRepoSource exists but tenants must explicitly opt in via customSources: [{SourceClass: 'RawRepoSource'}] or a new rawRepoSource: true flag. Explicit. |

Option A is closer to the user's "drop in as-is parser, and that should be the default" framing — but raises a small risk that operators forget they're using it and don't author a better CustomSource later. Option B requires more explicit operator opt-in, more boilerplate-y, but never surprises.

Contract Ledger

Decision Record impact

aligned-with ADR 0014 — completes the tenant-ingestion ergonomic story under existing cloud-topology authority. No ADR successor risk.

Acceptance Criteria

• New Source class exists under ai/services/knowledge-base/source/ that enumerates an arbitrary repo tree and emits chunk-sources per file.
• Defaulting behavior chosen + documented (Option A or B per Fix section).
• At least one configurable file-filter (allow/deny list or extension filter) with sensible defaults that skip .git, node_modules, common binary formats.
• kb-config.yaml can configure a tenant to use it (whether implicitly or explicitly per chosen Option).
• Smoke test: configure a tenant pointing at any small public repo (or neomjs/create-app test fixture), run ingestion, verify chunks present in KB with sourcePath reflecting the repo tree.
• CustomSources.md updated with a "when to use built-in raw-ingest vs author a custom Source" decision section.

Out of Scope

• Per-format parser coverage — that's #11735. This ticket adds an ENUMERATOR; parser-layer raw-text fallback already handles unknown formats.
• Sophisticated file filtering (gitignore-respect, .dockerignore-respect, etc.) — basic deny-list is enough for v1; revisit when operators report friction.
• Tenant-specific path overrides via sourcePaths — the existing override surface stays for Neo-shape-aligned tenants; raw-ingest is for arbitrary-shape tenants.
• Server-side clone / fetch concerns — deferred to #11731.

Avoided Traps

• Naming as GenericSource — rejected; "generic" doesn't communicate the as-is/walk-the-tree intent. RawRepoSource or WalkRepoSource more accurately describes the enumerator behavior.
• Conflating Source-enumeration with Parser-content-parsing — rejected; they're separate layers (per Overview.md "Source vs Parser" split). The fix is at the Source layer.
• Coupling to the Default-Source registry too tightly (Option A only) — rejected for blanket adoption; defaults need careful thought + may belong in a separate defaultRawIngest config field.

Related

• ADR 0014 — cloud-deployment topology authority.
• #11658 (Phase 0/1B registry) — established the SourceRegistry; this ticket adds a new registered class.
• #11735 — sibling parser-coverage-deepening (distinct concern; parser depth, not source enumeration).
• #11726 — tenant-repo ingestion operational model (consumer of this).
• #11791 — operator docs + health/telemetry for tenant-repo ingestion (downstream documentation consumer).
• #12014, #12015, #12016, #12017, #12019, #12022, #12025, #12026, #12028 — sibling friction tickets from the first-real-world cloud-deployment exercise.

Handoff Retrieval Hints

• Grep: ls ai/services/knowledge-base/source/ — pre-fix shows 10 typed Sources; post-fix adds one generic.
• Semantic: "raw-ingest Source default tenant arbitrary repo shape", "drop-in Source no custom authoring".
• File anchor: ai/services/knowledge-base/source/ directory.