Context
The operator reported a current KB regression suspicion: semantic search appears to fall into fallback/noisy paths after the repo added vast amounts of synced pull-request conversations. The requested V-B-A focus is pre-query weighting, not release-note polish or post-answer wording.
Live latest-open sweep: checked the latest 20 open issues at 2026-06-08T01:31:49Z via GitHub API; no equivalent open ticket found. Closest adjacent work:
- #12715 / PR #12718 reduces code-term lexical rescue I/O cost, but does not change Chroma candidate budgets or historical-source weighting.
#10057 added PullRequestSource when PR conversations were missing from the KB; this ticket addresses the opposite pressure now that PR conversation text is large enough to compete with source/docs.#7474 is historical query scoring work, but it predates the current PullRequestSource corpus pressure.
Memory-mining sweep: queried for knowledge base semantic search regression pull conversations query weights fallback, ask knowledge base answers got worse after indexing pull requests, and pre query weights source docs should outrank conversations; no prior session mapped this exact fix.
The Problem
type='all' currently starts from a single Chroma query over the full KB collection. That means large historical/conversation corpora can consume the candidate set before Neo source files and documentation reach the reranker.
Empirical evidence from the current checkout:
dist/ai-knowledge-base.jsonl contains pull: 689 chunks, but those chunks carry 7,343,216 content bytes, larger than src and guide bytes combined in the generated JSONL snapshot.ai/services/knowledge-base/source/PullRequestSource.mjs:86-94 embeds each PR markdown conversation as a single type: 'pull' chunk with full content.ai/services/knowledge-base/QueryService.mjs:131-153 builds one type='all' Chroma query with no source-tier filter.ai/services/knowledge-base/QueryService.mjs:201-202 only applies explicit post-query penalties for ticket and release; pull chunks are neutral once returned.ai/mcp/server/knowledge-base/config.mjs:343-358 has no pullPenalty or candidate-pool configuration.
A pure post-rank penalty is insufficient if the semantic top-k is already starved before source/docs appear.
The Architectural Reality
The owned substrate is the Knowledge Base query service and its MCP config surface:
QueryService.queryDocuments() owns Chroma candidate retrieval and hybrid scoring.PullRequestSource is valid signal, but it is historical execution/conversation evidence, not primary current source-of-authority material.learn/agentos/KnowledgeBase.md documents queryScoreWeights as a tuning surface; the new behavior should extend that tuning model rather than hard-coding an invisible special case.type='src' currently misses Agent OS implementation files indexed as type: 'ai-infrastructure', so source-like typed searches can fall into unrelated framework source results instead of KB service files.
The Fix
Implement candidate-stage weighting for QueryService.queryDocuments():
- For
type='all', issue bounded Chroma candidate queries by source tier before reranking:
- primary/current: source-like and documentation types such as
src, ai-infrastructure, guide, concept, and skill;
- secondary: app/example/test/raw-like material;
- historical:
ticket, pull, discussion, and release with a much smaller candidate budget.
- Preserve explicit typed searches, but make
type='src' include ai-infrastructure so Agent OS service source is reachable through the documented source-code filter.
- Add
queryScoreWeights.pullPenalty for post-query scoring so returned PR conversation chunks remain lower priority than current source/docs in type='all' results.
- Add unit coverage around candidate pools, source-like
src expansion, and historical pull deweighting.
Contract Ledger Matrix
| Target Surface |
Source of Authority |
Proposed Behavior |
Fallback |
Docs |
Evidence |
QueryService.queryDocuments({type:'all'}) |
QueryService.mjs |
Uses source-tier candidate pools before scoring, so historical conversations cannot monopolize top-k candidates |
If a pool is empty, remaining pools still return results; no answer should fail solely because pulls are deweighted |
Update learn/agentos/KnowledgeBase.md scoring/config section if public config names change |
Unit test asserts primary pool budget exceeds historical pool budget and results combine correctly |
queryScoreWeights.pullPenalty |
KB config files |
Applies only during broad type='all' reranking, analogous to existing ticketPenalty |
Explicit pull-focused future tool/type can bypass broad-search penalty |
Config comment / KnowledgeBase docs |
Unit test asserts a returned pull chunk ranks below source/doc evidence when both are available |
type='src' source filter |
QueryService.mjs + ApiSource type map |
Includes ai-infrastructure with src because Agent OS source files are implementation source |
Explicit internal callers can still request ai-infrastructure if they bypass public enum |
Tool description may need later schema broadening; not required for this PR |
Unit test asserts the Chroma where filter uses $in: ['src', 'ai-infrastructure'] |
Decision Record Impact
Decision Record impact: none. This is aligned with the existing KB hybrid-ranking architecture and the #10057 PullRequestSource premise, but changes its broad-search weighting now that corpus proportions changed.
Acceptance Criteria
Out of Scope
- Re-embedding the KB collection.
- Removing PullRequestSource or deleting PR conversation files.
- Changing Memory Core query_summaries behavior from #12450.
- Final v13 release-note narrative work.
Avoided Traps
- Only add a post-rank penalty: rejected because it cannot rescue source/docs absent from Chroma top-k.
- Exclude pull conversations entirely: rejected because PR reviews remain valuable historical execution evidence for architectural archaeology.
- Increase global
nResults: rejected because it raises latency/cost and still gives no semantic source-tier guarantee.
Related
Origin Session ID: e8f07ef9-ef7e-4815-8ff4-7abe13720621
Handoff Retrieval Hints: knowledge base pre-query candidate weighting, PullRequestSource full conversation chunks, queryScoreWeights pullPenalty, type src ai-infrastructure.
Context
The operator reported a current KB regression suspicion: semantic search appears to fall into fallback/noisy paths after the repo added vast amounts of synced pull-request conversations. The requested V-B-A focus is pre-query weighting, not release-note polish or post-answer wording.
Live latest-open sweep: checked the latest 20 open issues at 2026-06-08T01:31:49Z via GitHub API; no equivalent open ticket found. Closest adjacent work:
#10057addedPullRequestSourcewhen PR conversations were missing from the KB; this ticket addresses the opposite pressure now that PR conversation text is large enough to compete with source/docs.#7474is historical query scoring work, but it predates the current PullRequestSource corpus pressure.Memory-mining sweep: queried for
knowledge base semantic search regression pull conversations query weights fallback,ask knowledge base answers got worse after indexing pull requests, andpre query weights source docs should outrank conversations; no prior session mapped this exact fix.The Problem
type='all'currently starts from a single Chroma query over the full KB collection. That means large historical/conversation corpora can consume the candidate set before Neo source files and documentation reach the reranker.Empirical evidence from the current checkout:
dist/ai-knowledge-base.jsonlcontainspull: 689chunks, but those chunks carry7,343,216content bytes, larger thansrcandguidebytes combined in the generated JSONL snapshot.ai/services/knowledge-base/source/PullRequestSource.mjs:86-94embeds each PR markdown conversation as a singletype: 'pull'chunk with full content.ai/services/knowledge-base/QueryService.mjs:131-153builds onetype='all'Chroma query with no source-tier filter.ai/services/knowledge-base/QueryService.mjs:201-202only applies explicit post-query penalties forticketandrelease;pullchunks are neutral once returned.ai/mcp/server/knowledge-base/config.mjs:343-358has nopullPenaltyor candidate-pool configuration.A pure post-rank penalty is insufficient if the semantic top-k is already starved before source/docs appear.
The Architectural Reality
The owned substrate is the Knowledge Base query service and its MCP config surface:
QueryService.queryDocuments()owns Chroma candidate retrieval and hybrid scoring.PullRequestSourceis valid signal, but it is historical execution/conversation evidence, not primary current source-of-authority material.learn/agentos/KnowledgeBase.mddocumentsqueryScoreWeightsas a tuning surface; the new behavior should extend that tuning model rather than hard-coding an invisible special case.type='src'currently misses Agent OS implementation files indexed astype: 'ai-infrastructure', so source-like typed searches can fall into unrelated framework source results instead of KB service files.The Fix
Implement candidate-stage weighting for
QueryService.queryDocuments():type='all', issue bounded Chroma candidate queries by source tier before reranking:src,ai-infrastructure,guide,concept, andskill;ticket,pull,discussion, andreleasewith a much smaller candidate budget.type='src'includeai-infrastructureso Agent OS service source is reachable through the documented source-code filter.queryScoreWeights.pullPenaltyfor post-query scoring so returned PR conversation chunks remain lower priority than current source/docs intype='all'results.srcexpansion, and historical pull deweighting.Contract Ledger Matrix
QueryService.queryDocuments({type:'all'})QueryService.mjslearn/agentos/KnowledgeBase.mdscoring/config section if public config names changequeryScoreWeights.pullPenaltytype='all'reranking, analogous to existingticketPenaltytype='src'source filterQueryService.mjs+ApiSourcetype mapai-infrastructurewithsrcbecause Agent OS source files are implementation sourceai-infrastructureif they bypass public enumwherefilter uses$in: ['src', 'ai-infrastructure']Decision Record Impact
Decision Record impact: none. This is aligned with the existing KB hybrid-ranking architecture and the
#10057PullRequestSource premise, but changes its broad-search weighting now that corpus proportions changed.Acceptance Criteria
type='all'KB queries use pre-rerank candidate pools or an equivalent source-tier budget so source/docs get guaranteed candidate budget ahead of historical pull conversations.type='src'queries include Agent OS implementation chunks indexed asai-infrastructure.srcexpansion.Out of Scope
Avoided Traps
nResults: rejected because it raises latency/cost and still gives no semantic source-tier guarantee.Related
Origin Session ID: e8f07ef9-ef7e-4815-8ff4-7abe13720621
Handoff Retrieval Hints:
knowledge base pre-query candidate weighting,PullRequestSource full conversation chunks,queryScoreWeights pullPenalty,type src ai-infrastructure.