The Blind Spot — Expanded After Fifth Observation
Originally framed as a strengthening against hardcoded numerical thresholds in ACs. After four more observations in session 62d6f155 (2026-04-19), the underlying failure class is broader: accepting written claims at face value without verifying against precedent code, runtime behavior, or the rule's actual grounding assumption. Numerical thresholds are one instance; architectural descriptions, tool-description framings, self-inferred policies, and misread grounding assumptions are four more.
Five observed instances this session
| # |
Claim codified |
Written source |
Actual precedent / truth |
Fix |
| 1 |
"Output under 100 lines for Tier 1-2" → hard test assertion |
#10033 AC checklist |
Production output = 53 lines; 100 was a 2× safety margin on a 50-line estimate from a smaller-context era |
PR #10078 polish commit 29fb2ac0b — observability log replaces hard cap |
| 2 |
"how does X work → ask, file discovery → query_documents" → codified in memory-mining anti-patterns |
ask_knowledge_base official tool description |
Empirical A/B test: ask returns synthesized answer plus same top-5 refs — strictly dominates query_documents |
PR #10076 polish commit e79b89a3b + openapi.yaml commit 7eadba80f |
| 3 |
"Universal dispatch at ai/mcp/ToolService.mjs" → proposed instrumentation site for #10081 FAQ telemetry |
CodebaseOverview.md description |
Canonical precedent is RecorderService at per-server services/toolService.mjs; tobi had even cued this in the original framing ("similar to the NL event recorder") |
#10081 body rewritten 2026-04-19 |
| 4 |
"Architectural Pillar = fresh session" → applied as general policy to #10035 |
No written source — self-inferred from tobi's specific comment "#9999 will require a fresh session" |
Specific user statement, not a general rule. Self-codification inverted the semantic scope: "this one requires" → "these all require" |
Memory ext: feedback_verify_written_claims_against_precedent.md §4 (self-inferred-policy class) |
| 5 |
"25-turn guardrail is context-scarcity thinking" → initially flagged as stale finding |
AGENTS.md §10.6 (correctly read; the phrase "single task" was in my own quote) |
Thrashing-detection rule ("getting stuck" prevention, e.g., Gemini grep-looping 50×), NOT context-budget. Projected wrong grounding assumption onto a correctly-read rule |
Memory ext: feedback_verify_written_claims_against_precedent.md §5 (wrong-grounding-assumption class) |
All five were caught by tobi's challenges, not by any skill firing preemptively. The common failure: treating documentation / specification / description language as spec-of-current-truth rather than evidence-of-intent, OR projecting the wrong grounding assumption onto a correctly-read rule.
The Proposed Strengthening (expanded scope)
Target files
.agent/skills/ticket-intake/references/ticket-intake-workflow.md §1.4.agent/skills/pr-review/references/pr-review-guide.md (cross-link).agent/skills/memory-mining/references/memory-mining-protocol.md (cross-link)
Specific changes
Enumerate the five high-risk written-claim patterns explicitly in ticket-intake §1.4:
Written claims come in five high-risk classes. For each class, the Hypothesis gate requires an empirical verification step before codification:
(a) Hardcoded numerical thresholds — "under N lines", "≤ N ms", "at most N items". Verify by measuring the current value. Ask where N came from; if undocumented, prefer observability + architectural-layer assertion over hard cap.
(b) Architectural-description claims — "uses X pattern", "universal dispatch at Y", "similar to Z". Verify by reading the actual file / class / precedent in the codebase before paraphrasing it into a proposal. Especially when the user cues "similar to X" — that's an instruction to read X.
(c) Tool/API description claims — "use query_documents for file paths", "should be your first tool for conceptual questions". Verify by running the tool empirically against a representative query before codifying the routing rule.
(d) Self-inferred policies from specific statements — user says "X requires fresh session" and agent generalizes to "all X-class requires fresh session". Verify by asking whether the generalization was intended, or by looking for the rule written down elsewhere. Specific statements are not universal policies.
(e) Wrong grounding assumption on a correctly-read rule — read the rule accurately but project the wrong motivation onto it (e.g., "this is about token budget" when it's actually about thrashing detection). Verify by asking "what specific failure mode does this rule prevent?" rather than assuming based on surface framing.
Add a concrete verification-action table mapping claim class → required verification before codification:
| Claim class |
Required verification |
| (a) Numerical threshold |
Measure current value + ask for derivation |
| (b) Architectural description |
view_file / Glob / grep_search for the named precedent |
| (c) Tool routing / API behavior |
Empirical invocation against representative input |
| (d) Self-inferred policy |
Check for the rule written down; ask whether generalization was intended |
| (e) Wrong grounding assumption |
Ask "what specific failure mode does this rule prevent?" before mapping to a motivation |
Cross-link from pr-review — add one line noting that self-reviews should flag any newly-codified written claim that lacks a cited verification step.
Cross-link from memory-mining — add one line noting that "similar to X" cues are high-priority memory-mining triggers.
Falsifiable test of success
Future sessions receiving tickets with any of the five claim classes invoke memory-mining / precedent-reading / empirical tests before codifying — observable in PR bodies citing the verification step.
Scope
- Purely additive edits to existing skill references
- No new skill; no refactor of existing skill internals
- ~40–60 line diff across 1–3 skill files (slightly larger than the original narrower proposal; still low-complexity)
- No production code changes
Related
- PR #10078 — first blind-spot surface (100-line threshold, class a)
- PR #10076 — second blind-spot surface (ask vs query_documents routing, class c)
- #10081 — third blind-spot surface (universal dispatch vs per-server precedent, class b)
- #10083 — fourth and fifth blind-spot surfaces consolidated into a concrete audit fix of AGENTS.md §9
- #10075 / PR #10076 —
memory-mining skill (adjacent to this gap)
feedback_challenge_numerical_thresholds_in_acs.md — durable memory covering claim class (a)feedback_verify_written_claims_against_precedent.md — durable memory covering claim classes (b) through (e)
Origin Session ID
62d6f155-e57f-4279-9b59-36c9e4ecbc5e
Revision Note
This ticket was originally scoped narrowly to numerical thresholds (claim class a). After four more in-session observations, expanded to cover all five written-claim classes. The expanded scope preserves the original intent and absorbs what would otherwise be four follow-up tickets.
The Blind Spot — Expanded After Fifth Observation
Originally framed as a strengthening against hardcoded numerical thresholds in ACs. After four more observations in session
62d6f155(2026-04-19), the underlying failure class is broader: accepting written claims at face value without verifying against precedent code, runtime behavior, or the rule's actual grounding assumption. Numerical thresholds are one instance; architectural descriptions, tool-description framings, self-inferred policies, and misread grounding assumptions are four more.Five observed instances this session
29fb2ac0b— observability log replaces hard capask_knowledge_baseofficial tool descriptione79b89a3b+ openapi.yaml commit7eadba80fai/mcp/ToolService.mjs" → proposed instrumentation site for #10081 FAQ telemetryCodebaseOverview.mddescriptionRecorderServiceat per-serverservices/toolService.mjs; tobi had even cued this in the original framing ("similar to the NL event recorder")feedback_verify_written_claims_against_precedent.md§4 (self-inferred-policy class)AGENTS.md §10.6(correctly read; the phrase "single task" was in my own quote)feedback_verify_written_claims_against_precedent.md§5 (wrong-grounding-assumption class)All five were caught by tobi's challenges, not by any skill firing preemptively. The common failure: treating documentation / specification / description language as spec-of-current-truth rather than evidence-of-intent, OR projecting the wrong grounding assumption onto a correctly-read rule.
The Proposed Strengthening (expanded scope)
Target files
.agent/skills/ticket-intake/references/ticket-intake-workflow.md§1.4.agent/skills/pr-review/references/pr-review-guide.md(cross-link).agent/skills/memory-mining/references/memory-mining-protocol.md(cross-link)Specific changes
Enumerate the five high-risk written-claim patterns explicitly in ticket-intake §1.4:
Add a concrete verification-action table mapping claim class → required verification before codification:
view_file/Glob/grep_searchfor the named precedentCross-link from pr-review — add one line noting that self-reviews should flag any newly-codified written claim that lacks a cited verification step.
Cross-link from memory-mining — add one line noting that "similar to X" cues are high-priority memory-mining triggers.
Falsifiable test of success
Future sessions receiving tickets with any of the five claim classes invoke memory-mining / precedent-reading / empirical tests before codifying — observable in PR bodies citing the verification step.
Scope
Related
memory-miningskill (adjacent to this gap)feedback_challenge_numerical_thresholds_in_acs.md— durable memory covering claim class (a)feedback_verify_written_claims_against_precedent.md— durable memory covering claim classes (b) through (e)Origin Session ID
62d6f155-e57f-4279-9b59-36c9e4ecbc5eRevision Note
This ticket was originally scoped narrowly to numerical thresholds (claim class a). After four more in-session observations, expanded to cover all five written-claim classes. The expanded scope preserves the original intent and absorbs what would otherwise be four follow-up tickets.