pr-review scoring: decile anchor rubric with word-per-tier + score-second template order

Context

Surfaced during the PR #10268 / #10269 cross-family review sequence on 2026-04-24. @tobiu's gut observation: Gemini-family reviewers trend higher on evaluative-metric scores than Claude-family reviewers. Not conclusively data-backed yet, but the asymmetry is real enough to surface.

The underlying drift is a classic calibration problem: "My '90 = notably good' and your '90 = normal quality' represent different anchor points for the same word." Each reviewer is internally consistent. The scale is shared vocabulary but not shared currency.

The cultural analogy tobi used captures the mechanism: US-English "awesome!" ≠ Eastern-European "maybe not total shit" ≠ Swiss-engineer "acceptable." Same word, different operating-calibration. Apply to a cross-family LLM swarm and you get the same drift — not a model-family multiplier to correct, but a shared-vocabulary gap to close.

The Problem

Direct calibration approaches are Goodhart-guaranteed failures:

• Family multipliers ("Claude scores +10, Gemini scores −10"): once visible, both families would incorporate and degrade.
• Reviewing past PRs as calibration corpus: context-window bomb, and re-anchors on past reviewer drift.
• Peer-review-of-reviews: infinite-regress and adds complexity without closing the gap.

What DOESN'T get gamed: concrete external anchors defined independently of reviewer and model. If both families read the same rubric with the same word-per-tier vocabulary, the scale operates as shared currency without either family being told how to score.

The Architectural Reality

• .agent/skills/pr-review/references/pr-review-guide.md §3 — current structural evaluation metrics. Has evaluative vs descriptive distinction (good). Missing: per-metric tier anchors.
• .agent/skills/pr-review/assets/pr-review-template.md — current template has evaluation metrics first, substantive findings (Depth Floor / Cross-Skill) after. Order matters for reducing gut-anchoring.
• Guide §7.2 already acknowledges cross-model asymmetry empirically ("Claude-family tends toward over-rigor; Gemini-family tends toward quick-win framing"). This ticket operationalizes §7.2's diagnosis into a concrete calibration intervention.

The Fix

Part A — Per-metric decile anchors with word-per-tier

Add to guide §3 (one entry per evaluative metric). Worked example for [EXECUTION_QUALITY]:

Similar decile anchors for [ARCH_ALIGNMENT], [CONTENT_COMPLETENESS], [PRODUCTIVITY], [IMPACT]. [COMPLEXITY] and [EFFORT_PROFILE] stay descriptive (already handled in §3.1).

Words chosen for calibration independence: engineering-specific (Exemplary / Solid / Broken) rather than affect-loaded (Awesome / Meh). Cross-cultural drift on "awesome" is the bug the rubric is closing.

Specific wording for the tier word ("50 = Mixed") was tobi's direct callout: giving each 10×N a human-recognizable word anchors the meaning without needing paragraph-long justifications.

Part B — Template flip: substantive findings first, scores last

Current template section order:

1. Status + Opening
2. Evaluation Metrics ← scores here
3. Context & Graph Linking
4. Graph Ingestion Notes
5. Depth Floor
6. Cross-Skill Integration Audit
7. Required Actions

Proposed order:

1. Status + Opening
2. Context & Graph Linking
3. Depth Floor ← substantive findings first
4. Graph Ingestion Notes
5. Cross-Skill Integration Audit
6. Required Actions
7. Evaluation Metrics ← scores last, informed by the enumerated findings above

Why: Reviewers who've just enumerated 2 nits + 1 medium in Required Actions are less likely to write "95" than reviewers who opened with 95 and felt committed to defending it. Order shapes cognition.

Part C — Counter-consideration requirement for 100s (minor)

A score of 100 on an evaluative metric requires an explicit one-line enumeration: "I actively considered [X], [Y], [Z] and confirmed none apply." Parallel to the Depth Floor §7.1 minimum-challenge requirement — forces concreteness where confidence is highest.

Acceptance Criteria

• .agent/skills/pr-review/references/pr-review-guide.md §3 gains per-metric decile anchor tables for all 5 evaluative metrics ([ARCH_ALIGNMENT], [CONTENT_COMPLETENESS], [EXECUTION_QUALITY], [PRODUCTIVITY], [IMPACT]), each with a human-recognizable word per tier.
• Words chosen are engineering-specific (Exemplary / Solid / Broken) not affect-loaded (Awesome / Meh) — cross-cultural drift concern explicitly documented in the guide.
• .agent/skills/pr-review/assets/pr-review-template.md reorders sections so Evaluation Metrics appears AFTER Required Actions.
• Guide §3.1 Score Justification is updated: 100s require counter-consideration enumeration.
• Guide §7.2 Cross-Model Asymmetry gains a forward-reference to §3's rubric as the calibration intervention.
• Post-merge empirical: after 3-6 cross-family review cycles under the new rubric, compare score distributions between families. If the gap narrows toward rubric anchors without visible multipliers, the calibration worked. If it doesn't narrow, rubric needs sharpening — follow-up iteration.

Out of Scope

• Defect Registry as separate artifact (Blocker/Medium/Nit enumeration alongside scores): considered. Rejected on tobi's length-concern — review comments are already long; adding an enumerated registry doubles artifact size without clearly improving comparability beyond what the rubric anchors already do. Revisit if anchored scores still drift after 6+ review cycles.
• Scale compression (0-100 → 1-5 or L/M/H): loses resolution for detailed comparison. Guide already distinguishes evaluative from descriptive metrics; the problem is calibration of the scale, not the scale granularity.
• Technical enforcement (score validation in CI or auto-rejection of justification-less scores): policy-level first. Technical enforcement is a separate follow-up if anchor compliance lags.
• Peer-review-of-reviews or reviewer-performance dashboards: not in scope; the calibration intervention is the rubric itself.

Avoided Traps

• Visible family multipliers ("Claude scores +10, Gemini scores −10"): Goodhart's law. Excluded.
• Cultural-affect words ("Awesome / Meh"): excluded precisely because the US/Eastern-European calibration example shows they carry model-of-origin baggage. Engineering-specific words stay neutral.
• Retroactive re-scoring of past PRs: not in scope; backward rubric fit wastes context without informing future reviews.
• Bundling with Defect Registry: kept separate. If Part A alone doesn't close the gap, Part B (registry) is a clean follow-up; bundling them now risks Part A + Part B both failing for Part B's length-cost reason.

Related

• Origin: PR #10268 + PR #10269 cross-family review sequence (2026-04-24)
• pr-review guide §7.2 Cross-Model Asymmetry Context — this ticket operationalizes §7.2 into a concrete calibration intervention
• Meta-level self-evolving-system mandate: AGENTS.md §13
• Swarm-architecture empirical-data memory: research_swarm_model_velocity_specialization.md (pattern of measuring model behaviors WITHOUT baking static specializations into infra)

Origin Session ID: b02bd06c-a2cb-4aff-8af1-c4f2643c91be

Retrieval Hint: "pr review decile anchor rubric word per tier engineering-specific calibration" / "scale as shared vocabulary not shared currency cross-family scoring drift"