LearnNewsExamplesServices
Frontmatter
id13547
titleEnforce pr-review template skeleton fidelity
stateClosed
labels
bugaitestingregressionarchitecturemodel-experience
assigneesneo-gpt
createdAtJun 19, 2026, 1:08 PM
updatedAtJun 19, 2026, 1:49 PM
githubUrlhttps://github.com/neomjs/neo/issues/13547
authorneo-gpt
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 19, 2026, 1:49 PM

Enforce pr-review template skeleton fidelity

Closed v13.1.0/archive-v13-1-0-chunk-4 bugaitestingregressionarchitecturemodel-experience
neo-gpt
neo-gpt commented on Jun 19, 2026, 1:08 PM

Context

Operator regression report on 2026-06-19: after PR #13541 compressed .agents/skills/pr-review/references/pr-review-guide.md, compare these two live reviews by the same reviewer identity:

The #13486 review preserves the full icon-bearing template scaffold (### 🪜 Strategic-Fit Decision, ### 🧭 Patch-Blind Premise Snapshot, ### 🕸️ Context & Graph Linking, ### 🔬 Depth Floor, ### 🧠 Graph Ingestion Notes, ### 🎯 Close-Target Audit, ### 🪜 Evidence Audit, ### 🧪 Test-Execution & Location Audit, ### 📋 Required Actions, ### 📊 Evaluation Metrics).

The #13545 review keeps much of the substantive audit content, but drops the canonical icon-bearing section skeleton and rewrites the headings into a flatter custom shape (### Strategic-Fit Decision, ### Patch-Blind Premise Snapshot, etc.). That is a visible quality downgrade and weakens graph-readable review consistency.

Operator correction during implementation: the normal follow-up review template needs the same visual/template quality. "Compact" follow-up means delta-oriented content, not plain-heading downgrade. The micro-delta cost-circuit-breaker artifact remains separate.

Duplicate sweep:

  • Live latest-open sweep: checked latest 20 open issues via list_issues(state=open, limit=20, projection=summary) at 2026-06-19T11:05Z; no equivalent open ticket found.
  • A2A in-flight sweep: checked latest 40 messages at 2026-06-19T11:05Z; no overlapping [lane-claim] / [lane-intent] found for pr-review template-fidelity repair.
  • KB/ticket sweep: queried open tickets about pr-review template fidelity review quality regression emoji headings manage_pr_review validator; returned only closed historical validator/template issues, no open equivalent.
  • Local exact sweep: rg "template fidelity|review quality|review body|pr-review.*template|emoji headings|icon-bearing|manage_pr_review|PR_REVIEW_TEMPLATE_VALIDATION_FAILED" found historical validator/template context but no open duplicate.

Release classification: current Agent OS hardening; boardless unless @tobiu explicitly puts review-substrate regressions on a release board.

The Problem

PR #13541 successfully reduced payload size, but it exposed a missing invariant: using a review template means preserving the selected template's canonical skeleton, not merely including enough semantic anchors to pass manage_pr_review.

Verified current source state at ticket creation:

  • origin/dev:.agents/skills/pr-review/assets/pr-review-template.md contains the icon-bearing full-review headings.
  • origin/dev:.agents/skills/pr-review/assets/pr-review-followup-template.md still used a compact follow-up shape with plain headings, but operator direction clarified that this is itself part of the quality gap.
  • ai/services/github-workflow/PullRequestService.mjs validates the body with seven visible metric tags plus three silent structural substrings: Depth Floor, Required Actions, and Strategic-Fit Decision.
  • test/playwright/unit/ai/services/github-workflow/PullRequestService.spec.mjs has a deliberately minimal VALID_REVIEW_BODY with only a few headings and metric tags; this is enough for tool-boundary validation, but not enough for review-quality fidelity.

So the failure has two layers:

  1. Execution failure: the #13545 reviewer loaded/should have used the full template but manually emitted a custom flattened skeleton.
  2. Substrate gap: after compression, the guide still says to use the template, but it does not make exact skeleton fidelity a non-negotiable invariant, and the tool-boundary validator does not catch this downgrade.

The Architectural Reality

Primary surfaces:

  • .agents/skills/pr-review/references/pr-review-guide.md — conditional review workflow payload; should gain only a compact invariant, not a restored long-form manual.
  • .agents/skills/pr-review/assets/pr-review-template.md — full cycle-1 skeleton; already has the desired heading quality.
  • .agents/skills/pr-review/assets/pr-review-followup-template.md — normal cycle-N skeleton; should keep delta-oriented content while matching the visual/template quality floor.
  • ai/services/github-workflow/PullRequestService.mjs — MCP tool-boundary validator for manage_pr_review.
  • test/playwright/unit/ai/services/github-workflow/PullRequestService.spec.mjs — focused validator tests.

This is not a request to undo the #13541 compression. The review guide was too heavy. The correct fix is a small fidelity invariant, a follow-up-template polish pass, and mechanical coverage for the class of regression demonstrated by #13545.

The Fix

Add a compact template-skeleton fidelity rule and enforce it where cheap:

  • In pr-review-guide.md §6, explicitly state that reviews must preserve the selected template's canonical heading text, emoji/icons, section order, and null-state wording unless the guide explicitly allows an omission/collapse.
  • Update the normal follow-up template to use icon-bearing canonical headings while keeping its compact delta-oriented content.
  • Tighten manage_pr_review validation or an adjacent helper so a body shaped like the #13545 review cannot pass while dropping canonical template headings/icons.
  • Add focused unit coverage with reduced #13545-shaped cycle-1 and old-follow-up bodies that contain all metric tags and old broad structural anchors, but lack the canonical icon-bearing headings.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
.agents/skills/pr-review/references/pr-review-guide.md §6 Full template in .agents/skills/pr-review/assets/pr-review-template.md; operator correction on follow-up quality; regression evidence from #13486 vs #13545 Template use means exact skeleton fidelity for the selected review cycle: canonical headings/icons/order retained unless explicitly omitted by template rules If wording growth is too high, use one compact sentence plus rely on validator tests for mechanics Same guide §6 byte delta; grep for fidelity invariant
.agents/skills/pr-review/assets/pr-review-followup-template.md Operator correction in this ticket; cycle-N template selection in pr-review-guide.md §6.2 Follow-up reviews keep compact delta content but use icon-bearing canonical headings Leave micro-delta cost-circuit-breaker artifact untouched Follow-up template diff + validator acceptance test
manage_pr_review body validation Tool description already mandates reading /pr-review; current PullRequestService.mjs anchor validation Reject review bodies that drop the selected template's canonical icon-bearing headings while still passing broad anchors Reject with existing generic template-validation message rather than listing hidden anchors Existing tool error message points to skill/template unit tests using #13545-shaped and old-follow-up-shaped bodies

Decision Record impact

Aligned with ADR 0008 and the progressive-disclosure Map/Atlas model: this should add a small rule and a mechanical validator, not a new book or restored verbose payload.

Acceptance Criteria

  • pr-review-guide.md §6 contains a compact template-skeleton fidelity invariant.
  • The normal follow-up review template uses icon-bearing canonical headings while preserving compact delta-oriented content.
  • The fix does not restore the pre-#13541 long-form guide; additions are narrowly bounded and justified as regression repair.
  • manage_pr_review or an equivalent validator rejects a #13545-shaped cycle-1 review body that has the broad anchors/metrics but drops the canonical full-template icon-bearing headings.
  • The validator rejects the old plain-heading follow-up review skeleton.
  • The validator accepts the updated icon-bearing follow-up review skeleton.
  • Add or update focused unit coverage in PullRequestService.spec.mjs for the rejection and pass paths.
  • Run the focused github-workflow unit spec and node ./ai/scripts/lint/lint-skill-manifest.mjs --base origin/dev.
  • PR body includes /turn-memory-pre-flight load-effect reasoning because .agents/skills/** is mutated.

Out of Scope

  • Reverting PR #13541 wholesale.
  • Restoring every deleted paragraph from the old 50KB pr-review-guide.md.
  • Redesigning the whole review-template system.
  • Changing the micro-delta Review-Loop Cost Circuit Breaker format.
  • Enforcing stylistic convergence across model families beyond canonical template skeleton fidelity.

Avoided Traps

  • Bloat reversal: rejected. The user explicitly accepted shortening; the bug is quality-regression side effect, not compression itself.
  • Anchor stuffing only: rejected. The #13545 artifact proves broad anchors plus metric tags can still lose the visible review scaffold.
  • Follow-up as lesser artifact: rejected by operator correction. Normal follow-up reviews should be compact but not visually/structurally downgraded.

Related

Handoff Retrieval Hints

Search patterns:

  • pr-review template skeleton fidelity icon headings
  • manage_pr_review validator Strategic-Fit Decision Required Actions Depth Floor
  • #13486 #13545 review quality regression after #13541

Origin Session ID: current Codex session, 2026-06-19 review-quality regression turn.