Problem
The RLAIF data pipeline collects action traces and session summaries, but has no actual reward function. Session quality metrics (quality, productivity, impact) are LLM-estimated at summarization time — the LLM judges effort and coherence, not outcomes.
A session where an agent produced 3 PRs that all got reverted would still receive high productivity scores because the summarizer has no visibility into what happened after the session ended.
Proposal
Implement a PR Outcome Tracker that retroactively tags session summaries based on the merge outcome of associated PRs.
Reward Signal Definition
| PR Outcome |
Reward |
Rationale |
| Merged without changes |
1.0 |
Gold standard — agent produced merge-ready code |
| Merged with requested changes |
0.7 |
Good work, minor polish needed |
| Closed without merge |
0.0 |
Wasted effort — approach was wrong |
| Reverted after merge |
-1.0 |
Actively harmful — introduced regression |
Implementation
- Data Source: Use
gh pr list --state merged --json number,mergedAt,closedAt to scan recent PRs.
- Session Linking: Each PR's commit message contains
(#TICKET_ID). Cross-reference the ticket ID with session summaries that reference the same ticket (via the Origin Session ID or memory metadata).
- Retroactive Tagging: Update the session summary's metadata in ChromaDB with
outcomeReward: float and prNumber: int.
- Integration Point: This could run as a periodic
DreamService task or a standalone daemon invoked by runSandman.mjs.
Why This Matters for RLAIF
Without outcome-based rewards, any future fine-tuning or retrieval weighting is based on the LLM's self-assessment, which is inherently biased toward "I did good work." Outcome-based rewards ground the learning signal in reality: did the code actually ship?
This also enables weighted memory retrieval — when an agent queries past approaches, results from high-reward sessions should rank higher than those from sessions where the PR was rejected.
A2A Context
Origin Session ID: fff6dc5b-ca7f-4c9b-8eca-41bd8a97ad5d
Problem
The RLAIF data pipeline collects action traces and session summaries, but has no actual reward function. Session quality metrics (
quality,productivity,impact) are LLM-estimated at summarization time — the LLM judges effort and coherence, not outcomes.A session where an agent produced 3 PRs that all got reverted would still receive high
productivityscores because the summarizer has no visibility into what happened after the session ended.Proposal
Implement a PR Outcome Tracker that retroactively tags session summaries based on the merge outcome of associated PRs.
Reward Signal Definition
Implementation
gh pr list --state merged --json number,mergedAt,closedAtto scan recent PRs.(#TICKET_ID). Cross-reference the ticket ID with session summaries that reference the same ticket (via theOrigin Session IDor memory metadata).outcomeReward: floatandprNumber: int.DreamServicetask or a standalone daemon invoked byrunSandman.mjs.Why This Matters for RLAIF
Without outcome-based rewards, any future fine-tuning or retrieval weighting is based on the LLM's self-assessment, which is inherently biased toward "I did good work." Outcome-based rewards ground the learning signal in reality: did the code actually ship?
This also enables weighted memory retrieval — when an agent queries past approaches, results from high-reward sessions should rank higher than those from sessions where the PR was rejected.
A2A Context
Origin Session ID:
fff6dc5b-ca7f-4c9b-8eca-41bd8a97ad5d