Context
Leaf 4 of Epic #14565 — the validation gate between computation and render. Blocked-by the velocity leaf. No rendered forecast exists until this leaf demonstrates skill — the #14430 falsifyingQuery rule applied to predictions: a forecast without a falsifying backtest is invalid by construction.
The Problem
Attribution + velocity produce confident numbers whether or not they mean anything. The five-step hindcast protocol (#14453 OQ3, DC…17507981) is the falsifier: reconstruct anchor-set-at-window-start (pre-#14442 eras = flagged proxy-anchors) → attribute window W's motion with only-in-W information → score against a labeled sample (operator + ≥2 agents, adjudicated) AND known outcomes → stability re-run after churn → skill-bounded rendering (error bars from miss-rate; no skill at horizon = no render at horizon).
The Fixtures (binding)
- June 2026 = the born-labeled fixture: the documented business/design-direction stall — operator post-mortem 2026-07-04: "design and UX got fully lost; board almost complete = misleading; team hunting scraps without realising" — is ground-truth
INTENT_STARVED. A run over June that misses it = FAIL, full stop. June is a gate, never a tuning set.
- May 2026 = the divergence holdout: narrative layer says shutdown; repo layer built 664 non-chore commits — exactly where volume-blind attribution misreads with confidence. Untouched during development; scored once.
- F2 (from the velocity leaf): if attributed
v_D misreads May's known outcomes WORSE than the volume-blind baseline, the attribute-then-aggregate fidelity claim fails and OQ2 reopens.
The Fix
A repeatable harness (fixture data in-repo beside learn/agentos/measurements/ precedent, #14454): window reconstruction, only-in-W attribution replay, scoring against labels + outcomes, churn-stability re-run, and a skill report artifact (per-horizon miss-rates) that the render leaf consumes as its gate. Degradation path honored: proxy-anchors too weak → v1 ships attribution-only and the render gate stays closed by design.
Scope split (PR review, steward-executed): this ticket = the HARNESS + the June gate; the ceremony = #14812
The harness merged with the June gate PASSING and the May door structurally locked. The May single-shot execution, the labeled-sample adjudication (operator + ≥2 agents — human-involving by construction), and the skill-report artifact are #14812 (the ceremony ticket, operator-scheduled). Rationale: the harness is code; the ceremony is a protocol run — conflating them would either automate what must be adjudicated or block the code on a calendar.
Acceptance Criteria (this leaf — the harness + gate)
Deferred to #14812 (the ceremony)
- May-2026 scored exactly once; result recorded regardless of outcome.
- Labeled-sample protocol executed (operator + ≥2 agents, adjudication recorded).
- Skill report artifact emitted (per-horizon miss rates) — #14570's input contract.
Out of Scope
The render itself (leaf 5); forecast semantics beyond the null-model projection; OQ7 path-level hindcast.
Related
Parent: Epic #14565 (blocked-by the velocity leaf). #14453 OQ3 · #14454/#14472 (measurement-floor precedent) · #14430.
Origin Session ID: a5af7cf6-45a3-42db-8a30-f04f4241a55c
Retrieval Hint: "hindcast protocol June born-labeled fixture INTENT_STARVED May divergence holdout skill-bounded"
Context
Leaf 4 of Epic #14565 — the validation gate between computation and render. Blocked-by the velocity leaf. No rendered forecast exists until this leaf demonstrates skill — the #14430
falsifyingQueryrule applied to predictions: a forecast without a falsifying backtest is invalid by construction.The Problem
Attribution + velocity produce confident numbers whether or not they mean anything. The five-step hindcast protocol (#14453 OQ3, DC…17507981) is the falsifier: reconstruct anchor-set-at-window-start (pre-#14442 eras = flagged proxy-anchors) → attribute window W's motion with only-in-W information → score against a labeled sample (operator + ≥2 agents, adjudicated) AND known outcomes → stability re-run after churn → skill-bounded rendering (error bars from miss-rate; no skill at horizon = no render at horizon).
The Fixtures (binding)
INTENT_STARVED. A run over June that misses it = FAIL, full stop. June is a gate, never a tuning set.v_Dmisreads May's known outcomes WORSE than the volume-blind baseline, the attribute-then-aggregate fidelity claim fails and OQ2 reopens.The Fix
A repeatable harness (fixture data in-repo beside
learn/agentos/measurements/precedent, #14454): window reconstruction, only-in-W attribution replay, scoring against labels + outcomes, churn-stability re-run, and a skill report artifact (per-horizon miss-rates) that the render leaf consumes as its gate. Degradation path honored: proxy-anchors too weak → v1 ships attribution-only and the render gate stays closed by design.Scope split (PR review, steward-executed): this ticket = the HARNESS + the June gate; the ceremony = #14812
The harness merged with the June gate PASSING and the May door structurally locked. The May single-shot execution, the labeled-sample adjudication (operator + ≥2 agents — human-involving by construction), and the skill-report artifact are #14812 (the ceremony ticket, operator-scheduled). Rationale: the harness is code; the ceremony is a protocol run — conflating them would either automate what must be adjudicated or block the code on a calendar.
Acceptance Criteria (this leaf — the harness + gate)
{singleShot, operatorProvenance}); no May data in-repo.Deferred to #14812 (the ceremony)
Out of Scope
The render itself (leaf 5); forecast semantics beyond the null-model projection; OQ7 path-level hindcast.
Related
Parent: Epic #14565 (blocked-by the velocity leaf). #14453 OQ3 · #14454/#14472 (measurement-floor precedent) · #14430.
Origin Session ID: a5af7cf6-45a3-42db-8a30-f04f4241a55c Retrieval Hint: "hindcast protocol June born-labeled fixture INTENT_STARVED May divergence holdout skill-bounded"