> **Author's Note:** This proposal was autonomously synthesized by **Claude Opus 4.7 (Claude Code)** during an Ideation session with @tobiu (session `b5a17132-7324-46e1-b73e-038825bb4d55`). Empirical origin: comparison findings during PR
#10317 review surfaced a substantial gap between Anthropic's skill-creator (heavy eval infrastructure) and Neo's `create-skill` (conventions document, zero eval substrate). @tobiu invited filing if the gap warranted ideation. Filed as Discussion (not Issue) per `ideation-sandbox` — exploratory, multiple OQs, no concrete prescription yet.
The Concept
Anthropic's /skill-creator ships a complete skill-development laboratory: evals/evals.json for test cases, iteration-N/ workspace for tracked revisions, benchmark.json + benchmark.md for quantitative metrics (pass rate, time, tokens with mean ± stddev), eval-viewer/generate_review.py for human-in-the-loop qualitative review, blind A/B comparison agents (agents/comparator.md, agents/analyzer.md), and a description-optimization loop (scripts/run_loop.py) that splits eval sets into 60/40 train/test and iterates up to 5 times to maximize trigger accuracy without overfitting.
Neo's /create-skill (post-PR-#10317) ships a conventions document: directive-redirect pattern, YAML frontmatter with name + description + triggers, references/ + assets/ bundling layout, Anchor & Echo discipline. Zero eval infrastructure. Skills are validated through real-world agent invocation; quality is assessed informally via "did the agent do the right thing in the moment."
Proposal under consideration (this Discussion does NOT pre-commit to it): adopt some form of empirical eval substrate for Neo skills, calibrated to Neo's directive-redirect architecture rather than ported wholesale from Anthropic's bundle-of-actions architecture.
The Rationale
Two empirical anchors from this very session
Self-correction #1 (caught pre-post): PR #10317 review — my reflexive Block-level concern "relative paths break Claude Code Read tool per tool docstring" turned out to be wrong; tool spec was stricter than implementation. Empirical isolation test (§5.1 of pr-review) caught it before posting.
Self-correction #2 (caught post-post): PR #10317 review — my Required Action "empty PR body violates pull-request §6" was mechanically correct on the body field but missed Gemini's Fat Ticket info already on the comment thread. tobi's relay caught it ~7 min after I posted.
Both corrections flowed from skill-shaped reasoning failing in distinct ways. Neither would have been caught by a pre-merge eval — both fired during real review work. But that's exactly the kind of pattern an eval substrate would detect at the skill design layer if we had it: synthesizing canonical "agent reviews PR with empty body / agent reads relative path from skill directive" test cases against a labeled expected behavior.
Skill quality scales with usage
pull-request, pr-review, ticket-intake, epic-review are invoked many times per day across both harnesses.- A subtle directive bug or under-trigger in those skills compounds: agent N hits the bug, files PR with the same gap, agent N+1 picks up the PR, hits the same bug downstream.
- Currently the only validation loop is post-hoc: tobi catches the drift in a relay, I file a calibration update. That works but doesn't scale, and "tobi catches it" is the load-bearing element — what happens when adoption widens beyond Neo core team?
MX (Model Experience) framing
Per #10137 and the recent MX framing memory: "design agent-facing infrastructure with an MX dimension parallel to DX" and "model-friction captured as tickets via Golden Path = production mechanism." An eval substrate at the skill layer is MX as a development-time discipline, not just a post-hoc capture mechanism.
What Anthropic's skill-creator actually validates (and what doesn't carry over)
| Anthropic-validated property |
Carries to Neo? |
Output-format conformance (skill produces .docx/.csv/etc as specified) |
Partial — Neo skills mostly don't produce structured outputs; they produce agent decisions (review verdicts, ticket bodies, commit messages) |
| Description-trigger accuracy (does Claude invoke the skill when it should?) |
Yes, fully — Neo skills face the same triggering challenge; under-triggering on pr-review / ticket-intake is the same failure mode |
| Output quality vs baseline-without-skill |
Yes — "does the skill make the agent do better than no skill?" is the same question regardless of skill substrate |
| Iteration improvement (skill-N+1 better than skill-N on held-out test set) |
Yes — directly applicable |
| Quantitative metrics (tokens, duration, pass rate) |
Partial — Neo's directive-redirect pattern means the skill itself is lightweight; the substantive cost is in the invocation chain (agent reads SKILL.md → reads references/.md → executes per directive). Eval would need to measure end-to-end, not just skill-internal. |
Open Questions
1. What's the right unit of evaluation for a Neo skill? Anthropic skills ship with evaluation harness because the skill IS the procedure. Neo skills point at procedures (the directive-redirect pattern). The unit of evaluation is the agent decision after consulting the skill — but that's harder to canonicalize than file output. [OQ_RESOLUTION_PENDING]
2. Which categories of skill benefit? Different Neo skill archetypes have different eval surfaces:
- Workflow skills (
pull-request, pr-review, ticket-intake, epic-review) — agent-decision-shape outputs (verdicts, structured comments). High eval ROI.
- Operational skills (
self-repair, debugging-antigravity, neural-link) — diagnostic-action outputs (commands run, conclusions reached). Medium eval ROI; canonical-output is harder to specify.
- Conventions skills (
create-skill, ticket-create, ideation-sandbox) — produce other artifacts (skill files, tickets, discussions). Eval ROI depends on whether we evaluate the produced artifact or the process by which it was produced.
- Reasoning skills (
memory-mining, tech-debt-radar, industry-friction-radar) — produce conceptual analyses. Hardest to canonicalize; subjective evaluation may be required.
Should the eval substrate be uniform across categories, or per-category-shaped? [OQ_RESOLUTION_PENDING]
3. Are eval sets agent-authored or human-curated? Anthropic's flow has the agent author test prompts and assertions in collaboration with the user. For Neo, where the agent IS often the consumer of the skill, having the agent author its own evals risks circular self-justification. Some independent third-party labeling (different model family, or human spot-check) might be needed for honest scoring. [OQ_RESOLUTION_PENDING]
4. Where do eval artifacts live? Anthropic puts them in evals/evals.json + <skill-name>-workspace/iteration-N/ next to the skill. For Neo, options:
- Sibling to skill:
.agent/skills/<name>/evals/
- Centralized:
.agent/eval/<skill>/
- Inverted: track in Memory Core (eval as a structured memory artifact rather than file artifact)
- Hybrid: spec in repo, runs in MC
The MC option aligns with substrate-query-first thinking (#10309) but couples eval discipline to MC availability. [OQ_RESOLUTION_PENDING]
5. Toolchain choice — adapt or invent? Anthropic ships eval-viewer/generate_review.py, blind comparison agents, description-optimization scripts. These work out of the box if we copy them — but they assume Anthropic's skill-as-procedure shape. Adapting them to directive-redirect skills may require rewriting the harness substantially. Net: is this a port-wholesale, learn-and-rewrite, or invent-Neo-native effort? [OQ_RESOLUTION_PENDING]
6. Cost vs current zero-eval status quo. Skills currently work "well enough." Eval infrastructure is heavy. Real cost surface:
- Authoring cost per new skill: write 8-20 test prompts + assertions + curate negative-trigger examples
- Iteration cost per skill change: rerun evals across with-skill + baseline-without-skill + grade + viewer review + interpret feedback
- Maintenance cost: keep eval sets honest as the skill evolves — stale evals create false confidence
- Compute cost:
claude -p calls × eval count × iterations × baselines = nontrivial token budget
What's the empirical break-even? Probably skill-by-skill: heavy-use workflow skills justify the cost; reasoning skills may not. [OQ_RESOLUTION_PENDING]
7. Trigger optimization specifically. Anthropic's run_loop.py 60/40 train/test description-optimization is the most concrete portable piece — it improves a single field (description) using a single metric (trigger rate) with low human-in-the-loop burden. Could be adopted in isolation, ahead of the rest of the eval substrate. Lowest-cost-highest-immediate-value subset to consider first? [OQ_RESOLUTION_PENDING]
8. Cross-harness eval portability. Skills run on Claude Code AND Antigravity (Gemini CLI). An eval that passes on one harness may fail the other due to model differences (Gemini-family vs Claude-family failure modes, per pr-review §7.2). Eval infrastructure that doesn't account for cross-harness asymmetry will mis-attribute drift. [OQ_RESOLUTION_PENDING]
9. Relationship to existing rubrics. pr-review already ships decile anchors (§3.1); epic-review already ships five-stage gating; ticket-intake already ships challenge chains. These ARE qualitative rubrics used in production. Is "eval infrastructure" really a parallel substrate, or is it just systematizing what these rubrics already prescribe with empirical measurement loops? Could be a refactor framing rather than a net-new addition. [OQ_RESOLUTION_PENDING]
10. Sandman handoff integration. If eval results live in Memory Core (OQ 4), they could surface in sandman_handoff.md as a skill-quality signal — "skill X has degraded N% on regression test set since last release." Closes a feedback loop the current substrate doesn't have. But couples eval to MC availability and DreamService cycle timing. [OQ_RESOLUTION_PENDING]
Per-Domain Graduation Criteria
This Discussion graduates to Epic once:
- OQ 1 settled — "agent decision" canonicalized for at least one skill category (likely workflow skills as the highest-ROI starting point).
- OQ 2 settled — at minimum, an explicit decision on which skill categories the eval substrate targets v1 (recommend workflow skills only as MVP scope).
- OQ 5 settled — port-wholesale vs invent-Neo-native vs hybrid, with rationale.
- OQ 7 settled — whether trigger-optimization-only is the v1 entry point (potentially independent epic, lower scope).
- OQ 9 settled — relationship to existing decile/stage/challenge rubrics articulated.
- Cost framing settled — at least one skill profiled empirically (author 5 evals, run them, measure the cost) before deciding to scale.
OQs 3, 4, 6, 8, 10 are implementation-shape decisions naturally folded into the Epic body's AC.
Out of Scope
- Skill content critique. This Discussion is about whether-and-how to evaluate skills, not whether any specific current skill needs revision.
- Replacing the directive-redirect pattern. The pattern is load-bearing for Progressive Disclosure (#10281); eval substrate must work with it, not replace it.
- Comparison eval to compare skills against each other. Different scope (
tech-debt-radar-adjacent), not what this Discussion proposes.
- General evaluation infrastructure for Neo as a runtime engine. This is skill-substrate-evaluation. Engine-level testing is
unit-test / whitebox-e2e / Playwright territory.
- Anthropic-skill-creator porting. Even if we adopt some of its patterns, a port-wholesale would impose Anthropic's skill-architecture assumptions on Neo's directive-redirect substrate. Adapt, don't port.
Avoided Traps
- Treating eval as universal mandate. Rejected — different skill categories have different eval surfaces; uniform mandate over-engineers reasoning skills and under-serves workflow skills.
- Equating eval count with skill quality. Rejected — 20 evals on a poorly-designed skill don't improve it; the rubric matters more than the volume.
- Pre-committing to a toolchain (Anthropic's vs Neo-native) before scope is clear. Rejected — that's premature optimization. Scope first (which skills), then toolchain.
- Conflating triggering accuracy with output quality. Rejected — Anthropic's
run_loop.py measures triggering (does the skill activate when it should?), which is necessary but not sufficient. Output quality (does the skill produce the right response once it activates?) is a separate axis.
- Assuming Memory Core integration is free. Rejected — coupling eval substrate to MC creates dependency on DreamService cycles and MC availability; an eval surface should function offline.
- Building an eval substrate before any skill empirically demonstrates need. Rejected — empirical need has surfaced this session (two self-corrections in three turns; cross-family Gemini-stale-state on #10311); but the form of the answer should be empirically driven, not architectural-aesthetic.
- Treating this Discussion as a commitment. Rejected — graduation criteria explicit; if cost analysis (graduation #6) reveals eval ROI doesn't justify lift, this Discussion can
[REJECTED_WITH_RATIONALE] and close cleanly. Filing isn't adoption.
Related
#10283 — Pre-Authoring Adjacency Sweep for Meta-Skills (invocation discipline, adjacent scope; this Discussion = skill-quality measurement, that Issue = pre-flight check enforcement)#10281 — ideation-sandbox refactor (Progressive Disclosure adherence; precedent for skill-quality-as-architectural-concern)#10273 — pr-review decile anchor rubric (precedent for explicit skill-shaped rubric)#10137 — MX (Model Experience) framing; this is MX-as-development-discipline at the skill layer#10309 — substrate-query-first boot (skill-eval results could surface as boot-time orientation if MC-integrated, OQ 10)#10074 — blog draft articulating "self-improvement is a protocol rather than a model behavior"; eval substrate is one mechanism that protocol could take#10317 — recent PR-review session that empirically surfaced two skill-shape failure modes (self-corrections #1 and #2 from Rationale)pr-review §3.1 decile anchors, epic-review §3 five-stage chain, ticket-intake challenge chain — existing qualitative rubrics that may be the foundation for empirical eval rather than parallel substrate (OQ 9)
Retrieval Hint: "Neo skill empirical evaluation eval substrate benchmark trigger-optimization Anthropic skill-creator comparison directive-redirect MX skill-quality measurement"
Origin Session ID: b5a17132-7324-46e1-b73e-038825bb4d55
The Concept
Anthropic's
/skill-creatorships a complete skill-development laboratory:evals/evals.jsonfor test cases,iteration-N/workspace for tracked revisions,benchmark.json+benchmark.mdfor quantitative metrics (pass rate, time, tokens with mean ± stddev),eval-viewer/generate_review.pyfor human-in-the-loop qualitative review, blind A/B comparison agents (agents/comparator.md,agents/analyzer.md), and a description-optimization loop (scripts/run_loop.py) that splits eval sets into 60/40 train/test and iterates up to 5 times to maximize trigger accuracy without overfitting.Neo's
/create-skill(post-PR-#10317) ships a conventions document: directive-redirect pattern, YAML frontmatter withname+description+triggers,references/+assets/bundling layout, Anchor & Echo discipline. Zero eval infrastructure. Skills are validated through real-world agent invocation; quality is assessed informally via "did the agent do the right thing in the moment."Proposal under consideration (this Discussion does NOT pre-commit to it): adopt some form of empirical eval substrate for Neo skills, calibrated to Neo's directive-redirect architecture rather than ported wholesale from Anthropic's bundle-of-actions architecture.
The Rationale
Two empirical anchors from this very session
Self-correction #1 (caught pre-post): PR #10317 review — my reflexive Block-level concern "relative paths break Claude Code Read tool per tool docstring" turned out to be wrong; tool spec was stricter than implementation. Empirical isolation test (§5.1 of pr-review) caught it before posting.
Self-correction #2 (caught post-post): PR #10317 review — my Required Action "empty PR body violates pull-request §6" was mechanically correct on the body field but missed Gemini's Fat Ticket info already on the comment thread. tobi's relay caught it ~7 min after I posted.
Both corrections flowed from skill-shaped reasoning failing in distinct ways. Neither would have been caught by a pre-merge eval — both fired during real review work. But that's exactly the kind of pattern an eval substrate would detect at the skill design layer if we had it: synthesizing canonical "agent reviews PR with empty body / agent reads relative path from skill directive" test cases against a labeled expected behavior.
Skill quality scales with usage
pull-request,pr-review,ticket-intake,epic-revieware invoked many times per day across both harnesses.MX (Model Experience) framing
Per
#10137and the recent MX framing memory: "design agent-facing infrastructure with an MX dimension parallel to DX" and "model-friction captured as tickets via Golden Path = production mechanism." An eval substrate at the skill layer is MX as a development-time discipline, not just a post-hoc capture mechanism.What Anthropic's skill-creator actually validates (and what doesn't carry over)
.docx/.csv/etc as specified)pr-review/ticket-intakeis the same failure modeOpen Questions
1. What's the right unit of evaluation for a Neo skill? Anthropic skills ship with evaluation harness because the skill IS the procedure. Neo skills point at procedures (the directive-redirect pattern). The unit of evaluation is the agent decision after consulting the skill — but that's harder to canonicalize than file output.
[OQ_RESOLUTION_PENDING]2. Which categories of skill benefit? Different Neo skill archetypes have different eval surfaces:
pull-request,pr-review,ticket-intake,epic-review) — agent-decision-shape outputs (verdicts, structured comments). High eval ROI.self-repair,debugging-antigravity,neural-link) — diagnostic-action outputs (commands run, conclusions reached). Medium eval ROI; canonical-output is harder to specify.create-skill,ticket-create,ideation-sandbox) — produce other artifacts (skill files, tickets, discussions). Eval ROI depends on whether we evaluate the produced artifact or the process by which it was produced.memory-mining,tech-debt-radar,industry-friction-radar) — produce conceptual analyses. Hardest to canonicalize; subjective evaluation may be required.Should the eval substrate be uniform across categories, or per-category-shaped?
[OQ_RESOLUTION_PENDING]3. Are eval sets agent-authored or human-curated? Anthropic's flow has the agent author test prompts and assertions in collaboration with the user. For Neo, where the agent IS often the consumer of the skill, having the agent author its own evals risks circular self-justification. Some independent third-party labeling (different model family, or human spot-check) might be needed for honest scoring.
[OQ_RESOLUTION_PENDING]4. Where do eval artifacts live? Anthropic puts them in
evals/evals.json+<skill-name>-workspace/iteration-N/next to the skill. For Neo, options:.agent/skills/<name>/evals/.agent/eval/<skill>/The MC option aligns with substrate-query-first thinking (#10309) but couples eval discipline to MC availability.
[OQ_RESOLUTION_PENDING]5. Toolchain choice — adapt or invent? Anthropic ships
eval-viewer/generate_review.py, blind comparison agents, description-optimization scripts. These work out of the box if we copy them — but they assume Anthropic's skill-as-procedure shape. Adapting them to directive-redirect skills may require rewriting the harness substantially. Net: is this a port-wholesale, learn-and-rewrite, or invent-Neo-native effort?[OQ_RESOLUTION_PENDING]6. Cost vs current zero-eval status quo. Skills currently work "well enough." Eval infrastructure is heavy. Real cost surface:
claude -pcalls × eval count × iterations × baselines = nontrivial token budgetWhat's the empirical break-even? Probably skill-by-skill: heavy-use workflow skills justify the cost; reasoning skills may not.
[OQ_RESOLUTION_PENDING]7. Trigger optimization specifically. Anthropic's
run_loop.py60/40 train/test description-optimization is the most concrete portable piece — it improves a single field (description) using a single metric (trigger rate) with low human-in-the-loop burden. Could be adopted in isolation, ahead of the rest of the eval substrate. Lowest-cost-highest-immediate-value subset to consider first?[OQ_RESOLUTION_PENDING]8. Cross-harness eval portability. Skills run on Claude Code AND Antigravity (Gemini CLI). An eval that passes on one harness may fail the other due to model differences (Gemini-family vs Claude-family failure modes, per
pr-review §7.2). Eval infrastructure that doesn't account for cross-harness asymmetry will mis-attribute drift.[OQ_RESOLUTION_PENDING]9. Relationship to existing rubrics.
pr-reviewalready ships decile anchors (§3.1);epic-reviewalready ships five-stage gating;ticket-intakealready ships challenge chains. These ARE qualitative rubrics used in production. Is "eval infrastructure" really a parallel substrate, or is it just systematizing what these rubrics already prescribe with empirical measurement loops? Could be a refactor framing rather than a net-new addition.[OQ_RESOLUTION_PENDING]10. Sandman handoff integration. If eval results live in Memory Core (OQ 4), they could surface in
sandman_handoff.mdas a skill-quality signal — "skill X has degraded N% on regression test set since last release." Closes a feedback loop the current substrate doesn't have. But couples eval to MC availability and DreamService cycle timing.[OQ_RESOLUTION_PENDING]Per-Domain Graduation Criteria
This Discussion graduates to Epic once:
OQs 3, 4, 6, 8, 10 are implementation-shape decisions naturally folded into the Epic body's AC.
Out of Scope
tech-debt-radar-adjacent), not what this Discussion proposes.unit-test/whitebox-e2e/ Playwright territory.Avoided Traps
run_loop.pymeasures triggering (does the skill activate when it should?), which is necessary but not sufficient. Output quality (does the skill produce the right response once it activates?) is a separate axis.[REJECTED_WITH_RATIONALE]and close cleanly. Filing isn't adoption.Related
#10283— Pre-Authoring Adjacency Sweep for Meta-Skills (invocation discipline, adjacent scope; this Discussion = skill-quality measurement, that Issue = pre-flight check enforcement)#10281— ideation-sandbox refactor (Progressive Disclosure adherence; precedent for skill-quality-as-architectural-concern)#10273— pr-review decile anchor rubric (precedent for explicit skill-shaped rubric)#10137— MX (Model Experience) framing; this is MX-as-development-discipline at the skill layer#10309— substrate-query-first boot (skill-eval results could surface as boot-time orientation if MC-integrated, OQ 10)#10074— blog draft articulating "self-improvement is a protocol rather than a model behavior"; eval substrate is one mechanism that protocol could take#10317— recent PR-review session that empirically surfaced two skill-shape failure modes (self-corrections #1 and #2 from Rationale)pr-review §3.1decile anchors,epic-review §3five-stage chain,ticket-intakechallenge chain — existing qualitative rubrics that may be the foundation for empirical eval rather than parallel substrate (OQ 9)Retrieval Hint:
"Neo skill empirical evaluation eval substrate benchmark trigger-optimization Anthropic skill-creator comparison directive-redirect MX skill-quality measurement"Origin Session ID:
b5a17132-7324-46e1-b73e-038825bb4d55