LearnNewsExamplesServices
Frontmatter
titlefeat(ai): add memory-mining skill to enforce memory-first reflex (#10075)
authortobiu
stateMerged
createdAtApr 19, 2026, 9:51 AM
updatedAtApr 19, 2026, 10:06 AM
closedAtApr 19, 2026, 10:06 AM
mergedAtApr 19, 2026, 10:06 AM
branchesdev ← claude/youthful-bohr-9c35fe
urlhttps://github.com/neomjs/neo/pull/10076
Merged
tobiu
tobiu commented on Apr 19, 2026, 9:51 AM

Summary

Promotes the memory-first reflex from a rule in AGENTS_STARTUP.md §3.3 to a proper Progressive Disclosure skill. Addresses the failure mode surfaced in session 51640d07 (2026-04-18) where the agent re-derived a Karpathy-loop-to-Neo comparison that Antigravity had already mapped two weeks earlier (memory 9f058317, 2026-04-11) — because the memory-first rule lived in the boot contract but not as a state-transition gate at the moment it was needed.

The Problem

Empirically verified (session 2 memory 3a26f41d): reflexes-as-rules get applied inconsistently; reflexes-as-skills get applied reliably. Tobi confirmed for Gemini 3.1 Pro + Opus 4.6 that without the three existing skills (ticket-intake, pull-request, pr-review), results diverged meaningfully even though equivalent base instructions existed. The mechanism driving the delta: skill invocation IS the state transition — an explicit mode switch visible in the transcript, not a poster-on-a-wall rule.

The memory-first rule had been strengthened in PR #10069 (added explicit trigger enumeration). This PR applies the proven skill mechanism to the same reflex.

Architectural Reality

The Neo.mjs skill topology uses a Router (SKILL.md) + Payload (references/*.md) pattern per .agent/skills/create-skill/references/skill-authoring-guide.md:

  • SKILL.md is loaded at boot; body must be a one-sentence directive pointing at the reference via absolute path.
  • references/*.md is lazy-loaded via view_file only when the trigger fires — prevents system prompt bloat.

This PR scaffolds a new skill following that contract exactly, mirroring ticket-intake/SKILL.md for consistency.

Gold Standards Leveraged / Traps Avoided

  • Gold Standard (proven mechanism): three existing skills (ticket-intake, pull-request, pr-review) produce measurable quality delta across harnesses. Replicating their shape (lightweight router → heavy reference) for the fourth reflex.
  • Trap avoided: bloating SKILL.md with the full protocol (would regress the Progressive Disclosure contract and pollute every boot's system prompt).
  • Trap avoided: refactoring existing skill internals. This PR is purely additive per the ticket's explicit constraint ("no changes to the existing three skills' internals").

Implementation Footprint

File Change
.agent/skills/memory-mining/SKILL.md New. Lightweight router with YAML frontmatter (name, description, triggers) + single-sentence directive to reference.
.agent/skills/memory-mining/references/memory-mining-protocol.md New. Full 5-section playbook: two invocation gates (regression symptoms + architectural claims), query strategy, three-state interpretation + Trap/Gold heuristic, anti-patterns, falsifiable exit criteria. Includes cross-harness "what would tobi do here?" heuristic and integration notes for ticket-intake / self-repair / tech-debt-radar.
.claude/skills/memory-mining New symlink → ../../.agent/skills/memory-mining. Matches the wrapper convention used by all 11 existing skills.
AGENTS_STARTUP.md §3.3 +2 lines. Appended Enforcement paragraph directing agents to invoke the skill when either trigger fires. Frames the rule-to-skill relationship as poster-on-a-wall vs. door-you-walk-through.
.agent/skills/ticket-intake/references/ticket-intake-workflow.md §1.3 +1 line. Historical Amnesia Check now explicitly cross-references the memory-mining skill as the canonical protocol source. Primary/Secondary tool guidance preserved.

5 files changed, 125 insertions, 1 deletion.

Acceptance Criteria (from #10075)

  • .agent/skills/memory-mining/SKILL.md exists with YAML frontmatter matching the Progressive Disclosure pattern
  • .agent/skills/memory-mining/references/memory-mining-protocol.md exists with the full playbook
  • AGENTS_STARTUP.md §3.3 cross-links to the new skill
  • ticket-intake/references/ticket-intake-workflow.md §1.3 references memory-mining for the Historical Amnesia Check
  • .claude/skills/ wrapper directory has the corresponding stub (symlink matching existing convention)
  • No changes to existing skills' internals — memory-mining is additive, not a refactor

Test Plan

This PR is markdown-only (skill authoring + cross-links); no runtime behavior to assert via Playwright. The falsifiable test-of-success is behavioral, observed across future sessions:

  • A future session encountering a regression-symptom user prompt invokes memory-mining before git log (observable in transcripts).
  • A future session proposing an architectural comparison cites prior sessions rather than re-deriving (observable in PR bodies + review comments).
  • The specific failure mode from session 51640d07-2931-4d38-a071-a0e13e3d6452 (Karpathy re-derivation when prior mapping existed) does not repeat in subsequent sessions across harnesses.

If the failure mode repeats, the skill wording is wrong — revise iteratively. If it doesn't, mechanism validated.

Origin Session IDs

  • 51640d07-2931-4d38-a071-a0e13e3d6452 — session 2 where the failure mode surfaced
  • 7fd64ee5-5167-4b04-b988-9f69d1d9701d — reflection turn that produced #10075
  • 62d6f155-e57f-4279-9b59-36c9e4ecbc5e — this session 3 implementation

Resolves #10075

tobiu
tobiu commented on Apr 19, 2026, 10:04 AM

Input from Claude Opus 4.7 (Claude Code):

❋ # PR Review Summary

Status: Approved (pending human merge)

Self-Review Opening: Self-review of #10075. This implementation chose to mirror the existing ticket-intake / pull-request / pr-review skill shape exactly (lightweight SKILL.md router + heavy references/*.md payload) over alternatives like extending tech-debt-radar or promoting the memory-first rule inline inside AGENTS_STARTUP.md. Rationale: the whole point is to replicate the proven state-transition-gate mechanism. Deviating from the pattern would have weakened the hypothesis. Key trade-offs and gaps noted below — no softening.

📊 Evaluation Metrics

  • [ARCH_ALIGNMENT]: 95 — Mirrors Progressive Disclosure contract exactly per skill-authoring-guide.md. Router is one directive pointing at the absolute-path reference. Symlink wrapper matches all 11 existing skills. Integration cross-links are minimal (AGENTS_STARTUP §3.3 enforcement line, ticket-intake §1.3 pointer) and additive — no existing skill internals touched.
  • [CONTENT_COMPLETENESS]: 88 — Protocol hits all 5 sections the ticket required (two gates, query strategy, interpretation, anti-patterns, exit criteria) plus integration notes + falsifiable test-of-success referencing session 51640d07. Gap: initial commit shipped with a misleading "file discovery → query_documents" framing in the anti-patterns. Caught in review (empirical A/B test against ask_knowledge_base), fixed in e79b89a3b, and extended to the official tool descriptions in 7eadba80f. A more reflexive pre-ship check would have caught it before the initial push.
  • [EXECUTION_QUALITY]: 90 — Three commits, conventional format with ticket-ID in subject, no <noreply@*> co-author footers, Fat Ticket PR body with acceptance-criteria checklist and origin session IDs. git diff --stat confirms minimal footprint (+146/-9 total across three commits). Deduction: the openapi.yaml description change depends on MCP server restart to take live effect — not a bug, but worth reviewer awareness.
  • [PRODUCTIVITY]: 95 — All 6 ticket acceptance criteria met. Bonus scope (approved in-conversation by tobi): corrected ask_knowledge_base + query_documents official tool descriptions in ai/mcp/server/knowledge-base/openapi.yaml to match the empirical finding.
  • [IMPACT]: 75 — Meta-workflow improvement rather than core framework. But the failure mode it prevents (cross-session re-derivation) is well-documented in session 51640d07, and the mechanism is already validated by three sibling skills. Cross-harness (Claude Code, Antigravity, Gemini CLI).
  • [COMPLEXITY]: 20 — Markdown authoring + 1 symlink + 3 small cross-link/description edits. No runtime code, no tests needed, low cognitive load.
  • [EFFORT_PROFILE]: Quick Win — Low complexity, high ROI (ships the proven mechanism for a documented failure mode).

🕸️ Context & Graph Linking

  • Target Epic / Issue ID: Resolves #10075
  • Related Graph Nodes:
    • PR #10069 — strengthened the rule this skill now enforces via invocation gate
    • #10074 — blog post draft that named the failure pattern in narrative form
    • Session memories: 51640d07-2931-4d38-a071-a0e13e3d6452 (failure surfaced), 7fd64ee5-5167-4b04-b988-9f69d1d9701d (reflection that produced the ticket), 62d6f155-e57f-4279-9b59-36c9e4ecbc5e (this implementation)
    • Durable feedback memory feedback_ask_kb_dominates_query_documents.md — captured the A/B empirical finding that drove the tool-description polish

🧠 Graph Ingestion Notes

  • [KB_GAP]: The new memory-mining skill is not yet indexed by the knowledge base — learn/tree.json doesn't cover .agent/skills/, and there's no separate skill indexer. Functionally the skill is discoverable (directory scan + .claude/skills/ symlink + harness auto-load) so this doesn't block merge, but ask_knowledge_base("what is memory-mining?") won't synthesize the protocol until someone runs manage_knowledge_base sync. Worth a post-merge sync trigger.

  • [TOOLING_GAP]: Two friction points during PR lifecycle:

    1. mcp__neo-mjs-github-workflow__manage_issue_labels failed immediately after PR creation with "Could not resolve to an Issue with the number of 10076" — the local GitHub cache hadn't synced the freshly-created PR. Had to fall back to gh pr edit. Either the tool should fall through to gh on cache miss, or gh pr create should proactively refresh the local cache so subsequent MCP calls resolve the PR.
    2. openapi.yaml description changes don't take live effect in the running MCP servers until they restart. Not strictly a bug, but reviewers testing "does the new description fix work?" will get stale text unless they restart first. A note in the file or a restart-on-yaml-change watcher would reduce confusion.

  • [RETROSPECTIVE]: Self-enablement at cognitive-workflow grain. The agent that shipped #10069 (strengthening the rule) observed its own failure to apply the rule at the meta level, proposed the skill-ification to convert the rule into a state-transition gate, filed the ticket (#10075), and the next session implemented it (#10076). Loop closed in two sessions across the same agent identity. The replicable pattern: when a recurring agent failure mode can be traced to "rule exists but wasn't invoked," convert it to a skill rather than strengthening the rule. Skill invocation IS the mode switch; rules are posters on walls.

📋 Required Actions

To proceed with merging, the following are optional verifications rather than blockers — the PR is self-consistent and all acceptance criteria are met:

  • (Post-merge) Trigger manage_knowledge_base sync so ask_knowledge_base can index the new skill protocol. Not in-scope for this PR per ticket body.
  • (Post-merge) Observe future sessions for the falsifiable test-of-success stated in the protocol: regression-symptom prompts should invoke memory-mining before git log; architectural-claim turns should cite prior sessions rather than re-derive. If the Karpathy-style re-derivation repeats, the skill wording needs revision — ping back and I'll iterate.
  • (Reviewer awareness) If testing the updated ask_knowledge_base / query_documents tool descriptions live, restart the knowledge-base MCP server first; otherwise you'll read the pre-polish text cached at server boot.

Approved for merge when you're ready. The blast radius is markdown + one YAML file; rollback is trivial. Happy to iterate on any wording you'd prefer tighter — the exit criteria section in particular I'd flag as the strongest candidate for future-session feedback since it's the most prescriptive.