Context

The Memory Core holds 8500+ verbatim Thought → Action → Response trajectories with cross-model validation (per-model GitHub identities, graph-linked outcomes, cross-family PR reviews documented). This corpus is a rare training-signal asset — higher fidelity than web-scraped data because the outcomes are validated by a multi-model review process, and the structure (graph edges, ticket citations, ADR references) is preserved.

ai/Agent.mjs has OllamaProvider configured with gemma4:31b as the local sub-agent inference target. The sub-agent profiles (Librarian, QA, Browser) currently run the stock model. Fine-tuning against the swarm's own trajectory corpus is the natural next step — sub-agent quality on Neo-shaped work goes up, which reduces frontier-model dependency for routine tasks.

This ticket is deliberately scoped to extraction and preparation only. The fine-tuning itself + benchmarking + routing is the follow-up, scoped after we see what the extracted dataset actually looks like.

The Problem

Three specific capabilities missing today, each a prerequisite for any downstream fine-tuning work:

1. No extraction tool — Memory Core's query_raw_memories / get_session_memories can read trajectories, but there's no pipeline that emits a training-shaped dataset (JSONL with standardized fields, redacted secrets, deduplicated, split into train/val).
2. No redaction pipeline — trajectories may contain .env leakage (empirically observed: session 0327771f had GH_TOKEN / GEMINI_API_KEY values appear in raw memory). Any dataset release requires a scrubbing pass with a recognized-secret matcher + human review.
3. No dataset schema — what fields does a training row carry? Proposed structure needs to be documented before extraction so downstream consumers (fine-tuning, evaluation, publication) know what to expect.

Without these three, fine-tuning is unauditable. With them, it's a reproducible pipeline.

The Architectural Reality

• Source: Memory Core ChromaDB neo-agent-memory collection (8500+ rows) + neo-agent-sessions collection (794 summaries). Also accessible via ai/services.mjs SDK (same substrate, different surface).
• Associated graph edges: Native Edge Graph provides outcome metadata (ticket merged/rejected, PR cross-family-reviewed, cited ADRs, tag ingestion [RETROSPECTIVE]/[KB_GAP]/[TOOLING_GAP]).
• Candidate extraction substrate: new ai/scripts/extractTrajectoryDataset.mjs + a TrajectoryRow shape defined in ai/graph/ or co-located.
• Redaction substrate: pattern-matcher for GH_*, *_TOKEN, *_API_KEY, *_SECRET, .env style lines, GitHub PAT shapes, Anthropic/Google API key shapes. Fails-loud on match (not silent-redact) during the scrub pass so false positives surface human review.
• Schema documentation: learn/agentos/TrajectoryDataset.md co-located with other agentos guides.

The Fix

Three landing items together:

1. Extraction script at ai/scripts/extractTrajectoryDataset.mjs:
   • CLI: node ai/scripts/extractTrajectoryDataset.mjs --since <iso-date> --out <path>
   • Pulls from Memory Core via the SDK (not MCP tool — bypass the tool-layer truncation limits)
   • Joins with graph outcome metadata (merge status, review status, citations)
   • Emits JSONL to the specified output path
2. Redaction pass built into the extraction script:
   • Default mode: fail-loud on any matched secret pattern; require explicit --include-secrets-at-own-risk flag to proceed (strictly for local inspection, never for publication)
   • Recognized patterns as a module-scope SECRET_PATTERNS Set — extensible
   • Each redacted row emits a report line to stderr with pattern type + session ID for audit
3. Schema documentation at learn/agentos/TrajectoryDataset.md:
   • Row shape: { prompt, thought, toolCalls: string[], response, sessionId, agent, model, timestamp, outcome: {merged, rejected, deferred, null}, crossFamilyReview: {status, reviewer, citations[]}, tags: {kbGap, toolingGap, retrospective}[] }
   • Redaction guarantees + limitations (human review recommended for any release)
   • Field provenance (which Memory Core / Graph source populates each field)
   • Register in learn/tree.json

Acceptance Criteria

• ai/scripts/extractTrajectoryDataset.mjs exists, CLI-invokable, emits valid JSONL
• Row schema matches the documented shape (Playwright unit test validates a fixture row)
• Redaction patterns catch at least these eight categories: GH_TOKEN, GEMINI_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY, NEO_*_SECRET, .env file contents fingerprint, GitHub PAT shape (ghp_*, github_pat_*), generic base64-encoded-secret heuristic. Each has a unit test.
• Default mode fails-loud on any match; --include-secrets-at-own-risk override documented + logged to stderr when used
• learn/agentos/TrajectoryDataset.md published, registered in learn/tree.json
• Post-merge validation: run the extraction against the current Memory Core, manually inspect a 100-row sample for redaction correctness + schema conformance
• No row in the sample output contains any recognized secret pattern (empirical verification of the redaction pass)

Out of Scope

• Fine-tuning itself (Ollama fine-tune commands, Gemma weights, training loop) — separate ticket once this lands
• Benchmark design (SWE-Bench, a Neo-specific benchmark, comparison to the stock model) — separate ticket
• Automated publication of the dataset to HuggingFace or similar — separate decision, after empirical review of the first extraction
• Routing sub-agent workloads to the fine-tuned model at runtime — separate ticket once the fine-tuned model exists
• Building a real-time streaming extraction (incremental updates to a published dataset) — v2 concern

Avoided Traps

• "Export everything, let consumers filter": rejected. Unredacted exports leak secrets. Fails-loud by default is the right discipline.
• "Use MCP tool surface instead of SDK": rejected. MCP tool-layer has budget/truncation limits intended for runtime queries; dataset extraction needs direct SDK access to bypass those.
• "Build fine-tuning first, figure out dataset later": rejected. Fine-tuning without documented schema produces an unreproducible artifact. Schema + extraction first, fine-tuning second.
• "Include only merged-PR trajectories, skip rejections": rejected. Rejected / Request-Changes trajectories are more valuable training signal (they teach the model what gets rejected and why). Full distribution preserves the preference signal.
• "One big publication-ready dataset": rejected for this phase. First extraction is for internal fine-tuning. Publication is a separate decision after empirical review.

Related

• Parent Discussion (source of the architectural rationale): #10137 (MX — Model Experience)
• Sub-agent provider substrate: ai/Agent.mjs + GeminiProvider / OllamaProvider
• Model target: gemma4:31b via Ollama (config: ai/mcp/server/memory-core/config.mjs:106-110)
• Memory Core source: 8500+ rows in neo-agent-memory, 794 summaries in neo-agent-sessions
• Security precedent: session 0327771f observed .env contents leaking into raw memory — empirical motivator for the redaction requirement

Origin Session ID: 24aa1fa1-9a22-498e-97f2-760c12e5a79d

Handoff Retrieval Hints

• query_raw_memories(query="trajectory dataset extraction local model fine-tuning")
• query_raw_memories(query="redaction secret pattern .env leak memory")
• query_summaries(query="swarm trajectory local inference sub-agent fine-tune")