ConceptDiscoveryService output schema — add extraction_metadata (missing_fields, ambiguous_references, confidence_score)

Summary

Post-merge follow-up to #10105. Empirical probe against live gemma-4-31b-it (documented in session memory, summarized in the handoff comment of #10105) surfaced the model's own proposed schema for reliable self-report: narrow objective markers it CAN produce consistently, not subjective friction it cannot.

When asked directly whether a generic feedback field would be reliable, gemma answered "No" and articulated the three failure modes itself (post-hoc rationalization / confabulation, RLHF politeness bias, correlation-vs-causation). Then it proposed this schema:

{
  "candidates": [...],
  "extraction_metadata": {
    "missing_fields": ["field_name"],
    "ambiguous_references": ["The PR mentions 'the module' but three modules are defined."],
    "confidence_score": 0.0
  }
}

The model's reasoning: "Asking me to list ambiguous references forces me to point to specific strings in the source text. This is a linguistic task I am actually designed for. You cannot verify if I felt slowed down. You can verify if a reference was actually ambiguous."

The Fix

1. Extend the Teaching-Test system prompt in ConceptDiscoveryService.extractConceptsFromSource to request extraction_metadata alongside candidates.
2. Update the response parser to capture + preserve extraction_metadata.
3. Persist metadata onto mined candidate rows in nodes.jsonl as optional fields (keep schema additive — legacy rows without metadata still parse).
4. Surface metadata downstream: GoldenPathSynthesizer could prioritize curator review by confidence_score; ambiguous_references flags source-text pointer candidates that may qualify as concepts themselves with more context.

Acceptance Criteria

• System prompt in ConceptDiscoveryService instructs gemma to return extraction_metadata with the three documented fields
• Response parser captures metadata and passes it through to appendCandidates
• JSONL schema extended with optional extraction_metadata block per candidate
• New test in ConceptDiscoveryService.spec.mjs covers metadata extraction from a mocked LLM response
• Existing 49 tests still pass
• ConceptService reader tolerates both metadata-present and metadata-absent rows (additive compatibility)

Out of Scope

• Downstream consumer implementations (prioritizing review by confidence_score, surfacing ambiguous_references in sandman_handoff.md) — file separate tickets if/when the pattern proves valuable with real data.
• Cross-session aggregation / Hebbian reinforcement feedback loop — orthogonal design tracked in session memory; file when discovery produces meaningful mined volume.

Related

• Origin PR: #10105 (ConceptDiscoveryService LLM-based extraction — merged)
• Parent epic: #10036 (Memory Core Concept Discovery)
• Empirical probe rationale: session memory feedback_llm_substrate_for_semantic_tasks.md

Origin Session ID: 1db25bbe-f39d-4dcd-bb0e-bc125ce91326