Context
Operator-directed (2026-06-22, divergence→convergence): the per-message inline concept-extraction was removed from the hot path (#13834 / PR #13836) because it fired a gemma4 call per A2A message. But the auto-extracted concepts are not just noise — they are a frequency signal for emerging MX/process vocabulary. Relocate (don't retire) the extraction to a scheduled drainer.
Why messages have unique signal (stratified): CODE concepts (multi-threading) land in PRs eventually — low message-unique value. PROCESS/MX concepts (how the swarm works) are born in A2A and frequently never become a code-PR. This session alone minted coordination↔saturation cycle, measure-cheap-first, skip-vs-choke, structural-bridge, idle-window-by-construction — none a code concept, none in any PR diff, all recurring vocabulary. The epic/PR corpus (ConceptDiscoveryService) structurally cannot catch these — it scans epics + PRs, not messages. This drainer harvests the swarm's self-understanding vocabulary = the friction→gold substrate.
The Design (converged)
- Unfiltered capture, scheduled — a new registry task (alongside dream/summary,
exclusive-heavy, idle-window-friendly) batch-processes un-harvested messages. NOT inline per-message (that was the #13834 hammer).
- Cheap frequency pre-filter (no gemma4) — rank by frequency over the free structured signal first: message subjects (
[lane-claim], [converge], …) + the curated taggedConcepts already on each message. The LLM is spent ONLY on the Teaching-Test promotion of the top-N candidates.
- Teaching-Test promotion — reuse
ConceptDiscoveryService's LLM semantic gate ("is this a real concept?") on the high-frequency candidates. (Reconciles the frequency-vs-Teaching-Test tension: frequency surfaces candidates; the LLM judges them.)
- Dedup-by-name (the key filter) — operator: "unfiltered => but in the end filtered by concept name." Normalize names (case / hyphen-underscore /
The-prefix / CONCEPT:-prefix) and collapse variants. The 19k CONCEPT population has rampant name-variant dups (observed in a runSandman log: V-B-A/VBA/VBA_FrictionGold; Review-Response-Protocol/Review_Response_Protocol; NeuralLink/TheNeuralLink).
Open Question (the real fork)
One ontology with a tagged process branch, or two ontologies? Process/MX concepts (measure-cheap-first) aren't the same ontology as code concepts (multi-threading). If they land in the code ontology, its gap-audit fires on process terms (noise in that audit). The drainer should feed a process/MX branch or layer, not dilute the code ontology.
Acceptance Criteria
Out of Scope
Related
- Builds on: #13834 (the inline removal that this relocates). Reuses:
ConceptDiscoveryService (the Teaching-Test gate; currently epics+PRs, manual). Sibling: #13835 (tri-state). Parent epic: #13624 (orchestrator) or #12065 (REM-SSOT) — steward's call.
Tooling note: the §1a duplicate sweep was partial — a pre-block grep + gh search found no existing concept-drainer ticket (closest are the orchestrator epics #13624/#13755); the full live gh latest-open sweep was blocked by a transient auto-classifier outage and should be re-run at pickup.
Origin Session ID: e96a0867-4c28-4877-b2f4-f56173ee9fd1
Context
Operator-directed (2026-06-22, divergence→convergence): the per-message inline concept-extraction was removed from the hot path (#13834 / PR #13836) because it fired a gemma4 call per A2A message. But the auto-extracted concepts are not just noise — they are a frequency signal for emerging MX/process vocabulary. Relocate (don't retire) the extraction to a scheduled drainer.
Why messages have unique signal (stratified): CODE concepts (
multi-threading) land in PRs eventually — low message-unique value. PROCESS/MX concepts (how the swarm works) are born in A2A and frequently never become a code-PR. This session alone mintedcoordination↔saturation cycle,measure-cheap-first,skip-vs-choke,structural-bridge,idle-window-by-construction— none a code concept, none in any PR diff, all recurring vocabulary. The epic/PR corpus (ConceptDiscoveryService) structurally cannot catch these — it scans epics + PRs, not messages. This drainer harvests the swarm's self-understanding vocabulary = the friction→gold substrate.The Design (converged)
exclusive-heavy, idle-window-friendly) batch-processes un-harvested messages. NOT inline per-message (that was the #13834 hammer).[lane-claim],[converge], …) + the curatedtaggedConceptsalready on each message. The LLM is spent ONLY on the Teaching-Test promotion of the top-N candidates.ConceptDiscoveryService's LLM semantic gate ("is this a real concept?") on the high-frequency candidates. (Reconciles the frequency-vs-Teaching-Test tension: frequency surfaces candidates; the LLM judges them.)The-prefix /CONCEPT:-prefix) and collapse variants. The 19k CONCEPT population has rampant name-variant dups (observed in a runSandman log:V-B-A/VBA/VBA_FrictionGold;Review-Response-Protocol/Review_Response_Protocol;NeuralLink/TheNeuralLink).Open Question (the real fork)
One ontology with a tagged process branch, or two ontologies? Process/MX concepts (
measure-cheap-first) aren't the same ontology as code concepts (multi-threading). If they land in the code ontology, its gap-audit fires on process terms (noise in that audit). The drainer should feed a process/MX branch or layer, not dilute the code ontology.Acceptance Criteria
exclusive-heavy, with aconceptHarvestedmarker (mirrorgraphDigested) so messages aren't re-processed.Out of Scope
Related
ConceptDiscoveryService(the Teaching-Test gate; currently epics+PRs, manual). Sibling: #13835 (tri-state). Parent epic: #13624 (orchestrator) or #12065 (REM-SSOT) — steward's call.Tooling note: the §1a duplicate sweep was partial — a pre-block grep +
ghsearch found no existing concept-drainer ticket (closest are the orchestrator epics #13624/#13755); the full liveghlatest-open sweep was blocked by a transient auto-classifier outage and should be re-run at pickup.Origin Session ID: e96a0867-4c28-4877-b2f4-f56173ee9fd1