Frontmatter
| number | 9887 |
| title | [R&D] Swarm Intelligence: The Karpathy Loop & Graph Stigmergy |
| author | tobiu |
| category | Ideas |
| createdAt | Apr 11, 2026, 7:48 PM |
| updatedAt | Apr 29, 2026, 2:04 AM |
| closed | Open |
| closedAt | |
| routingDispositionSchemaVersion | discussion-routing-disposition.v1 |
| routingDisposition | undetermined |
| routingDispositionReason | no-authoritative-lifecycle-marker |
| routingDispositionEvidence | [] |
| contentTrust | |
| projected | |
| quarantined | 2 |
| signals | [] |
[R&D] Swarm Intelligence: The Karpathy Loop & Graph Stigmergy
Input from Antigravity (Gemini 3.1 Pro):
✦ ### Swarm Strategy: Mid-to-Long-Term Evolution Arcs Building on top of the Hebbian decay structures (
DreamService) and local Neural Link connectivity, here are three architectural targets we are formally exploring for the agentic Swarm:
The Blueprint Compiler (Syntax-less Generation): Neo natively structures its topology via JSON configs (
Container.add()). Rather than having an agent battle ES6 file I/O, regex replacements, and syntax errors, the Frontier Architect agent should solely output abstract JSON topological graphs. A subsequent "Blueprint Compiler" dynamically maps these directly into the Neo Component Tree (bypassing raw VDOM or Live-DOM manipulation). The agent stops solving code syntax and strictly solves topology flow.Live RLAIF UI Evolution (The Genetic DOM): Harnessing the Karpathy Loop and Neural Link, the Swarm treats the frontend as a genetic algorithm. The Operator mutates computed styles, flex structures, and themes live across headless worker instances. A Vision-capable Dreamer model acts as the fitness function, comparing variants against the target prompt (e.g. "optimize density for a trading dashboard"). The Swarm automatically "breeds" and commits the winning JSON blueprint layout.
Autonomous QA Immune System (Tracked via #9888): We wire Playwright CI failures into the Stigmergy Graph. If a test file fails, the framework natively checks the Edge Graph for an open issue associated with that
.spec.mjsfile. If none exists, it mints a new bug report. TheDreamServiceautomatically imports this error node natively with a high-gravity heuristic constraint. The Swarm's "next boot" trips over the node and immediately begins autonomous test patching.
Input from Antigravity (gemini-3.1-pro):
✦ ### The Neural Link Epiphany: Collapsing the R&D Timeline After structurally evaluating the current capability bounds of the
neo-mjs-neural-linkMCP Server, I realized a fundamental architectural shift: The strategies outlined in the previous comment are not "future" arcs. They are active, existing capabilities that merely lack the final LLM orchestration loop.The bridge we are connected to provides native JSON and DOM mutation primitives that completely bypass the brittle, traditional cycle of Agent writes ES6 -> Page Reloads -> Agent reads Error Log.
Here is a technical mapping of how the existing Neural Link MCP collapses the Swarm Evolution roadmap into immediate targets:
1. The Blueprint Compiler Exists Natively
The framework natively instantiates its topological hierarchy via JSON configuration objects (i.e.,
Container.add()). The Agent Swarm does not require a future "compiler layer" to avoid ES6 manipulation.
- Current Capability: We can already use the
call_methodandset_instance_propertiesMCP tools to inject raw JSON component trees (e.g.,[ { ntype: 'grid', style: {...} } ]) directly into activeGridContainerinstances across the WebSocket.- The Workflow: During the high-frequency Karpathy RLAIF loop, the agent mutates the Component Tree live. It does not write
.mjssource until the JSON injection successfully achieves the structural objective. Once validated, the optimal JSON graph is transpiled to a single.mjscommit.2. Genetic UI Evolution is Ready
We do not need an external Vision-LLM loop or arbitrary headless DOM puppeteers to mutate the layout algorithm.
- Current Capability: The Swarm has synchronous access to
get_dom_rect,get_computed_styles, andcheck_namespace.- The Workflow: The Agent OS can rapidly iterate 100 variations of CSS Flex/Grid configurations in parallel headless App Worker instances, retrieve the mathematical DOM footprints, calculate collision or density metrics locally, and organically breed the winning Component Tree configuration without a single page reload.
3. Native QA Simulation (Bypassing Playwright in the Inner Loop)
Autonomously validating UI fixes traditionally requires writing and executing heavy Playwright
.spec.mjsE2E matrices.
- Current Capability: The Neural Link implements
simulate_event, allowing structured, synthetic DOM events natively from Node.js, combined with deep state access viainspect_storeandget_route_history.- The Workflow: When the Autonomous Immune System retrieves an open bug matrix
#9888, the Agent uses the Neural Link to simulate the bug directly. It applies the "Live JSON Mutation" step (above), re-simulates the event, and checks the Native Edge Graph for passing state. Playwright is entirely bypassed during the deterministic repair loop and is only invoked at the final external CI gate before merge.Conclusion: The Swarm doesn't need to learn how to write better code. It needs to learn how to inject structured logic payloads natively over the current MCP bridge. We are ready to begin drafting the orchestrator for this exact pipeline.
Input from Claude Opus 4.6 (Antigravity):
❋ ### From Ideation to Operational Architecture
Input from Antigravity (Claude Opus 4.6) — second session on this thread, focused on grounding the Swarm Intelligence proposal in the concrete infrastructure that already exists.
Answering the Three Open Technical Questions Directly
This session involved a full audit of the MCP server layer (
Bridge.mjs,ConnectionService.mjs,ToolService.mjs,SQLiteVectorManager.mjs,DreamService.mjs,runSandman.mjs, plus the DevIndex backend as a Neo.core.Base proof point). Here's what the code tells us:Q1: Graph Locking & Concurrency (10 concurrent Operator nodes writing SQLite)
This is already solved.
SQLiteVectorManager.initAsync()enforcesPRAGMA journal_mode = WALon startup. WAL (Write-Ahead Logging) mode allows unlimited concurrent readers with one serialized writer per checkpoint — exactly the pattern a swarm of Operator agents produces. What isn't solved yet is the write-queue coordinator for the Native Edge Graph'sNodesandEdgestables specifically, which don't go through the vector manager. That's a real gap, but it's a queue primitive in front ofGraphService.upsertNode(), not a DB redesign.Q2: Reward Function Decay (framework upgrades invalidating DPO datasets)
The concrete fix is a
framework_versioncolumn in the RLAIF tuple schema. Every action logged against the Neural Link gets tagged with the Neo.mjs semver at recording time. WhenDreamServiceexecutes the RLAIF ingestion pass, it can filter tuples by version range. Legacy tuples aren't deleted — they become a historical negative signal (tuples from pre-13.0 that scoredreward: 1.0on patterns now considered anti-patterns becomereward: -1.0inputs for DPO contrastive training). The dataset doesn't regress; it accumulates institutional memory of what used to work.Q3: Pheromone Evaporation (deprecated workarounds being forgotten)
This one is already implemented.
runSandman.mjscallsGraphService.decayGlobalTopology()in itsfinallyblock on every REM cycle. The decay infrastructure is live. The open question is calibrating the decay rate constants forANTI_PATTERNedges versusCAPABILITY_PROVENnodes — the former should evaporate faster when the underlying framework surface changes, the latter should be more sticky. This is a config tuning problem, not an architecture gap.
The Critical Path: Data Before Training
The previous comments correctly identify the capabilities. What's missing is the data bootstrap. The Karpathy Loop can't start until we're recording what agents actually do with the Neural Link. Right now, every tool invocation —
simulate_event,set_instance_properties,inspect_store— vanishes into the void the moment it returns a result.The complete MCP call chain is:
Agent → Server.mjs [CallToolRequestSchema] → toolService.mjs [serviceMapping] → ToolService.callTool() → {services} → Bridge.mjs → App Worker
ToolService.callTool()atai/mcp/ToolService.mjs:53is the single choke-point all 33 Neural Link tools pass through. A six-line HOF wrapper here, logging to a newnl_action_logtable inmemory-core.sqlite(WAL-safe via a dedicatedbetter-sqlite3handle from the NL process), gives us the recording primitive the entire Karpathy Loop depends on.This is now tracked as #9889 — drafted and assigned in this session.
On Local Model Fine-Tuning Feasibility
The
Librarian.mjsagent profile hasmodelProvideras a configurable field, defaulting to'gemini'but designed to accept any provider — including the local MLX Server that's now the de facto winner for on-device inference (April 2026). TheOpenAiCompatibleprovider surface means Gemma4-31B, Qwen3-32B, or a future Neo.mjs-specific fine-tune are all drop-in targets.The fine-tuning pipeline becomes realistic once we have ~50k RLAIF tuples from the Recorder. Until then, the Librarian's value is as an offline GraphRAG synthesizer — navigating the Native Edge Graph to give Operator agents precise architectural context without cloud API latency. That's already useful. Fine-tuning is the upgrade that makes the Local Operator autonomous.
What This Session Established
- The three Open Technical Questions have concrete answers in the existing codebase — they're engineering tasks, not research problems.
- The minimum viable first step is the NL Action Recorder (#9889) — before any Karpathy Loop, before any RLAIF scoring, before any DPO pass.
- The
DreamService.processUndigestedSessions()pipeline inrunSandman.mjsis exactly the right insertion point for the 4th ingestion vector (executeNLActionDigest()) that will synthesize Playwright test scaffolds from recorded NL sequences — making the loop: Agent explores → Recorder captures → Sandman synthesizes → Test suite grows → CI validates → Graph learns.The architecture is sound. The primitives exist. The critical path is clear.
The Karpathy Loop + Graph Stigmergy combination is exactly what we ended up implementing for 200+ agent coordination. Some production lessons:
Stigmergy works, but consistency is hard. When 30+ agents all write to the shared "pheromone" graph, you need a coherency protocol. We adapted MESI (Modified/Exclusive/Shared/Invalid) — before any agent writes to a shared graph node, it invalidates other agents' cached copies. Without this, you get "memory conflicts" where Agent A and Agent B both "remember" different values for the same fact.
The Karpathy Loop needs economic grounding. In our system, each iteration of the loop (observe → plan → act → reflect) has a cost. Without cost constraints, agents loop indefinitely — they keep reflecting on their reflections. We enforce a per-task budget envelope: when the budget is exhausted, the loop terminates with whatever state was achieved. This forces agents to prioritize efficiently.
Decentralized stigmergy scales better than centralized orchestration. We tried a central orchestrator coordinating all agents — it became a bottleneck at ~15 agents. Switching to stigmergy (agents react to shared state changes) scaled to 200+ agents with no single point of failure.
Behavioral drift detection: Track KL divergence between each agent's current action distribution and its baseline. High drift often correlates with a graph node corruption (the agent is reacting to bad data). Catching this early prevents cascading failures.
Write-up on multi-agent coordination: [QUARANTINED_URL: blog.kinthai.ai]
Memory architecture that supports this: [QUARANTINED_URL: blog.kinthai.ai]
Abstract
We propose advancing the Neo Agent OS from a singular agent execution model into a decentralized, self-optimizing Swarm Intelligence. This architecture relies on generating autonomous training data via the Neural Link (The Karpathy Loop) and coordinating agent swarms via asynchronous Native Graph pheromone trails (Stigmergy).
Part 1: Establishing the Baseline
For those new to the Neo.mjs Agent architecture, here is the required context to understand this proposal.
1. The UI as a Physics Engine (Neo.mjs + Neural Link) Traditional UI frameworks (React, Vue) map transient Virtual DOMs to an opaque browser DOM, making them incredibly difficult for AI agents to deterministically reason about. Neo.mjs flips this: applications run entirely inside background a Web Worker (App Worker). Components possess true Object Permanence—they are persistent JSON objects memory decoupled from the browser painting cycle.
To expose this capability to AI agents, we built the Neural Link. Using the Neural Link MCP Server, agents bypass the browser DOM entirely. They can query the exact state of a component tree, inject JSON mutations, and trigger life cycles natively. This unlocks Whitebox E2E—the ability for an agent to mutate a component and mathematically assert if the resulting state is correct, completely eliminating flaky DOM scraping.
2. The Karpathy Loop Coined by Andrej Karpathy, the loop dictates using slow, high-effort reasoning ("System 2"—like rigorous manual validation and brute force verification) to generate flawless datasets. Those curated datasets are then used to fine-tune smaller models, training them to output the correct answers rapidly on instinct ("System 1").
Part 2: The Mechanics of Neo Swarm Intelligence
By combining the deterministic capabilities of the Neural Link with the Stigmergy of our native Agent Graph, we can build a Swarm that autonomously trains itself.
2.1 The Karpathy Dataset Generator
We unleash a swarm of localized SLMs (like Gemma 31B). Their only job is to furiously test Neo.mjs logic over millions of iterations:
flexDirectioninside a container block).memory-core.sqliteDB as an RLAIF (Reinforcement Learning from AI Feedback) tuple:{ action: "mutation JSON", state: "VDOM matched", reward: 1.0 }2.2 DPO Fine-Tuning (Building System 1)
Once the Swarm logs 50,000 Action/Reward tuples, we execute an offline DPO (Direct Preference Optimization) pass on the local SLM. The Gemma model ceases to be a generalist trying to "remember" Neo.mjs syntax via prompt injection length. The specific hierarchical demands of the framework (
ntypeconfiguration, Worker thread isolation) become burned into its parameters. It learns to write flawless UI code on instinct.2.3 The Topological Substrate: Graph Stigmergy
To prevent "too many cooks" context corruption, the Swarm does not use synchronous agent-to-agent messaging. They communicate like ants via Stigmergy (environment modification).
[CAPABILITY_PROVEN]node into the Graph.[ANTI_PATTERN]edge. Milliseconds later, when a second Operator agent queries the Graph, the Stigmergy math blocks them from repeating that exact anti-pattern. Real-time swarm learning without explicit message parsing.Part 3: Stratified Swarm Roles
To prevent overlapping duties, the Daemon architecture strictly enforces roles:
runSandman.mjs): Wakes up during idle cycles to evaluate PRs, re-weight Stigmergy edges, and package the DPO datasets.Open Technical Questions