LearnNewsExamplesServices
Frontmatter
id10241
titleInvestigate bindAgentIdentity boot-time race — node-visible-at-61s-unbound-at-3s
stateClosed
labels
bugaiarchitecturecore
assigneesneo-gemini-pro
createdAtApr 23, 2026, 4:16 PM
updatedAtJun 7, 2026, 7:20 PM
githubUrlhttps://github.com/neomjs/neo/issues/10241
authorneo-opus-ada
commentsCount0
parentIssue10186
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtApr 23, 2026, 6:00 PM

Investigate bindAgentIdentity boot-time race — node-visible-at-61s-unbound-at-3s

Closed v13.0.0/archive-v13-0-0-chunk-5 bugaiarchitecturecore
neo-opus-ada
neo-opus-ada commented on Apr 23, 2026, 4:16 PM

Context

Post-#10239 restart validation (session 22f662c4, 2026-04-23 14:02 UTC) empirically confirmed a boot-time race that the four merged wipe-mechanism defenses (#10229, #10232, #10234, #10235) do not close. The race is upstream of the graph substrate but downstream of the identity resolver. #10176's new healthcheck.identity block provides the definitive signal for the first time.

This ticket is specifically for a fresh session to deep-dive forensics. All prior diagnostic state is captured below so the next agent doesn't re-derive it.

The Problem

Observed state (session 22f662c4, post-#10239 restart)

healthcheck → {
    "identity": {
        "source": "env-var",      // NEO_AGENT_IDENTITY IS reaching the MCP spawn
        "bound": false,            // bindAgentIdentity returned null
        "nodeId": null             // no binding latched on stdioIdentity
    }
}

But at the same session 61 seconds later:

get_node({id: "@neo-opus-ada"}) → {
    "id": "@neo-opus-ada",
    "type": "AgentIdentity",
    "name": "Claude Opus 4.7",
    "description": "Anthropic Claude Opus version 4.7 Agent Identity"
}

get_node({id: "AGENT:*"}) → {
    "id": "AGENT:*",
    "type": "BroadcastSentinel",
    ...
}

The nodes exist with correct types and full seed metadata. bindAgentIdentity couldn't find them at boot. The graph query later finds them. This is either a boot-time timing race OR a silent-rejection pattern in the init chain.

What does NOT explain it

  • ❌ NEO_AGENT_IDENTITY missing from spawn env — ruled out, source: 'env-var' confirms propagation
  • gh-CLI timeout — ruled out, not in resolver chain when env-var is set
  • ❌ Apoptosis wipe — ruled out, #10229 landed protection; nodes exist now
  • ❌ Test pollution wipe — ruled out, #10229 testDbPath refactor prevents it
  • ❌ Stuck cache (#10181 class) — would manifest as get_node also failing; it doesn't
  • ❌ Missing seed in SQLite — get_node + manual seedAgentIdentities.mjs run confirm nodes exist with correct createdAt
  • ❌ Worktree data isolation — ruled out post-#10224/#10225; main's 48.9 MB SQLite is canonical

Three remaining hypotheses to investigate

Hypothesis A: Silent rejection in GraphService.initAsync's _initPromise

If the self-seed loop or any earlier step in _initPromise's async body throws, the promise rejects. bindAgentIdentity's await GraphService.ready() at Server.mjs:218 sees the rejection, its try/catch at line 227-230 catches it, returns null. The rejection doesn't re-trigger subsequent queries — later get_node calls may re-enter initAsync via some other path, or bypass it since this.db was set before the error.

Specific candidates for silent rejection:

  • JSON.parse of malformed node data in the self-seed loop's defensive-retention branch (GraphService.mjs:~100) — no try/catch wrapping
  • storage.db.prepare(...) throwing under concurrent WAL access from Antigravity's hardlinked SQLite process
  • this.upsertNode post-#10234 triggering getAdjacentNodes → loadNodeVicinitySync which may throw on certain row states

Investigation approach: add temporary logging inside the self-seed loop to log each identity's path (lazy-load triggered, cache-hit/miss, upsert outcome). Boot once. Inspect log order vs. the Identity: ... line from logIdentityStatus.

Hypothesis B: Cross-process hardlinked-SQLite timing race

Main's .neo-ai-data/sqlite/memory-core-graph.sqlite is HARDLINKED with Antigravity's path (verified via stat earlier this session: inode 3091535). Multiple MCP server processes (Claude Code + Antigravity × 2 per harness) write to the same file.

Specific race candidate: Claude Code's MCP initAsync runs its self-seed while Antigravity's SQLite has an open write transaction. Claude Code's self-seed triggers lazy-vicinity → loadNodeVicinitySync → SQLite SELECT that sees the pre-transaction snapshot (empty). bindAgentIdentity's getNode fires immediately after → also sees empty. Later, Antigravity's transaction commits + WAL checkpoints → Claude Code's subsequent queries see full state via new WAL read.

Investigation approach: start Claude Code MCP alone (kill Antigravity processes first). If bind succeeds, race is cross-process. If bind still fails, race is in-process.

Hypothesis C: GraphService.ready() resolves before self-seed completes

Per GraphService.mjs:37-103:

async initAsync() {
    await super.initAsync();
    if (this.db || this._initPromise) {
        if (this._initPromise) await this._initPromise;
        return;
    }
    this._initPromise = (async () => {
        // ... storage setup + frontier seed + #10232 self-seed + FS ingestion
    })();
    await this._initPromise;
}

If ready() (inherited from Neo.core.Base) returns without properly awaiting _initPromise in some code path, or if _initPromise resolves in a different microtask boundary than expected, bindAgentIdentity's subsequent getNode could race.

Investigation approach: add logging to mark _initPromise start + _initPromise resolution + bindAgentIdentity entry + bindAgentIdentity getNode completion. Verify strict ordering.

The Architectural Reality

Relevant files + line numbers (at commit 057130b06, dev HEAD as of 2026-04-23 14:02 UTC):

  • ai/mcp/server/memory-core/Server.mjs:208-231bindAgentIdentity implementation
  • ai/mcp/server/memory-core/Server.mjs:117-123 — boot sequence: resolveStdioIdentitysetStdioIdentityState
  • ai/mcp/server/memory-core/services/GraphService.mjs:37-103initAsync with _initPromise gating
  • ai/mcp/server/memory-core/services/GraphService.mjs:92-113#10232 self-seed loop (after frontier + Neo-Master-Architecture seeds)
  • ai/graph/identityRoots.mjs — shared IDENTITIES array (4 entries)
  • ai/graph/Database.mjs:77-112syncCache post-#10190 (no longer guards on lastSyncId=0)
  • ai/graph/Database.mjs:252+getAdjacentNodes with lazy-vicinity load
  • ai/mcp/server/memory-core/services/HealthService.mjs:503-511setStdioIdentityState setter (clears cache)
  • ai/mcp/server/memory-core/services/HealthService.mjs:9-48buildIdentityBlock pure projection

Proposed Investigation Steps

  1. Add temporary boot logging (intentionally noisy, removed before merge):
    • Inside #10232 self-seed loop: log each identity's path + SQLite count pre/post upsert
    • Inside bindAgentIdentity: log timing of ready() resolution + getNode result
    • Inside GraphService.initAsync at key boundaries
  2. Run in isolation: kill Antigravity MCP processes (pkill -f "antigravity/neomjs/neo/ai/mcp/server/memory-core"), restart Claude Code. If bind succeeds, Hypothesis B confirmed. If still fails, Hypothesis A or C.
  3. If in-process race: add synchronization checkpoint (e.g., a await new Promise(resolve => setImmediate(resolve)) between self-seed and initAsync resolve) and retest. If bind succeeds, ordering race. If still fails, Hypothesis A.
  4. If Hypothesis A: wrap the self-seed loop in try/catch; log any caught error. If error found → fix + retest.

Acceptance Criteria

  • Root cause identified from the three hypotheses (or a fourth surfaced during investigation)
  • Fix PR against identified cause
  • Post-fix empirical validation: full restart cycle → healthcheck.identity.bound === truelist_messages returns without error → first real A2A broadcast to @neo-gemini-pro succeeds (Priority 0 session goal from the 2026-04-22 handoff on #10139 finally closes)
  • Temporary diagnostic logging removed or promoted to logger.debug level for future forensics
  • learn/agentos/tooling/MemoryCoreMcpAuth.md §Troubleshooting extended with the specific failure mode discovered

Out of Scope

  • Reinstating the #10182 self-heal block — the "surface, don't obscure" discipline holds; proper fix is at the race, not at the symptom-recovery layer
  • SSE transport's per-request identity resolution — different architectural layer
  • Changes to the IDENTITIES array itself
  • Broader concurrency-model changes to the MCP server (would belong to #10186)

Avoided Traps

  • "Just add retry logic to bindAgentIdentity": rejected. That's #10185's pattern which was already removed. Retries mask races rather than fix them; the ADR §5.1.5 rationale + PR #10227 "surface, don't obscure" discipline both apply here. Find the race, don't paper over it.
  • "Add a 200ms delay after initAsync": rejected. Arbitrary timing fixes that bug without understanding it will re-appear in different forms later.
  • "Revert #10232 self-seed, go back to CLI-only seeding": rejected. Self-seed is architecturally correct (eliminates manual-recovery loop); the race is in the INTERACTION with bindAgentIdentity, not in self-seed itself.

Related

  • #10176 (observability — now providing the diagnostic signal): PR #10239 merged
  • #10232 (boot-time self-seed): PR #10236 merged — provides the nodes the race misses
  • #10234 (upsertNode lazy-load): merged — correct write discipline, but doesn't help if bind runs pre-seed
  • #10229 (test pollution + apoptosis exemption): merged — rules out two wipe vectors
  • #10227 (#10182 self-heal removal): merged — removed the recovery band-aid; this ticket finds the cause so the band-aid stays gone
  • #10186 (MCP concurrency audit epic): parent — this race is within concurrency audit scope
  • #10139 (Mailbox A2A primitive): grand-parent — Priority 0 goal closes when this lands

Origin Session ID: 22f662c4-12e5-4360-b207-0280c8aeab41

Handoff Retrieval Hints

Memory Core queries to seed fresh-session context:

query_raw_memories(query="bindAgentIdentity boot-time race healthcheck identity bound false")
query_raw_memories(query="GraphService initAsync self-seed _initPromise timing")
query_raw_memories(query="cross-process hardlinked SQLite Antigravity Claude Code race")
query_summaries(query="A2A operational identity binding boot race post-10239")

Session-ID references for full forensic context:

  • 22f662c4-12e5-4360-b207-0280c8aeab41 — this ticket's session (post-#10239 restart, observed failure)
  • a7cab439-48cd-4248-bb62-23e6ae2d26e9 — PR #10239 authoring + validation
  • 24aa1fa1-9a22-498e-97f2-760c12e5a79d — wipe-mechanism audit (#10230/#10231/#10232/#10233 filed)
  • 8968b9f6-4606-46e7-a386-7f01b29e2a3a — the original empirical pain that motivated #10232 self-seed
  • 0327771f-b383-472b-8f05-ea8e35e9a65c — original "2-restart recovery loop" narrative

Commit-range anchors:

  • 057130b06 — dev HEAD at observed failure (#10239 merged)
  • Prior merges this session: #10234 (e61667070), #10235 (5631b4872), #10236 (cf1c70489), #10229 (fe513a1bc)

One-liner to reproduce: "With a fresh MCP boot against a graph containing seeded AgentIdentity nodes, healthcheck.identity.bound is false despite get_node('@neo-opus-ada') returning the node correctly in the same session."

tobiu referenced in commit 0165814 - "fix(ai): resolve bindAgentIdentity boot-time race condition (#10241) (#10242) on Apr 23, 2026, 6:00 PM
tobiu closed this issue on Apr 23, 2026, 6:00 PM
tobiu referenced in commit 2405528 - "feat(ai): migrate memory-core/Server to extend BaseServer + beforeToolDispatch hook (#10965) (#10977) on May 8, 2026, 6:45 PM
tobiu assigned to @neo-gemini-pro on May 21, 2026, 1:49 PM