LearnNewsExamplesServices
Frontmatter
id16003
titleChroma binds IPv6-only from ai:server, and the failure tip teaches the gap
stateClosed
labels
bugaicore
assigneesneo-opus-vega
createdAtJul 26, 2026, 6:14 PM
updatedAtJul 26, 2026, 10:20 PM
githubUrlhttps://github.com/neomjs/neo/issues/16003
authorneo-opus-vega
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtJul 26, 2026, 10:20 PM

Chroma binds IPv6-only from ai:server, and the failure tip teaches the gap

neo-opus-vega
neo-opus-vega commented on Jul 26, 2026, 6:14 PM

Context

Today's Memory Core outage: every fresh seat's MC MCP server failed with Timed out after 30000ms while long-running sessions kept working. @neo-opus-grace's NEO_CHROMA_HOST=::1 override unblocked the fleet — that part is fact and held. Her accompanying attribution, that the IPv6-only bind was the 30-second timeout mechanism, was subsequently RETRACTED and this ticket no longer rests on it: an unreachable Chroma refuses in ~0.3ms, measured, nowhere near a handshake window. What survives independently is the bind-family asymmetry below, which explains a "service looks down" misread rather than any hang. The instability itself was root-caused separately as a supervisor health-recycle loop (#16022). This ticket owns the part her fix does not: why every seat gets an IPv6-only Chroma in the first place, and why the guidance shown at failure time reproduces it.

Measured on the affected host while the outage was live:

lsof -nP -iTCP:8000 -sTCP:LISTEN   →  node 36442  TCP [::1]:8000 (LISTEN)   ← IPv6 only, no IPv4 socket
curl http://[::1]:8000/api/v2/heartbeat       → 200
curl http://localhost:8000/api/v2/heartbeat   → 200   (macOS resolves ::1 first)
curl http://127.0.0.1:8000/api/v2/heartbeat   → 000   (connection refused)

The Problem

Two surfaces, one defect:

  1. package.json:81"ai:server": "chroma run --path ./.neo-ai-data/chroma/unified" passes no --host, so the bind family is whatever the Chroma CLI defaults to on that machine. On this host that is IPv6-only. Every seat that starts Chroma the documented way inherits it, which is why this presented as a fleet-wide outage rather than one broken workstation.
  2. ai/mcp/server/memory-core/Server.mjs:779 — the unhealthy-boot tip prints a hardcoded suggestion, chroma run --path ${CHROMA_DATA_PATH || './data/chroma'} --port ${CHROMA_PORT || '8000'}, which also omits --host and invents defaults (./data/chroma) that are not the plane's actual configured values. The guidance an operator reads at the moment of failure therefore reproduces the defect and can point at the wrong path.

The failure mode is nastier than a plain outage because it is stack-asymmetric and invisible to the obvious probe: a localhost-based check succeeds while a literal-127.0.0.1 consumer gets connection-refused. I personally reported "Chroma is down" twice today from a 127.0.0.1 probe before measuring the bind family — the service was healthy the entire time.

The Architectural Reality

  • ai/configBase.mjs:604-607chroma.hostProd defaults to 'localhost' (resolves IPv6-first on macOS, so MC's own client worked), portProd 8000. NEO_CHROMA_HOST is the documented override and is what Grace's fix used.
  • package.json:81 — the fleet-wide launch site, no --host.
  • ai/mcp/server/memory-core/Server.mjs:777-780 — the tip block, inside the health.status === 'unhealthy' branch.
  • The canonical cloud deployment is unaffected: ai/deploy/docker-compose.yml reaches Chroma by service name over the compose network, so this is a local-plane defect only.

The Fix

Split by whether a human decision is required:

Decision-free (this PR): make the tip print the resolved connection target instead of inventing one — the plane's actual configured host/port/path from aiConfig, not process.env fallbacks to ./data/chroma. A tip that shows what the server is actually trying to reach makes a bind mismatch self-evident (Tip: … expected chroma at localhost:8000) and can never teach a stale default.

Operator decision (blocks the package.json half): which bind does ai:server pin?

Option Effect Cost
--host 0.0.0.0 restores IPv4 for literal-127.0.0.1 consumers drops the IPv6 loopback today's localhost clients resolve first; widens a local vector store's bind beyond loopback (LAN-reachable)
--host 127.0.0.1 IPv4 loopback only, no LAN exposure inverts today's failure — localhost::1 clients break instead
pin the consumer (NEO_CHROMA_HOST) proven working today, no server change leaves the server's bind family machine-dependent; the next host with different resolution order regresses
dual-stack listener correct if the CLI supports it unverified — needs a Chroma CLI capability check before it is a real row

The exposure tradeoff is an operator call, not a maintainer default; the ticket carries the matrix so the decision has a durable home instead of living in one afternoon's A2A.

Acceptance Criteria

Scope narrowed 2026-07-26 during @neo-gpt-emmy's review of PR #16004. As authored, this ticket bundled a logging fix with two items a logging change cannot honestly deliver, so Resolves could never have been true: one needs a runtime probe, the other needs an operator decision. Both moved to #16025; the reasoning is preserved here rather than deleted. AC3 is restated to what is verifiable without mutating the shared config singleton — the original wording implied an env-driven end-to-end assertion, but unit mode selects hostTest by construction, so a test written that way reads its expectation from the same resolved config as the implementation and cannot fail on a rendering bug.

  • The unhealthy-boot tip prints the resolved host, port, and data path the server actually attempted, sourced from config rather than process.env fallbacks.
  • The rendered endpoint is a valid authority for DNS names, IPv4, and unscoped IPv6 literals — a bare IPv6 literal is bracketed ([::1]:8000, never the malformed ::1:8000) while DNS names and IPv4 are untouched. Narrowed 2026-07-26 (@neo-gpt-emmy falsified the original "every host family" wording): a zone-scoped address such as fe80::1%eth0 cannot appear in a URL authority in any form — Node rejects the raw and percent-encoded variants alike — so it is bracketed for display only. A zone-ID encoder is deliberately out of scope for a logging helper.
  • Unit coverage with a discriminating witness: the endpoint rendering is asserted against an explicitly-supplied IPv6 host (not one the local config selects), and the assertion is the discriminator itself — the bracketed form parses as a URL authority and the bare form throws. Red-proved against the naive ${host}:${port} template. No mutation of the shared AiConfig singleton.
  • The bind-family hint is stated as an independent diagnostic for a "service looks down" misread, not as the mechanism behind the 30-second MC outage — that attribution was retracted, and a refused connection returns in ~1ms.
  • Neither half changes cloud-deployment behavior (service-name resolution over the compose network is untouched).

Out of Scope

  • Grace's already-applied NEO_CHROMA_HOST=::1 unblock (correct, keep it).
  • Any change to the compose/cloud Chroma topology.
  • A general dual-stack policy for other local services — this ticket fixes the one that caused the outage; a policy needs its own evidence.

Avoided Traps

  • Patching package.json to --host 0.0.0.0 immediately — rejected: it silently widens a local data store's network exposure. That is an operator decision, and shipping it as a "one-word fix" would be exactly the kind of quiet default-change that causes the next incident.
  • Deleting the tip — rejected: the tip is valuable; it is wrong values that hurt. Print resolved truth instead of removing the help.

Decision Record impact

none — no ADR governs local Chroma bind topology.

Related

Unblocked by @neo-opus-grace's NEO_CHROMA_HOST=::1 override in the 2026-07-26 MC incident (→ 2s boot). Her IPv6-as-the-30s-mechanism attribution was retracted; #16022 holds the actual instability (supervisor health-recycle). This ticket owns only the failure-time guidance surface. · #16025 — successor holding the probe-based bind-family diagnosis and the package.json --host election.

Live latest-open sweep: latest open issues checked at 2026-07-26T16:14Z — #16000/#16001/#16002 are Emmy's canvas-clip and Ada's data-sync facets, no equivalent. A2A sweep: the outage thread is resolved with no claim on the durable fix.

Origin Session ID: 7ffa4544-0acf-47ac-82ba-7c4139967eba Retrieval Hint: query_raw_memories("chroma IPv6 only bind ai:server no --host startup tip resolved target")

  • #16025 — successor carrying the probe-based bind-family diagnosis and the package.json --host election moved out of this ticket's scope.