Isolate live-remote-AI tests: explicit opt-in, not skip(!KEY)

Context

The 2026-06-08 €50 Gemini incident surfaced a latent test hazard: QA.spec.mjs + Librarian.spec.mjs gate on test.skip(!process.env.GEMINI_API_KEY) and run the real agent cognitive loop — so a present real key (in a dev .env) makes them run live against the billed API on every test pass. The safety is inverted.

The Problem

• QA.spec.mjs:58 / Librarian.spec.mjs:44 — test.skip(!GEMINI_API_KEY) → a present key means run live, not skip.
• They invoke the real loop (Librarian.spec:77 primaryAgent.loop.processEvent); ai/agent/Loop.mjs:357 fires up to 10 provider.generate calls/event + a retry path → each test = many billed calls.
• No test fakes/isolates GEMINI_API_KEY; with modelProvider defaulting 'gemini', the live calls hit Flash.
• CI sets NEO_TEST_SKIP_CI=true (CI safe) — but dev-local is exposed every run.

The Architectural Reality

• The skip(!KEY) gate (QA / Librarian specs); ai/agent/Loop.mjs (the agentic loop); the absent local/mock isolation.
• ADR 0019 B4: tests isolate by construction (UNIT_TEST_MODE → local/mock), never depend on env-key presence to gate billed calls. Sibling: #12435 (aiConfig-singleton test mutation — its snapshot/restore framing is superseded by by-construction).

The Fix

• Invert the gate — live-remote-AI tests require an explicit opt-in (NEO_RUN_LIVE_AI_TESTS=1), never "run because a key exists."
• By-construction local/mock — force modelProvider → a local/mock provider in the unit config so the agent-loop tests resolve local, not remote (ADR 0019; not runtime singleton mutation).
• Optionally bound loop iterations in test mode.

Decision Record impact

aligned-with ADR 0019 (by-construction isolation; B4). Relates #12435.

Acceptance Criteria

• QA / Librarian (+ any live-agent specs) do NOT run against a remote API merely because GEMINI_API_KEY is set — they require explicit NEO_RUN_LIVE_AI_TESTS=1.
• Default unit-test runs resolve a local/mock provider by construction (no remote billed calls) — verified.
• No runtime mutation of the shared AiConfig singleton (B4 / ADR 0019).

Out of Scope

• Runtime provider default — #12742. Backfill loop-safety — #12746. KB-ask / key hygiene — gpt's #12741 / #12744.
• The aiConfig-singleton snapshot/restore mechanics — #12435 (related; coordinate so both land by-construction, not snapshot/restore).

Avoided Traps

• skip(!KEY) — inverts safety (present key = live billed). Use explicit opt-in.
• Snapshot/restore (#12435's stale title) — restore() is itself a B4 singleton mutation; isolate by construction.

Related

• #12740 (epic parent). #12435 (aiConfig test-isolation sibling). #12742 (provider default). #12746 (backfill loop-safety). €50 Gemini cost incident, 2026-06-08.

Live latest-open sweep: checked latest 10 open + targeted search at 2026-06-08T08:10Z; no live-AI-test-isolation dup; #12435 is the related aiConfig-singleton sibling (distinct).

Origin Session ID: 1cf2ad9f-28af-4242-9069-fd14c04e1b62 Retrieval Hint: query_raw_memories("live agent test skip GEMINI_API_KEY inverts safety opt-in by-construction")

Authored by Claude Opus 4.8 (Claude Code), @neo-opus-grace.