Agent OS local-first AI provider defaults and cost-safety

Context

On 2026-06-08, a Gemini API cost spike exposed a provider-routing and maintenance-loop safety gap in the Agent OS. Local cloud metrics showed high-volume Gemini traffic against a direct API key; cross-agent forensics then tied the onset to Memory Core mini-summary backfill activity scheduled by the orchestrator.

Verified evidence from the incident session:

• Local orchestrator logs showed memory miniSummary backfill (pending-memory-minisummary:50) starting at 2026-06-08T06:43:24Z and repeatedly launching 50-row batches in the same window as the cost spike.
• ai/config.template.mjs:127 and ai/config.template.mjs:136 default chatProvider / modelProvider to gemini when NEO_MODEL_PROVIDER is unset.
• ai/config.template.mjs:262 defaults the remote Gemini chat model name to gemini-3.5-flash.
• ai/services/memory-core/MemoryService.mjs:713 reads aiConfig.modelProvider for mini-summary generation and passes process.env.GEMINI_API_KEY into buildChatModel().
• ai/daemons/orchestrator/scheduling/memorySummaryBackfill.mjs:39 schedules a backfill task whenever any pending mini-summary row exists; it has no cooldown/no-back-to-back guard at the trigger layer.
• ai/services/knowledge-base/SearchService.mjs:54 directly constructs Gemini for KB ask synthesis when GEMINI_API_KEY is present.
• .codex/config.template.toml:38 and .codex/config.template.toml:57 pass GEMINI_API_KEY into KB and Memory Core MCP server environments by default.

Live latest-open sweep: checked latest 60 open issues at 2026-06-08T07:57:30Z via live GitHub and GitHub Workflow MCP. Closest hits were #12456 (broad AiConfig SSOT cleanup parent), #12713 (miniSummary model-call hang root cause), #12737 (degraded KB refs on synthesis failure), and #12719 (KB candidate weighting). None is the cost-safety parent for this incident. Targeted live searches for Gemini incident local first provider defaults, ask knowledge base provider configurable local capable, and GEMINI_API_KEY pass through spend cap found no equivalent open ticket.

Semantic duplicate sweep: used non-synthesis query_documents(type='ticket') for incident/provider/key-pass-through queries. The normal ask_knowledge_base semantic sweep is intentionally avoided here because KB ask synthesis is one of the incident surfaces being ticketed; the non-synthesis sweep surfaced related historical provider/default tickets (#12573, #10103, #10097) but no equivalent incident parent.

The Problem

Cheap, bounded Agent OS summarization and knowledge lookup paths can silently become remote paid Gemini workloads when a local harness or daemon process inherits GEMINI_API_KEY while NEO_MODEL_PROVIDER remains unset. That is structurally wrong for local development and for the MX loop: compressing one memory turn into a tweet-sized miniSummary should default to local inference or fail-soft fallback, with remote APIs explicitly opted in.

The incident was not just a single bad key. It exposed a cluster:

• Remote chat defaults are still reachable by omission (NEO_MODEL_PROVIDER unset).
• Maintenance scheduling can repeatedly launch backfill batches while backlog remains above the limit.
• KB ask synthesis is still Gemini-hardwired instead of reading the same provider-routing substrate as Memory Core.
• Local harness config templates pass remote keys into server processes by default.
• Live-AI tests can be gated by key presence instead of an explicit opt-in.
• Cost/spend safety is not visible in the local restart/key-clear procedure.

This needs an epic because the fixes belong to multiple leaf surfaces: provider defaults, orchestrator maintenance safety, test isolation, KB synthesis provider routing, and harness/operator cost hygiene. A single PR would either under-fix the incident or become a monolith.

Intended Solution Shape

Make Agent OS local-first for cheap synthesis by default, and make remote paid AI explicit.

The intended shape is not to remove Gemini support. Gemini remains a valid remote provider for cloud or explicit local opt-in, but local harnesses and maintenance paths must not spend remote tokens merely because an API key exists in the inherited environment. Summary-sized work should default to local provider routing or fail-soft raw fallback. Backfill work should be paced so a large backlog cannot become a back-to-back paid-call loop. KB ask synthesis should share provider routing rather than keeping a separate Gemini-only constructor.

The epic also captures operational hardening: local harness templates, restart/key-clear runbooks, and spend-cap checks need to make remote-AI exposure visible and reversible after incidents.

Architectural Reality

This work touches Agent OS infrastructure, not the application-engine Body layer:

• Tier-1 AiConfig provider leaves (ai/config.template.mjs) govern deployment-wide chat/provider defaults.
• Memory Core summary and mini-summary paths consume aiConfig.modelProvider through buildChatModel().
• Knowledge Base ask synthesis currently bypasses that routing and constructs Gemini directly.
• Orchestrator maintenance scheduling decides when lifecycle children run.
• Codex/Claude/Gemini harness config templates decide which secrets and provider selectors child MCP servers inherit.
• Existing #12456 remains the broad AiConfig SSOT cleanup parent; this epic is narrower and incident-driven.

Decision Record impact

aligned-with ADR 0019. The work should read resolved AiConfig leaves at use sites and avoid introducing parallel provider aliases unless a later ADR explicitly supersedes that constraint.

Out of Scope

• Removing Gemini as a supported remote provider.
• Replacing the entire Knowledge Base ranking stack; #12719 owns candidate weighting.
• Degraded-reference fallback for KB synthesis failure; #12737 already owns that leaf.
• Closing #12456; that parent remains blocked only by #12435 per its latest resolution review.
• Human billing-console work beyond documenting spend-cap / key-clear expectations.

Avoided Traps

• Treating the deleted key as the fix. Rotation stopped immediate spend but left the same routing hazard in the templates and defaults.
• Making one giant incident PR. The affected surfaces are separable and should resolve through linked leaf tickets.
• Retrying remote providers inside cheap summary paths as the first response. Retry can amplify quota/cost failure; local-first/default-fail-soft is the safer base.
• Adding a second summaryProvider SSOT casually. ADR 0019 requires using the resolved provider leaf unless a deliberate successor decision changes that model.

Related

• #12456 — adjacent AiConfig SSOT cleanup parent, not the incident parent.
• #12713 — miniSummary model-call hang root-cause investigation.
• #12737 — degraded KB refs when ask synthesis fails.
• #12719 — KB candidate weighting below source/docs.
• #12673 / PR #12676 — miniSummary backfill substrate that surfaced the incident class.

Origin Session ID: e8f07ef9-ef7e-4815-8ff4-7abe13720621 Retrieval Hint: "Gemini cost spike miniSummary backfill local-first provider defaults KB ask provider configurable GEMINI_API_KEY pass-through"