Reconcile Gemini incident spend with backfill volume and token math

Context

The 2026-06-08 Gemini incident has a verified call-source/timing lead: the GitHub checkout orchestrator repeatedly launched memory miniSummary backfill (pending-memory-minisummary:50) during the cost-spike window. The remaining question is quantitative: does the observed/logged backfill volume plausibly explain the reported ~€50 charge, or did additional surfaces amplify the spend?

Operator challenge that triggered this ticket: Memory Core has roughly 15k memories. A chunked backfill over all of them could produce roughly 15k Flash model calls, but token pricing for tweet-sized summaries is usually not high enough to assume €50 without math. That is exactly the V-B-A gap.

V-B-A evidence already collected:

• /Users/Shared/github/neomjs/neo/.neo-ai-data/orchestrator-daemon/orchestrator.log shows memory miniSummary backfill (pending-memory-minisummary:50) starting at 2026-06-08T06:43:24Z and repeated through the sampled incident window.
• A day-window parse of that same log for 2026-06-08 found 73 backfill starts and 72 completions, with pending-memory-minisummary:50 every time: at most 3,650 model-call attempts represented by that single orchestrator log for the UTC day up to the captured tail.
• Google AI Developer API pricing page, checked 2026-06-08, prices Gemini Flash-family text calls per 1M input/output tokens and notes output pricing includes thinking tokens. Exact incident SKU must still come from Cloud Billing export/service metrics; do not infer it from the repo model string alone.
• MemoryService.buildMiniSummary() prompt shape is one prompt containing User: <prompt> and Agent: <response>, so per-call cost depends on actual stored prompt/response size, not just the target 280-character output.

Live latest-open sweep: checked latest 60 open issues at 2026-06-08T07:57:30Z via live GitHub and GitHub Workflow MCP. Targeted live search for Gemini cost spike billing token math returned no equivalent open issue. Related but non-equivalent: #12713 investigates why a miniSummary call hung; #12740 is the incident epic; #12741 handles KB ask provider routing; #12737 handles degraded KB refs on synthesis failure.

Semantic duplicate sweep: used non-synthesis query_documents(type='ticket') because ask_knowledge_base synthesis is part of the cost-safety incident surface. Exact corpus sweep found no equivalent billing/token reconciliation ticket.

The Problem

The team has a strong source/timing diagnosis, but the charge magnitude still needs falsifiable accounting.

A plausible incident reconstruction must reconcile all of these quantities:

• Cloud Billing / service metrics request count for the deleted direct API key.
• Cloud Billing SKU/model/tier actually charged (gemini-3.5-flash repo string may not map 1:1 to the public pricing page label).
• Logged orchestrator backfill starts and pending counts across all running checkouts, not just the GitHub checkout log.
• Actual input/output/thinking token counts per miniSummary call, including large stored prompts/responses and hidden thinking-token output if charged.
• Retry behavior in the Google SDK / provider path, including whether 429s are billable, retried, or merely counted in request volume.
• Other inherited-key call surfaces during the same window, especially KB ask synthesis and live-AI tests.

Without this reconciliation, we risk two wrong fixes: under-fixing because "it was only 15k cheap calls," or over-fitting to miniSummary backfill while another remote-AI surface contributed the majority of spend.

The Architectural Reality

• ai/services/memory-core/MemoryService.mjs:709 builds miniSummary prompts from stored turn prompt/response content and routes through buildChatModel().
• ai/daemons/orchestrator/scheduling/memorySummaryBackfill.mjs:39 schedules a backfill whenever pending rows exist, but the log only proves scheduled batches for a given checkout's orchestrator.
• Multiple local clones/harnesses can exist simultaneously, each with its own logs and inherited environment.
• Cloud Billing is the source of truth for spend amount, SKU, region, and key attribution; local logs are source of truth for repo-side scheduling and candidate call surfaces.
• Public pricing is only a sanity-check input; the actual billed SKU can differ by model, region, tier, and Vertex vs Developer API route.

The Fix

Create a reproducible incident cost ledger that reconciles billing, logs, and source-level prompt sizes.

Expected investigation shape:

• Export or manually capture Cloud Billing / service metrics for the incident key over the spike window: request count, SKU/model, region, token counts if available, and cost.
• Parse orchestrator logs across all local checkouts for backfill starts/completions and pending counts.
• Sample actual memory prompt/response character lengths from the same Memory Core collection, convert to a bounded token estimate, and compute low/base/high spend bands.
• Inspect provider/SDK retry behavior and whether retries can multiply billable calls.
• Attribute residual spend to additional surfaces if backfill math cannot explain the bill.
• Post the reconciliation table on this ticket before closing it.

Contract Ledger Matrix

Decision Record impact

none. This is incident forensics and cost-accounting; it may inform later ADR/provider-default changes but does not itself change architecture.

Acceptance Criteria

• Cloud Billing / service metrics for the incident key and spike window are summarized: request count, region, model/SKU, and cost.
• Backfill starts/completions are counted across all relevant local checkout orchestrator logs, not just one checkout.
• Actual Memory Core prompt/response size distribution is sampled without exposing private memory content.
• Low/base/high token-cost estimates are computed for miniSummary backfill and compared with the reported ~€50.
• Retry/429 behavior is classified as billable, non-billable, or unknown with evidence.
• Any residual spend is either attributed to verified sibling surfaces or explicitly left as unresolved with next probes.
• Final comment states whether miniSummary backfill alone is sufficient, insufficient, or sufficient only under specified assumptions.

Out of Scope

• Implementing provider-default changes (#12740 sibling leaves).
• Fixing KB ask provider routing (#12741).
• Root-causing the miniSummary hang itself (#12713).
• Publishing raw secrets, API keys, or private memory content.

Avoided Traps

• Equating "15k memories" with "15k billable calls" without checking logs and retry behavior.
• Equating "logged backfill starts" with exact billable calls; a batch can defer, fail, retry, or run in multiple clones.
• Using public pricing as the final answer when Cloud Billing SKU data is available.
• Copying private memory prompt/response text into public issue comments.

Related

Parent: #12740 Related: #12713, #12737, #12741, #12673, #12676 Official pricing sanity-check source: https://ai.google.dev/gemini-api/docs/pricing

Origin Session ID: e8f07ef9-ef7e-4815-8ff4-7abe13720621 Retrieval Hint: "Gemini incident cost reconciliation miniSummary backfill token math billing SKU request count"