Sub 8: Benchmarking gemma4 + context-window reuse research

Parent Epic

#12065 — Sub 8 of 9. NEW per operator-direct addition 2026-05-27 ~01:36Z post-STEP_BACK. Parallel to Sub 1.

Premise

Operator: "we need benchmarking for gemma4 => creating context windows is the most expensive task. if there is any way to REUSE created context windows => big win."

Each SemanticGraphExtractor.executeTriVectorExtraction invocation today opens a fresh gemma4 context for the LLM call (the OpenAI-compat surface via Ollama). With session payloads up to 256K tokens + 10 sessions per REM cycle, the cumulative context-creation cost may dominate the actual inference cost. If Ollama / OpenAI-compat backend supports context-cache reuse (KV-cache persistence across calls within the same session OR shared system-prompt cache across calls), we could reduce per-cycle wall-clock from MINUTES × N → MINUTES + tiny-deltas.

Prescription

Two parts:

Part A — Benchmarking baseline:

• Measure wall-clock + token-count per phase of executeTriVectorExtraction:
  • Time to first token (TTFT) — proxy for context-creation cost
  • Time to last token (TTLT) — proxy for inference cost
  • Tokens-per-second during generation
• Capture for sessions of different sizes: small (~5K tokens), medium (~30K), large (~100K), max-band (~200K)
• Run via ai:run-sandman against fresh + warmed states; isolate the per-call context-creation overhead
• Document baseline in learn/agentos/gemma4-rem-benchmark.md or similar

Part B — Context-window-reuse research:

• Investigate Ollama API: does keep_alive parameter enable context reuse across calls? (Ollama docs: keep_alive controls how long the model stays in memory after a request; affects KV-cache retention)
• Investigate OpenAI-compat API: any reuse semantics in the proxy surface?
• Test empirically: 2 back-to-back Tri-Vector calls — does TTFT drop on call 2?
• Document findings: if reuse works, propose Sub 7 chunking enhancement to keep chunks under one keep_alive window
• If reuse NOT possible at Ollama layer, propose alternative: longer-running gemma server process owning the context

Acceptance Criteria

• AC1: Benchmark suite lands (script at ai/scripts/benchmark/gemma4-rem-benchmark.mjs or equivalent) — measures TTFT/TTLT/tps per session size
• AC2: Baseline measurement document published at learn/agentos/gemma4-rem-benchmark.md — captures pre-optimization numbers per the 4 size buckets
• AC3: Ollama keep_alive + KV-cache reuse investigation documented — empirical V-B-A of context-reuse capability
• AC4: If reuse works: integration recommendation for Sub 3 (executeRemCycle configures keep_alive for batch processing) + Sub 7 (chunk batching within one keep-alive window)
• AC5: If reuse doesn't work at Ollama layer: alternative documented (e.g., longer-running gemma server lifecycle owned by orchestrator)
• AC6: Cost-asymmetry quantification: fresh-context vs reused-context per invocation (expected: 5-20× speedup if reuse works)
• AC7: Benchmark integration with Sub 7's chunking strategy — verify 2-pass + reduce cost-asymmetry is acceptable (2-3× single-pass cost, NOT 10-20×)

Avoided Traps

• ❌ Skip baseline measurement before proposing optimization — operator-explicit "benchmarking" framing requires empirical numbers, not theoretical claims
• ❌ Assume KV-cache reuse works without V-B-A — Ollama documentation is ambiguous; needs empirical test
• ❌ Optimize for gemma4 specifically without abstraction — keep findings provider-agnostic where possible (operator may switch to gemma-5 / qwen / etc.)

Related

• Epic #12065
• Operator directive 2026-05-27 ~01:36Z: "we need benchmarking for gemma4 => creating context windows is the most expensive task."
• Discussion #12062 §2.4.1 (cap-raise hot-fix enables benchmarking with realistic payload sizes)
• Sub 3 (consumer integration if reuse works), Sub 7 (chunk-batching enhancement)
• ADR 0014 cloud-deployment-topology (cloud deployments use different gemma server; reuse findings may differ)