an ask-model comparison that measures only latency would ship a model that fabricates instead of abstaining

Refs #16998

Problem

The ask model must be chosen by measurement, and the two axes are not equally easy to measure — which is how a regression gets shipped.

Latency is trivial to measure. Answer honesty is not. Asked about a concept absent from the corpus, the current model answered "I do not have enough information to answer this question" and named what it did find, rather than fabricating a plausible one. A faster model that invents an answer instead of abstaining is a regression, and nothing in a latency comparison would show it.

Measured baseline (local plane, uncontended): google/gemma-4-26b-a4b generates at 9.6 tok/sec (12,340 ms for 119 tokens, 28-token prompt); a limit: 5 ask end-to-end is 42,973 ms; limit: 3 answers in 10.2 s.

⚠️ The baseline is confounded and the harness must record it as such: the host is paging (128 GB physical, container VM configured at 48 GiB, swap 96% full). A 4-bit MLX model with 4B active parameters at 9.6 tok/sec is not evidence about model size. Any comparison run before the host allocation is settled records that condition alongside the numbers, or a later reader will attribute a memory artifact to a model.

Candidates. Gemma 4 ships five sizes — verified, not recalled: E2B (5.1B total / 2.3B active), E4B (8B), 12B dense, 26B A4B (26B total / 4B active MoE), 31B dense; 256K context. 31B is excluded by operator decision. The family is not a constraint — non-Gemma candidates are in scope and should come from research rather than habit.

⚠️ A concurrency cohort run today measures ADMISSION, not the model — do not attribute it to the model. Ask synthesis is serialized in-process: SearchService.mjs:176 → buildChatModel → chatRequestQueue.enqueue (buildChatModel.mjs:160) → the process-wide sharedLocalChatRequestQueue, documented as "a single-lane request scheduler that runs async tasks one at a time" with no concurrency option. So two overlapping asks serialize regardless of endpoint or replica count, and the second's latency lands near 2× the first. That signature is the queue admitting one, not a model degrading under load — the harness must record queue wait separately from generation time so the two can never be confused. #17000 owns the fix.

Acceptance criteria

• A repeatable harness measures, per candidate model, on a fixed query set: end-to-end latency, generation rate, and prompt size.
• The same harness scores abstention-honesty: for queries whose answer is absent from the corpus, an abstention scores above a fabricated answer, and the scoring is inspectable rather than a single opaque number.
• The query set includes at least one known-absent concept and one known-present concept, so honesty and capability are separable.
• Every recorded run captures host conditions (swap state, VM allocation, models resident) beside the numbers.
• A baseline for the incumbent model is recorded, so any candidate is compared against a measurement rather than a memory.
• The harness reports without selecting — the model decision is the parent lane's, informed by this output.
• Runs execute against the bounded context contract (#16999) and the dedicated ask endpoint (#17000) once those land, so a candidate's numbers are not confounded by an unbounded prompt or a contended endpoint. A run predating either records which one was absent.
• A bounded concurrency cohort — at minimum a single ask and two overlapping asks — separates queue wait from generation time, plus throughput and resident-memory conditions. A cohort reporting only end-to-end latency cannot distinguish serialized admission from a slow model, which is the one confusion this cohort exists to prevent.
• The separation is a ledger READ, not new instrumentation — it is an existing SELECT (@neo-opus-ada, verified at source; this corrects my original wording, which implied work that does not need doing). providerActivityLedger.mjs:357-360 already returns avg/max_queue_wait_ms and avg/max_execution_ms in one aggregate row, already grouped by priority; :446-449 is the live waiting/executing form. Supporting chain: :147 computes the wait, :120-121 schema, :258 defaults queueDisposition to 'neo-queued' and the ask path never overrides it, :416 reads it back. The cohort reads existing rows and instruments nothing, so it cannot perturb what it measures.
• Priority is held constant across the cohort. InteractiveBatchQueue.enqueue(task, priority='interactive', …) prefers any waiting interactive task over batch, so a cohort mixing priorities measures the lane selector rather than admission. Ask dispatches at interactive (SearchService.mjs:553).
• The output names an initial safe parallelism envelope for today's deployment alongside the model recommendation, and states that it is sized to current RAM and dev-agent count — not a promise of enterprise scale or 75 simultaneous requests.

Contract Ledger

Consumed from elsewhere, not owned here

• A bounded context (#16999) and the dedicated endpoint (#17000) — this leaf measures through them; it does not build either. Both are independently deliverable and neither blocks harness authoring, only clean numbers.
• Host allocation — @neo-gpt-sol's lane; recorded, never changed.

Out of scope

• Picking the model (parent lane decides).
• Provider isolation and the context budget — sibling leaves.
• Fixing host memory — @neo-gpt-sol's lane; this leaf records the condition, never changes it.

Post-Merge Validation

None deferred.

Authored by @neo-opus-vega

Deltas

A benchmark harness under the KB diagnostics surface, plus a recorded incumbent baseline.

Evidence: L2 (harness arms over a fixed query set) → L2 required. No residuals.

Test Evidence

To be recorded on the implementing PR.