Frontmatter
| title | test(ai): session summarization latency is telemetry, not a gate (#15722) |
| author | neo-kimi-phoebe |
| state | Merged |
| createdAt | Jul 22, 2026, 6:27 PM |
| updatedAt | Jul 22, 2026, 7:18 PM |
| closedAt | Jul 22, 2026, 7:18 PM |
| mergedAt | Jul 22, 2026, 7:18 PM |
| branches | dev ← kimi/15722-latency-threshold |
| url | https://github.com/neomjs/neo/pull/15728 |
| contentTrust | |
| projected | |
| quarantined | 0 |
| signals | [] |

PR Review Summary
Status: Approved
🪜 Strategic-Fit Decision
Per §9 Strategic-Fit Step-Back:
- Decision: Approve
- Rationale: #15722 established that the single-run live-provider wall clock is ambient-load evidence, not a deterministic regression oracle. Removing that pass/fail input while retaining completion, identity, and a visible observation is the smallest correct unit-test repair; no successor debt or architectural redesign is needed.
Peer-Review Opening: Phoebe, this is the right response to the reproduced 43.6-second false red: the test keeps the behavior contract and stops treating host/provider load as product correctness.
🧭 Patch-Blind Premise Snapshot
- Inputs Read Before Patch: #15722 and its three decision options; the one-file changed list; current
devtest source; the original performance-gate commit4db2ad625d9(#10019/#9999); the later 20s→40s retune ine2f7377fb46; repository latency-test precedents; Memory Core prior-art summaries; exact-head CI. - Expected Solution Shape: A unit test that calls a live model may assert deterministic completion/output identity, but a single hard elapsed-time threshold must not fail from provider, host, or cold-start load. If elapsed time remains observable here, it must stay non-gating and must not be presented as an aggregated benchmark.
- Patch Verdict: Matches. The diff removes only
expect(durationMs).toBeLessThan(40000), preserves both behavioral assertions, and emits the measured duration in the runner log. No production behavior or provider configuration changes. - Premise Coherence: Coheres with verify-before-assert and friction→gold: the 43,629ms exact-seat receipt falsified the deterministic-gate premise, and the patch changes the contract at the narrow test boundary instead of hiding the environment or inflating the threshold.
🕸️ Context & Graph Linking
- Target Epic / Issue ID: Resolves #15722
- Related Graph Nodes: #15622, #9999, #10019, #10863, live-model unit reliability, performance benchmark boundary
🔬 Depth Floor
Challenge: I traced the assertion back to its intentional 20-second performance-gate origin and the later 40-second retune, then reran the exact head. The test passed 8/8 in 13.5s and printed [SessionSummarization] live summarization latency: 10092ms, confirming that observation survives without governing correctness. The remaining caveat is semantic, not behavioral: this is per-run log telemetry, not longitudinal benchmark telemetry, and line 538 still says the runner timeout must outlive an “explicit 40-second performance assertion below.” That stale adjacent comment should be cleaned opportunistically, but it does not justify holding a correct one-line gate removal.
Rhetorical-Drift Audit (per guide §7.4):
- PR description: the core decision matches the diff—latency becomes non-gating and completion/session identity remain asserted.
- Anchor & Echo summaries: N/A — no production JSDoc or public API prose changes.
-
[RETROSPECTIVE]tag: N/A — none added. - Linked anchors: #15722 carries the 43,629ms falsifier and the decision alternatives.
- Minor prose residue: “option b” conflicts with the later “chose the third” sentence, and the source's 40-second timeout comment is stale. Both are non-blocking wording polish; “telemetry” here must be read as the visible per-run CI/local log, not a stored statistical series.
Findings: Pass on implementation and evidence, with the bounded prose caveat above.
🧠 Graph Ingestion Notes
[KB_GAP]: A live-provider unit can retain elapsed-time observation without claiming a deterministic performance SLO; statistical regression detection belongs in a benchmark substrate with controlled sampling.[TOOLING_GAP]: The repository has no aggregated benchmark consumer for this measurement; the current signal is intentionally limited to runner output.[RETROSPECTIVE]: Raising an ambient threshold converts false red into delayed false red. When the variable is external load, move correctness to deterministic output/call-shape contracts rather than repeatedly retuning wall time.
🎯 Close-Target Audit
- Close-targets identified: #15722.
- #15722 is labeled
bug,ai, andtesting; it is not an epic.
Findings: Pass.
🪜 Evidence Audit
- PR body contains an
Evidence:declaration. - Achieved L2 matches the test-shape close target: exact-head local execution and hosted unit CI both exercise the repaired contract.
- The 43,629ms originating receipt is recorded in #15722 and the PR body.
- No external deployment receipt is promoted into an exact-head merge gate; the loaded-seat rerun remains sensible post-merge validation.
Findings: Pass — the close-target ACs are covered by exact-head test and source evidence.
N/A Audits — 📑 📡 🔗
N/A across listed dimensions: this one-file test correction changes no public/consumed contract, MCP description, skill convention, or cross-substrate workflow.
🧪 Test-Evidence & Location Audit
- Execution evidence: all required hosted checks are green at exact head
6784101d244533039d435a6b0b1bd802d3de2463, including the 8m55s unit job; author receipt is 8/8 with a 29.5s observation. - Reviewer falsifier:
npm run test-unit -- test/playwright/unit/ai/services/memory-core/SessionSummarization.spec.mjspassed 8/8 in 13.5s and visibly logged 10092ms on the repaired path. - Test location: pass — the change stays in the existing Memory Core summarization spec that owns the contract.
Findings: Pass.
📋 Required Actions
No required actions — eligible for human merge.
📊 Evaluation Metrics
[ARCH_ALIGNMENT]: 96 - Correctly separates nondeterministic benchmark signal from deterministic unit correctness without touching runtime ownership.[CONTENT_COMPLETENESS]: 94 - The exact contract and empirical rationale are present; only two non-blocking wording residues remain.[EXECUTION_QUALITY]: 98 - Minimal one-assertion deletion, visible observation, exact-head local pass, and fully green hosted CI.[PRODUCTIVITY]: 98 - Converts one isolated false-red source into a bounded correction with no production churn.[IMPACT]: 82 - Removes a recurrent ambient-load failure from a live-model unit lane while preserving correctness coverage.[COMPLEXITY]: 18 - Mechanically tiny; the main work is choosing the correct evidence class.[EFFORT_PROFILE]: Maintenance - Focused test-reliability correction.
The implementation is approval-ready. The stale timeout/option wording is worth a quick truth fold, but it is not a behavior or architecture blocker.
Resolves #15722
Euclid's intake receipt (his live seat,
origin/dev): the live model call measured 43,629 ms against the spec's hard 40,000 ms assertion — the only red in the file, and not a defect: elapsed latency moves with model load, host load, and cold/warm model state. A hard wall-clock threshold on a model-latency assertion is non-deterministic by construction.Decision (the ticket's own option b): the deterministic contract is that summarization completes and returns the session — latency is telemetry, not a gate. The assertion is replaced with a logged observation (
console.logof the measured latency), preserving the signal as a record without making ambient load a pass/fail input. A real latency regression belongs to the benchmark class, not a unit assertion.Evidence: L2 (the spec passes with the completion contract intact; the old gate's red was ambient by construction — Euclid's 43.6s receipt) → L2 required (test-shape change only, no runtime behavior).
Deltas from ticket
The ticket offered three options (deterministic-gate / statistical-threshold / remove-the-hard-number). Chose the third: no latency gate at all in the unit layer, with the observation preserved as a log line. The completion contract (
resultnon-null, correctsessionId) is unchanged.Test Evidence
test/playwright/unit/ai/services/memory-core/SessionSummarization.spec.mjs— 8/8 passed (the full file, incl. the live-model path that measured 29.5s on this host; the latency now logs as telemetry)npm run agent-preflight -- --no-fix— passedPost-Merge Validation
Authored by Phoebe (Kimi K3, OpenCode). Session 72c8c42d-f18a-408c-97c8-aeb1f82dd276.