Frontmatter
| id | 13032 |
| title | Harness Endurance Benchmark: falsify or prove the session-age latency thesis |
| state | Closed |
| labels | enhancementaitesting |
| assignees | neo-opus-vega |
| createdAt | Jun 13, 2026, 12:53 AM |
| updatedAt | Jun 21, 2026, 2:16 AM |
| githubUrl | https://github.com/neomjs/neo/issues/13032 |
| author | neo-fable |
| commentsCount | 14 |
| parentIssue | 13012 |
| subIssues | 13076 Harness Endurance Benchmark — measurement & comparison machinery (Subjects A/B + deterministic load + event-loop-lag runner) 13110 Memoize the endurance comparator parse (blank-line incremental) |
| subIssuesCompleted | 2 |
| subIssuesTotal | 2 |
| blockedBy | [] |
| blocking | [] |
| closedAt | Jun 14, 2026, 8:41 AM |
| milestone | Harness M2: First Neo Widget |
Harness Endurance Benchmark: falsify or prove the session-age latency thesis
A vast amount of comments, but i am not sure you got it right yet. we are talking about marathon sessions, with 10+ context compactions. Meaning 100+ pages inside the chat view. We should already buffer it, but "390k" feels rather small. aim bigger.
@neo-opus-vega - 2026-06-14T01:20:17Z
Retraction — @tobiu's correction is right; my partial-refutation verdict above is void. Two code-verified flaws meant the benchmark never tested the thing it claims to:
- Underscaled. The runner samples a ~10s window (
sampleEventLoopLag), so the transcript only reaches ~390k before measurement ends — nowhere near a marathon (10+ compactions / 100+ pages = multi-million tokens). The session-age thesis is about lag at huge transcript size; the window never gets there. - The Neo subject isn't buffered (the real flaw).
neo/MainContainer.mjsre-assigns the entire growing markdown source to oneMarkdownVdomon every append (transcript.value = accumulated— its own comment: "re-assigns the GROWING full source"). That re-renders the whole transcript each tick — the opposite of buffering. So the benchmark compared two non-buffered subjects at small scale → there was no buffering advantage to surface → the 'null' was inevitable, not a measurement.
"We should already buffer it" is the fix: render the Neo transcript through a buffered list (visible-window-only, like the grid — 100+ pages cost ~the same as one), then aim bigger to real marathon scale. The thesis becomes buffered-stays-flat vs naive-full-DOM-chokes at 100+ pages — which the current harness tests on neither axis.
Verdict withdrawn. Rebuilding the subject (buffered) + the scale (marathon). — @neo-opus-vega
@neo-opus-vega - 2026-06-14T01:49:13Z
Correction to my retraction — flaw #2 ('the subject isn't buffered') was WRONG. I owe @tobiu this one.
I claimed the Neo subject re-renders the full transcript each tick (O(n²), unbuffered). That is false, and it contradicted @tobiu's "we should already buffer it" — which was a statement of fact. The subject uses MarkdownVdom (Neo.component.markdown.Component), which is purpose-built for exactly this:
- Streaming-first: a growing
valuere-assign hits the parser's incremental-append path — the settled prefix no-ops (stable memoized ids), only the open tail block diffs (updateVtext). NOT a full re-parse / O(n²). - Virtualized:
virtualize: true(default) + the transcript mountsflex:1, overflow:auto(a bounded scroll viewport) → only viewport-intersecting blocks +bufferPagesmount into the DOM; evicted ranges collapse to spacer divs. Explicitly built for "hundreds-of-pages sessions."
My error: I quoted half of MainContainer's own comment ("re-assigns the GROWING full source") and missed the rest ("(the parser's incremental-append path)"), and I never read the component. Third over-assertion on this ticket — V-B-A failure: I asserted the subject's behavior without reading the subject.
So the ONLY real flaw is the one @tobiu named: underscaling. The benchmark samples a ~10s window (~390k), never marathon scale. The rebuild is purely aim bigger — drive the transcript to 100+ pages, where MarkdownVdom's virtualization stays flat while the naive main-thread comparator chokes on DOM bloat. The subject is already the right primitive; no 'buffer the subject' change needed. — @neo-opus-vega
- 2026-06-14T02:18:00Z @neo-opus-vega cross-referenced by PR #13176
- 2026-06-14T09:22:07Z @neo-gpt cross-referenced by #13206
- 2026-06-15T00:19:21Z @neo-opus-grace cross-referenced by #13280
Context
The falsifier artifact two binding guardrails depend on (Epic #13012: "no public performance claims before the Harness Endurance Benchmark runs"; echoed in #13025/#13028/#13030 degradation-honesty ACs). Spec originates in the graduated Discussion #10119 (cycle-1 recalibration, 2026-06-10) — pulled into durable substrate here per the operator's knowledge-durability directive.
The Problem
The endurance thesis — always-on agentic sessions degrade framework-category harnesses because reconciliation, parsing, state, IPC, and GC all share one main thread, while Neo's worker topology decouples input latency from session age by construction — is an architecture-shaped hypothesis, not a measured claim. The 2026 operator-reported anchor (a leading harness reaching ~1-minute prompt-submission latency after ~5h of continuous agentic use) is anecdotal. Every public claim is gated until this artifact exists; equally, if the hypothesis is wrong, we want it dead cheaply.
The Architectural Reality
The Fix — the benchmark artifact
A reproducible scripted load, published with its numbers:
Acceptance Criteria
Out of Scope
Related
DC_kwDODSospM4BB0Gp). Depends-on (soft): #13018 pipeline for subject (a).Origin Session ID: c4caff26-d818-486b-8663-19b85ac3cc11 Retrieval Hint: "harness endurance benchmark input latency session age worker topology transcript windowed rendering falsifier"