LearnNewsExamplesServices
Frontmatter
id13076
titleHarness Endurance Benchmark — measurement & comparison machinery (Subjects A/B + deterministic load + event-loop-lag runner)
stateClosed
labels
enhancementaitesting
assigneesneo-opus-vega
createdAtJun 13, 2026, 9:31 AM
updatedAtJun 13, 2026, 4:26 PM
githubUrlhttps://github.com/neomjs/neo/issues/13076
authorneo-opus-vega
commentsCount0
parentIssue13032
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 13, 2026, 4:26 PM

Harness Endurance Benchmark — measurement & comparison machinery (Subjects A/B + deterministic load + event-loop-lag runner)

Closed v13.1.0/archive-v13-1-0-chunk-2 enhancementaitesting
neo-opus-vega
neo-opus-vega commented on Jun 13, 2026, 9:31 AM

Parent: #13032 · Epic: #13012

The reusable measurement & comparison machinery for the Harness Endurance Benchmark — the falsifier scaffold #13032 needs before it can render a calibrated verdict. Split out from #13032 so the machinery can land now while #13032 stays open for the load-calibration → conclusive verdict → published numbers it still tracks.

Scope (what this delivers)

  • examples/harnessEndurance/shared/LoadProfile.mjs — deterministic, seeded (mulberry32) load: zero-dep markdown-append stream + keystroke cadence, byte-identical per {seed, config} so both subjects + the runner consume the IDENTICAL sequence.
  • Subject A (examples/harnessEndurance/neo/) — the Neo transcript surface (off-thread parse/diff via Neo.component.markdown.Component), self-driving the append stream.
  • Subject B (examples/harnessEndurance/comparator/) — an honest single-main-thread vanilla comparator (full re-parse each tick), at security parity (hostile input rendered inert).
  • Runner (test/playwright/e2e/HarnessEnduranceBenchmark.spec.mjs) — drives both subjects under the same load + samples main-thread event-loop lag (per-tick re-anchored; NOT rAF, which free-runs without vsync in headless Chromium) + heap, logging the cross-subject delta.

Acceptance Criteria

  • Deterministic load generator with unit-proven reproducibility.
  • Both subject apps boot + stream the load, manually runnable standalone.
  • One-command e2e runner samples a credible main-thread-health metric for both subjects + logs the delta.
  • Metric-credibility documented (event-loop-lag rationale; rAF-headless + accumulator-drift traps caught + fixed).

Out of scope (remains on #13032)

  • Load-calibration → the conclusive cross-subject verdict (no p-hacking-to-Neo-win).
  • Published numbers + methodology on public surfaces (negative result publishes with equal prominence).
  • keystroke→echo / frame-time / task-queue-depth metric layers.

Delivered by PR #13066.