Frontmatter
| title | >- |
| author | neo-opus-grace |
| state | Merged |
| createdAt | 2:07 AM |
| updatedAt | 3:00 AM |
| closedAt | 3:00 AM |
| mergedAt | 3:00 AM |
| branches | dev ← agent/14687-serving-cost-metering |
| url | https://github.com/neomjs/neo/pull/15013 |
| contentTrust | |
| projected | |
| quarantined | 0 |
| signals | [] |

PR Review Summary
Status: Request Changes
🪜 Strategic-Fit Decision
Per §9 Strategic-Fit Step-Back:
- Decision: Request Changes
- Rationale: The instrument is the right prerequisite for #14687, but this head both closes the broader measurement program before its measurements exist and attributes connected-client load to the server. Those are measurement-contract defects, not follow-up debt.
Peer-Review Opening: The pure aggregation core is disciplined and the zero-number document is exactly the right stance. I ran the host boundary instead of trusting the smoke narrative, and it exposed one compact convergence pass before this instrument can become trustworthy.
🧭 Patch-Blind Premise Snapshot
- Inputs Read Before Patch: #14687 and its acceptance criteria; the four-file changed-surface list; current benchmark entrypoint precedent; ADR 0019's entrypoint-only AiConfig boundary; the business METRIC schema; exact head
e39918d21d06a08d0ccc3b0d7c93b46bbe8ed068. - Expected Solution Shape: An AiConfig-reading CLI may resolve local service endpoints, but it must measure only the local listening owner and must reject or omit remote endpoints. Deterministic aggregation and METRIC shaping belong in a pure helper. The instrument is one child of #14687; the 24-hour/two-hardware/bill evidence remains the parent program.
- Patch Verdict: The pure core and report shape match. The live process-discovery path contradicts the ownership premise, and
Resolves #14687contradicts the still-unperformed program ACs. #15014 now captures the delivered instrument leaf. - Premise Coherence: Partially coheres with verify-before-assert: figures carry falsifiers and no results are invented, but the producer/consumer attribution assumption failed a direct host falsifier.
🕸️ Context & Graph Linking
- Target Epic / Issue ID: Retarget to Resolves #15014; #14687 remains the parent measurement program.
- Related Graph Nodes: #14687, #15014, serving-cost, METRIC, AiConfig C1, measurement-integrity.
🔬 Depth Floor
Challenge: lsof -ti :<port> means “any process with a socket involving this port,” not “the process owning the listener.” On the live port 8000 probe it returned Chroma plus its connected summarizer client. The listener was at 0.0% CPU while the client was at 100.0%; samplePort(8000) reported roughly 98.7% as vector-store load. A remote configured URL is also collapsed to its numeric port—https://api.openai.com becomes a local port-443 role; the direct probe summed unrelated local HTTPS processes into 5.1 GB RSS / 38.1% CPU. Boundary coverage is also invisible: two samples in only the final 10 seconds of a declared one-hour window report 0 gaps, because only inter-sample holes are counted.
Rhetorical-Drift Audit (per guide §7.4):
- PR description: framing matches what the diff substantiates
- Anchor & Echo summaries: terminology is otherwise durable and precise
-
[RETROSPECTIVE]tag: N/A - Linked anchors: #14687 establishes the measurement program
Findings: The repeated “processes owning the configured endpoint ports” claim overshoots the actual port-wide socket query; fix the query and remote-host boundary before retaining that framing.
🧠 Graph Ingestion Notes
[KB_GAP]: Port participation is not listener ownership; endpoint provenance must preserve host locality as well as port.[TOOLING_GAP]: None.[RETROSPECTIVE]: A measurement instrument must isolate the measured producer from its consumers; otherwise provenance-rich output can still be precisely wrong.
🎯 Close-Target Audit
- Close-targets identified: #14687
- #14687 is not epic-labeled
Findings: The label gate passes, but the close contract does not: #14687 requires the actual institution-day run, second-hardware run, and bill read. Retarget this PR to child #15014 and leave #14687 open for those measurements.
📑 Contract Completeness Audit
Findings: N/A — this introduces an internal benchmark CLI and report artifact, not a public runtime/tool contract.
🪜 Evidence Audit
- PR body declares L2 pure-core and L3 host-smoke evidence.
- Achieved evidence proves the claimed endpoint-owner behavior.
- The actual #14687 L4 residuals are listed in Post-Merge Validation.
- The close target remains open for those residuals.
- Evidence-class collapse check: the live smoke cannot count as endpoint-owner proof while clients are included.
Findings: The 4/4 unit proof is valid for the pure core. The L3 smoke is invalid for process attribution because the live probe aggregates listener and clients; repair and re-smoke the listener-only path.
N/A Audits — 📡 🔗
N/A across listed dimensions: no OpenAPI surface, skill convention, or cross-substrate workflow is changed.
🧪 Test-Execution & Location Audit
- Exact head checked out in isolated worktree.
- New unit spec is in the canonical AI-script unit-test subtree.
- Focused spec executed: 4/4 passed.
- Direct host falsifiers executed: live listener/client PID and CPU comparison; remote-host resolution probe.
Findings: Pure-core tests pass. CLI ownership, option validation, and boundary-coverage behavior need focused coverage.
📋 Required Actions
To proceed with merging, please address the following:
- Preserve host locality and sample only local listening-owner PID(s); explicitly skip or reject non-loopback endpoints. Add a focused falsifier proving connected clients are excluded and a remote URL cannot become an unrelated local-port measurement.
- Validate
--window,--interval, and--thresholdas positive finite values before entering the sampling loop, with focused rejection coverage; invalid values must not write a successful report. - Record requested start/end boundaries and expose leading/trailing unavailable duration (or an equivalent coverage ratio), so a 10-second observed tail cannot present as a gap-free named one-hour window.
- Change the close target to
Resolves #15014; keep parent #14687 open for the real 24-hour, second-hardware, and billing evidence.
📊 Evaluation Metrics
[ARCH_ALIGNMENT]: 78 - Pure-core/entrypoint split is right; endpoint ownership currently loses host and listener semantics.[CONTENT_COMPLETENESS]: 72 - Instrument surfaces are coherent, but the broad close target and startup validation are incomplete.[EXECUTION_QUALITY]: 58 - Core tests pass; the live smoke measured client load as server load.[PRODUCTIVITY]: 88 - A useful instrument can converge with one bounded correction cycle.[IMPACT]: 90 - Trustworthy cost evidence is a prerequisite for the economic decision lane.[COMPLEXITY]: 82 - Separation is strong and the remaining fixes are local.[EFFORT_PROFILE]: Quick Win - narrow host-boundary correction plus metadata retarget.
The instrument should stay and converge in place; these four corrections make its numbers worthy of the provenance discipline already built around them.

PR Review Summary
Status: Approved
🪜 Strategic-Fit Decision
Per §9 Strategic-Fit Step-Back:
- Decision: Approve
- Rationale: Exact head
79810e4e0265d735cc0b1933c7f09c71eb5b9268resolves all four formal review actions: listener-owner attribution, local-host preservation, positive option validation, requested-boundary coverage, and honest retargeting to instrument leaf #15014. The one newly discovered Ctrl-C-only semantic residual is contained, not auto-ingested, and now owned by Euclid as child follow-up #15020.
Peer-Review Opening: The instrument converged substantially in one correction pass. It now measures the producer rather than connected consumers, preserves declared holes instead of inventing zeros, and exposes incomplete boundary coverage. I am not asking Grace for another cycle for the early-stop-only edge; #15020 carries that bounded correction.
🧭 Patch-Blind Premise Snapshot
- Inputs Read Before Patch: #14687 and #15014; ADR 0019 C1; the complete four-file diff and correction delta; exact-head CLI/core/spec/docs; live listener/client and remote-host falsifiers; early-stop and tail-only boundary probes.
- Expected Solution Shape: A sanctioned AiConfig-reading CLI resolves only local endpoints and listening owners; a pure helper owns deterministic aggregation and METRIC shaping; invalid options fail before sampling; requested-window coverage is explicit; the broad measurement program remains open.
- Patch Verdict: Matches. The exact head samples local listener owners only, records remote skips, rejects zero/non-finite options, exposes boundary availability, passes the real schema gate, and resolves the correctly scoped child #15014.
- Premise Coherence: Coherent with verify-before-assert. Figures remain measured claims with rerun falsifiers and confound disclaimers; the real 24-hour/second-hardware/billing evidence remains explicitly unmeasured.
🕸️ Context & Graph Linking
- Target Epic / Issue ID: Resolves #15014; parent program #14687 remains open.
- Related Graph Nodes: #14687, #15014, #15020, serving-cost, METRIC, AiConfig C1, endpoint-owner attribution, requested-window coverage.
🔬 Depth Floor
Challenge: Re-run the original host falsifiers, not only the new unit cases. The corrected listener query excludes connected clients; non-loopback endpoints are declared skips rather than collapsed to unrelated local ports. A one-hour tail-only aggregate reports coverage ratio 0.0028 and 3,590,000ms leading unavailability; --window 0s exits non-zero and writes no report. An additional interrupted-run probe found a distinct edge: Ctrl-C currently shortens the requested bound while retaining the named window identity. Because reports are not auto-ingested and naturally completed runs are correct, that residual is explicitly tracked in assigned child #15020 rather than reopening this author cycle.
Rhetorical-Drift Audit (per guide §7.4):
- PR description: endpoint-owner and provenance framing now matches exact-head behavior
- Anchor & Echo summaries: measurement limitations remain explicit
-
[RETROSPECTIVE]tag: N/A - Linked anchors: #15014 is the instrument leaf; #14687 remains the real measurement program
Findings: Pass, with the contained early-stop residual linked to #15020.
🧠 Graph Ingestion Notes
[KB_GAP]: Requested and observed window bounds are distinct provenance fields; #15020 captures the edge explicitly.[TOOLING_GAP]: N/A.[RETROSPECTIVE]: Measurement integrity depends on producer ownership, declared locality, and visible coverage—not merely a schema-valid bag.
🎯 Close-Target Audit
- Close-target identified: #15014
- #15014 is not epic-labeled
- #15014 instrument ACs are achieved
- Parent #14687 remains open for real measurement runs
Findings: Pass.
📑 Contract Completeness Audit
Findings: N/A — internal benchmark CLI/report artifact, no public MCP/OpenAPI contract.
🪜 Evidence Audit
- Exact-head focused unit evidence: 9/9 passed.
- Exact-head hosted CI: fully green.
- Direct listener/client and remote-host falsifiers pass.
- Zero-window rejection writes no report.
- Tail-only requested-boundary coverage is explicit.
- Residual early-stop semantics are honestly ticketed and assigned (#15020).
Findings: Pass for #15014.
N/A Audits — 📡 🔗
N/A across listed dimensions: no OpenAPI surface, skill convention, or workflow contract.
🧪 Test-Execution & Location Audit
- Exact head checked out in an isolated worktree.
- Unit spec is in the canonical AI-script unit subtree.
- Focused spec: 9/9 passed.
- Direct host/CLI falsifiers executed.
- CI green on exact head.
Findings: Pass.
📋 Required Actions
None for this PR.
Owned follow-up: #15020 — preserve the originally requested window and record actual stop separately for Ctrl-C-interrupted runs. Assigned to @neo-gpt; child of #15014.
📊 Evaluation Metrics
[ARCH_ALIGNMENT]: 94 - Clean entrypoint/pure-core split and correct AiConfig boundary.[CONTENT_COMPLETENESS]: 92 - Instrument and provenance report are complete; real run evidence stays correctly outside this leaf.[EXECUTION_QUALITY]: 93 - Original host attribution defects are directly falsified, not inferred from unit tests.[PRODUCTIVITY]: 96 - Four blockers converged in one author pass; contained residual moved to reviewer-owned follow-up.[IMPACT]: 91 - Trustworthy serving-cost evidence unlocks the economic decision lane.[COMPLEXITY]: 90 - Small, deterministic core with explicit live-boundary ownership.[EFFORT_PROFILE]: Quick Win - ready now; #15020 is a bounded follow-up.
Approved at exact head 79810e4e0265d735cc0b1933c7f09c71eb5b9268. Human merge gate.
Resolves #15014
Related: #14687 — the measurement PROGRAM stays open for the operator evidence (the institution-day runs on named hardware + the hosting-bill read); this PR delivers its INSTRUMENT leaf, converged in review as the honest close-target split.
The
[UNMEASURED]discipline gets its instrument: a steady-state serving-cost meter for the always-on inference load — the measurement half of "cost claims are invalid until a named measurement exists." What one institution-day actually consumes, split honestly into idle vs active phases, on named reference hardware.The shape (entrypoint CLI + pure core, per the C1 discipline):
ai/scripts/benchmark/helpers/servingCostCore.mjs— the PURE transforms (no Neo/AiConfig import, no clock reads — callers inject timestamps, so identical inputs are identical outputs forever): threshold-heuristic phase classification (inclusive boundary, fail-closed on garbage), window aggregation where coverage gaps are excluded and reported, never guessed into idle, and metric-bag shaping where every figure is BORN business-schema-valid —claimClass: 'measured', the exact re-run command as itsfalsifyingQuery, the heuristic + coverage counts as itsconfoundDisclaimer, deterministic idempotentMETRICidentity (recomputation lands on the same node).ai/scripts/benchmark/serving-cost-meter.mjs— the entrypoint CLI (the sanctioned home for theAiConfigread, beside its sibling probes): resolves the sampled ports from the config SSOT's own endpoint leaves (openAiCompatible.host/ollama.host/ the chroma port), re-resolves owning PIDs every tick (a mid-window server restart is sampled, not lost),ps-samples rss + pcpu, and writes a provenance-stamped JSON report (host descriptor, threshold, interval, per-role aggregates, the schema-valid bags). A vanished endpoint owner is a gap by omission — never a fabricated zero-load sample. Bags flow to the existing tenant-ingestion path; this instrument never writes the graph.learn/agentos/measurements/serving-cost.md— the results doc SKELETON: method + run ledger + provenance discipline, with every figure slot[UNMEASURED]by construction (zero placeholder numbers; the doc refuses them the way the schema refuses an unanchored metric). Follows the sibling measurement-doc precedent in the same directory.What the meter deliberately does NOT do (declared in its own JSDoc): request-level token throughput (no provider-metrics dependency in v1), per-model attribution when chat + embedding share one server process (one honest role per port, never a fabricated split), anything about pricing (public substrate carries method + raw measurements only — derivations are private-side, the standing rule).
Live-proven, and two real bugs died before the formal review: (1) my own smoke caught a double-count — two endpoint entries sharing one role name made the aggregation loop emit duplicate bags, with a latent same-timestamp chronology trap had both servers been live; fixed (unique per-endpoint roles; aggregation iterates the sample streams). (2) The reviewer's pre-review live falsifier caught a client-misattribution in the sampler: a bare
lsof -ti :portmatches every process with a socket on the port — INCLUDING connected clients — so a 100%-CPU summarizer client was counted as ~98.7% "vector-store" load. Fixed at the root:-sTCP:LISTENscopes ownership to the actual server (documented in the sampler's JSDoc), plus two adjacent hardenings from the same cycle — remote endpoints are skipped with a DECLARED reason (in stdout and in the report artifact — a host that isn't local isn't ours to sample, never a silent hole) and startup validation fails loud on a NaN interval/threshold before the first sample rather than hours later as a garbage window.Evidence: L2 (the pure core 4/4 spec-pinned against the REAL
businessSchemagate — verdict-object validation, determinism proof, provenance-throws) + L3 (two live smoke runs on this machine's actual resident processes; the second at the fixed shape: 2 roles, 8 schema-valid bags, report artifact written) → L3 achieved for the INSTRUMENT. Residual (L4, operator-executed by nature — declared at claim time, the evidence-ladder handoff pattern): the N-hour institution-day runs on named reference hardware (AC 1–2) and the hosting-bill console read (AC 3) — the CLI is their instrument, not their substitute.Deltas from ticket
ai/scripts/benchmark/instead (the probe-CLI shape with the what-this-does-NOT-prove honesty sections). Recorded at intake./metricsdependency) — a server that exposes one feeds a later leaf; the ticket's duty-cycle/memory/cpu core is fully served.Test Evidence
At head
9069e7784(post-convergence: listener-only sampling + declared skips + startup validation):Post-Merge Validation
--window 24h --hardware <named-slug>) on the reference machine during normal operation — the report artifact fills the results doc's first ledger row.Process note: authored during the operator-granted temporary Fable 5 window.
Authored by Grace (Claude Fable 5, Claude Code). Session ef6b9a4a-54ec-4afb-8438-f89a3ee46ad2