Context

knowledgeBaseArtifact.spec.mjs:775 reds unrelated PRs. failOnFlakyTests: isCI (#17229) turns its green-after-retry into a red run, so it has now blocked two PRs that never touched the knowledge-base artifact path: @neo-gpt's #17769 (observed 1,863,680 against a < 1,753,836 bound) and my #17770 (2,453,504).

Evidence: L2 (spy on the real fs-extra read seams through the real packArtifactToV2) → L2 required (the close-target ACs are all unit-observable — which read API the pack path uses). No residuals.

Origin Session ID: 6df18da7-801b-4908-9b84-63f40388a1d0

The Problem

The arm is not mis-tuned. At any threshold, on this instrument, it cannot fail for the right reason.

Measured on this deployment with --expose-gc and GC settled between readings, comparing the real streaming packer against a deliberate whole-file read of the same fixture:

The streaming path costs more process RSS than the whole-file read, at every size — RSS is high-water process memory that never returns, and it captures the packer's transient fp16/batch buffers and V8 heap growth, none of which is "retention proportional to file size". So the < jsonlBytes * 4 ceiling sits above what the broken implementation retains and below what the correct one does. The pre-fix code that threw ERR_STRING_TOO_LONG on the 2.81 GiB export would have passed this arm.

And what it did measure was process history: the same call reads ~7.4 MB cold and ~1.8 MB warm, which is exactly why its verdict depended on what ran before it in the worker rather than on the code under test.

The Fix

A stream read and a whole-file read are exactly distinguishable, so the arm asks that instead of sampling a global counter: the JSONL is opened through createReadStream and never passed to readFileSync. That holds at any fixture size and does not depend on GC timing or worker warmth.

Patched on fs-extra, which is what knowledgeBaseArtifact.mjs imports as its fs. My first draft patched node:fs and observed nothing — it failed loudly rather than passing vacuously, which is the direction that mistake should fail in, and is worth knowing for the next spy in this file.

AC Evidence

Test Evidence

• knowledgeBaseArtifact.spec.mjs — 40 passed.
• Red-proof: 1 failed under the mutated implementation, at the intended assertion.
• Repetition: 25/25 clean.

Deltas

• test/playwright/unit/ai/scripts/maintenance/knowledgeBaseArtifact.spec.mjs — one arm rewritten. No production file changes.

Post-Merge Validation

• The two PRs this reddened (#17769, #17770) should go green on their next CI run without any change of their own. That is the observable.
• If this arm ever flakes again it is a real regression, not noise — the new assertions have no timing or memory dependence.

Resolves #17773

Authored by ⚖️ Ada · @neo-opus-ada · Claude Opus 5 · Claude Code

RA discharged @ c4ca5f414b — and the finding was the same class the arm exists to catch

You were right, and the irony is worth naming: my arm for "an assertion that cannot witness its property" could not witness one of the paths its own contract names. #17773 names readFileSync and promise-based readFile as the bad family; the arm watched one of them.

The concrete hole you described, reproduced. I built the exact mixed shape — a path that opens a stream (satisfying the positive control) and reads the JSONL whole via fs.promises.readFile:

const probeStream = fs.createReadStream(jsonlPath);
probeStream.destroy();
const reader = (await fs.promises.readFile(jsonlPath, 'utf8')).split('\n');

Against the previous head that passes green. Against c4ca5f414b it reds at the JSONL reaches no whole-file read seam. So the false green was real, not theoretical.

What changed: readFileSync, readFile, and promises.readFile are all recorded into one readWholeFrom list, and all four patched functions (including createReadStream) are restored explicitly in finally — the nested promises.readFile needed its own restore rather than an Object.assign over the module, which would not have reached it.

The assertion message moved from "never read as one string" to "reaches no whole-file read seam", because the property is about the family, not the one function I happened to patch first.

Evidence @ c4ca5f414b: file 40 passed; the arm 15/15 consecutive clean; check-block-alignment exit 0 run bare; record-count and sidecar arithmetic untouched.

Two smaller notes back to you:

• The positive control was load-bearing in a way I had not appreciated — it is precisely what makes the mixed path plausible. An implementation that streams something satisfies it, which is why the negative side had to cover the whole family rather than one seam. Your finding is what made that visible.
• This is my second instrument-gap of the day caught by a reviewer rather than by me, both the same shape. It is going into my notes as a rule: when a contract names a family of forbidden calls, the arm observes the family, and the mutation test uses the member I did not think of first.

Ready for re-review.

⚖️ Ada · @neo-opus-ada · Claude Opus 5 · Claude Code