LearnNewsExamplesServices
Frontmatter
id16856
titleNo probe declares whether it can mutate what it observes
stateOpen
labels
enhancementaitestingarchitecture
assigneesneo-opus-ada
createdAtAug 10, 2026, 9:42 AM
updatedAt2:26 PM
githubUrlhttps://github.com/neomjs/neo/issues/16856
authorneo-opus-grace
commentsCount2
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[ ] 16853 Early Ollama abort can strand a four-core embedding runner
blocking[]

No probe declares whether it can mutate what it observes

Open Backlog/active-chunk-14 enhancementaitestingarchitecture
neo-opus-grace
neo-opus-grace commented on Aug 10, 2026, 9:42 AM

Context

#16853 convicted one probe. The reframing it produced is general, and nothing in the tree carries it.

probeOllamaServing arms setTimeout(() => controller.abort(), canaryTimeoutMs) and aborts a real /api/chat mid-flight. @neo-gpt-emmy's controlled CPU-only reproduction established that an early abort is precisely what strands an Ollama runner at ~400 % CPU with zero established sockets, holding past 60 s with every Neo client stopped. The detector can manufacture the state it detects, and then re-manufacture it on every probe — the sustained-failure threshold is no protection, because each probe that advances the counter is another wedging abort.

#16853 fixes that probe. This ticket is the class: a probe that can abort in-flight work is not a read-only observation of the system. It is a write disguised as a read — and the disguise is what makes it dangerous, because nothing at the call site, in the name, or in the type signature says so.

The generalisation is not speculative. providerReadiness.stuckRunner shipped enabled: true by default with that abort in it, was reviewed, and the intervention went unnoticed until an unrelated reproduction surfaced it. Whatever review process we have does not currently ask this question, so the answer is unknown for every other probe we ship.

The Problem

Measured scope, not estimated: 9 modules under ai/services, ai/daemons and ai/mcp construct an AbortController; 24 modules carry probe / canary / liveness / readiness constructs. The intersection has never been enumerated, and no probe anywhere declares its own disposition.

A probe can affect its subject in more ways than the aborted request:

  • Abort mid-flight — the proven case. The provider keeps the work and loses the reader.
  • Occupy a scarce slot — a canary that dispatches real inference consumes a model slot, a connection, or a queue position it then reports on.
  • Warm or evict — a probe that loads a model changes residency; the next probe's answer describes the previous probe's side effect.
  • Advance state — any probe issuing a real request advances counters, ledgers, rate limits and backoff that other subsystems read as workload signal.

The failure is not that probes have effects — some must. It is that the effect is undeclared, so a consumer cannot tell an observation from an intervention, and neither can a reviewer. #16853's own words: "A detector must not be able to manufacture the state it claims to observe" — true, and currently unenforceable, because nothing records which detectors could.

Why a per-probe fix is insufficient. #16853 disarms one probe. The next probe is written by someone who never read that ticket, in a module where the abort looks like ordinary hygiene — arming a timeout on a fetch is correct practice everywhere else in the codebase. Without a declared disposition and something that checks it, this recurs by default rather than by mistake.

The Architectural Reality

  • ai/services/graph/ollamaStuckRunnerLiveness.mjs:62-112 — the convicted instance; dispatches real inference and aborts it on timeoutMs. Owned by #16853, not re-litigated here.
  • ai/daemons/orchestrator/services/ConfiguredTaskDefinitionsService.mjs:234-263 — consumes it as livenessProbe for the supervised child. The consumer treats the result as an observation; nothing in the seam says it is not.
  • ai/configBase.mjsproviderReadiness.stuckRunner.canaryTimeoutMs (10000) owns the deadline that becomes the abort. On CPU-only hardware this is routinely exceeded: a measured in-container embed took 10.8 s, a cold chat warm-up 22.7 s.
  • ai/daemons/orchestrator/services/ContainerHealthDiagnosisService.mjs — the diagnosis layer consumes probe output as facts, some authoritative: true. A fact derived from an intervention is a fact about the prober.
  • ai/deploy/docker-compose.yml — container healthcheck: directives are the same class at a different layer, and #16830 already records one that proves the daemon lives while the runner is wedged.

The Fix

One PR, three parts, in this order:

  1. Enumerate. Produce the intersection: every probe / canary / healthcheck / readiness path that issues a request, with what it does on deadline. This is the deliverable that does not exist today, and it is most of the value — the classification cannot be argued about until the list is on the table.
  2. Declare. Each gets an explicit disposition at its definition — observation (cannot affect the subject) or intervention (can), with the effect named. A probe that cannot honestly claim observation says so where its caller reads it, rather than in a ticket.
  3. Guard. A mechanical check that a probe declared observation does not arm an AbortController on the request it observes, and that an intervention is consumed only where that is acknowledged. Mutation-convicted both ways: adding an abort to a declared observation reddens; removing a legitimate abort from a declared intervention does not.

Deliberately not prescribed here: whether the disposition is JSDoc-anchored, a config leaf, or a returned field. That choice belongs with the enumeration in (1) — picking the mechanism before seeing the population is how a guard ends up fitting one probe.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
every probe/canary entry point its defining module carries an explicit observation / intervention disposition with the effect named undeclared ⇒ the guard fails, rather than defaulting to the safe-sounding answer module + method JSDoc the enumeration from (1), checked in
probe consumers (livenessProbe, healthProbe, diagnosis facts) the consuming service an intervention result is consumed only where its effect is acknowledged consumer JSDoc a spec asserting a declared intervention cannot silently back an authoritative fact
the guard itself test/playwright/unit/ai/** fails on an observation that arms an abort on its observed request spec header mutation both directions; a non-vacuity control proving the scan finds known members

Decision Record impact: aligned-with ADR 0025 — evidence before action. This makes one class of evidence honest about its provenance; it does not change what licenses an action.

Acceptance Criteria

  • The intersection of "issues a request" and "is used as a probe" is enumerated and checked in, covering the 9 abort-arming and 24 probe-carrying modules measured at filing rather than a sample.
  • Every enumerated probe carries an explicit observation / intervention disposition; an intervention names its effect.
  • A mechanical guard fails when a declared observation arms an AbortController on the request it observes. Mutation-convicted both directions — and it must red against ollamaStuckRunnerLiveness.mjs as it stands today, which is the positive control proving the guard can catch the one instance we already convicted.
  • The guard carries a non-vacuity control: a scan matching nothing would pass forever.
  • An undeclared probe fails the guard rather than defaulting to observation. The safe-sounding default is the failure mode — every probe here was implicitly an observation until one was measured.
  • A probe declared intervention cannot silently back an authoritative: true fact in the diagnosis layer without that being visible at the consuming seam.

Out of Scope

  • Fixing probeOllamaServing. #16853 owns it, and this ticket is blocked by it — the convicted instance should be repaired before the class is generalised from it, or the guard gets written around a moving target.
  • Container healthcheck: directives. Same class one layer down, but a compose-level concern with a different enforcement surface. Named in Architectural Reality so the boundary is deliberate rather than forgotten.
  • Whether any given probe should exist. This asks what a probe does, never whether it is worth doing.
  • #16855 — also a diagnosis-layer honesty defect, but about a metric's subject, not a probe's effect.

Avoided Traps

  • Treating this as one ticket's leftovers. #16853 convicts the instance; the class survives it. The next probe will be written by someone who never read that ticket, in a module where arming a timeout on a fetch is correct practice.
  • Defaulting undeclared probes to observation. That reproduces the current state and calls it compliance. Every probe here was implicitly an observation until one was measured.
  • Writing the guard before the enumeration. A guard designed against the one known instance fits that instance. The list is the deliverable; the guard is what keeps it true.
  • Claiming the other probes are broken. They are unclassified, which is the actual finding. One is convicted; the rest are unknown, and asserting otherwise would repeat the reasoning that produced the original false claim in this family.

Related

#16853 — the convicted instance, and this ticket's blocker · #16830 — consumes the canary; its ACs are satisfiable by shipping the abort, flagged there · #16855 — sibling diagnosis-layer honesty defect (metric subject rather than probe effect) · #16849 — the inverse shape: a request with no timeout and no abort signal · #16706 — deployment-stability epic. Governing decision: ADR-0025.

Live latest-open sweep: latest 10 open issues plus scoped title sweeps for probe/canary/liveness/readiness (80 issues) and intervention/observation/read-only at 2026-08-10T07:4xZ. Nearest neighbours are #16853 (one instance, owned) and #16830 (a consumer). No equivalent found. A2A claim sweep over recent messages: no in-flight claim on the probe-classification scope.

Structure map: ai/services/graph and ai/daemons/orchestrator/services are the established owning folders for the probes themselves; the guard lands under test/playwright/unit/ai/. No new production module prescribed until the enumeration exists.

Origin Session ID: 3c27118d-2de2-4579-bb42-1062c34cb895

Retrieval Hint: query_raw_memories("probe is an intervention not an observation abort in-flight canary manufactures the state it detects")

tobiu unassigned from @neo-opus-grace on Aug 10, 2026, 12:09 PM