LearnNewsExamplesServices
Frontmatter
id16380
titleFleet stall telltales: three receipt-observed defects
stateClosed
labels
bugai
assigneesneo-fable-clio
createdAtAug 2, 2026, 6:13 PM
updatedAtAug 2, 2026, 9:13 PM
githubUrlhttps://github.com/neomjs/neo/issues/16380
authorneo-fable-clio
commentsCount0
parentIssue14560
subIssues[]
subIssuesCompleted0
subIssuesTotal0
contentTrust
projected
quarantined0
signals[]
blockedBy[]
blocking[]
closedAtAug 2, 2026, 9:13 PM

Fleet stall telltales: three receipt-observed defects

neo-fable-clio
neo-fable-clio commented on Aug 2, 2026, 6:13 PM

Context

The Lane A leaf-1 headed receipts (#16324 / PR #16329) surfaced three cockpit-truthfulness defects around the stall/liveness telltales, recorded in that PR's cycle-1 review response as "tickets to file". They are bundled deliberately: three small fixes on one verification surface — a single cockpit receipt run proves all three — so a split would fragment one PR-sized delivery (ticket-create §8 bundle-by-default).

Live latest-open sweep: checked latest 20 open issues at 2026-08-02T16:12:09Z; no equivalent found. A2A in-flight claim sweep (~60 min window): claims cover #14418/#16223/#16375/#16376/#16350 — no scope overlap.

The Problem

Three independent observations from live cockpit receipts, one shared symptom class: the stall/liveness axes render less truthfully than the data they carry.

D1 — stall chips render [object Object]. Stall rows in the ActivityStream display the literal string [object Object] where a human-readable label belongs — the stall finding's payload reaches the chip as an object where the label path expects a string surface.

D2 — stall events flood the merged head-50. Every fleet snapshot re-emits work-graph stall findings stamped with the SNAPSHOT time: createStallActivityEvents(stallFindings, {capturedAt: observedAt}) at ai/services/fleet/fleetPrLaneActivityAdapter.mjs:73. The composer ranks by occurredAt (ai/services/fleet/fleetActivityComposer.mjs:181), so unchanged stalls re-float to the top of the merged head-50 on every poll and drown genuinely fresh activity. A stall that has existed for a day permanently outranks a review posted a minute ago.

D3 — tenant probes abort on a healthy-but-loaded plane. ai/services/fleet/FleetTenantService.mjs:606, :647, :685 bound the tenant-side plane probes with AbortSignal.timeout(10_000). Leaf-1 measured the containerized plane under WAL load at ~17s (initialize) / ~25s (list_messages) — which is why planeMailboxClient shipped with a 60s bound. The 10s literals predate those measurements: a healthy-but-loaded plane reads as aborted/degraded tenant state — a false-degraded fabrication class.

The Architectural Reality

  • apps/agentos/view/fleet/EventChip.mjs + apps/agentos/view/fleet/kindRegistry.mjs — the chip label path and the kind→presentation mapping (D1's home; the exact offending field resolved at implementation).
  • ai/services/fleet/fleetPrLaneActivityAdapter.mjs:73 + :192+ (createStallActivityEvents) — the stall emitter; findings come from buildWorkGraphStallFindings and carry their own temporal facts.
  • ai/services/fleet/fleetActivityComposer.mjs:181 — the merged-ranking authority (sort by occurredAt desc); correct as-is, D2 is producer-side stamping.
  • ai/services/fleet/FleetTenantService.mjs:606/:647/:685 — three hand-written 10_000 literals on one concern (plane-probe patience).
  • App-work gate: D1 touches apps/** — the implementing session loads the neo-core contracts per AGENTS.md intake 9.6 before editing view code.

The Fix

  • D1: the stall kind's chip label renders a string surface derived from the finding (title/reason/ticket ref), never a raw object; unknown/shapeless payloads degrade to a named fallback label, not [object Object].
  • D2: stall events carry a STABLE occurredAt derived from the finding's own temporal facts (its detected-at/anchor timestamp), with snapshot time kept only as capture metadata — unchanged stalls keep their rank; fresh real events sort above stale stalls. No composer change.
  • D3: measure the healthy rebuilt plane FIRST (record initialize/list latencies in the PR — the plane is one commit behind dev and healthy as of 2026-08-02 ~15:00Z), then replace the three literals with ONE leaf-declared bound (ADR-0019 clean: leaf(default, env, type), read at the use sites) calibrated ≥ measured healthy tail with margin.

Contract Ledger Matrix

Target Surface Source of Authority Proposed Behavior Fallback Docs Evidence
tenant probe-timeout config leaf fleet config base (exact home resolved at implementation per ADR 0019) one declared bound replaces three 10_000 literals leaf default = the calibrated value; env override per leaf machinery leaf JSDoc + measurement record in the PR measured-healthy receipt + unit coverage on the consuming probes
stall event occurredAt fleetPrLaneActivityAdapter stable finding-derived timestamp finding lacking temporal facts → capture time WITH a named degraded marker adapter JSDoc spec: repeated snapshots preserve rank; degraded marker witnessed
stall chip label kindRegistry/EventChip string surface, named fallback for shapeless payloads fallback label view JSDoc spec/receipt: no [object Object]

Decision Record impact: none (D3 aligned-with ADR 0019).

Acceptance Criteria

  • D1: a cockpit receipt shows stall chips with human-readable labels; a shapeless payload renders the named fallback, never [object Object].
  • D2: two consecutive snapshots with unchanged stall findings do not re-rank them; a fresh non-stall event outranks all stale stalls; witnessed by a composer/adapter spec.
  • D3: healthy-plane latencies measured and recorded BEFORE calibration; the three literals replaced by one ADR-0019-clean leaf; bound ≥ measured healthy tail with named margin; false-degraded class absent in a post-change receipt.
  • One PR resolves all three; existing fleet specs stay green.

Out of Scope

  • Wake-axis semantics (#16347 / PR #16368 — the serial predecessor).
  • Host receiver observability; new telltale kinds; composer ranking changes.
  • planeMailboxClient bounds (already calibrated in leaf 1).

Related

#14560 (parent: FM cockpit epic) · #16324 / PR #16329 (origin receipts) · #16347 / PR #16368 (serial predecessor — implementation starts after its merge) · D#16304 (plane rebuild context for D3's measurement)

Origin Session ID: 5892cb00-bb3f-467b-9346-81c509834503

Retrieval Hint: query_raw_memories("fleet stall telltale EventChip occurredAt flood tenant probe timeout receipt"); commit anchor: PR #16329 merge.

tobiu referenced in commit 3bea805 - "feat: fleet stall telltales tell the truth (#16380) (#16386) on Aug 2, 2026, 9:13 PM
tobiu closed this issue on Aug 2, 2026, 9:13 PM