LearnNewsExamplesServices
Frontmatter
id12886
titleNeural Link: drag-motion observability — agents must SEE live drag choreography
stateClosed
labels
enhancementaiarchitecture
assigneesneo-fable
createdAtJun 11, 2026, 3:52 AM
updatedAtJun 11, 2026, 8:48 AM
githubUrlhttps://github.com/neomjs/neo/issues/12886
authorneo-fable
commentsCount1
parentIssuenull
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtJun 11, 2026, 8:48 AM

Neural Link: drag-motion observability — agents must SEE live drag choreography

Closed v13.0.0/archive-v13-0-0-chunk-17 enhancementaiarchitecture
neo-fable
neo-fable commented on Jun 11, 2026, 3:52 AM

Context

Operator PRIO 0 (2026-06-11, verbatim: "PRIO 0 => new NL tools to enable you to see it"): tonight's grid-DnD regression cluster (#12883 — wrong mid-drag header motion, broken overdrag auto-scroll, column duplication on drop) shipped through FOUR blind verification layers — unit specs (visual write path stubbed), two independent live probes (final-order assertions only), and two formal cross-family reviews. Agents currently have NO way to perceive the motion layer: the thing a human user experiences first is the thing the swarm cannot see at all. This is the substrate fix: give agents eyes on live drag choreography, then fix #12883 WITH sight.

The Problem

Three perception gaps, each mapping to a regression class that escaped tonight:

  1. Mid-drag geometry over time — headers animate via wrapperStyle.left on absolutely-positioned items (container.SortZone#updateItem); nothing can sample where elements ACTUALLY are during a drag window. (Regression class: "headers move to wrong positions, not even close".)
  2. Drag-pipeline trajectoryget_drag_state is a point-in-time snapshot; there is no trace of the drag lifecycle (per-move currentIndex transitions, switchItems calls with arguments, indexMap evolution, overdrag/scroll activations, final moveTo arguments). (Regression classes: burst-walk misbehavior, broken overdrag scroll — and it permanently supersedes the temp-logging instrumentation #12880's AC1 wanted.)
  3. Engine↔DOM consistency — no tool asserts that a container's items[], vdom.cn, and the REAL DOM children agree. (Regression class: drop-duplication — a column existing twice is exactly a three-surface disagreement, currently invisible until a human looks.)

The Fix (tool surfaces — shapes for intake validation, split at implementation if needed)

  1. observe_motion{componentIds|selector, durationMs, intervalMs} → time-series of client rects (+ relevant style/transform), sampled main-thread-side during the window, returned as a trace. Agent drives the drag (any dispatch path) while the recorder runs; the trace IS the mid-drag assertion surface.
  2. get_drag_trace — ring-buffer instrumentation in container.SortZone (+ grid subclass): per drag, record move-events (clientX/Y, computed target), currentIndex transitions, switchItems(i,j) calls, scroll activations, onDragEnd resolution (fromIndex/toIndex/locked verdict). Tool returns the last N drags' traces. Negligible cost when idle; permanent falsifier substrate.
  3. verify_component_consistency{componentId} → diff report across items[] / vdom.cn / live DOM childNodes (ids, order, count). Generic beyond grids — the duplication detector for any container.

All three route through the existing app-worker NL channel (no dependency on #12884's window-registration gap, which remains the simulate_event routing fix).

Acceptance Criteria

  • AC1: observe_motion returns a usable trace for a devindex header drag (manual or dispatched), revealing the #12883 wrong-position motion
  • AC2: get_drag_trace captures a full drag lifecycle incl. burst-walk switch sequences and overdrag scroll activations
  • AC3: verify_component_consistency detects a synthetically induced items/vdom/DOM mismatch (and the live duplication case while it exists)
  • AC4: tool descriptions within MCP budget (§5.3); whitebox-e2e + neural-link skill payloads updated to route to these tools
  • AC5: the #12883 fix PR uses these tools as its evidence layer (the doctrine: no more blind motion fixes)

Out of Scope

  • The #12883 fix itself (separate, gated on these eyes per operator directive).
  • simulate_event routing for non-shared apps (#12884).
  • Screenshot/pixel-level perception (rect-trace is the right granularity for this defect class; pixel tools are a later layer if rects prove insufficient).

Decision Record impact

none (extends the existing NL tool surface; ADR 0019-conformant config reads).

Related

Operator directive thread: #12883 (the consumer). #12884 (sibling NL gap). #12807 (@neo-opus-ada's simulatability — dispatch side; this is the perception side). Umbrella: the whitebox-e2e doctrine proven tonight.

Release classification: ON the board — operator PRIO 0; gates the #12883 fix which gates the cut.

Origin Session ID: 567191c3-16f6-4235-914c-b51fc94d1514 Retrieval Hint: query_raw_memories("drag motion observability NL eyes PRIO 0 duplication")

Live latest-open sweep: latest 12 open checked 01:48Z + #12883/#12884 filed by me since; no equivalent. A2A sweep: clean; lane operator-routed to @neo-fable.

tobiu referenced in commit ae22396 - "feat(ai): add NL drag-motion observability tools (#12886) (#12890) on Jun 11, 2026, 8:48 AM
tobiu closed this issue on Jun 11, 2026, 8:48 AM