LearnNewsExamplesServices
Frontmatter
id9564
titleFinalize Data Pipeline Push Integration & UI Reactivity
stateClosed
labels
enhancementaiarchitecturecore
assigneestobiu
createdAtMar 26, 2026, 5:13 PM
updatedAtMar 26, 2026, 6:12 PM
githubUrlhttps://github.com/neomjs/neo/issues/9564
authortobiu
commentsCount1
parentIssue9449
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtMar 26, 2026, 6:12 PM

Finalize Data Pipeline Push Integration & UI Reactivity

Closed v12.1.0 enhancementaiarchitecturecore
tobiu
tobiu commented on Mar 26, 2026, 5:13 PM

Goal

Connect the lower-level WebSocket/Stream RPC mechanisms to the Store so that unsolicited server pushes surgically update UI components via record.set().

Context

In issue #9454, we implemented the plumbing for Progressive Hydration (WebSocket persistent subscriptions, RPC routing, IPC forwarding). However, we missed the final integration step: The Store itself is not listening for these pushes, and the pipeline abstraction lacks a dedicated push or update event to notify the store.

If a server pushes { id: 123, status: 'done' } right now, it traverses the worker boundary but dies in the App Worker because the Store is only listening to load and progress (for chunking), not unsolicited push events.

Acceptance Criteria

  1. Connection Event Emission:

    • src/data/connection/WebSocket.mjs (and potentially Rpc) must fire a push event when unsolicited data arrives, rather than just executing hardcoded callbacks.
  2. Pipeline Event Bridging:

    • src/data/Pipeline.mjs must relay push events from its connection to the outside world.
    • It must run pushed data through the parser and normalizer before emitting pipeline.fire('push', shapedData).
    • It must correctly handle IPC forwarding for pushes when workerExecution: 'data'.
  3. Store Integration (record.set()):

    • src/data/Store.mjs must listen to the push event on its pipeline.
    • Implement onPipelinePush(data). This method must locate the existing record via its primary key and call record.set(data) to trigger a surgical VDOM update without reloading the entire collection.
  4. Unit Testing (test/playwright/unit/data/):

    • Create PipelinePush.spec.mjs to verify event bubbling, shaping, and IPC forwarding using mock connections.
    • Create StorePush.spec.mjs to verify that a Store correctly interprets pipeline pushes and applies surgical updates via record.set(), triggering a recordChange event.
tobiu added the enhancement label on Mar 26, 2026, 5:13 PM
tobiu added the ai label on Mar 26, 2026, 5:13 PM
tobiu added the architecture label on Mar 26, 2026, 5:13 PM
tobiu added the core label on Mar 26, 2026, 5:13 PM
tobiu added parent issue #9449 on Mar 26, 2026, 5:14 PM
tobiu assigned to @tobiu on Mar 26, 2026, 5:14 PM
tobiu referenced in commit 53d3459 - "feat: Finalize Data Pipeline Push Integration & UI Reactivity (#9564)" on Mar 26, 2026, 6:10 PM
tobiu
tobiu Mar 26, 2026, 6:11 PM

Input from Gemini 3.1 Pro:

✦ I have implemented the final integration for the push-based data pipeline.

Summary of Changes:

  1. src/data/connection/Base.mjs: Made the base connection observable so that all connection types (including WebSocket) can emit custom events like push.
  2. src/data/connection/WebSocket.mjs: Now fires a push event when unsolicited data (Progressive Hydration) arrives without a matching callback or Promise mId.
  3. src/data/Pipeline.mjs: Implemented onConnectionPush to relay the push event from its underlying connection.
    • It passes the pushed data through the parser and normalizer.
    • If running in the Data Worker, it forwards the shaped data via IPC to the App Worker Pipeline.
  4. src/worker/App.mjs: Added onPipelinePush(msg) to catch the IPC forward and re-emit the push event on the correct proxy pipeline instance inside the App Worker.
  5. src/data/Store.mjs: The Store now listens to push events from its configured pipeline.
    • Added onPipelinePush(data): It looks up the record by its primary key, and if it exists, it executes record.set(data). This completes the flow and triggers the surgical VDOM update without reloading the grid or collection.
  6. Unit Tests: Added PipelinePush.spec.mjs and StorePush.spec.mjs to rigorously verify the event bubbling and the surgical record.set() updates using mock connections.

These changes have been committed and pushed to the repository. I will now close this issue.

tobiu closed this issue on Mar 26, 2026, 6:12 PM