Epic: Unified Data Pipeline Architecture (Pipeline -> Connection -> Parser -> Normalizer)

Goal

Modernize the framework's data architecture by implementing a unified, thread-agnostic data pipeline orchestrated by a new cornerstone class: Neo.data.Pipeline. This epic resolves the fragmentation between local data fetching (Fetch/XHR) and the RPC API, ensuring all incoming data—whether pulled via REST, returned from an RPC call, or pushed spontaneously via WebSockets—flows through a standardized transformation and ingestion process.

Context & The Architectural Schism

Currently, Neo.mjs has two competing data layers:

1. The New Pipeline: A modern, shaping-focused flow (Connection -> Parser -> Normalizer -> Store) designed to handle complex datasets (like Tree Grids). However, parser.Stream incorrectly handles fetching natively, breaking the single responsibility principle. Furthermore, the Store is currently hardcoded to orchestrate remote Data Worker instantiation for its Normalizer.
2. The RPC API: A powerful system that generates typed proxy functions, handles WebSocket multiplexing, and buffers Ajax requests. If a Store uses the api config (RPC), it bypasses any Parsers or Normalizers.

The Solution: Introduce Neo.data.Pipeline as an architectural cornerstone. The Store will exclusively aggregate a Pipeline instance. The Pipeline encapsulates the cross-worker orchestration, and the RPC API becomes a Transport Mechanism (Connection.Rpc).

The "Merged Universe": RPC + Pipelines

Data shaping (Parsers and Normalizers) should not be exclusive to Stores. A ViewController making a direct RPC call shouldn't have to manually format raw JSON. We are merging these concepts:

• remotes-api.json configurations will be enhanced to optionally define parser and normalizer configs for specific endpoints.
• When Neo.remotes.Api detects these configs, the generated proxy function will automatically pipe the raw JSON response through a Pipeline inside the Data Worker before returning the perfectly shaped data to the App Worker.

Progressive Hydration & Delta-Aware Pipelines

Modern applications require high Time-To-Interactive (TTI). A common backend pattern is to push "Quick Wins" (lightweight data like IDs and titles) immediately, and then stream the heavy, processed fields (like complex summaries or aggregations) later via WebSockets as Operational Transforms (Opcodes / Deltas).

To support this natively:

1. Pipelines are Delta-Aware: They are not just for read() (Full Loads). They must support continuous stream handlers that output partial record updates.
2. The Parser's Dual Role: For a read(), the Parser shapes a bulk array. For a stream, a specialized Parser translates proprietary backend Opcodes into standardized Neo.mjs Deltas (Insert, Update, Delete).
3. Surgical Updates: When a Store receives a parsed Delta from the Pipeline, it uses record.set({ field: 'new value' }). The RecordFactory increments the record version, and the VDOM Worker calculates a minimal patch, updating just a single grid cell without replacing the whole row or losing local UI state.

Implementation Phasing

1. Phase 1: The Pipeline Cornerstone (#9451)

   • Create Neo.data.Pipeline to manage workerExecution state.
   • Refactor Store to delegate to this.pipeline.read().

2. Phase 2: Connection Foundation & Refactoring (#9452)

   • Extract fetch logic out of Neo.data.parser.Stream into Connections.

3. Phase 3: IPC & Remote Execution (#9453)

   • Implement the IPC routing so an App Worker Pipeline can remotely execute a Data Worker Pipeline instance.

4. Phase 4: RPC Integration & The Merged Universe (#9455)

   • Create Connection.Rpc to wrap proxy calls.
   • Enhance Neo.remotes.Api to support pipelines for standalone RPC calls.

5. Phase 5: Push-Based WebSocket Integration (#9454)

   • Introduce streams to remotes-api.json.
   • Implement persistent stream subscriptions in manager.rpc.Message.
   • Support "Delta" routing from Connections through Parsers directly to Stores or ViewControllers.