LearnNewsExamplesServices
Frontmatter
id9551
titleExamples: Implement unified Data Pipeline showcases
stateClosed
labels
enhancementaicore
assigneestobiu
createdAtMar 25, 2026, 5:37 PM
updatedAtMar 25, 2026, 8:27 PM
githubUrlhttps://github.com/neomjs/neo/issues/9551
authortobiu
commentsCount1
parentIssue9449
subIssues[]
subIssuesCompleted0
subIssuesTotal0
blockedBy[]
blocking[]
closedAtMar 25, 2026, 8:27 PM

Examples: Implement unified Data Pipeline showcases

Closed v12.1.0 enhancementaicore
tobiu
tobiu commented on Mar 25, 2026, 5:37 PM

Goal

Create new working examples in the examples/data/ folder to demonstrate the unified Data Pipeline architecture.

Description

Since the data architecture has been overhauled to use a Pipeline -> Connection -> Parser -> Normalizer flow, we need to provide concrete, working examples for developers.

These examples should showcase:

  1. App Worker Pipeline: An example using a lightweight Grid driven by an App Worker pipeline.
  2. Data Worker Pipeline: An example showing remote execution of a pipeline inside the Data Worker.
  3. Connection Types: Examples covering different connection types like Fetch and the new Rpc connection.
  4. Data Shaping: Demonstrations of Parser and Normalizer configurations.

These examples will serve as a reference for migrating existing stores and implementing new data flows.

tobiu assigned to @tobiu on Mar 25, 2026, 5:37 PM
tobiu added the enhancement label on Mar 25, 2026, 5:37 PM
tobiu added the ai label on Mar 25, 2026, 5:37 PM
tobiu added the core label on Mar 25, 2026, 5:37 PM
tobiu added parent issue #9449 on Mar 25, 2026, 5:39 PM
tobiu referenced in commit e872704 - "feat: Implement folder-scoped dynamic loading for Data Worker pipelines (#9551)" on Mar 25, 2026, 8:26 PM
tobiu
tobiu Mar 25, 2026, 8:27 PM

Input from Gemini 3.1 Pro:

✦ Completed the implementation of the Unified Data Pipeline showcases and framework enhancements.

Summary of Work:

  1. Framework: Folder-Scoped Dynamic Loading
    • Enhanced Neo.worker.Data with loadDataModule({className}). This method uses strictly scoped dynamic imports (e.g., ../data/connection/${name}.mjs) to satisfy the "rip apart" strategy, preventing Webpack bundle bloat while allowing the Data Worker to load its own dependencies on demand.
  2. Core: Pipeline IPC Orchestration
    • Updated Neo.data.Pipeline to utilize loadDataModule during its initRemoteExecution phase.
  3. Core: Instance Lookup Stabilization
    • Imported Neo.manager.Instance into the Data Worker to allow resolution of instance IDs for IPC calls.
    • Refactored RemoteMethodAccess to use the Neo.get(id) alias for better resilience.
  4. Example: Data Pipeline Showcase
    • Created a new tabbed example in examples/data/pipeline/ demonstrating both App Worker and Data Worker pipeline execution.
    • Registered the example in docs/examples.json.

Verification: Both tabs in the new example are correctly loading data using their respective pipeline execution modes.

tobiu closed this issue on Mar 25, 2026, 8:27 PM