Enhanced Store & Grid Performance with Lazy Record Instantiation and Configurable Chunking

Is your feature request related to a problem? Please describe.

Prior to this enhancement, handling large datasets in Neo.data.Store and Neo.grid.Container presented several performance challenges:

1. Eager Record Instantiation: All Neo.data.Record instances were created upfront, leading to significant initial load times and memory consumption for large datasets.
2. UI Freezes during Large Data Adds: Synchronously adding extremely large numbers of raw data objects to the store could cause noticeable UI freezes.
3. VDom Reconciliation Errors with Chunking: The existing chunking mechanism (intended to mitigate UI freezes) could lead to VDom reconciliation errors (e.g., RangeError, infinite loops) due to component ID collisions in Neo.grid.column.Component.
4. Component Instance Overhead: Component columns could create and retain numerous component instances, contributing to memory overhead and rendering performance issues.

Describe the solution you'd like

This feature introduces a comprehensive set of enhancements to Neo.data.Store and Neo.grid.Container to address the aforementioned performance bottlenecks and improve overall responsiveness when dealing with large datasets.

Key aspects of the solution include:

1. Lazy Record Instantiation (GET-driven approach):

   • Neo.data.Store now stores raw data objects directly. Neo.data.Record instances are created lazily, only when an item is explicitly accessed via store.get(), store.getAt(), store.first(), store.last(), store.find(), store.findBy(), or store.forEach().
   • Once a raw data object is converted to a Record instance, the Record instance replaces the raw data in the store's internal collection for subsequent faster access.

2. Configurable Data Chunking:

   • The Neo.data.Store now includes an initialChunkSize config (Number, default 0). When set to a non-zero value, Store.add() will add data in chunks, mitigating UI freezes for very large synchronous data loads.
   • The postChunkLoad event is used to signal the completion of a chunked load.

3. Robust Component ID Management:

   • Neo.grid.column.Component.mjs#getComponentId now conditionally generates component IDs. If the store is in a chunking phase (store.chunkingTotal is truthy), it uses the rowIndex directly for unique IDs, preventing collisions. Otherwise, it uses the modulo-based ID generation for virtual scrolling.

4. Automated Component Instance Cleanup:

   • Neo.grid.Body now includes clearComponentColumnMaps() and cleanupComponentInstances() methods.
   • clearComponentColumnMaps() is called when the grid body is destroyed or when the store is completely cleared, ensuring all component instances are destroyed.
   • cleanupComponentInstances() is called after a chunked load (postChunkLoad) to destroy component instances that are no longer visible or needed (i.e., outside the mountedRows range), reducing memory overhead.
   • Grid.Body#afterSetStore also triggers clearComponentColumnMaps() when the store changes, ensuring proper cleanup of components from the old store.

Describe alternatives you've considered

(No specific alternatives considered as this is a comprehensive set of optimizations addressing multiple interconnected performance aspects.)

Additional context

This refactoring significantly improves the performance and memory footprint of applications dealing with large datasets by deferring the creation of Record instances and optimizing VDom reconciliation for chunked data.

Benchmark Results (from examples/grid/bigData):

Note: "Record creation" refers to the time taken for Neo.data.Record instantiation. "Data gen + add" refers to the time taken to generate raw data and add it to the store's collection.

The benchmarks demonstrate a dramatic reduction in initial data processing time due to lazy instantiation. While synchronous adding of 1M rows still causes a UI freeze, the configurable chunking mechanism provides a way to manage this for perceived performance. The resolution of VDom errors and component cleanup further enhance stability and efficiency.