Performance & Serialization Enhancements#20
Merged
afshawnlotfi merged 22 commits intomainfrom Mar 9, 2026
Merged
Conversation
…nce SchemaUtils for list handling
…le reconstruction
…oved type annotation preservation
…kable serialization with orjson and caching
- Changed data types for index sizes and cell types from uint32 to uint8 in various classes and functions to optimize memory usage. - Updated the ElementUtils and ElementData classes to handle the new data types and added a method to create ElementData from polygons. - Modified the Mesh class to accommodate the new data types and adjusted related methods for consistency. - Updated tests to reflect changes in data types, ensuring compatibility with the new structure. - Adjusted TypeScript interfaces to align with the changes in the Python implementation, including updates to JSON schema definitions.
…rministic gzip compression
- Updated all test files to replace direct instantiation of Mesh and its subclasses with the new factory method `create()`. - Ensured that the conversion methods in tests now utilize the updated mesh creation approach. - Adjusted tests for handling JAX arrays to align with the new instantiation method. - Modified tests for dictionary arrays and dynamic models to reflect the changes in mesh creation. - Ensured that all instances of `Mesh` and `CustomMesh` are created using the `create()` method for consistency and to leverage any internal optimizations.
…nd optimized extraction
…c model instantiation
… handling in mesh creation
…root directory structure
…g of Packable objects
…se dynamic extracted directory
…mic model resolution
…re; add PackableCache for improved caching; update tests and documentation
…ove xxhash dependency
…ted documentation and utilities
…a dict as parameter
afshawnlotfi
added a commit
that referenced
this pull request
Mar 10, 2026
* feat: Add _ListAnnotation for inline JSON list serialization and enhance SchemaUtils for list handling * fix: Improve asset handling in SchemaUtils by refining resource and array type checks * feat: Enhance pickle serialization support in Packable class with a helper function * feat: Add PackableWorker and worker entry point for offloading Packable reconstruction * feat: Refactor extract_value method to use extract_basemodel for improved type annotation preservation * feat: Optimize array encoding and decoding in ArrayUtils; enhance Packable serialization with orjson and caching * Refactor meshly to use uint8 for index sizes and cell types - Changed data types for index sizes and cell types from uint32 to uint8 in various classes and functions to optimize memory usage. - Updated the ElementUtils and ElementData classes to handle the new data types and added a method to create ElementData from polygons. - Modified the Mesh class to accommodate the new data types and adjusted related methods for consistency. - Updated tests to reflect changes in data types, ensuring compatibility with the new structure. - Adjusted TypeScript interfaces to align with the changes in the Python implementation, including updates to JSON schema definitions. * feat: Integrate xxhash for improved checksum computation and add deterministic gzip compression * Refactor mesh creation to use factory methods - Updated all test files to replace direct instantiation of Mesh and its subclasses with the new factory method `create()`. - Ensured that the conversion methods in tests now utilize the updated mesh creation approach. - Adjusted tests for handling JAX arrays to align with the new instantiation method. - Modified tests for dictionary arrays and dynamic models to reflect the changes in mesh creation. - Ensured that all instances of `Mesh` and `CustomMesh` are created using the `create()` method for consistency and to leverage any internal optimizations. * feat: Enhance Packable and SerializationUtils with checksum support and optimized extraction * feat: Enhance SchemaUtils with nested model reconstruction and dynamic model instantiation * feat: Update markers parameter to accept Nx3 numpy arrays and enhance handling in mesh creation * refactor: Rename constants and update PackableStore to use a unified root directory structure * feat: Implement PackableCache for efficient in-memory and disk caching of Packable objects * refactor: Remove EXTRACTED_DIR constant and update PackableStore to use dynamic extracted directory * feat: Add discriminator matching support in SchemaUtils for enhanced model resolution * feat: Enhance PackableStore with root and extracted directory structure; add PackableCache for improved caching; update tests and documentation * refactor: Replace xxhash with SHA256 for checksum computation and remove xxhash dependency * feat: Add InlineArray type for inline JSON serialization; update related documentation and utilities * refactor: Update extract_checksums method to be static and accept data dict as parameter * chore: Bump version to 3.2.0-alpha in pyproject.toml and package.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR introduces significant performance optimizations, improved serialization capabilities, and enhanced caching mechanisms for the Meshly library across both Python and TypeScript implementations.
Key Changes
🚀 Performance Optimizations
ForkPoolutility for parallel processing on POSIX systems with copy-on-write memory sharingArrayUtils💾 Caching System
get_many()andput_many()with parallelized disk I/O via ForkPool🔧 Serialization Enhancements
_ListAnnotation: Added support for inline JSON list serialization🏭 API Improvements
create()factory methods for consistent instantiation🌐 Web Worker Support (TypeScript)
meshly/workermodule for easy worker instantiation📁 Storage Refactoring
PackableStoreto use a unified root directory structureEXTRACTED_DIRconstant in favor of dynamic path resolution🐛 Bug Fixes
extract_valueto useextract_basemodelfor better type preservationFiles Changed
Breaking Changes
create()factory method instead of direct instantiationDependencies
xxhash,orjson