rosidl: generate, cache, and distribute pre-parsed ASTs as JSON - #989
rosidl: generate, cache, and distribute pre-parsed ASTs as JSON#989wjwwood wants to merge 4 commits into
Conversation
cdcccdb to
6d05f6d
Compare
|
Pulls: #989
|
|
Pulls: #989
|
|
Pulls: #989
|
|
Pulls: #989
|
- Add serialization and deserialization for rosidl_parser.definition ASTs to/from JSON.
- Add in-memory AST caching and fast loading from .idl.json files in parse_idl_file.
- Save .idl.json AST during interface generation in the build tree.
- Install .idl.json files alongside .idl files into share/${PROJECT_NAME}/.
- Add unit tests for serialization roundtrip and pre-parsed AST loading.
Signed-off-by: William Woodall <wjwwood@google.com>
- Add dict_to_nestable_type helper for Array and Sequence types - Explicitly type elements list in idl_content_to_dict - Narrow Message element types in test_parser.py Signed-off-by: William Woodall <wjwwood@google.com>
…or pre-parsed ASTs Signed-off-by: William Woodall <wjwwood@google.com>
Signed-off-by: William Woodall <wjwwood@google.com>
0cee86a to
2de66c8
Compare
|
Pulls: #989
|
1 participant
Description
Currently, ROS 2 IDL parsing via
rosidl_parser.parse_idl_file()parses.idlfiles from raw text with Lark every time an IDL is accessed. This pr trys to improve that by saving the result of that parse and installing it to the disk next to the idl files themselves, so future accesses can use that. This is similar to.pycfiles in python or precompiled headers in C++.In typical workflows:
rosidl_generator_c,rosidl_generator_cpp,rosidl_typesupport_*, etc.) runs as an independent Python subprocess, resulting in 9–11 redundant parse cycles per message file.sensor_msgs,nav_msgs, or user workspace packages) must repeatedly re-parse the.idlfiles of all upstream interface dependencies (std_msgs,geometry_msgs,builtin_interfaces).This PR implements distributed pre-parsed AST caching and JSON serialization:
rosidl_parser.serialization: Adds bidirectional conversion betweenrosidl_parser.definition.IdlContentAST data structures and JSON-serializable dictionaries (idl_content_to_dict,dict_to_idl_content,save_ast_json,load_ast_json).rosidl_parser.parser.parse_idl_file():_ast_cache).<name>.idl.jsonpre-parsed AST file alongside the.idlfile. If present and up to date, it deserializes the AST directly, bypassing the slower grammar parser entirely.rosidl_generator_type_description&rosidl_pycommon: Saves<name>.idl.jsonalongside adapted.idlfiles during interface generation in the build tree.rosidl_cmake: Installs<name>.idl.jsonalongside.idlfiles intoshare/${PROJECT_NAME}/(OPTIONAL), ensuring pre-parsed ASTs are available for all downstream package builds and binary package installs.Empirical Benchmarks
1. Clean Rebuild Time (
rm -rf build/<pkg>)rolling)test_msgssensor_msgspx4_msgs2. Repeated (No-Op) and Incremental Rebuilds
rolling)test_msgstest_msgssensor_msgssensor_msgspx4_msgspx4_msgsNote: No-op and single-file incremental rebuilds are dominated by CMake dependency checking and remain unchanged.
3. Installed Artifact Footprint & Disk Space Impact
std_msgstest_msgssensor_msgspx4_msgsThe pre-parsed AST JSON files add < 1.5% to the installed size of interface packages.
Related to #931
Is this user-facing behavior change?
No, this is an internal build performance optimization.
Did you use Generative AI?
Yes, Gemini.