Repository navigation
[C++] can we use simdjson to replace rapidjson #35460
Description
Activity
If simdjson is faster than RapidJSON for our use case too, I'm OK with this.
Could you try this and share our benchmark result?
https://github.com/apache/arrow/blob/main/cpp/src/arrow/json/parser_benchmark.ccWe are available to help.
Note that simdjson is used by Apache Doris and ClickHouse.
Great!
Seems that writer can still use original logic, but parser can make full use of simdjson?
Is there any merit to use both RapidJSON and simdjson?
I think that using either RapidJSON or simdjson will reduce our maintenance cost.
- changed the title
[-]can we use simdjson to replace rapidjson[/-][+][C++] can we use simdjson to replace rapidjson[/+]on May 15, 2023 I'm skeptical switching to simdjson would improve performance a lot, btw. Parsing is only a small part of the work necessary to convert JSON to Arrow.
I just want to revive this discussion again. #45459 will make
RapidJsonas a required dependency to parquet. However, RapidJson has been poorly maintained since 2016: Tencent/rapidjson#2321. It also seems to be a blocker to support CMake 4: #45985Reacted by Dewey DunningtonLet's try simdjson for better maintainability!
Reacted by Dewey DunningtonFor what it's worth, I think DuckDB uses yyjson, although I don't know if simdjson was considered. I seem to remember some past issues with homebrew packaging with respect to buildtime vs. runtime SIMD support (although I imagine simdjson has considered this as well!).
Reacted by Gang WuI seem to remember some past issues with homebrew packaging with respect to buildtime vs. runtime SIMD support (although I imagine simdjson has considered this as well!).
The simdjson library uses runtime dispatching. It is currently used by Node.js, ClickHouse and other important systems. In turn, Node.js is part of several important systems.
Optionally, we also support fancy C++20 features, and we will be integrating C++26 features (e.g., static reflection and the like) as soon as possible.
Suggested reference:
- On-Demand JSON: A Better Way to Parse Documents? Software: Practice and Experience 54 (6), 2024
Documentation: https://github.com/simdjson/simdjson/blob/master/doc/basics.md
I agree we need to do something about our RapidJSON dependency. simdjson is a reasonable contender. We should just have to check it doesn't reduce the current platform compatibility (especially when SIMD isn't available/supported by simdjson).
Reacted by Gang Wu64 remaining items
@Reranko05 on main we're getting this:
/usr/bin/ld: /build/arrow-install/lib/libparquet.so.2600.0.0: undefined reference to `arrow::json::JsonWriter::String(std::basic_string_view<char, std::char_traits<char> >)' /usr/bin/ld: /build/arrow-install/lib/libparquet.so.2600.0.0: undefined reference to `arrow::json::JsonWriter::GetString() const'
Is this something one of these issues will address or should we open a new one?
Is this something one of these issues will address or should we open a new one?
@Krishnanand-G is working on the issue, we can try the approach suggested by @pitrou #50900 (comment)
- added a commit that references this issue
on Aug 26, 2026 - added a commit that references this issue
on Sep 21, 2026 - added a commit that references this issue
on Sep 28, 2026
Describe the enhancement requested
As the performance result mentioned in simdjson community: the simdjson library uses three-quarters less instructions than state-of-the-art parser RapidJSON. And the throughput of simdjson is much higher than that of rapidjson:
So can we replace rapidjson with simdjson to implement json parser?
Component(s)
C++