GH-51037: [C++] Replace RapidJSON with simdjson in JSON parser (#51038)

### Rationale for this change

This PR continues the simdjson migration by replacing the RapidJSON-based parsing implementation used by the JSON parser.

The existing parser uses RapidJSON's SAX/handler interface to parse JSON values and populate Arrow builders. This change replaces that implementation with simdjson's ondemand API while retaining the existing builder and type-inference logic.

### Changes

* Replace the RapidJSON parser and handler interface with simdjson's ondemand API.
* Parse JSON documents using `simdjson::ondemand::parser::iterate_many`.
* Use `ResolveSimdjsonResult()` consistently when handling simdjson results.
* Preserve support for nested objects and arrays.
* Preserve explicit-schema and inferred-field behavior.
* Preserve unexpected-field handling for `Error`, `Ignore`, and `InferType`.
* Continue storing numeric values as raw JSON tokens.
* Trim trailing whitespace from numeric raw tokens to preserve existing behavior.
* Preserve JSON parse error propagation through `Status::Invalid`.
* Remove the parser's RapidJSON-specific dependencies.

### Are there any user-facing changes?

No

### Was AI used for this PR?

In accordance to the [AI generation guidelines](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code), please disclose below whether and how AI was used in this PR.

**PR code and description written by:**

- [X] Human
- [X] AI

**Reviewed before submission by:**

- [X] Human
- [X] AI
- [ ] Not reviewed

Fixes: #51037

* GitHub Issue: #51037

Authored-by: Aaditya Srinivasan <aadityasri03@gmail.com>
Signed-off-by: Antoine Pitrou <antoine@python.org>

Author: Reranko05

cpp/cmake_modules/ThirdpartyToolchain.cmake

@@ -2869,6 +2869,10 @@ function(build_simdjson)
 
   prepare_fetchcontent()
 
+  # Keep simdjson's threading configuration consistent with Arrow's,
+  # which is required for Emscripten where Arrow threading is disabled.
+  set(SIMDJSON_ENABLE_THREADS ${ARROW_ENABLE_THREADING})
+
   # simdjson enables precompiled headers unconditionally.
   # Recompiling simdjson.cpp against it produces differing artifacts
   # Disable precompiled headers to avoid reproducible build failures.

cpp/src/arrow/json/chunker.cc

@@ -32,16 +32,6 @@ namespace arrow {
 namespace json {
 namespace {
 
-// XXX We could try to SIMD-accelerate this routine but it's called only
-// once per chunk and also will presumably examine a minimal amount of bytes.
-int64_t ConsumeWhitespace(std::string_view view) {
-  const auto ws_count = view.find_first_not_of(" \t\r\n");
-  if (ws_count == std::string_view::npos) {
-    return view.size();
-  }
-  return static_cast<int64_t>(ws_count);
-}
-
 // A BoundaryFinder implementation that assumes JSON objects can contain raw newlines,
 // and uses the structural indexes computed by simdjson to delimit them.
 class ParsingBoundaryFinder : public BoundaryFinder {
@@ -156,7 +146,8 @@ class ParsingBoundaryFinder : public BoundaryFinder {
     if (consumed_length > 0) {
       // If we found at least one document, also consume its trailing whitespace
       // to avoid stray bytes at the end of the stream.
-      consumed_length += ConsumeWhitespace(input.substr(consumed_length));
+      consumed_length += internal::ConsumeJsonWhitespace(input.substr(consumed_length),
+                                                         /*trailing=*/false);
     }
     return consumed_length;
   }