Evaluate adding sonic-rs for JSON serialization and deserialization

[ifsheldon]

Context

Agentics currently uses serde_json as the workspace JSON implementation. JSON parsing/serialization appears in several potentially hot or contract-sensitive paths, including:

• backend/shared challenge bundle and zip project manifest parsing
• backend/shared DB JSON conversion for specs, evaluation payloads, leaderboard data, and metrics
• backend/api-server request/response DTO handling through Axum and serde_json::Value
• frontend CLI API response parsing and pretty JSON output

cloudwego/sonic-rs is a fast Rust JSON library based on SIMD, with Apache-2.0 licensing. Latest checked release: v0.5.8.

Proposal

Evaluate whether adding sonic-rs can improve JSON serialization/deserialization performance in Agentics without weakening the existing JSON contracts.

This should not be a blanket mechanical replacement. The first pass should identify where serde_json is actually on a meaningful path, then adopt sonic-rs only where it gives value and keeps the code clear.

Candidate areas to inspect

• Challenge and solution manifests:
  • backend/shared/src/challenge_bundle.rs
  • backend/shared/src/zip_project.rs
• DB JSON conversion helpers and evaluation payload handling:
  • backend/shared/src/db/*.rs
• API response parsing in the CLI:
  • frontends/agentics-cli/src/api.rs
• CLI JSON rendering:
  • frontends/agentics-cli/src/output.rs

Requirements

• Preserve the existing optional-field JSON contract. Response DTOs should continue omitting absent optional fields rather than emitting explicit null.
• Keep generated frontend schemas stable. If shared Rust DTOs change, follow the existing schema-generation workflow in frontends/web/.
• Avoid adding compatibility layers before MVP. If a cleaner internal API requires changing call sites, update the call sites directly.
• Do not silently change error behavior. Current parse failures should still report useful context.
• Keep any adoption scoped and measurable rather than replacing every serde_json use by default.

Suggested implementation plan

1. Confirm sonic-rs serde compatibility and feature flags for this workspace.
2. Add a small benchmark or targeted measurement for the largest JSON payloads we care about, such as challenge specs, run manifests, evaluation payloads, and leaderboard responses.
3. Replace only the proven hot-path parse/serialize calls, or introduce a narrow workspace JSON helper if that makes call sites clearer.
4. Keep serde_json::Value usage where it remains the best fit, especially around SQLx JSON columns and tests, unless there is a clear improvement.
5. Run the relevant Rust checks/tests before landing.

Acceptance criteria

• A short measurement summary explains whether sonic-rs is worth adopting and for which paths.
• Any adopted use preserves existing API/CLI JSON output and error semantics.
• Existing JSON contract tests still pass.
• The dependency choice and scope are documented in the PR description or a short code comment/helper module where appropriate.