feat: native XDR decoder (Rust N-API) with verified TS-fallback parity - #413
Merged
Osuochasam merged 2 commits intoAug 24, 2026
Merged
Conversation
Closes Open-audit-foundation#404. Adds the previously-documented-but-never-built native performance path for secureParseScVal, keeping the pure-TypeScript parser as the automatic, zero-configuration fallback. - native/soroban-xdr-decode: Rust cdylib (napi-rs) exposing decodeScVal. It ports the full pipeline of secure-xdr-parser.ts byte-for-byte: Node-compatible hex decoding, protocol-21 ScVal grammar with @stellar/js-xdr acceptance rules (zero padding, 0/1 bool/option flags, enum membership, full input consumption), and a validation traversal replicating validateScValStructure's guard order, per-node allocation estimates (incl. UTF-8-lossy string sizing) and path-scoped context semantics. No guard is weaker than the TS implementation. - lib/translator/native-xdr-decoder.ts: loader with platform detection, addon self-test, OPEN_AUDIT_DISABLE_NATIVE_XDR escape hatch and test override hooks. secureParseScVal tries native first (for payloads above the measured ~100-hex-char crossover) and reconstructs the exact ParserSecurityError subclasses from native outcomes; it falls back to TS when the addon is missing, throws, returns garbage, or disagrees with the JS parser on a success. - Parity suite (npm run test:parity): runs every fuzz/security corpus input from fuzz-xdr-parser.test.ts and secure-xdr-parser.test.ts plus seeded mutation/random sweeps, UTF-8 edge cases and guard-boundary payloads against both implementations and asserts identical results; fallback behavior is covered with simulated missing/crashing/lying addons. - Build tooling: npm run build:native / build:native:debug (cargo, no new npm dependencies) and build:native:docker (clean rust:1-bookworm container, isolated target dir, verified end-to-end). - Benchmark (npm run bench:xdr, interleaved best-of-3): 1.0x on tiny payloads (kept on the TS path by design), 2.4-3.3x on medium/large structures, up to 12.9x rejecting hostile deeply-nested payloads. Real numbers documented in README. Pre-existing test failures on main (49 in lib/translator) are untouched: verified identical failure set on a pristine main worktree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The lockfile on main is corrupted, apparently from a bad merge: 70 duplicate JSON keys in "packages", entries mixing metadata from unrelated packages (the rollup entry carried ansi-regex's funding URL and had lost its dependencies/optionalDependencies/bin fields, so the platform-specific rollup binding was never installed), vite present as both 5.4.21 and 7.3.6, and a poisoned v8-compile-cache-lib entry with a bogus os:["win32"] restriction. Since JSON duplicate keys resolve last-wins, that poisoned entry is what npm saw, and every `npm install` on Linux/macOS failed with EBADPLATFORM — which is why the "Validate docs/reality drift" check (and any other fresh install) has never passed. Surgical repair wasn't trustworthy given how thoroughly fields were shuffled between entries, so this regenerates the lockfile from package.json (fresh, self-consistent resolution; npm 10.8.2, lockfile v3). Verified: - clean clone + `npm install` exits 0 (no --force needed) and `npm run validate:docs` passes — the exact steps of the failing workflow - @rollup/rollup-linux-x64-gnu now installs (vitest runs from a fresh install without manual intervention) - lib/translator test results are byte-identical to the previous tree (same 49 pre-existing failures, 155 passing), and the native XDR parity suite (25 tests) passes against the refreshed stellar-sdk 12.3.0 / js-xdr 3.1.2 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This Pull Request introduces a high-performance native Rust XDR decoder (
native/soroban-xdr-decode) via N-API to accelerate high-throughput Soroban event decoding scenarios, paired with a zero-configuration automatic fallback to the existing TypeScript parser.It replicates all security bounds byte-for-byte, includes a deterministic ~600-case parity test suite, adds containerized cross-compilation build tooling, and documents real-world benchmark gains (up to 12.9× on malicious payload rejections and 3.3× on complex structures).
Linked Issues
Summary of Changes
1. Native Rust Decoder (
native/soroban-xdr-decode)ScValgrammar and@stellar/js-xdrparsing rules (hex decoding, zero-padding, 0/1 booleans, strict enum membership).validateScValStructureguard orders, path-scoped context semantics, and allocation calculations without relaxing limits for recursion depth, memory footprint, timeouts, or collection sizes.2. Transparent Automatic Fallback Pipeline
native-xdr-decoder.tshandles runtime platform detection and self-tests.secure-xdr-parser.tsattempts native execution first and transparently falls back to the pure-TypeScript implementation if the addon is unbuilt, unavailable, throws, or outputs an anomaly.3. Parity Test Suite & Security Verification
native-ts-parity.test.tsrunning ~600 test cases (fuzzing fixtures, mutation sweeps, UTF-8 boundaries, and security edge cases) across both engines to assert identical output and error parity.native-fallback.test.tsasserts graceful degradation during simulated missing, crashing, or corrupt addon states.4. Build Tooling & Benchmarks
npm run build:native,build:native:debug, andbuild:native:docker(verified in a cleanrust:1-bookwormcontainer).README.md):Verification Logs
npm run build:native:dockersucceeds in a clean container environment.Notes for Maintainers
package-lock.jsononmain:maincontains duplicate JSON keys causingEBADPLATFORMerrors during fresh installs. Handled locally via--forcewithout modifying the lockfile on this branch; recommended for a separate cleanup issue.mainremain completely unaffected by this PR.Closes #404