Rework serialization to support pydantic codecs

[nikitakuklev]

Summary

Replace the recursive walker / JSON_ENCODERS serialization machinery with context-aware codecs in a new xopt.types module. Serialization behavior is now driven by a single SerializationOptions pydantic context:

• array_mode: list | b64 (binary numpy/torch payloads)
• module_mode: drop | file (.pt next to the dump, load looks in same location (not always cwd)) | inline
• df_mode: dict | b64
• optional gzip/zstd compression

Encoding is done with special prefixes, and compression detected through magic bytes, following apsopt implementation. We move to using pure pydantic functionality, using advanced hooks like __get_pydantic_core_schema__. See example notebook for usage.

Backward compatibility

Old saves should load correctly. Legacy base64: inline torch tag works but will be swapped on save.

Format-affecting changes:

• Explicit null for non-Optional fields now raises instead of being silently stripped.
• supports_* capability flags are frozen and no longer serialized. (bugfix)
• DataFrame reloads restore the integer index (previously string keys).

Other

• orjson dependency dropped (pydantic already does fast json in Rust); zstandard added; requires pydantic>=2.12
• A few incidental bugfixes related to serialization

Fixes #444

[codecov]

Codecov Report

❌ Patch coverage is 92.30769% with 84 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
xopt/types.py	86.24%	30 Missing and 26 partials ⚠️
xopt/pydantic.py	90.80%	6 Missing and 2 partials ⚠️
xopt/base.py	81.25%	3 Missing and 3 partials ⚠️
xopt/generators/bayesian/bayesian_generator.py	82.35%	3 Missing and 3 partials ⚠️
xopt/generators/bayesian/models/standard.py	87.50%	2 Missing and 2 partials ⚠️
xopt/tests/test_pydantic.py	98.85%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[nikitakuklev]

@electronsandstuff opinion on overall arch? This would override #443 changes.

[electronsandstuff]

I just gave it a skim. Overall, this feels like the most modern way to handle it in pydantic and I support.

My understanding is that you make a "codec" and annotated type,

NDArray = Annotated[np.ndarray, NDArrayCodec()]

and pydantic knows to use __get_pydantic_core_schema__ natively?

---

Only other thoughts from quickly reading:

• Do we already support all of these encoding options in SerializationOptions? If not, I tend to err on the side of minimizing the features needed to support. Ie if we could look for an OK set of options which are the "Xopt choice" and drop the rest it will save us from having to maintain 2^n variations moving forward (unless there's a strong reason for some). Users probably don't need to worry about gzip vs zstd for instance and it's just another thing to continue to test and support.
• I saw some round-trip tests for Bayesian. I wonder if we are standardized enough now that we can make a parameterized test that does round-trip saving on every generator.
• Another test idea: confirm every generator has the support_flag fields frozen and exclued. IE double check that users subclass correctly. Parametrized pytest check