Designing configuration infrastructure for mlcast python package

[leifdenby]

@franchg and I have just been having a discussion about how to design the configuration in mlcast. I used ChatGPT to summarise our conversion and make summary of the two approaches and tables of pros and cons below:

Summary of the Two Configuration Approaches

We are considering two alternative approaches for handling configuration in the project:

1. Hydra + Pydantic
   Uses Hydra to manage hierarchical YAML configurations and dynamically instantiate Python objects via _target_. Pydantic is used for schema validation. This approach treats the YAML config as both configuration and a class instantiation template, enabling highly flexible and composable setups.

2. Dataclasses + dataclasses-wizard
   Treats configuration as Python code expressed through dataclasses, with dataclasses-wizard providing YAML serialization/deserialization. YAML is a passive storage format rather than an active driver of object creation. This approach emphasizes type safety, static tooling support, and conceptual simplicity by avoiding dynamic class instantiation in configuration files.

Hydra + Pydantic

Category	Pros	Cons
Flexibility	Highly flexible configuration system; _target_ allows substituting whole classes without code changes	Flexibility comes at cost of complexity; config effectively becomes executable code
Modularity	Strong support for hierarchical config composition and experiment overrides	Hard to trace which classes are instantiated until runtime
Ecosystem	Mature tooling and widely used in ML training frameworks	Requires learning Hydra concepts (composition, instantiation, overrides) in addition to PyTorch/Lightning
User Experience	Powerful for advanced users who need dynamic architecture changes	Steep learning curve; difficult for newcomers and harder to teach (as observed in AIWCAS school)
Type Safety	Pydantic provides runtime validation	In practice, config typing often degrades to Any, reducing discoverability and static guarantees
Tooling Support	Hydra CLI utilities, sweeps, config stores	IDEs can't reliably infer instantiated types → limited autocomplete and code navigation
Separation of Concerns	YAML can define whole object graphs	Mixing Python class references into YAML blurs boundary between config and code

Examples:

• anemoi:
  • config in yaml: https://github.com/ecmwf/anemoi-core/blob/main/training/src/anemoi/training/config/lam.yaml
  • validation schema: https://github.com/ecmwf/anemoi-core/blob/main/training/src/anemoi/training/schemas/base_schema.py#L94
  • object instantiation: https://github.com/ecmwf/anemoi-core/blob/main/models/src/anemoi/models/models/encoder_processor_decoder.py#L36
• general purpose pytorch-lightning + hydra setups:
  • https://github.com/ashleve/lightning-hydra-template
  • https://github.com/franchg/yalht

Dataclasses + dataclasses-wizard

Category	Pros	Cons
Mental Model	Configuration is plain Python code; easier to reason about	Lacks Hydra’s built-in config composition and sweeping mechanisms out of the box
Type Safety	Strong typing and inheritance via dataclasses makes config structure explicit	Requires disciplined design if config grows large or complex
Tooling Support	Excellent IDE support: autocomplete, type hints, static analysis, navigation	Fewer higher-level utilities; some features must be implemented manually
User Experience	Lower cognitive load; no hidden instantiation magic	Less dynamic than _target_-based systems if users expect full pluggability
Serialization	dataclasses-wizard handles YAML round-tripping seamlessly	Not as feature-rich for configuration lifecycle management as Hydra
Debuggability	No dynamic class instantiation hidden in config files; code paths are explicit	Fewer "batteries included" for large-scale experiment management
Separation of Concerns	Config remains config; no Python identifiers or imports leaked into YAML	If the project requires dynamic class selection at runtime, patterns must be implemented intentionally

Examples:

• mllam:
  • config in yaml: https://github.com/mllam/mllam-data-prep/blob/main/example.danra.yaml
  • dataclasses for config (validation through type annotations): https://github.com/mllam/mllam-data-prep/blob/main/mllam_data_prep/config.py#L301
  • use of config values (no object instantiation based on class-names in config): https://github.com/mllam/mllam-data-prep/blob/main/mllam_data_prep/create_dataset.py#L117

If anyone has further thoughts on this please join in the discussion here by posting comments ☺️

[leifdenby]

Since writing the above I have come across fiddle and used it to develop what I'm calling a GNN-based forecasting "scaffold", I've called it weatherduck🌦️🦆: https://github.com/leifdenby/weatherduck. What I like about fiddle is that using it leads to a very clear separation of concerns, it is easy to replace sub-structures in an architecture, it is based on python (so our editors can do static code analysis) but still allows for representing the config in for example yaml-form. And finally, you can easily render your config is a diagram with inbuilt configuration in fiddle, here's an example from weatherduck:

In essence what fiddle does is adds a layer of indirection between the python code you write to instantiate a hierarchy of python classes and the actually instantiation. You get a "config" representation of the actual instantiation operation that would take place (if you call fiddle.build(config)) but gives you an option to modify any part of this instantiation operation (both what classes are instantiated and the keyword arguments used during instantiation) by simply assigning to the config object (e.g. config.pl_module.data.batch_size = 32 or config.pl_module.model.encoder = MyNewPyramidEncoder(...).

Also, fiddle is built by the Google team lead by Stephan Hoyer (of xarray fame), so you know it's going to be great 🚀 ☺️

Fiddle-based configs

Category	Pros	Cons
Mental Model	Configuration is pure Python; configs are first-class objects that can be inspected, modified, and reasoned about	Requires thinking in terms of Python object graphs and partial construction, which may be unfamiliar
Type Safety	Uses normal Python typing; config values correspond directly to constructor arguments	Type checking is not enforced by the framework itself; relies on discipline and type hints. But we could use type hints to ensure deserialised yaml-based configs use correct types
Tooling Support	Strong IDE support (autocomplete, navigation, refactoring) since everything is Python	Fewer dedicated IDE helpers compared to Hydra’s CLI-driven workflows
User Experience	Clear separation between configuration and execution; no YAML-as-code	Learning the Fiddle API (Config, build, partials) adds an extra abstraction
Modularity	Easy to refactor and reuse config fragments via normal Python functions	No built-in hierarchical config composition like Hydra defaults/groups
Debuggability	Very good: configs are inspectable objects, no hidden runtime instantiation	Debugging requires understanding Fiddle’s deferred construction model
Serialization	Designed primarily for code-based configs; good visualization/debug printing	YAML/JSON round-tripping is not a core goal or first-class feature
Separation of Concerns	Strong: config logic lives in Python, execution is explicit	Less suitable if users expect config-only (YAML-only) workflows
Scalability	Well-suited for large ML systems with complex parameter graphs	Less standardized across projects; smaller community than Hydra

My current suggestion is that we go with this. It feels more lightweight than the other two options, and avoids the duplication of logic in config schemas and code, but config schema in a sense is generated from the code (by inspecting the call arguments of classes instantiated) and so we don't have to write config schemas and ensure these are in sync with the code either.

[franchg]

Hi @leifdenby! Now that I have a bit of time I would like to chime in here. I have been experimenting a bit with fiddle too in our recent effort to make the ConvGRU model ensemble capable (see also #7). I do find fiddle overall better than wrestling with a yaml file tree in hydra. Everything is python code, including configurations, but we can still override the config values via command line. If I have to nitpick, the only drawback I found is that you can only override basic types via cli (strings and numbers), but not more complex objects (like dictionaries), but I may have not fiddled (phun intended) enough and maybe there is a way :)

All of this to say that I support to go fiddle as a configuration management solution for mlcast.