Fix/cmip7 pipeline fixes#263
Open
JanStreffing wants to merge 6 commits intofix/cmip7-use-metadata-not-cmip6-tablesfrom
Open
Fix/cmip7 pipeline fixes#263JanStreffing wants to merge 6 commits intofix/cmip7-use-metadata-not-cmip6-tablesfrom
JanStreffing wants to merge 6 commits intofix/cmip7-use-metadata-not-cmip6-tablesfrom
Conversation
The DefaultPipeline had both handle_unit_conversion and convert steps. The convert function has wrong signature for pipeline use (requires from_unit, to_unit parameters). The handle_unit_conversion wrapper already handles all unit conversion, making the second step redundant and broken. Fixes ParameterBindError: missing required argument 'to_unit'
Everett ConfigManager requires default values to be strings.
Changed dimension_mapping defaults from dict/bool to None/string
and convert to appropriate types after retrieval.
Fixes ConfigurationError: default value {} is not a string
Changed from _pycmor_cfg() to rule.get() since dimension_mapping is not a registered PycmorConfig option. This allows users to optionally provide dimension_mapping in their YAML rules without requiring it to be in the global config schema. Also updated experiment_id from 'spinup' to 'picontrol' to match CMIP7 CV experiment definitions.
Root cause: When using with_namespace('pycmor'), everett ConfigDictEnv
expects keys with the namespace prefix (e.g., PYCMOR_DASK_CLUSTER),
not just uppercase keys (DASK_CLUSTER).
Fixed by modifying _create_environments() to add namespace prefix:
uppercased_cfg[f'{cls._NAMESPACE.upper()}_{key.upper()}'] = value
This allows pycmor config from YAML to be properly loaded and applied.
Verified: SLURM cluster now activates with dask_cluster: slurm in YAML.
Also includes:
- Removed duplicate convert step in pipeline
- Fixed activity_id lookup (spinup -> picontrol)
- Fixed dimension_mapping config access
- Added SLURM jobqueue config to test YAML
Only log when rules match, not for every mismatch. With hundreds of CMIP7 variables and few rules, the non-match logs create excessive noise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.