[ENH]: Introduce `junifer.api.generate_yaml` by synchon · Pull Request #498 · juaml/junifer

synchon · 2026-05-29T16:04:42Z

description of feature/fix
tests added/passed
add an entry for the latest changes

This PR adds generate_yaml under api to generate feature YAML from metadata. Its primary use-case is in julio's feature addition to registry.

codecov · 2026-05-29T16:05:38Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (a8e9f31) to head (ca6cc8a).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #498   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            1         1           
  Lines            1         1           
=========================================
  Hits             1         1

Flag	Coverage Δ
docs	`100.00% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fraimondo · 2026-06-03T14:20:23Z

I have the impression that this way (dump_exclude) adds a lot of maintenance work: if something changes in the superclass (new internal variable), we need to go and update every subclass, including non-junifer ones.

Can't we just look at what fields are defined in the class (not the superclass) and just pass those ones to the constructor?

synchon · 2026-06-08T13:08:26Z

I have the impression that this way (dump_exclude) adds a lot of maintenance work: if something changes in the superclass (new internal variable), we need to go and update every subclass, including non-junifer ones.

That's a fair argument and I see your point.

Can't we just look at what fields are defined in the class (not the superclass) and just pass those ones to the constructor?

Not with how I understand the thing works. A model's fields consist of its own fields and superclass' fields (if it has one). So apart from defining what to exclude (or include), I don't see other way. I'll push some updates to make it better.

synchon · 2026-07-21T12:51:13Z

@fraimondo I've updated the datagrabber dumping logic as discussed. Kindly review #499 before this.

fraimondo · 2026-07-22T09:07:34Z

Do we know how this function works when we have external imports (with statements in the YAML)?

synchon · 2026-07-22T09:25:25Z

Do we know how this function works when we have external imports (with statements in the YAML)?

It copies exactly what is passed. For julio, we copy exactly what is stored in h5.

fraimondo · 2026-07-22T09:56:20Z

Do we know how this function works when we have external imports (with statements in the YAML)?

It copies exactly what is passed. For julio, we copy exactly what is stored in h5.

In order to create the yaml, we parse the metadata and instantiate the object. I'm not sure this will be possible if the definition of a datagrabber/marker is in an external package which is part of the with statement.

synchon · 2026-07-22T11:57:46Z

Do we know how this function works when we have external imports (with statements in the YAML)?

It copies exactly what is passed. For julio, we copy exactly what is stored in h5.

In order to create the yaml, we parse the metadata and instantiate the object. I'm not sure this will be possible if the definition of a datagrabber/marker is in an external package which is part of the with statement.

Let me lay it down w.r.t. julio:

junifer YAML gets parsed by junifer.api.parse_yaml which loads the external modules and registers componenets as needed.
If with is present in the junifer YAML, julio adds it directly to the generated YAML.
As the external components are already registered, junifer.api.generate_yaml can load them from component registry and dump as needed.

For using junifer.api.generate_yaml without julio, one would need to already register the components beforehand. Now, as it's part of junifer.api, we can presume that the user will do it like so.

Does that solve your concern?

fraimondo · 2026-07-22T13:22:46Z

Do we know how this function works when we have external imports (with statements in the YAML)?

It copies exactly what is passed. For julio, we copy exactly what is stored in h5.

In order to create the yaml, we parse the metadata and instantiate the object. I'm not sure this will be possible if the definition of a datagrabber/marker is in an external package which is part of the with statement.

Let me lay it down w.r.t. julio:
* junifer YAML gets parsed by `junifer.api.parse_yaml` which loads the external modules and registers componenets as needed.

* If `with` is present in the junifer YAML, julio adds it directly to the generated YAML.

* As the external components are already registered, `junifer.api.generate_yaml` can load them from component registry and dump as needed.
For using junifer.api.generate_yaml without julio, one would need to already register the components beforehand. Now, as it's part of junifer.api, we can presume that the user will do it like so.

Does that solve your concern?

junifer.api.generate_yaml has one parameter that is a meta dict. This meta dict can be extracted from and HDF5 file, without parsing any yaml. We need to be able to support generating a YAML from any meta dict, no only after parsing the respective yaml.

Even if this means adding all variables int he meta dict and a huge comment stating that "since this is an external datagrabber/marker/etc not part of junifer core, some of this variables might not be needed or should definitely be removed."

synchon · 2026-07-22T14:23:54Z

Do we know how this function works when we have external imports (with statements in the YAML)?

It copies exactly what is passed. For julio, we copy exactly what is stored in h5.

In order to create the yaml, we parse the metadata and instantiate the object. I'm not sure this will be possible if the definition of a datagrabber/marker is in an external package which is part of the with statement.

Let me lay it down w.r.t. julio:
* junifer YAML gets parsed by `junifer.api.parse_yaml` which loads the external modules and registers componenets as needed.

* If `with` is present in the junifer YAML, julio adds it directly to the generated YAML.

* As the external components are already registered, `junifer.api.generate_yaml` can load them from component registry and dump as needed.
For using junifer.api.generate_yaml without julio, one would need to already register the components beforehand. Now, as it's part of junifer.api, we can presume that the user will do it like so.
Does that solve your concern?
junifer.api.generate_yaml has one parameter that is a meta dict. This meta dict can be extracted from and HDF5 file, without parsing any yaml. We need to be able to support generating a YAML from any meta dict, no only after parsing the respective yaml.

Even if this means adding all variables int he meta dict and a huge comment stating that "since this is an external datagrabber/marker/etc not part of junifer core, some of this variables might not be needed or should definitely be removed."

I don't follow. What happens if one adds "with" key to the "meta" dictionary passed without loading the external components?

fraimondo · 2026-07-22T14:41:38Z

Do we know how this function works when we have external imports (with statements in the YAML)?

It copies exactly what is passed. For julio, we copy exactly what is stored in h5.

In order to create the yaml, we parse the metadata and instantiate the object. I'm not sure this will be possible if the definition of a datagrabber/marker is in an external package which is part of the with statement.

Let me lay it down w.r.t. julio:
* junifer YAML gets parsed by `junifer.api.parse_yaml` which loads the external modules and registers componenets as needed.

* If `with` is present in the junifer YAML, julio adds it directly to the generated YAML.

* As the external components are already registered, `junifer.api.generate_yaml` can load them from component registry and dump as needed.
For using junifer.api.generate_yaml without julio, one would need to already register the components beforehand. Now, as it's part of junifer.api, we can presume that the user will do it like so.
Does that solve your concern?
junifer.api.generate_yaml has one parameter that is a meta dict. This meta dict can be extracted from and HDF5 file, without parsing any yaml. We need to be able to support generating a YAML from any meta dict, no only after parsing the respective yaml.
Even if this means adding all variables int he meta dict and a huge comment stating that "since this is an external datagrabber/marker/etc not part of junifer core, some of this variables might not be needed or should definitely be removed."
I don't follow. What happens if one adds "with" key to the "meta" dictionary passed without loading the external components?

Let's asume you open and HDF5 file, you load the meta and then you pass it to generate_yaml. At no point there was a parse_yaml call, so all external modules will not be loaded. This code will fail as it needs to instantiate the elements in the meta to generate the yaml.

This use case should be considered. In the case that the object can't be instantiated, the fields should be extracted from the meta dict and a comment in the yaml should be added.

synchon · 2026-07-23T09:33:37Z

Do we know how this function works when we have external imports (with statements in the YAML)?

It copies exactly what is passed. For julio, we copy exactly what is stored in h5.

In order to create the yaml, we parse the metadata and instantiate the object. I'm not sure this will be possible if the definition of a datagrabber/marker is in an external package which is part of the with statement.

Let me lay it down w.r.t. julio:
* junifer YAML gets parsed by `junifer.api.parse_yaml` which loads the external modules and registers componenets as needed.

* If `with` is present in the junifer YAML, julio adds it directly to the generated YAML.

* As the external components are already registered, `junifer.api.generate_yaml` can load them from component registry and dump as needed.
For using junifer.api.generate_yaml without julio, one would need to already register the components beforehand. Now, as it's part of junifer.api, we can presume that the user will do it like so.
Does that solve your concern?
junifer.api.generate_yaml has one parameter that is a meta dict. This meta dict can be extracted from and HDF5 file, without parsing any yaml. We need to be able to support generating a YAML from any meta dict, no only after parsing the respective yaml.
Even if this means adding all variables int he meta dict and a huge comment stating that "since this is an external datagrabber/marker/etc not part of junifer core, some of this variables might not be needed or should definitely be removed."
I don't follow. What happens if one adds "with" key to the "meta" dictionary passed without loading the external components?
Let's asume you open and HDF5 file, you load the meta and then you pass it to generate_yaml. At no point there was a parse_yaml call, so all external modules will not be loaded. This code will fail as it needs to instantiate the elements in the meta to generate the yaml.

This use case should be considered. In the case that the object can't be instantiated, the fields should be extracted from the meta dict and a comment in the yaml should be added.

Do the latest commits address your concern?

fraimondo · 2026-07-23T09:36:20Z

+    # Set datagrabber
+    meta_dg = meta["datagrabber"].copy()
+    a = meta_dg.pop("class")
+    try:


I would explicity check for the datagrabber being in the registry that relying on a ValueError.

Could be that because of versions mismatchs, some parameters are renamed and then we do have errors but because of other reasons.

The check is updated to be precise now. Also, open to go the non-idiomatic route as well.

We still rely on a ValueError. It should be something like

if component is registered:
Instantiate and dump
else:

add ALL variables in the meta to the yaml

Add the note: " is not a built-in component and thus could not be properly regenerated. Some of these entries in the YAML section might be redundant and not needed. Please check the documentation/implementation of this specific datagrabber and remove the unnecesary entries."

fraimondo · 2026-07-23T09:36:33Z

+            meta_p = [meta_p]
+        for mp in meta_p:
+            b = mp.pop("class")
+            try:


Same here, explicit check

fraimondo · 2026-07-23T09:36:42Z

+    c = meta_m.pop("class")
+    y["markers"] = []
+    try:
+        m = PipelineComponentRegistry().get_class(step="marker", name=c)


fraimondo · 2026-07-23T09:37:20Z

+        "- `datadir` is ignored and not reproduced. "
+        "If `datadir` used was not a temporary directory, you will have to "
+        "manually edit this YAML.\n"
+        "- In case the dataset was 'dirty', there is no guarantee that the "


This note should only appear if the dataset was dirty (the meta said so)

Updated now.

fraimondo · 2026-07-23T10:36:31Z

+    # Set datagrabber
+    meta_dg = meta["datagrabber"].copy()
+    a = meta_dg.pop("class")
+    try:


We still rely on a ValueError. It should be something like

if component is registered:
Instantiate and dump
else:

add ALL variables in the meta to the yaml

Add the note: " is not a built-in component and thus could not be properly regenerated. Some of these entries in the YAML section might be redundant and not needed. Please check the documentation/implementation of this specific datagrabber and remove the unnecesary entries."

synchon · 2026-07-23T10:59:14Z

We still rely on a ValueError.

ValueError is only raised if the component is not registered. try...except is the "idiomatic" way to do it. I understand if that doesn't work and it needs to be superfluous.

fraimondo · 2026-07-23T11:01:04Z

We still rely on a ValueError.

ValueError is only raised if the component is not registered. try...except is the "idiomatic" way to do it. I understand if that doesn't work and it needs to be superfluous.

In that case, if any part of the instantiation raises a ValueError (like would happen if a parameter changes options, or using an old junifer version), then we go to:

add ALL variables in the meta to the yaml
Add the note: " is not a built-in component or it failed to instantiate and thus could not be properly regenerated. Some of these entries in the YAML section might be redundant and not needed. Please check the documentation/implementation of this specific datagrabber and remove the unnecesary

synchon · 2026-07-23T11:11:04Z

We still rely on a ValueError.

ValueError is only raised if the component is not registered. try...except is the "idiomatic" way to do it. I understand if that doesn't work and it needs to be superfluous.

In that case, if any part of the instantiation raises a ValueError (like would happen if a parameter changes options, or using an old junifer version), then we go to:

add ALL variables in the meta to the yaml

Add the note: " is not a built-in component or it failed to instantiate and thus could not be properly regenerated. Some of these entries in the YAML section might be redundant and not needed. Please check the documentation/implementation of this specific datagrabber and remove the unnecesary

model_construct does not raise an exception so there will be no error during the model construction.

fraimondo · 2026-07-23T11:18:15Z

Can we validate? I'm worried about using different junifer versions than the one that generated the meta. Or we either go full strict and not allow any mismatch (which will create a problem with julio later on), or we validate the model. Otherwise, variables that do not match will be "ignored" and not "dumped", which might yield a different yaml than expected.

I prefer to have a YAML with a note saying "check your datagrabber/marker/preprocessor due to possible changes in the API" than one without any message that actually works differently than expected.

synchon · 2026-07-23T13:58:50Z

Can we validate? I'm worried about using different junifer versions than the one that generated the meta. Or we either go full strict and not allow any mismatch (which will create a problem with julio later on), or we validate the model. Otherwise, variables that do not match will be "ignored" and not "dumped", which might yield a different yaml than expected.

I prefer to have a YAML with a note saying "check your datagrabber/marker/preprocessor due to possible changes in the API" than one without any message that actually works differently than expected.

model_construct is now replaced with model_validate which will fail for most due to the nature of the metadata and the models. Necessary comments will be added on generation.

fraimondo · 2026-07-23T14:07:39Z

Can we validate? I'm worried about using different junifer versions than the one that generated the meta. Or we either go full strict and not allow any mismatch (which will create a problem with julio later on), or we validate the model. Otherwise, variables that do not match will be "ignored" and not "dumped", which might yield a different yaml than expected.
I prefer to have a YAML with a note saying "check your datagrabber/marker/preprocessor due to possible changes in the API" than one without any message that actually works differently than expected.

model_construct is now replaced with model_validate which will fail for most due to the nature of the metadata and the models. Necessary comments will be added on generation.

I still don't understand why it will fail for "most". As long as you choose the dump_fields and pass it to the constructor, this should recreate the same object.

It should fail in case of:

External components
Components that changed API and the meta is from a non-compatible version.

All the rest should not fail. Otherwise we are dumping the wrong variables.

synchon · 2026-07-23T14:50:10Z

Can we validate? I'm worried about using different junifer versions than the one that generated the meta. Or we either go full strict and not allow any mismatch (which will create a problem with julio later on), or we validate the model. Otherwise, variables that do not match will be "ignored" and not "dumped", which might yield a different yaml than expected.
I prefer to have a YAML with a note saying "check your datagrabber/marker/preprocessor due to possible changes in the API" than one without any message that actually works differently than expected.

model_construct is now replaced with model_validate which will fail for most due to the nature of the metadata and the models. Necessary comments will be added on generation.

I still don't understand why it will fail for "most". As long as you choose the dump_fields and pass it to the constructor, this should recreate the same object.

It should fail in case of:

External components

Components that changed API and the meta is from a non-compatible version.

All the rest should not fail. Otherwise we are dumping the wrong variables.

Ignore my previous reply's "fail" part, it works as intended.

fraimondo · 2026-07-24T09:19:29Z

+        },
+    ],
+)
+def test_generate_yaml(m: dict) -> None:


So this tests that the actual function works. Can we test for correctness?

The latest commit should check for basic correctness.

This reverts commit 99aab92.

… in generate_yaml

…rate_yaml

…s and checks in generate_yaml

synchon requested a review from fraimondo May 29, 2026 16:06

synchon mentioned this pull request May 29, 2026

[REG]: Introduce feature files addition to registry juaml/julio#3

Open

synchon force-pushed the feat/generate-yaml-api branch from a5c4e20 to 7e49b70 Compare June 15, 2026 09:04

github-actions Bot added the Stale label Jul 16, 2026

synchon removed the Stale label Jul 16, 2026

synchon force-pushed the feat/generate-yaml-api branch from 7e49b70 to 70c439b Compare July 21, 2026 12:12

synchon added this to the 0.0.8 milestone Jul 21, 2026

synchon force-pushed the feat/generate-yaml-api branch from 83f88ed to 56c1a85 Compare July 21, 2026 14:06

fraimondo reviewed Jul 21, 2026

View reviewed changes

Comment thread junifer/configs/juseless/datagrabbers/ucla.py Outdated

Comment thread junifer/datagrabber/aomic/id1000.py

Comment thread junifer/datagrabber/hcp1200/datalad_hcp1200.py

Comment thread junifer/datagrabber/datalad_base.py

Comment thread junifer/datagrabber/pattern_datalad.py

fraimondo requested changes Jul 23, 2026

View reviewed changes

fraimondo reviewed Jul 24, 2026

View reviewed changes

synchon added 21 commits July 28, 2026 13:26

feat: add junifer.api.generate_yaml

0f9ed1a

feat: add _dump_exclude class variables to datagrabbers

845bd49

docs: add generate_yaml documentation

5087add

chore: add changelog 498.feature

1d2e2f6

Revert "feat: add _dump_exclude class variables to datagrabbers"

3daca38

This reverts commit 99aab92.

update: improve generate_yaml datagrabber module dump exclusion

f94ce52

docs: update generate_yaml.rst

1386278

update: add dump_fields class method for datagrabber fields dumping

07ba047

chore: correct dump_fields for JuselessUCLA

5fe3936

update: add comments about datadir to generate_yaml output

acad9c2

refactor: improve external component handling and comments generation…

2555c0a

… in generate_yaml

chore: add test for generate_yaml

c4109f6

update: make external component check exception more targeted in gene…

b3a0104

…rate_yaml

update: conditional comment addition in generate_yaml

a59f96e

update: add parameters for external components in generate_yaml

73effdf

update: improve external component comment in generate_yaml

cdd7d8b

update: make external component check non-idiomatic in generate_yaml

64d052e

refactor: replace model_construct with model_validate and add comment…

12332b0

…s and checks in generate_yaml

fix: correct usage of model_validate in generate_yaml

cb88686

chore: update test for generate_yaml

ec63d26

chore: lint

ca6cc8a

synchon force-pushed the feat/generate-yaml-api branch from 97a5c10 to ca6cc8a Compare July 28, 2026 10:26

Uh oh!

Conversation

synchon commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

fraimondo commented Jun 3, 2026

Uh oh!

synchon commented Jun 8, 2026

Uh oh!

synchon commented Jul 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fraimondo commented Jul 22, 2026

Uh oh!

synchon commented Jul 22, 2026

Uh oh!

fraimondo commented Jul 22, 2026

Uh oh!

synchon commented Jul 22, 2026

Uh oh!

fraimondo commented Jul 22, 2026

Uh oh!

synchon commented Jul 22, 2026

Uh oh!

fraimondo commented Jul 22, 2026

Uh oh!

synchon commented Jul 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

synchon commented Jul 23, 2026

Uh oh!

fraimondo commented Jul 23, 2026

Uh oh!

synchon commented Jul 23, 2026

Uh oh!

fraimondo commented Jul 23, 2026

Uh oh!

synchon commented Jul 23, 2026

Uh oh!

fraimondo commented Jul 23, 2026

Uh oh!

synchon commented Jul 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

synchon commented May 29, 2026 •

edited

Loading

codecov Bot commented May 29, 2026 •

edited

Loading