where to find inputs of a job

[zhubonan]

I am wondering where the inputs of a job gets stored. It seems that the job store only stores the output of a job in which is also what JobStoreDocument suggests.
When I use jobflow-remote, the inputs are stored in the collection of the latter, though. But it does not seem to be a collection aimed for long term storage (or I could be wrong).

(sorry I accidentally hit Ctrl+Enter while tying previously)

[utf]

Currently, the inputs aren't stored if using run_locally. There is a WIP PR that aimed to store the input references only (not the associated args/kwargs that these inputs correspond to). See #425.

The jobflow-remote storage should be seen as robust however. I can see an argument for storing the job inputs fully although this will result in a lot of duplicated data if using Fireworks or Jobflow-remote managers.

[zhubonan]

Hmm, I see. The potential use case I have in mind is to reconstruct the flow after it is run, which can be handy for reproducibility, and practically to make it easier to rerun a flow with minimum changes without the original submission script.

Is it something that is already possible with jobflow-remote?

[utf]

Yes, I believe that is possible using jobflow remote. Although perhaps @gpetretto could comment?

[gpetretto]

I am not entirely sure if I undestood correctly the use case you have in mind.
If you just want to see the inputs used for each Job these are indeed stored in jobflow-remote and you can see the connections between the jobs.
Once a Flow is submitted (and executed) it would also be possible to modify some inputs and rerun the same job. However, if instead you would like to generate a new flow with modified inputs and submit it, this is not possible at the moment. It should be possible to reconstruct the inputs and connections from the data in the DB to generate a new Flow object but I see some potential issues:

• for each job the in the DB the inputs are available. But it might not be possible to distinguish between jobs that are there from the beginning and those that have been dynamically generated.
• It will not be possible to get back the Flow Maker, so it will only be possible to modify the Flow object.

I think it might be interesting to add such a functionality to reconstruct the initial Flow object, but I am wondering if we should consider optionally storing the whole Flow input in a separate collection.

[zhubonan]

I see. Thanks for the detailed answer!