Approaches To Statefulness

[TimothyWillard]

It would be helpful to have some context about invocations of a configuration file. Particularly for process actions which could benefit from having context about recent optimization/simulation runs. There are two open questions, 1) how should the history be stored at all, and 2) what should the history data structure look like. I think getting (1) down is the most important and (2) will be solvable via iteration. Some preliminary options (taken from #55):

1. User has to explicitly provide information to following commands. Pros are that its explicitly clear and let's flepimop2 be stateless, but cons are forces users to think about state which would make it painful to script a series of flepimop2 commands together.
2. flepimop2 manages state in a hidden cache that allow flepimop2 to track the series of actions that have been performed on a configuration. Pros are that when it works its seamless, but cons are difficult to implement on our side and when things do not work it becomes painful for the user quickly.
3. flepimop2 places state in the configuration files via an --out-config option (or perhaps a better name). Pros are that its clear what the state is and it's editable directly by users when needed to work around an issue, but con is having to keep track of this output configuration file.

PR #71 is a brief demonstration of the third approach. Currently just storing the action, timestamp, and a name for the history although would probably need more details in future iterations such as a target and perhaps a result of the command.

[pearsonca]

Additional thought re 3: the resulting config file might not be trustworthy. For example, if some results get deleted, some inputs updated, etc, etc - so whatever the history-writing solution is, it may also have to be cheap to check

I don't like option 2 - see "two hardest problems".

As a riff on 1 (which also requires the note above re 3): if a configuration file deterministically implied a checkable state, then state could be "stored" by the presence/absence of that state. For example, if the results specification is sufficient to completely determine where files would be => state can easily be checked if those files are there vs not. There's minimally a mild usability hit (the user can't just look at the config and see the state), and possibly a large one (i.e. the library will have to expose state checking).

[TimothyWillard]

As a riff on 1 (which also requires the note above re 3): if a configuration file deterministically implied a checkable state, then state could be "stored" by the presence/absence of that state. For example, if the results specification is sufficient to completely determine where files would be => state can easily be checked if those files are there vs not. There's minimally a mild usability hit (the user can't just look at the config and see the state), and possibly a large one (i.e. the library will have to expose state checking).

Okay, so practically what does this mean? What kind of information would flepimop2 need to ask of the user/store in configuration to confirm that outputs expected do exist and are accessible for further use? How does this apply to say process modules that may themselves want to provide state, i.e. a filtering process module that then feeds into a plotting process module.