Implement restart feature - #268
Conversation
|
Hello @Nunkyl! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:
Comment last updated at 2024-03-22 11:33:01 UTC |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #268 +/- ##
==========================================
- Coverage 72.71% 72.62% -0.09%
==========================================
Files 142 142
Lines 8450 8525 +75
==========================================
+ Hits 6144 6191 +47
- Misses 2306 2334 +28 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
насчет сохранения каждые 60 секунд -- будто бы время это не оптимальный критерий для определения интервала сохранения, лучше делать это в поколениях. Например, каждое или каждое пятое. Можно делать выбор исходя из времени, затраченного на первое поколение, -- если оно большое, то сохранять каждое последующее, если нет -- каждое N |
| graph_generation_params: GraphGenerationParams, | ||
| graph_optimizer_params: GPAlgorithmParameters, | ||
| use_saved_state: bool = False, | ||
| saved_state_path: str = 'saved_optimisation_state/main/evo_graph_optimiser', |
There was a problem hiding this comment.
'saved_optimisation_state/main' повторяется, можно вынести в отдельную константу. и вообще все строки
| if os.path.isfile(saved_state_file): | ||
| current_saved_state_path = saved_state_file | ||
| else: | ||
| raise SystemExit('ERROR: Could not restore saved optimisation state: ' |
There was a problem hiding this comment.
можно наверно просто писать лог, мол начать с сохраненного состояния не удалось, оптимизация начинается с нуля
| saved_state_path, **custom_optimizer_params) | ||
|
|
||
| # Restore state from previous run | ||
| if use_saved_state: |
There was a problem hiding this comment.
лучше вынести всю restore optimisation related логику в отдельный приватный метод
| break | ||
| # Adding of new population to history | ||
| self._update_population(new_population) | ||
| delta = datetime.now() - last_write_time |
There was a problem hiding this comment.
отдельный метод, код будет более читаемым
| if self.use_saved_state: | ||
| bar = tqdm(total=self.requirements.num_of_generations, desc='Generations', unit='gen', | ||
| initial=self.current_generation_num - 2) | ||
| else: | ||
| bar = tqdm(total=self.requirements.num_of_generations, desc='Generations', unit='gen', initial=0) |
There was a problem hiding this comment.
У этого класса нет атрибута self.current_generation_num и self.use_saved_state. Можно не менять код этой функции, а в PopulationalOptimizer в optimize сетить нужное значение. Что-то типа:
pbar.n = self.current_generation_num
pbar.refresh()
| assert time1 > 2 | ||
| assert time2 < 1 |
There was a problem hiding this comment.
Эта проверка не совсем ясна - почему именно 1 и 2?
| initial_population = [generate_labeled_graph('tree', 5, node_types) for _ in range(10)] | ||
|
|
||
| # Setup optimization parameters | ||
| requirements_run_1 = GraphRequirements(timeout=timedelta(minutes=timeout), |
There was a problem hiding this comment.
Действительно ли нужно создавать два отдельных объекта?
В идеале, наверное, сохранять фиксированную небольшую долю времени, затрачиваемого на сохранение. |
- resolve conflicts with current main
- address review comments:
* base GraphOptimizer no longer references subclass-only attributes:
the progress bar of a restored run is adjusted in PopulationalOptimizer.optimise
via pbar.n / pbar.refresh()
* repeated saved-state path literals replaced with the SAVED_STATE_BASE_DIR
constant and a per-class default derived from the class name
* state restoration logic extracted into a private method _try_restore_saved_state
* failed restoration logs a warning and falls back to starting from scratch
instead of raising SystemExit
* state saving logic extracted into a private method _save_state_if_needed
- saving is aligned with generation boundaries; the default mode is now adaptive:
the interval is derived from the duration of the previous save so that no more
than SAVE_STATE_TIME_FRACTION (5%) of the run time is spent on saving
- stop conditions are rebuilt after restoration so they use the new run requirements
- fix timedelta.seconds -> total_seconds() in the save interval check
- guard against None timeout during restoration
- add dill to requirements.txt
- rewrite test_saved_state: deterministic assertions with explanations,
no wall-clock timing assertions, cleanup of state files, plus a test
for the fallback-to-scratch behaviour
7 participants
Implements the optimisation restart feature. During optimisation the state of the optimiser object will be continuously saved to disk. If optimisation fails at some point it will be possible to continue the optimisation process starting with the last saved configuration, instead of starting over. Right now the function was implemented for
PopulationalOptimizerandEvoGraphOptimizerclassesImplementation details
The optimiser object's state will automatically always get saved to a .pkl file
The path is
/default_data_dir/saved_state_path/run_id/timestamp.pkl, wheredefault_data_dir- default_data_dir() from golem.core.pathssaved_state_path- folder path insidedefault_data_dir(its default value can be changed by the user)run_id- a unique id generated for every runtimestamp- the timestamp of when the file is written to diskBy default the state will be saved every 60 seconds. This can be changed in the
save_state_deltaparameter inoptimise()To restore saved state -> while creating an optimiser object (i.e. EvoGraphOptimizer) set class parameter
use_saved_stateto True and specify the path to the file if necessary (otherwise the last available file will be used)New params in the optimiser class:
use_saved_state[optional] true or falsesaved_state_path[optional] path to location of files (string)saved_state_file[optional] path to a specific file that will be used for restoration (string)New parameter in the optimise() function:
save_state_delta[optional] the amount of seconds to wait before saving the next file with the state (int), default = 60Examples:
Important:
While the saved state is restored all settings for the optimisation will be taken from the saved state file except for two:
It is possible to change their values for the second run of the optimiser
A new run using saved state will write data to the same folder the saved state file is in