Implement novelty_threshold decay in ProximityArchive

[amarrerod]

Description

Implements issue #540: novelty_threshold decay in ProximityArchive after X iterations without inserting any new novel solution.

TODO

• Update ProximityArchive to consider two new optional arguments: iterations_without_imp and threshold_decay. iterations_without_imp it is either None or a positive integer. When None, ProximityArchive works as expected with a fixed novelty_threshold value. Otherwise, it defines the maximum number of iterations without inserting novel solutions, before decreasing the novelty_threshold by a factor of threshold_decay. Note that threshold_decay is a float value in the range [0.0, 1.0], and the new novelty_threshold is calculated following

$$ max(0.0, nthreshold * decay)$$

where nthreshold is the current value of novelty_threshold.

• Create __maybe_update_threshold in ProximityArchive: A private method of the ProximityArchive that, if the decay strategy is enabled, is called every time n_novel_enought is zero in add. When the number of iterations reaches the user-defined maximum, the novelty_threshold is decreased.

• Create test_add_non_novel_solution_threshold_decay: Inside proximity_archive_test, a new test case is included to test the new functionality. The test tries to insert the same solution X iterations and, after that, checks that the novelty_threshold has been updated to the new expected value.

Questions

• The __maybe_update_threshold is a function that receives no parameters and acts over the state of the ProximityArchive object. I thought about including this logic directly in the addmethod. However, I believe it is cleaner to have all the logic inside a private method and call it from every branch of the addmethod when necessary.

Status

• I have read the guidelines in
  CONTRIBUTING.md
• I have linted and formatted my code with ruff and pylint
• I have tested my code by running pytest
• I have added a description of my change to the changelog in HISTORY.md
• This PR is ready to go

[btjanaka]

Hi @amarrerod, thanks for the PR! I have some minor edits that I can make later, but since you seem quite familiar with Novelty Search, I wanted to check with you if this is a known way to decay the novelty threshold? When I wrote the issue in #540, the idea I proposed was off the top of my head; I am not sure if it is grounded in any paper.

[amarrerod]

Hi @btjanaka, basically, I took the idea from parameter/restriction reduction, which is used in many optimisation algorithms. Apparently, someone has already tried something similar: https://ala2018.cs.universityofgalway.ie/papers/ALA_2018_paper_7.pdf (Eq. 3), but they decrease the threshold in every iteration, whether there are new novel solutions or not.

[btjanaka]

HI @amarrerod, thanks again for sending in the PR! I made some tiny changes and also left some comments below. Let me know if you have any questions!

[btjanaka]

Hi @btjanaka, basically, I took the idea from parameter/restriction reduction, which is used in many optimisation algorithms. Apparently, someone has already tried something similar: https://ala2018.cs.universityofgalway.ie/papers/ALA_2018_paper_7.pdf (Eq. 3), but they decrease the threshold in every iteration, whether there are new novel solutions or not.

Sounds good! Let's proceed with this implementation, then.

[btjanaka]

Hi @amarrerod, thanks again! I made some changes and submitted the PR. These were my changes:

• Moved threshold_decay_rate before threshold_decay_itrs and threshold_decay_min, and I gave threshold_decay_itrs a default of 1. This way, a user can just pass in the threshold_decay_rate, and threshold decay will be turned on.
• Moved the check for threshold_decay_rate is None to _maybe_update_threshold.
• Rearranged comments for _maybe_update_threshold.
• Ran 5 trials on the sphere benchmark to check for any performance regressions; after 5 trials I got:

  |   Algorithm |              QD Score |      Coverage |   Trials |
  |------------:|----------------------:|--------------:|---------:|
  |      ns_cma | 155,384.75 ± 7,831.25 | 19.08 ± 1.05% |        5 |
  

  Which is close to the current results; hence I see no regression.
• Made minor edits to the tests and also added another test.