Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions template.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Please enter the information relevant to your problem/suite/generator.
#
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General comment:
I'd prefer to have the number of fields as small as possible to just not discourage people from filling the template. That's why I question every entry and try to keep the template as easy as possible.
I also think the template should not address optimisation experts but domain experts who might not be aware of our terminology.

# Instructions:
# - Fields that are not relevant can be removed (delete the whole line), or left empty.
# - Enter 'unknown' for fields where the value is unknown.
# - If multiple suggested values are relevant, include all those that fit.
# - Suggested values for fields are recommendations. If they are not a good fit, feel free to add something else (and ideally, explain why).
# - If some property is present, it does not have to apply to all problems/variables.
- name:
short: BBOB # Short name or abbreviation
full: Real-Parameter Black-Box Optimization Benchmarking # Full name
suite/generator/single: suite # Whether this is a single problem, a problem suite, or problem generator
objectives:
number: '1' # (list of) fixed numbers, or ranges (5-11,19,85), scalable if it can be any value
variables: # Information about the input variables
types: continuous # Can be one or more of (continuous, integer, binary, categorical, mixed) in case of mixed, describe which mixes are possible
conditional: 'no' # Whether there are conditional dependencies between variables, 'yes' or 'no'
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unsure whether this information is needed. I would always say "yes" since I can't imagine a situation where there are no dependencies but I guess there surely are. Anyhow, I wonder whether this has any value?!?

dimensionality:
scalable: 'yes' # Is the dimensionality scalable yes or no
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the number is a range (or "any"), it's scalable, if not, it's not.
Isn't it that simple?
And if not, is the field really required or just for experts?

number: '2-40' # (list of) fixed numbers, or ranges (5-11,19,85), or 'any' if it can be any value
constraints:
present: 'no' # Whether there are constraints yes or no
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the number is 0 then we expect a "no", if not, we expect "yes", or?
Do we need the field?

soft: 'no' # Whether there are soft constraints yes or no
hard: 'no' # Whether there are hard constraints yes or no
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can fields "soft" and "hard" be merged to "hardness" or the like?
Possible entries would then be "hard", "soft", or maybe "both"?

number: '0' # How many constraints there are in total, fixed numbers or range, scalable if it can by any value
boundary/box: 'yes' # Is the problem box-constrained yes, no, or some (if only some variables are bounded)
permutation: 'no' # Variables must be a permutation yes, no, or some
dynamic: # Dynamic properties, e.g., changing objective functions
present: 'no' # Enter, yes, no, or optional
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again I think the field might be economised ;)
If the type is empty, presence would be "no" if not, "yes".

types: '' # Types of dynamic behaviour that are present
noise: # Noise on the objective function(s)
present: 'no' # Enter, yes, no, or optional
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as for "dynamic".

types: '' # Types of noise that are present
modality:
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's complex ...
What do we need this information for? Do we need it?

I agree that modality is important but there is way more to it than just "uni" or "multi" and just this doesn't really help IMO.

types: 'unimodal, multimodal' # Enter all that are present
evaluations:
multi-fidelity: 'no' # Whether evaluations can be performed at multiple fidelities, yes, no, or optional
partial possible: 'no' # Whether evaluation of subset(s) of decision variables are possible
independent objectives: 'no' # Can objective functions be evaluated independently of each other yes or no
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or "some" in case of different groups of objectives who share evaluation properties?

reference:
links:
- https://doi.org/10.1080/10556788.2020.1808977 # URL to source or closely related research article, preferably DOI based. Add a new line below for each additional article.
authors: 'Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, Dimo Brockhoff'
contact person: ''
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a possible value could be "any" in the sense of "any of the above" but leaving this field empty is confusing. (please don't contact us?)

implementations:
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plural? Can people leave multiple implementations? Do we want this?

- name: COCO # Name of the tool/library/package
link: https://github.com/numbbo/coco
languages: 'C/C++, Python, Java, Matlab/Octave' # Programming languages the implementation can be used with
evaluation time: 'less than a second' # Choose: 'less than a second', 'seconds', 'minutes', 'hours', 'days'
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it clear that we refer to a single fitness/objective function evaluation time here? (not a whole opt. run)

And I guess we expect the maximum in case of "independent objectives" (different evaluation times) above? Maybe better put a field "time" in the group of "evaluations"? I don't know ...

specific requirements: 'no' # Can be things like memory/GPU requirements, or the need for a specific simulator. Enter 'no' if none.
source:
origin: 'artificial' # Is the problem artificial, real-world, or real-world-like
real-world: # In case it is real-world (like), provide details
Copy link
Copy Markdown

@bn12 bn12 Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't expect any entry here, right?
Can't we just put a comment to the two (sub-) fields following like
" in case of "real-world" or "real-world-like" "
?

domain: '' # E.g., automotive
open/closed: '' # Is the problem open source yes or no
textual description:
general info: ''
motivation: 'evaluate algorithm performance for typical difficulties that occur in continuous problems'
challenge/key characteristics: ''
limitations: ''
example links: '' # URLs to examples where it was used (e.g., scientific articles) - Note: URLs to where it was proposed should go under the 'reference' field.
other info: ''