-
Notifications
You must be signed in to change notification settings - Fork 3
111 create yaml template for problem submission #121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
950c447
43fbc89
cabadca
d91f74b
833a593
19c0635
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| # Please enter the information relevant to your problem/suite/generator. | ||
| # | ||
| # Instructions: | ||
| # - Fields that are not relevant can be removed (delete the whole line), or left empty. | ||
| # - Enter 'unknown' for fields where the value is unknown. | ||
| # - If multiple suggested values are relevant, include all those that fit. | ||
| # - Suggested values for fields are recommendations. If they are not a good fit, feel free to add something else (and ideally, explain why). | ||
| # - If some property is present, it does not have to apply to all problems/variables. | ||
| - name: | ||
| short: BBOB # Short name or abbreviation | ||
| full: Real-Parameter Black-Box Optimization Benchmarking # Full name | ||
| suite/generator/single: suite # Whether this is a single problem, a problem suite, or problem generator | ||
| objectives: | ||
| number: '1' # (list of) fixed numbers, or ranges (5-11,19,85), scalable if it can be any value | ||
| variables: # Information about the input variables | ||
| types: continuous # Can be one or more of (continuous, integer, binary, categorical, mixed) in case of mixed, describe which mixes are possible | ||
| conditional: 'no' # Whether there are conditional dependencies between variables, 'yes' or 'no' | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm unsure whether this information is needed. I would always say "yes" since I can't imagine a situation where there are no dependencies but I guess there surely are. Anyhow, I wonder whether this has any value?!? |
||
| dimensionality: | ||
| scalable: 'yes' # Is the dimensionality scalable yes or no | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If the number is a range (or "any"), it's scalable, if not, it's not. |
||
| number: '2-40' # (list of) fixed numbers, or ranges (5-11,19,85), or 'any' if it can be any value | ||
| constraints: | ||
| present: 'no' # Whether there are constraints yes or no | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If the number is 0 then we expect a "no", if not, we expect "yes", or? |
||
| soft: 'no' # Whether there are soft constraints yes or no | ||
| hard: 'no' # Whether there are hard constraints yes or no | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can fields "soft" and "hard" be merged to "hardness" or the like? |
||
| number: '0' # How many constraints there are in total, fixed numbers or range, scalable if it can by any value | ||
| boundary/box: 'yes' # Is the problem box-constrained yes, no, or some (if only some variables are bounded) | ||
| permutation: 'no' # Variables must be a permutation yes, no, or some | ||
| dynamic: # Dynamic properties, e.g., changing objective functions | ||
| present: 'no' # Enter, yes, no, or optional | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Again I think the field might be economised ;) |
||
| types: '' # Types of dynamic behaviour that are present | ||
| noise: # Noise on the objective function(s) | ||
| present: 'no' # Enter, yes, no, or optional | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same as for "dynamic". |
||
| types: '' # Types of noise that are present | ||
| modality: | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's complex ... I agree that modality is important but there is way more to it than just "uni" or "multi" and just this doesn't really help IMO. |
||
| types: 'unimodal, multimodal' # Enter all that are present | ||
| evaluations: | ||
| multi-fidelity: 'no' # Whether evaluations can be performed at multiple fidelities, yes, no, or optional | ||
| partial possible: 'no' # Whether evaluation of subset(s) of decision variables are possible | ||
| independent objectives: 'no' # Can objective functions be evaluated independently of each other yes or no | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Or "some" in case of different groups of objectives who share evaluation properties? |
||
| reference: | ||
| links: | ||
| - https://doi.org/10.1080/10556788.2020.1808977 # URL to source or closely related research article, preferably DOI based. Add a new line below for each additional article. | ||
| authors: 'Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, Dimo Brockhoff' | ||
| contact person: '' | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. a possible value could be "any" in the sense of "any of the above" but leaving this field empty is confusing. (please don't contact us?) |
||
| implementations: | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Plural? Can people leave multiple implementations? Do we want this? |
||
| - name: COCO # Name of the tool/library/package | ||
| link: https://github.com/numbbo/coco | ||
| languages: 'C/C++, Python, Java, Matlab/Octave' # Programming languages the implementation can be used with | ||
| evaluation time: 'less than a second' # Choose: 'less than a second', 'seconds', 'minutes', 'hours', 'days' | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is it clear that we refer to a single fitness/objective function evaluation time here? (not a whole opt. run) And I guess we expect the maximum in case of "independent objectives" (different evaluation times) above? Maybe better put a field "time" in the group of "evaluations"? I don't know ... |
||
| specific requirements: 'no' # Can be things like memory/GPU requirements, or the need for a specific simulator. Enter 'no' if none. | ||
| source: | ||
| origin: 'artificial' # Is the problem artificial, real-world, or real-world-like | ||
| real-world: # In case it is real-world (like), provide details | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We don't expect any entry here, right? |
||
| domain: '' # E.g., automotive | ||
| open/closed: '' # Is the problem open source yes or no | ||
| textual description: | ||
| general info: '' | ||
| motivation: 'evaluate algorithm performance for typical difficulties that occur in continuous problems' | ||
| challenge/key characteristics: '' | ||
| limitations: '' | ||
| example links: '' # URLs to examples where it was used (e.g., scientific articles) - Note: URLs to where it was proposed should go under the 'reference' field. | ||
| other info: '' | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
General comment:
I'd prefer to have the number of fields as small as possible to just not discourage people from filling the template. That's why I question every entry and try to keep the template as easy as possible.
I also think the template should not address optimisation experts but domain experts who might not be aware of our terminology.