Joe's Chart on GALE2

[vivekaxl]

[vivekaxl]

My comment on these charts are: Can't really say anything about how good or bad GALE2 is.
@timm @ginfung What do you think?

[vivekaxl]

Another idea is to tune how many points should be mutated or generated. For these results 25% of the population is mutated. We can try working with different ratios.

[WeiFoo]

can you make your legend consistent with algorithms?

[vivekaxl]

Old Results: #27

[vivekaxl]

@WeiFoo As requested. Great suggestion! Thanks

[WeiFoo]

It's better and much clear now.

my read of these pics is no significant difference in GALE and GALE2, except for GALE seems better than GALE2 in EIS, cellpone, web_portal, eshop, in terms of spread.

suggestions for your next expriment if you have, but now it's totally fine:

keep your order of x-axis, legend, and algorithms in for each data set, consistent
for example,

• order all legends like, MOEAD, GALE, GALE2, DE, NSGA3 from top to bottom for all pictures.
• order algorithms for each data set, from left to right, like MOEAD, GALE, GALE2, DE, NSGA3 across all pictures.
• order data sets for different goals(hypervolume and spread), like EIS, cellphone, Web_Portal, eshop for the both bottom two pics.

[timm]

lose numbers?
runtimes?
#evals?

[timm]

for the spread and hypervolume stuff, are these means or medians (want medians).

and what is the IQR?

and for the product line stuff, is this with 5 goals?

[timm]

And the loss values?

[timm]

hey @vivekaxl,
Is @ginfung smiling at the nasty runtimes in the feature models. NOW he has something to try his constraint propagation tricks on

hey @ginfung:

• for a quick tutorial on constraint satisfaction, please see http://aima.cs.berkeley.edu/2nd-ed/newchap05.pdf
• by the way, that chapter is from the SECOND edition of norvig's AI text; fyi-there is a third edition from 2010 http://aima.cs.berkeley.edu/ that you should look up in the library, see if this chapter has added any more heuristics .
• I take it you were showing me a min-conflicts trick the other day?
• btw, once you debug faster GALE on medium sized stuff (e.g. web_portal) then things get nasty since you have to work then on the linux kernel stuff

[timm]

hey @vivekaxl
please confirm: LESS spread is better and MORE hypervolume is worse?

[timm]

hey @vivekaxl
looks like were going to have to defend GALE/GALE2 on the basis of "darn fast, results not too bad".

for each model:
   for each objective:      
         let pop1 be all the objective scores in baseline across all N repeats with  sd of s
         let a small effect be s*0.4 (see https://goo.gl/9t7wYH)
         let pop2 be all the objective scores in last generation across all N repeats with mean of n1
         let pop3 be GALE2's  objective scores in last generation across all N repeats with mean of n2
         generate a table show the difference in the n1-n2.
              if n1-n2 is less than a small effect, show n1-n2 in gray
              otherwise, show in black

add a summary table counting up the percent gray (where GALE2 was no worse than a small effect different to another optimizer)

[ginfung]

@vivekaxl runtime for EIS/eshop, GALE>>DE ? This confused me. Other tendency are the same as I did.