Q: Would "Adversarial Policy" break the top players?

[TomLucidor]

This is a very weird question that I would like to note. If phenotype convergence (behavioral convergence) happen but not genotype (code structure), then it means everyone could very well be making the same mistakes, and new agents (or older discarded mutations) can come back and beat the elites of the current crowd. The "Adversarial Policy" papers like to talk about non-transitivity, how could that be resolved?

[akarshkumar0101]

It depends on the domain, but I'm not sure it can always be resolved. For example some domains naturally have rock-paper-scissors dynamics. I believe Core War is one of them! You can create specialized policies to exploit most policies. However, these specialist policies may not be general with respect to a wider set of other policies.

[TomLucidor]

Here is a thought, maybe instead of a single point of convergence, the algorithm can show multiple optimization points with non-transitive dynamics between them, but they are are generally better than most policies. How can that be done?

[akarshkumar0101]

Yea that would be cool! I think ideas from quality-diversity (MAP-Elites style) algorithms could be very useful for that.
Maybe the behavior space you search over is the phenotype vector from our paper!