Train the PPO

[WeihangGuo]

I ran python ac_solver/agents/ppo.py --exp-name demo --wandb-log to train the PPO, but it doesn’t seem to be solving any of the problems. Did I do something wrong?

wandb: Run history:
wandb:                 charts/episode ▁▁▂▂▂▂▃▃▃▄▄▄▅▅▅▅▆▆▆▇▇▇▇██
wandb:             charts/global_step ▁▁▂▂▂▂▃▃▃▄▄▄▅▅▅▅▆▆▆▇▇▇▇██
wandb:          charts/highest_solved ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:           charts/learning_rate ██▇▇▇▇▆▆▆▅▅▅▅▄▄▄▃▃▃▂▂▂▂▁▁
wandb: charts/normalized_lengths_mean ▁▄▅▆▆▇▇▇▇▇▇▇▇▇▇▇▇████████
wandb: charts/normalized_returns_mean █▅▄▃▃▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁
wandb:                  charts/solved ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:                charts/unsolved ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:          debug/advantages_mean ▅█▄▅▁▃▃▄▃▃▃▃▃▄▃▄▂▄▄▅▄▄▄▄▃
wandb:           debug/advantages_std █▅▄▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁
wandb:               losses/approx_kl █▄▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:                losses/clipfrac ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:            losses/entropy_loss █▇▇▆▅▅▄▄▄▃▃▃▃▂▂▂▂▁▂▂▁▂▂▂▂
wandb:      losses/explained_variance ▇█▇██▄▄▄▅▃▂▃▄▄▄▅▃▄▁▄▂▄▂▄▄
wandb:             losses/policy_loss ▁▇▁▇▅▅▆▆▇▆█▅▅▇▆▆▆▇▆▇▆▆▆▆▆
wandb:              losses/value_loss █▄▃▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁

[shehper]

Hi, python ac_solver/agents/ppo.py will only train the PPO agent for some set of hyperparameters defined in args. But the hyperparameters we used to train agents in the paper are different. For example, you could try the hyperparameters given in Appendix A of https://arxiv.org/pdf/2408.15332

I hope that you start seeing the number of solved presentations go up. In case you do not, please let me know and I will look into it.