freeze and band #67
freeze and band #67
Conversation
| discriminator_optimizer.zero_grad() | ||
| self.manual_backward(discriminator_loss["loss"], retain_graph=True) | ||
| discriminator_optimizer.step() | ||
| if self.global_step >= 10000: |
There was a problem hiding this comment.
I know this draft, but can we make this configurable / enableable?
There was a problem hiding this comment.
worse than draft, this hack and yes, we could/should
There was a problem hiding this comment.
A funny note / observation:
global steps are per optimizer step; so with three datasets, it shows up in the graph as 10000 / 3 😅
There was a problem hiding this comment.
we should put that in a comment in the docstring
| discriminator_optimizer.zero_grad() | ||
| self.manual_backward(discriminator_loss["loss"], retain_graph=True) | ||
| discriminator_optimizer.step() | ||
| if self.global_step >= self.num_frozen_steps: |
There was a problem hiding this comment.
I was looking at the loss graphs, and they didn't seem quite right, so I went back and compared this to how it's done in bigvgan:
https://github.com/NVIDIA/BigVGAN/blob/main/train.py#L486
I their implementation (different than here), the discriminator_optimizer.zero_grad() is always called, but backwards and step are optionally called.
I think the way you have it here we may be implicitly stepping the discriminator through the generator because it's not zeroing the gradients.
No description provided.