I noticed that in actor_loss, the Q‑guidance term (q_loss) is computed by taking argmax over the actor’s logits and then one‑hot encoding the result before passing it to the Q‑network. Since argmax is non‑differentiable, the gradient from q_loss would not propagate back to the actor's parameters. This could mean the policy is not actually being updated to maximize Q‑values, and only bc_flow_loss and distill_loss affect the actor.
Could you please confirm whether this is intentional, or if there might be a missing gradient estimator in this part of the code?
Thank you for your time
I noticed that in actor_loss, the Q‑guidance term (q_loss) is computed by taking argmax over the actor’s logits and then one‑hot encoding the result before passing it to the Q‑network. Since argmax is non‑differentiable, the gradient from q_loss would not propagate back to the actor's parameters. This could mean the policy is not actually being updated to maximize Q‑values, and only bc_flow_loss and distill_loss affect the actor.
Could you please confirm whether this is intentional, or if there might be a missing gradient estimator in this part of the code?
Thank you for your time