[rl] Fix batch invariant logprob calculation by forcing vllm use trainer's function#3629
Open
wwwjn wants to merge 2 commits into
Open
[rl] Fix batch invariant logprob calculation by forcing vllm use trainer's function#3629wwwjn wants to merge 2 commits into
wwwjn wants to merge 2 commits into
Conversation
meta-cla
Bot
added
the
CLA Signed
tianyu-l
reviewed
Jun 11, 2026
Comment on lines
+120
to
+121
| # Broadcast 2D logits over token_ids' num_logprobs columns so the | ||
| # trainer's compute_logprobs (one token per position) works unchanged. |
Contributor
There was a problem hiding this comment.
say more. I'm confused by when we need to expand anything. The docstring of https://github.com/pytorch/torchtitan/blob/main/torchtitan/experiments/rl/actors/trainer.py#L45 is not great, either. But at least I could see the the BSV stands for bsz, seqlen, vocab size.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rational: The logp calculation should be more transparent to us