test: add 7 tests for PromptBakingTrainer.training_step - #26
Conversation
ReviewGood test coverage for Conflict with #17: This PR mocks Please update the tests to mock Will merge once updated for the batched generation interface. |
Covers: - delegates to super when responses already present - calls _generate_trajectory num_trajectories times - populates inputs['responses'] before delegating - filters blank/whitespace-only trajectories - returns zero-loss tensor when all trajectories blank - warns when trajectory count exceeds threshold (64) - no warning when count is at or below threshold Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PR marksverdhei#17 replaces the _generate_trajectory loop in training_step with a single _generate_trajectories_batched call returning (msg, response) tuples. Update all test mocks accordingly so tests remain valid after marksverdhei#17 merges. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
c4884c2 to
a12bc9c
Compare
marksverdhei
left a comment
There was a problem hiding this comment.
The updated mocks targeting _generate_trajectories_batched look correct. Clean test structure covering delegation, blank filtering, zero-loss path, and warning thresholds.
However, this PR depends on #17 (which introduces _generate_trajectories_batched). #17 currently has merge conflicts with main after #20 was merged and needs a rebase first. Once #17 lands, this is ready to merge. Holding approval until then.
marksverdhei
left a comment
There was a problem hiding this comment.
Reviewed. Good decision reverting to mock _generate_trajectory (current main API) instead of coupling to #17's unreleased batched method. Tests cover the key training_step branches: super delegation with existing responses, trajectory generation, blank filtering, zero-loss on all-blank, and warning thresholds. LGTM.
Summary
Adds test coverage for
PromptBakingTrainer.training_step, the method that generates trajectories on-the-fly when no precomputed responses are in the batch.Tests added (
tests/test_training_step.py)test_training_step_with_existing_responses_delegates_to_super— wheninputs["responses"]is already populated, confirmssuper().training_step()is called without generating new trajectoriestest_training_step_generates_trajectories_when_no_responses— verifies_generate_trajectoryis called exactlynum_trajectoriestimes per prompttest_training_step_populates_inputs_with_generated_responses— checks that generated responses are written intoinputs["responses"]before callingsuper()test_training_step_filters_blank_trajectories— blank/whitespace-only responses are excluded from the batchtest_training_step_returns_zero_when_all_trajectories_blank— all-blank responses → returns a zero-loss tensor withrequires_grad=True,super()not calledtest_training_step_warns_when_many_trajectories— a WARNING is logged whenlen(user_messages) * num_trajectories > 64test_training_step_no_warn_below_threshold— no warning emitted when at or below the 64-trajectory thresholdTest plan
All 7 tests pass on CPU with tiny GPT-2 + LoRA (same fixture as existing
test_trainer.py).🤖 Generated with Claude Code