Skip to content

Fix dynamic inference batching for variable-length tokens - #69

Open
zjn20030811 wants to merge 1 commit into
OpenGalaxea:mainfrom
zjn20030811:fix/pad-dynamic-token-batches
Open

Fix dynamic inference batching for variable-length tokens#69
zjn20030811 wants to merge 1 commit into
OpenGalaxea:mainfrom
zjn20030811:fix/pad-dynamic-token-batches

Conversation

@zjn20030811

Copy link
Copy Markdown

Summary

  • right-pad variable-length input_ids, labels, and attention_mask before default collation
  • preserve tokenizer, loss-mask, and attention padding values used by the model input preprocessor
  • add regression coverage for dynamic token batches and the documented batched inference path

Fixes #67.

Validation

  • pytest -q tests/test_serve_policy_dynamic_batching.py (2 passed)
  • python -m py_compile src/g05/utils/data/data_utils.py tests/test_serve_policy_dynamic_batching.py
  • git diff --check

@zjn20030811

Copy link
Copy Markdown
Author

Implemented and validated the fix for #67.

collate_fn_pad_sequences now right-pads each 1-D token field independently before default_collate: input_ids uses the supplied tokenizer pad id, labels uses -100, and attention_mask uses 0. Equal-length fields and non-token fields remain unchanged. The regression test checks all three pad values, and the existing variable-length batched-vs-sequential inference test passes.

Validation: pytest -q tests/test_serve_policy_dynamic_batching.py (2 passed), py_compile, and git diff --check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

collate_fn_pad_sequences never pads, so test_serve_policy_dynamic_batching fails on clean main

1 participant