Skip to content

Guard short token datasets in ANE and dynamic training paths#38

Open
TastyHeadphones wants to merge 1 commit intomaderix:mainfrom
TastyHeadphones:tastyheadphones/short-dataset-guards-ane-dynamic
Open

Guard short token datasets in ANE and dynamic training paths#38
TastyHeadphones wants to merge 1 commit intomaderix:mainfrom
TastyHeadphones:tastyheadphones/short-dataset-guards-ane-dynamic

Conversation

@TastyHeadphones
Copy link
Contributor

Summary

  • add a token dataset length guard in training/train_large_ane.m
  • add the same guard in training/training_dynamic/train.m
  • fail early with a clear error when the dataset is too short for one (input,target) training window

Why

Both paths sample with:

max_pos = n_tokens - SEQ - 1

When n_tokens <= SEQ + 1, this unsigned subtraction can underflow, producing a huge random range and potentially leading to out-of-bounds reads.

Validation

  • make -C training train_large_ane
  • make -C training/training_dynamic train

dev-erik added a commit to dev-erik/ANE that referenced this pull request Mar 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant