Hello,
The main input sequence x_BLC is constructed by concatenating:
A prefix sos derived from the text context, expanded to length first_l (300).
The embedded ground truth image tokens x_BLCv_wo_first_l (length L - first_l = 9151).
This results in x_BLC having a sequence length of 9451. Shape: (B, 9451, C).
However, other components seem based on a different sequence length:
The level index buffer self.lvl_1L appears to be initialized with length L + context_token - 1 = 9451 + 300 - 1 = 9750.
The random_masking function generates a boolean mask (mask_full_output internally) also corresponding to this 9750 length.
The mask tuple used for indexing x_BLC is derived from this 9750-length boolean mask.
Conflict: The code attempts to index x_BLC (length 9451) using the mask tuple, whose sequence indices can range up to 9749 (derived from the 9750 length).
My debugging confirms this: x_BLC.shape is (B, 9451, C), but the mask tuple contains sequence indices with max() values greater than or equal to 9451 (e.g., up to 9743 in my test).
This leads to an indexing-error; Specifically, (RuntimeError: CUDA error: device-side assert triggered) during the indexing step;
x_BLC = x_BLC[mask].reshape(B, -1, x_BLC.shape[-1])
There appears to be an inconsistency in the sequence lengths used for the main data tensor (x_BLC, length 9451) and the tensors used for masking/level-indices (mask_full, lvl_1L, length 9750). This prevents the model from running the forward pass correctly.
Could you please clarify the intended sequence length for these operations and advise on how to resolve this discrepancy? It seems the length calculation L + context_token - 1 might need adjustment to align with x_BLC's length L.
Hello,
The main input sequence
x_BLCis constructed by concatenating:A prefix sos derived from the text context, expanded to length
first_l(300).The embedded ground truth image tokens
x_BLCv_wo_first_l(length L - first_l = 9151).This results in
x_BLChaving a sequence length of 9451. Shape: (B, 9451, C).However, other components seem based on a different sequence length:
The level index buffer
self.lvl_1Lappears to be initialized with length L + context_token - 1 = 9451 + 300 - 1 = 9750.The random_masking function generates a boolean mask (mask_full_output internally) also corresponding to this 9750 length.
The mask tuple used for indexing x_BLC is derived from this 9750-length boolean mask.
Conflict: The code attempts to index
x_BLC(length 9451) using the mask tuple, whose sequence indices can range up to 9749 (derived from the 9750 length).My debugging confirms this: x_BLC.shape is (B, 9451, C), but the mask tuple contains sequence indices with max() values greater than or equal to 9451 (e.g., up to 9743 in my test).
This leads to an indexing-error; Specifically, (RuntimeError: CUDA error: device-side assert triggered) during the indexing step;
x_BLC = x_BLC[mask].reshape(B, -1, x_BLC.shape[-1])There appears to be an inconsistency in the sequence lengths used for the main data tensor (x_BLC, length 9451) and the tensors used for masking/level-indices (mask_full, lvl_1L, length 9750). This prevents the model from running the forward pass correctly.
Could you please clarify the intended sequence length for these operations and advise on how to resolve this discrepancy? It seems the length calculation L + context_token - 1 might need adjustment to align with x_BLC's length L.