Thanks for the fascinating paper and for open-sourcing the data generation code — really interesting idea. We tried reproducing the Appendix D.2/D.3 binary classification experiment and ran into a result that diverges quite a bit from what's reported, so wanted to share our setup in case it's useful (or in case we're missing something on our end).
Summary
We attempted to reproduce the VJEPA-2 full fine-tuning binary classification experiment described in Appendix D.2/D.3, following the paper's stated hyperparameters as closely as possible. The paper reports near-chance performance: val accuracy "remained at exactly 50% for the majority of training" with loss plateauing near ln 2 ≈ 0.693 (D.2), and Table 1 reports training accuracy of only 52.3–56.1% across the 100/400/1,100-video scales (D.3). In our reproduction, the model instead converges to ~100% train and validation accuracy. We describe our exact setup below, several rounds of confound-elimination we went through (the result still converges each time), and specific ambiguities we found in the paper and the public repository that may explain the discrepancy.
Our setup
- Checkpoint:
facebook/vjepa2-vitl-fpc64-256 (HuggingFace transformers, VJEPA2ForVideoClassification / VJEPA2VideoProcessor), randomly-initialized attentive pooler + linear classifier head, entire model unfrozen (full fine-tuning, no layer-wise LR decay).
- Optimization: AdamW, lr=1e-4, weight_decay=0.01, 100 epochs, batch_size=4 per GPU (effective batch 32 across 8 GPUs via DDP), bf16 autocast + gradient checkpointing.
- Dataset generation: built on top of the public
generate_dynamic_videos.py / noise_generator.py from this repository. Fixed rendering parameters: 960×540, 60 fps, 5.0 s/clip (300 raw frames), noise density 0.5, speckle size 1×1 px, speed 2 px/raw-frame, direction horizontal, movement_type="static" (object mask fixed at frame center).
- Positive (label=1): one of 62 SpookyBench silhouette images embedded via
generate_image_video, use_same_noise=False (two independently-generated noise fields for foreground/background, per Algorithm 1's Nbg/Nfg description).
- Negative (label=0): no equivalent generator exists anywhere in the public repository, so we implemented our own from Appendix D.1's text description ("same noise generation pipeline but without any foreground mask applied, i.e., uniform random noise with no embedded temporal pattern"). Final version: an independently re-sampled random noise frame drawn fresh every raw frame (no persistence across frames, no directional roll).
- Encoding: H.264,
crf=0 (lossless) — see "confounds ruled out" below for why this mattered.
- Scales: 100 / 400 / 1,100 videos (nested subsets, 80/20 train/val split, class-balanced).
- Model input: 64 contiguous raw frames (stride=1) sampled from the 300-frame clip (random start for training, centered window for validation), then processed with
VJEPA2VideoProcessor (shortest_edge=292, center-crop 256×256, ImageNet normalization).
Result
Scale=100 (80 train / 220 val clips, class-balanced): val accuracy reaches 1.0000 by ~epoch 20 and stays there through epoch 90+ (run still in progress at time of writing). Runs at scale=400 and scale=1,100 under identical settings are also in progress.
Wrap-up & Questions for the authors
We'd love to hear your thoughts on this — is there something in our setup that might explain the gap (e.g. the negative-class generation, frame-sampling strategy, or encoding settings)? If possible, could you share the exact configuration/scripts used for the Appendix D.2/D.3 experiment (especially the negative-class generator, which we couldn't find in the public repo)? That would help us pin down where the discrepancy is coming from.
Thanks again for the great paper — really appreciate you taking the time to look into this!
Thanks for the fascinating paper and for open-sourcing the data generation code — really interesting idea. We tried reproducing the Appendix D.2/D.3 binary classification experiment and ran into a result that diverges quite a bit from what's reported, so wanted to share our setup in case it's useful (or in case we're missing something on our end).
Summary
We attempted to reproduce the VJEPA-2 full fine-tuning binary classification experiment described in Appendix D.2/D.3, following the paper's stated hyperparameters as closely as possible. The paper reports near-chance performance: val accuracy "remained at exactly 50% for the majority of training" with loss plateauing near ln 2 ≈ 0.693 (D.2), and Table 1 reports training accuracy of only 52.3–56.1% across the 100/400/1,100-video scales (D.3). In our reproduction, the model instead converges to ~100% train and validation accuracy. We describe our exact setup below, several rounds of confound-elimination we went through (the result still converges each time), and specific ambiguities we found in the paper and the public repository that may explain the discrepancy.
Our setup
facebook/vjepa2-vitl-fpc64-256(HuggingFacetransformers,VJEPA2ForVideoClassification/VJEPA2VideoProcessor), randomly-initialized attentive pooler + linear classifier head, entire model unfrozen (full fine-tuning, no layer-wise LR decay).generate_dynamic_videos.py/noise_generator.pyfrom this repository. Fixed rendering parameters: 960×540, 60 fps, 5.0 s/clip (300 raw frames), noise density 0.5, speckle size 1×1 px, speed 2 px/raw-frame, direction horizontal,movement_type="static"(object mask fixed at frame center).generate_image_video,use_same_noise=False(two independently-generated noise fields for foreground/background, per Algorithm 1'sNbg/Nfgdescription).crf=0(lossless) — see "confounds ruled out" below for why this mattered.VJEPA2VideoProcessor(shortest_edge=292, center-crop 256×256, ImageNet normalization).Result
Scale=100 (80 train / 220 val clips, class-balanced): val accuracy reaches 1.0000 by ~epoch 20 and stays there through epoch 90+ (run still in progress at time of writing). Runs at scale=400 and scale=1,100 under identical settings are also in progress.
Wrap-up & Questions for the authors
We'd love to hear your thoughts on this — is there something in our setup that might explain the gap (e.g. the negative-class generation, frame-sampling strategy, or encoding settings)? If possible, could you share the exact configuration/scripts used for the Appendix D.2/D.3 experiment (especially the negative-class generator, which we couldn't find in the public repo)? That would help us pin down where the discrepancy is coming from.
Thanks again for the great paper — really appreciate you taking the time to look into this!