Having Trouble Recreating Results in the Paper

[yulun-rayn]

Hi,

Thanks for releasing the source code! I did a quick run on ETTh1 using the exact configs in here and did not manage to create the results in the paper. I got 0.438 MSE and 0.432 MAE on PL192. Run met early stopping criteria on epoch 40. Before I proceed to run more experiments, could you clarify if the configs in here are the same as what were used for the paper? Am I missing something? Thanks a lot!

[Sachithx]

Thank you, @yulun-rayn, for your interest in our work! 🙏
The issue you mentioned has already been discussed and resolved in this thread. Please have a look there for the detailed explanation.
If you continue to encounter similar issues or have any other questions, feel free to open a new issue. I’d be happy to look into it and help further.

[Sachithx]

Below is the resulting outcome based on the scripts in the repository.

[yulun-rayn]

Thanks! Does resolved mean if I run the latest main branch after your PR in that thread, I would be able to reproduce the results? Because I am running on the latest commit and this is what I got.

[yulun-rayn]

@Sachithx is this plot you show run on the exact same script in the latest commit? Like the author in the other thread said, I thank you for publishing this work and am looking forward to cite this work if I can reproduce the results.

[Sachithx]

@yulun-rayn I just re-ran the script and was able to reproduce the same outcome as shown in the figure above. However, this could be a bit confusing. Would you mind sharing the environment dependencies and package versions you’re using?
I previously noticed that some libraries didn’t seem to function correctly when running the dynamic patching module, and different GPUs also produced slightly different MSE/MAE values.

[yulun-rayn]

I don't think how the threshold determines patch size would be affected by any dependencies or GPUs? The current etth1 entropy checkpoint & settings just give me all false patch_start_mask and then max_patch_length makes all patch size 12.

Edit: nevermind that was during get_model_complexity_info, in actual forward it looks fine. So I don't know why the discrepancy between results are that big. I'll send you the dependencies shortly

[yulun-rayn]

Is it possible that the entropy model you're using is not the same as the one currently resides in the repo?

[Sachithx]

I don't think how the threshold determines patch size would be affected by any dependencies or GPUs? The current etth1 entropy checkpoint & settings just give me all false patch_start_mask and then max_patch_length makes all patch size 12.

Edit: nevermind that was during get_model_complexity_info, in actual forward it looks fine. So I don't know why the discrepancy between results are that big. I'll send you the dependencies shortly

Dynamic patch boundary works fine, I printed for your previous query; that's fine.

[Sachithx]

Is it possible that the entropy model you're using is not the same as the one currently resides in the repo?

Yes, I took a fresh clone and executed all. Local-global repositories are properly updated. I encourage you to test in a new environment with below dependencies.

absl-py==2.3.1
accelerate==1.10.1
aiohappyeyeballs==2.6.1
aiohttp==3.13.1
aiosignal==1.4.0
annotated-types==0.7.0
antlr4-python3-runtime==4.9.3
anyio==4.9.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==3.0.0
async-lru==2.0.5
async-timeout==5.0.1
attrs==25.3.0
babel==2.17.0
beautifulsoup4==4.13.3
bleach==6.2.0
certifi==2025.10.5
cffi==1.17.1
charset-normalizer==3.4.1
chronos-forecasting==1.5.3
click==8.3.0
comm==0.2.2
contourpy==1.3.2
cycler==0.12.1
datasets==4.2.0
debugpy==1.8.13
decorator==5.2.1
defusedxml==0.7.1
dill==0.4.0
evaluate==0.4.6
exceptiongroup==1.2.2
executing==2.2.0
fastjsonschema==2.21.1
filelock==3.20.0
fonttools==4.60.1
fqdn==1.5.1
frozenlist==1.8.0
fsspec==2025.9.0
gitdb==4.0.12
GitPython==3.1.45
gluonts==0.16.2
grpcio==1.75.1
h11==0.14.0
hf-xet==1.1.10
httpcore==1.0.7
httpx==0.28.1
huggingface-hub==0.35.3
idna==3.11
ipykernel==6.29.5
ipython==8.34.0
isoduration==20.11.0
jedi==0.19.2
Jinja2==3.1.6
joblib==1.5.2
json5==0.11.0
jsonpointer==3.0.0
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
jupyter-events==0.12.0
jupyter-lsp==2.2.5
jupyter_client==8.6.3
jupyter_core==5.7.2
jupyter_server==2.15.0
jupyter_server_terminals==0.5.3
jupyterlab==4.3.6
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.3
kiwisolver==1.4.9
Markdown==3.9
markdown-it-py==4.0.0
MarkupSafe==3.0.3
matplotlib==3.10.7
matplotlib-inline==0.1.7
mdurl==0.1.2
mistune==3.1.3
mpmath==1.3.0
multidict==6.7.0
multiprocess==0.70.16
narwhals==2.8.0
nbclient==0.10.2
nbconvert==7.16.6
nbformat==5.10.4
nest-asyncio==1.6.0
networkx==3.4.2
notebook==7.3.3
notebook_shim==0.2.4
numpy==2.1.3
nvidia-cublas-cu12==12.8.4.1
nvidia-cuda-cupti-cu12==12.8.90
nvidia-cuda-nvrtc-cu12==12.8.93
nvidia-cuda-runtime-cu12==12.8.90
nvidia-cudnn-cu12==9.10.2.21
nvidia-cufft-cu12==11.3.3.83
nvidia-cufile-cu12==1.13.1.3
nvidia-curand-cu12==10.3.9.90
nvidia-cusolver-cu12==11.7.3.90
nvidia-cusparse-cu12==12.5.8.93
nvidia-cusparselt-cu12==0.7.1
nvidia-nccl-cu12==2.27.5
nvidia-nvjitlink-cu12==12.8.93
nvidia-nvshmem-cu12==3.3.20
nvidia-nvtx-cu12==12.8.90
omegaconf==2.3.0
overrides==7.7.0
packaging==24.2
pandas==2.3.3
pandocfilters==1.5.1
parso==0.8.4
pillow==12.0.0
platformdirs==4.3.7
plotly==6.3.1
prometheus_client==0.21.1
prompt_toolkit==3.0.50
propcache==0.4.1
protobuf==6.33.0
psutil==7.1.0
ptflops==0.7.5
pure_eval==0.2.3
pyarrow==21.0.0
pycparser==2.22
pydantic==2.12.3
pydantic_core==2.41.4
Pygments==2.19.1
pyparsing==3.2.5
python-dateutil==2.9.0.post0
python-json-logger==3.3.0
pytz==2025.2
PyYAML==6.0.3
pyzmq==26.3.0
referencing==0.36.2
regex==2025.9.18
requests==2.32.3
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rich==14.2.0
rpds-py==0.24.0
safetensors==0.6.2
scikit-learn==1.7.2
scipy==1.15.3
seaborn==0.13.2
Send2Trash==1.8.3
sentry-sdk==2.42.0
six==1.17.0
smmap==5.0.2
sniffio==1.3.1
soupsieve==2.6
stack-data==0.6.3
sympy==1.14.0
tensorboard==2.20.0
tensorboard-data-server==0.7.2
terminado==0.18.1
threadpoolctl==3.6.0
tinycss2==1.4.0
tokenizers==0.22.1
tomli==2.2.1
toolz==0.12.1
torch==2.9.0
tornado==6.4.2
tqdm==4.67.1
traitlets==5.14.3
transformers==4.57.1
triton==3.5.0
types-python-dateutil==2.9.0.20241206
typing-inspection==0.4.2
typing_extensions==4.15.0
tzdata==2025.2
uri-template==1.3.0
urllib3==2.5.0
wandb==0.22.2
wcwidth==0.2.13
webcolors==24.11.1
webencodings==0.5.1
websocket-client==1.8.0
Werkzeug==3.1.3
xxhash==3.6.0
yarl==1.22.0

[yulun-rayn]

and the datasets you used are the exact same ones as these? https://github.com/zhouhaoyi/ETDataset/tree/main/ETT-small

[Sachithx]

and the datasets you used are the exact same ones as these? https://github.com/zhouhaoyi/ETDataset/tree/main/ETT-small

yes.
Can you check with 5 different seeds? @yulun-rayn

[yulun-rayn]

Ok I will. Thanks for your help! I just tested on another device. Just to make sure I'm looking at the right number, are the MSE, MAE, RSE printouts after early stopping not the ones to look at? I guess if you look at lowest test loss among all epochs, you can get as low as 0.430, which is better than the printouts even though not at the level of those reported in paper. Also, if we record best test loss in any epoch, how do we get the corresponding MAE?

Epoch: 40 cost time: 15.153807640075684
Epoch: 40, Steps: 65 | Train Loss: 0.4155461 Vali Loss: 1.0248138 Test Loss: 0.4295133
EarlyStopping counter: 20 out of 20
Early stopping
>>>>>>>testing: EntroPE_ETTh1_SL96_PL192_GL1_EL1_DL1_GD16_DD16_ED16_GH4_EH4_DH4<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
test 2689
MSE: 0.43724003434181213, MAE: 0.4320778250694275, RSE: 0.6279306411743164
wandb: 
wandb: Run history:
wandb:            batch_loss ▇█▇▇▅▅▄▄▃▂▂▃▂▃▂▂▂▁▃▁▂▂▂▂▂▂▂▂▂▁▂▂▂▂▂▁▁▂▁
wandb:         best_val_loss ▁
wandb:  early_stopping_epoch ▁
wandb:                 epoch ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb:        epoch_duration █▃▂▂▂▂▂▂▂▂▂▁▂▂▁▁▂▂▂▂▃▂▂▂▃▂▂▂▂▂▂▂▂▄▃▅▄▄▄▄
wandb:       epoch_test_loss █▇▆▅▄▃▃▂▂▂▂▂▂▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:      epoch_train_loss ██▇▇▆▅▄▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:        epoch_val_loss █▇▆▅▄▃▃▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb: estimated_time_left_s █▇▇▇▆▆▆▆▆▆▅▅▅▅▄▄▄▄▄▄▄▃▃▃▃▂▂▂▂▂▂▃▂▁▁▂▁▁▁
wandb:         learning_rate ▁▁▁▁▁▁▂▂▂▂▂▂▃▃▃▃▄▄▄▅▅▅▅▆▆▆▆▇▇▇▇▇▇██████
wandb:                   +13 ...
wandb: 
wandb: Run summary:
wandb:            batch_loss 0.41733
wandb:         best_val_loss 1.00811
wandb:  early_stopping_epoch 40
wandb:                 epoch 40
wandb:        epoch_duration 15.15381
wandb:       epoch_test_loss 0.42951
wandb:      epoch_train_loss 0.41555
wandb:        epoch_val_loss 1.02481
wandb: estimated_time_left_s 146115.71646
wandb:         learning_rate 0.001
wandb:                   +13 ...
wandb: 

[Sachithx]

Ok I will. Thanks for your help! I just tested on another device. Just to make sure I'm looking at the right number, are the MSE, MAE, RSE printouts after early stopping not the ones to look at? I guess if you look at lowest test loss among all epochs, you can get as low as 0.430, which is better than the printouts even though not at the level of those reported in paper. Also, if we record best test loss in any epoch, how do we get the corresponding MAE?

Epoch: 40 cost time: 15.153807640075684
Epoch: 40, Steps: 65 | Train Loss: 0.4155461 Vali Loss: 1.0248138 Test Loss: 0.4295133
EarlyStopping counter: 20 out of 20
Early stopping
>>>>>>>testing: EntroPE_ETTh1_SL96_PL192_GL1_EL1_DL1_GD16_DD16_ED16_GH4_EH4_DH4<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
test 2689
MSE: 0.43724003434181213, MAE: 0.4320778250694275, RSE: 0.6279306411743164
wandb: 
wandb: Run history:
wandb:            batch_loss ▇█▇▇▅▅▄▄▃▂▂▃▂▃▂▂▂▁▃▁▂▂▂▂▂▂▂▂▂▁▂▂▂▂▂▁▁▂▁
wandb:         best_val_loss ▁
wandb:  early_stopping_epoch ▁
wandb:                 epoch ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███
wandb:        epoch_duration █▃▂▂▂▂▂▂▂▂▂▁▂▂▁▁▂▂▂▂▃▂▂▂▃▂▂▂▂▂▂▂▂▄▃▅▄▄▄▄
wandb:       epoch_test_loss █▇▆▅▄▃▃▂▂▂▂▂▂▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:      epoch_train_loss ██▇▇▆▅▄▃▃▃▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:        epoch_val_loss █▇▆▅▄▃▃▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb: estimated_time_left_s █▇▇▇▆▆▆▆▆▆▅▅▅▅▄▄▄▄▄▄▄▃▃▃▃▂▂▂▂▂▂▃▂▁▁▂▁▁▁
wandb:         learning_rate ▁▁▁▁▁▁▂▂▂▂▂▂▃▃▃▃▄▄▄▅▅▅▅▆▆▆▆▇▇▇▇▇▇██████
wandb:                   +13 ...
wandb: 
wandb: Run summary:
wandb:            batch_loss 0.41733
wandb:         best_val_loss 1.00811
wandb:  early_stopping_epoch 40
wandb:                 epoch 40
wandb:        epoch_duration 15.15381
wandb:       epoch_test_loss 0.42951
wandb:      epoch_train_loss 0.41555
wandb:        epoch_val_loss 1.02481
wandb: estimated_time_left_s 146115.71646
wandb:         learning_rate 0.001
wandb:                   +13 ...
wandb: 

I am only recording the final saved model based on the validation loss, as a result of early stopping or the best model. Finally script auto-loads the saved model and tests again; those result is saved in the result.txt file (which is equal to the final line of your snap). I extract the result from there. Example as below:.

[Sachithx]

@yulun-rayn I noticed that your epoch duration is listed as 15.15 seconds, which is significantly higher than my device’s 2.66 seconds. This discrepancy could be due to a hardware dependency.

[yulun-rayn]

yeah I noticed that too. didn't think it'd make that much of a difference but I could be wrong. I'm just running interactively on a10g instance to test it out for now. what is your device?