while running python -m src.inference experiment=sem_embeds_inference_flat data_dir=data/amazon_data/beauty,it exits without any errors.
[2025-11-03 14:13:19,593][src.utils.inference_utils][INFO] - Global Rank: 0 wrote 5 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_0_20251103T061319578.pkl.
[2025-11-03 14:13:19,594][src.utils.inference_utils][INFO] - Rank 0 finished writing predictions.
[2025-11-03 14:13:20,582][src.utils.inference_utils][INFO] - Global Rank: 4 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_4_20251103T061320530.pkl.
[2025-11-03 14:13:20,587][src.utils.inference_utils][INFO] - Global Rank: 5 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_5_20251103T061320531.pkl.
[2025-11-03 14:13:20,610][src.utils.inference_utils][INFO] - Global Rank: 3 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_3_20251103T061320574.pkl.
[2025-11-03 14:13:20,612][src.utils.inference_utils][INFO] - Global Rank: 7 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_7_20251103T061320538.pkl.
[2025-11-03 14:13:20,626][src.utils.inference_utils][INFO] - Global Rank: 2 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_2_20251103T061320557.pkl.
[2025-11-03 14:13:20,652][src.utils.inference_utils][INFO] - Global Rank: 1 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_1_20251103T061320592.pkl.
[2025-11-03 14:13:20,685][src.utils.inference_utils][INFO] - Global Rank: 6 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_6_20251103T061320642.pkl.
[2025-11-03 14:13:21,903][src.utils.inference_utils][INFO] - Global Rank: 4 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_4_20251103T061321844.pkl.
[2025-11-03 14:13:21,906][src.utils.inference_utils][INFO] - Global Rank: 5 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_5_20251103T061321848.pkl.
[2025-11-03 14:13:21,911][src.utils.inference_utils][INFO] - Global Rank: 7 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_7_20251103T061321874.pkl.
[2025-11-03 14:13:21,924][src.utils.inference_utils][INFO] - Global Rank: 3 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_3_20251103T061321868.pkl.
[2025-11-03 14:13:21,929][src.utils.inference_utils][INFO] - Global Rank: 2 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_2_20251103T061321890.pkl.
[2025-11-03 14:13:21,948][src.utils.inference_utils][INFO] - Global Rank: 1 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_1_20251103T061321912.pkl.
[2025-11-03 14:13:21,987][src.utils.inference_utils][INFO] - Global Rank: 6 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_6_20251103T061321951.pkl.
[2025-11-03 14:13:23,205][src.utils.inference_utils][INFO] - Global Rank: 4 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_4_20251103T061323165.pkl.
[2025-11-03 14:13:23,207][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,207][src.utils.inference_utils][INFO] - Rank 4 finished writing predictions.
[2025-11-03 14:13:23,220][src.utils.inference_utils][INFO] - Global Rank: 3 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_3_20251103T061323181.pkl.
[2025-11-03 14:13:23,222][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,222][src.utils.inference_utils][INFO] - Rank 3 finished writing predictions.
[2025-11-03 14:13:23,227][src.utils.inference_utils][INFO] - Global Rank: 7 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_7_20251103T061323173.pkl.
[2025-11-03 14:13:23,228][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,229][src.utils.inference_utils][INFO] - Rank 7 finished writing predictions.
[2025-11-03 14:13:23,245][src.utils.inference_utils][INFO] - Global Rank: 1 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_1_20251103T061323204.pkl.
[2025-11-03 14:13:23,246][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,247][src.utils.inference_utils][INFO] - Rank 1 finished writing predictions.
[2025-11-03 14:13:23,247][src.utils.inference_utils][INFO] - Global Rank: 2 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_2_20251103T061323189.pkl.
[2025-11-03 14:13:23,249][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,249][src.utils.inference_utils][INFO] - Rank 2 finished writing predictions.
[2025-11-03 14:13:23,307][src.utils.inference_utils][INFO] - Global Rank: 6 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_6_20251103T061323248.pkl.
[2025-11-03 14:13:23,308][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,309][src.utils.inference_utils][INFO] - Rank 6 finished writing predictions.
[2025-11-03 14:13:23,357][src.utils.inference_utils][INFO] - Global Rank: 5 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_5_20251103T061323166.pkl.
[2025-11-03 14:13:23,358][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,359][src.utils.inference_utils][INFO] - Rank 5 finished writing predictions.
[2025-11-03 14:13:23,359][src.utils.inference_utils][INFO] - Merging pickle files on main process.
[2025-11-03 14:13:31,361][src.utils.inference_utils][INFO] - Merged 12101 rows into merged_predictions.pkl.
[2025-11-03 14:13:35,443][src.utils.inference_utils][INFO] - Merged 12101 rows into merged_predictions_tensor.pt. as pytorch tensor
Predicting DataLoader 0: | | 169/? [00:45<00:00, 3.74it/s]
while running python -m src.inference experiment=sem_embeds_inference_flat data_dir=data/amazon_data/beauty,it exits without any errors.
[2025-11-03 14:13:19,593][src.utils.inference_utils][INFO] - Global Rank: 0 wrote 5 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_0_20251103T061319578.pkl.
[2025-11-03 14:13:19,594][src.utils.inference_utils][INFO] - Rank 0 finished writing predictions.
[2025-11-03 14:13:20,582][src.utils.inference_utils][INFO] - Global Rank: 4 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_4_20251103T061320530.pkl.
[2025-11-03 14:13:20,587][src.utils.inference_utils][INFO] - Global Rank: 5 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_5_20251103T061320531.pkl.
[2025-11-03 14:13:20,610][src.utils.inference_utils][INFO] - Global Rank: 3 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_3_20251103T061320574.pkl.
[2025-11-03 14:13:20,612][src.utils.inference_utils][INFO] - Global Rank: 7 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_7_20251103T061320538.pkl.
[2025-11-03 14:13:20,626][src.utils.inference_utils][INFO] - Global Rank: 2 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_2_20251103T061320557.pkl.
[2025-11-03 14:13:20,652][src.utils.inference_utils][INFO] - Global Rank: 1 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_1_20251103T061320592.pkl.
[2025-11-03 14:13:20,685][src.utils.inference_utils][INFO] - Global Rank: 6 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_6_20251103T061320642.pkl.
[2025-11-03 14:13:21,903][src.utils.inference_utils][INFO] - Global Rank: 4 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_4_20251103T061321844.pkl.
[2025-11-03 14:13:21,906][src.utils.inference_utils][INFO] - Global Rank: 5 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_5_20251103T061321848.pkl.
[2025-11-03 14:13:21,911][src.utils.inference_utils][INFO] - Global Rank: 7 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_7_20251103T061321874.pkl.
[2025-11-03 14:13:21,924][src.utils.inference_utils][INFO] - Global Rank: 3 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_3_20251103T061321868.pkl.
[2025-11-03 14:13:21,929][src.utils.inference_utils][INFO] - Global Rank: 2 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_2_20251103T061321890.pkl.
[2025-11-03 14:13:21,948][src.utils.inference_utils][INFO] - Global Rank: 1 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_1_20251103T061321912.pkl.
[2025-11-03 14:13:21,987][src.utils.inference_utils][INFO] - Global Rank: 6 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_6_20251103T061321951.pkl.
[2025-11-03 14:13:23,205][src.utils.inference_utils][INFO] - Global Rank: 4 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_4_20251103T061323165.pkl.
[2025-11-03 14:13:23,207][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,207][src.utils.inference_utils][INFO] - Rank 4 finished writing predictions.
[2025-11-03 14:13:23,220][src.utils.inference_utils][INFO] - Global Rank: 3 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_3_20251103T061323181.pkl.
[2025-11-03 14:13:23,222][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,222][src.utils.inference_utils][INFO] - Rank 3 finished writing predictions.
[2025-11-03 14:13:23,227][src.utils.inference_utils][INFO] - Global Rank: 7 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_7_20251103T061323173.pkl.
[2025-11-03 14:13:23,228][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,229][src.utils.inference_utils][INFO] - Rank 7 finished writing predictions.
[2025-11-03 14:13:23,245][src.utils.inference_utils][INFO] - Global Rank: 1 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_1_20251103T061323204.pkl.
[2025-11-03 14:13:23,246][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,247][src.utils.inference_utils][INFO] - Rank 1 finished writing predictions.
[2025-11-03 14:13:23,247][src.utils.inference_utils][INFO] - Global Rank: 2 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_2_20251103T061323189.pkl.
[2025-11-03 14:13:23,249][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,249][src.utils.inference_utils][INFO] - Rank 2 finished writing predictions.
[2025-11-03 14:13:23,307][src.utils.inference_utils][INFO] - Global Rank: 6 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_6_20251103T061323248.pkl.
[2025-11-03 14:13:23,308][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,309][src.utils.inference_utils][INFO] - Rank 6 finished writing predictions.
[2025-11-03 14:13:23,357][src.utils.inference_utils][INFO] - Global Rank: 5 wrote 64 rows to /mnt/nanjgrowth-train-ks3/ganzijie/GRID/logs/inference/runs/2025-11-03/14-10-57/pickle/predictions_5_20251103T061323166.pkl.
[2025-11-03 14:13:23,358][src.utils.inference_utils][INFO] - Buffer is empty, nothing to flush.
[2025-11-03 14:13:23,359][src.utils.inference_utils][INFO] - Rank 5 finished writing predictions.
[2025-11-03 14:13:23,359][src.utils.inference_utils][INFO] - Merging pickle files on main process.
[2025-11-03 14:13:31,361][src.utils.inference_utils][INFO] - Merged 12101 rows into merged_predictions.pkl.
[2025-11-03 14:13:35,443][src.utils.inference_utils][INFO] - Merged 12101 rows into merged_predictions_tensor.pt. as pytorch tensor
Predicting DataLoader 0: | | 169/? [00:45<00:00, 3.74it/s]