Hello,
I've been trying to run train the model on 4 GPUs but I am having some problems with the rank of the GPUs. In fact it cannot see the arguement. More specifically:
bash train.sh
/home/gtzelepis/miniconda3/envs/dformer/lib/python3.10/site-packages/torch/distributed/launch.py:181: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use-env is set by default in torchrun.
If your script expects `--local-rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions
warnings.warn(
[2024-06-13 17:39:19,500] torch.distributed.run: [WARNING]
[2024-06-13 17:39:19,500] torch.distributed.run: [WARNING] *****************************************
[2024-06-13 17:39:19,500] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2024-06-13 17:39:19,500] torch.distributed.run: [WARNING] *****************************************
[W socket.cpp:436] [c10d] The server socket cannot be initialized on [::]:29575 (errno: 97 - Address family not supported by protocol).
[W socket.cpp:663] [c10d] The client socket cannot be initialized to connect to [::ffff:127.0.0.1]:29575 (errno: 97 - Address family not supported by protocol).
import local models
usage: train.py [-h] [--dataset NAME] [--train-split NAME] [--val-split NAME] [--dataset-download] [--class-map FILENAME] [--model MODEL] [--pretrained] [--initial-checkpoint PATH] [--resume PATH]
[--no-resume-opt] [--num-classes N] [--gp POOL] [--img-size N] [--in-chans N] [--input-size N N N N N N N N N] [--crop-pct N] [--mean MEAN [MEAN ...]] [--std STD [STD ...]]
[--interpolation NAME] [-b N] [-vb N] [--channels-last] [--torchscript | --aot-autograd] [--fuser FUSER] [--fast-norm] [--grad-checkpointing] [--opt OPTIMIZER] [--opt-eps EPSILON]
[--opt-betas BETA [BETA ...]] [--momentum M] [--weight-decay WEIGHT_DECAY] [--clip-grad NORM] [--clip-mode CLIP_MODE] [--layer-decay LAYER_DECAY] [--sched SCHEDULER] [--sched-on-updates]
[--lr LR] [--lr-base LR] [--lr-base-size DIV] [--lr-base-scale SCALE] [--lr-noise pct, pct [pct, pct ...]] [--lr-noise-pct PERCENT] [--lr-noise-std STDDEV] [--lr-cycle-mul MULT]
[--lr-cycle-decay MULT] [--lr-cycle-limit N] [--lr-k-decay LR_K_DECAY] [--warmup-lr LR] [--min-lr LR] [--epochs N] [--epoch-repeats N] [--start-epoch N]
[--decay-milestones MILESTONES [MILESTONES ...]] [--decay-epochs N] [--warmup-epochs N] [--warmup-prefix] [--cooldown-epochs N] [--patience-epochs N] [--decay-rate RATE] [--no-aug]
[--scale PCT [PCT ...]] [--ratio RATIO [RATIO ...]] [--hflip HFLIP] [--vflip VFLIP] [--color-jitter PCT] [--aa NAME] [--aug-repeats AUG_REPEATS] [--aug-splits AUG_SPLITS] [--jsd-loss]
[--bce-loss] [--bce-target-thresh BCE_TARGET_THRESH] [--reprob PCT] [--remode REMODE] [--recount RECOUNT] [--resplit] [--mixup MIXUP] [--cutmix CUTMIX]
[--cutmix-minmax CUTMIX_MINMAX [CUTMIX_MINMAX ...]] [--mixup-prob MIXUP_PROB] [--mixup-switch-prob MIXUP_SWITCH_PROB] [--mixup-mode MIXUP_MODE] [--mixup-off-epoch N] [--smoothing SMOOTHING]
[--train-interpolation TRAIN_INTERPOLATION] [--drop PCT] [--drop-connect PCT] [--drop-path PCT] [--drop-block PCT] [--bn-momentum BN_MOMENTUM] [--bn-eps BN_EPS] [--sync-bn] [--dist-bn DIST_BN]
[--split-bn] [--model-ema] [--model-ema-force-cpu] [--model-ema-decay MODEL_EMA_DECAY] [--seed S] [--worker-seeding WORKER_SEEDING] [--log-interval N] [--recovery-interval N]
[--checkpoint-hist N] [-j N] [--save-images] [--amp] [--amp-dtype AMP_DTYPE] [--amp-impl AMP_IMPL] [--no-ddp-bb] [--pin-mem] [--no-prefetcher] [--output PATH] [--experiment NAME]
[--eval-metric EVAL_METRIC] [--tta N] [--local_rank LOCAL_RANK] [--use-multi-epochs-loader] [--log-wandb]
DIR
train.py: error: unrecognized arguments: --local-rank=0
import local models
import local models
usage: train.py [-h] [--dataset NAME] [--train-split NAME] [--val-split NAME] [--dataset-download] [--class-map FILENAME] [--model MODEL] [--pretrained] [--initial-checkpoint PATH] [--resume PATH]
[--no-resume-opt] [--num-classes N] [--gp POOL] [--img-size N] [--in-chans N] [--input-size N N N N N N N N N] [--crop-pct N] [--mean MEAN [MEAN ...]] [--std STD [STD ...]]
[--interpolation NAME] [-b N] [-vb N] [--channels-last] [--torchscript | --aot-autograd] [--fuser FUSER] [--fast-norm] [--grad-checkpointing] [--opt OPTIMIZER] [--opt-eps EPSILON]
[--opt-betas BETA [BETA ...]] [--momentum M] [--weight-decay WEIGHT_DECAY] [--clip-grad NORM] [--clip-mode CLIP_MODE] [--layer-decay LAYER_DECAY] [--sched SCHEDULER] [--sched-on-updates]
[--lr LR] [--lr-base LR] [--lr-base-size DIV] [--lr-base-scale SCALE] [--lr-noise pct, pct [pct, pct ...]] [--lr-noise-pct PERCENT] [--lr-noise-std STDDEV] [--lr-cycle-mul MULT]
[--lr-cycle-decay MULT] [--lr-cycle-limit N] [--lr-k-decay LR_K_DECAY] [--warmup-lr LR] [--min-lr LR] [--epochs N] [--epoch-repeats N] [--start-epoch N]
[--decay-milestones MILESTONES [MILESTONES ...]] [--decay-epochs N] [--warmup-epochs N] [--warmup-prefix] [--cooldown-epochs N] [--patience-epochs N] [--decay-rate RATE] [--no-aug]
[--scale PCT [PCT ...]] [--ratio RATIO [RATIO ...]] [--hflip HFLIP] [--vflip VFLIP] [--color-jitter PCT] [--aa NAME] [--aug-repeats AUG_REPEATS] [--aug-splits AUG_SPLITS] [--jsd-loss]
[--bce-loss] [--bce-target-thresh BCE_TARGET_THRESH] [--reprob PCT] [--remode REMODE] [--recount RECOUNT] [--resplit] [--mixup MIXUP] [--cutmix CUTMIX]
[--cutmix-minmax CUTMIX_MINMAX [CUTMIX_MINMAX ...]] [--mixup-prob MIXUP_PROB] [--mixup-switch-prob MIXUP_SWITCH_PROB] [--mixup-mode MIXUP_MODE] [--mixup-off-epoch N] [--smoothing SMOOTHING]
[--train-interpolation TRAIN_INTERPOLATION] [--drop PCT] [--drop-connect PCT] [--drop-path PCT] [--drop-block PCT] [--bn-momentum BN_MOMENTUM] [--bn-eps BN_EPS] [--sync-bn] [--dist-bn DIST_BN]
[--split-bn] [--model-ema] [--model-ema-force-cpu] [--model-ema-decay MODEL_EMA_DECAY] [--seed S] [--worker-seeding WORKER_SEEDING] [--log-interval N] [--recovery-interval N]
[--checkpoint-hist N] [-j N] [--save-images] [--amp] [--amp-dtype AMP_DTYPE] [--amp-impl AMP_IMPL] [--no-ddp-bb] [--pin-mem] [--no-prefetcher] [--output PATH] [--experiment NAME]
[--eval-metric EVAL_METRIC] [--tta N] [--local_rank LOCAL_RANK] [--use-multi-epochs-loader] [--log-wandb]
DIR
train.py: error: unrecognized arguments: --local-rank=3
usage: train.py [-h] [--dataset NAME] [--train-split NAME] [--val-split NAME] [--dataset-download] [--class-map FILENAME] [--model MODEL] [--pretrained] [--initial-checkpoint PATH] [--resume PATH]
[--no-resume-opt] [--num-classes N] [--gp POOL] [--img-size N] [--in-chans N] [--input-size N N N N N N N N N] [--crop-pct N] [--mean MEAN [MEAN ...]] [--std STD [STD ...]]
[--interpolation NAME] [-b N] [-vb N] [--channels-last] [--torchscript | --aot-autograd] [--fuser FUSER] [--fast-norm] [--grad-checkpointing] [--opt OPTIMIZER] [--opt-eps EPSILON]
[--opt-betas BETA [BETA ...]] [--momentum M] [--weight-decay WEIGHT_DECAY] [--clip-grad NORM] [--clip-mode CLIP_MODE] [--layer-decay LAYER_DECAY] [--sched SCHEDULER] [--sched-on-updates]
[--lr LR] [--lr-base LR] [--lr-base-size DIV] [--lr-base-scale SCALE] [--lr-noise pct, pct [pct, pct ...]] [--lr-noise-pct PERCENT] [--lr-noise-std STDDEV] [--lr-cycle-mul MULT]
[--lr-cycle-decay MULT] [--lr-cycle-limit N] [--lr-k-decay LR_K_DECAY] [--warmup-lr LR] [--min-lr LR] [--epochs N] [--epoch-repeats N] [--start-epoch N]
[--decay-milestones MILESTONES [MILESTONES ...]] [--decay-epochs N] [--warmup-epochs N] [--warmup-prefix] [--cooldown-epochs N] [--patience-epochs N] [--decay-rate RATE] [--no-aug]
[--scale PCT [PCT ...]] [--ratio RATIO [RATIO ...]] [--hflip HFLIP] [--vflip VFLIP] [--color-jitter PCT] [--aa NAME] [--aug-repeats AUG_REPEATS] [--aug-splits AUG_SPLITS] [--jsd-loss]
[--bce-loss] [--bce-target-thresh BCE_TARGET_THRESH] [--reprob PCT] [--remode REMODE] [--recount RECOUNT] [--resplit] [--mixup MIXUP] [--cutmix CUTMIX]
[--cutmix-minmax CUTMIX_MINMAX [CUTMIX_MINMAX ...]] [--mixup-prob MIXUP_PROB] [--mixup-switch-prob MIXUP_SWITCH_PROB] [--mixup-mode MIXUP_MODE] [--mixup-off-epoch N] [--smoothing SMOOTHING]
[--train-interpolation TRAIN_INTERPOLATION] [--drop PCT] [--drop-connect PCT] [--drop-path PCT] [--drop-block PCT] [--bn-momentum BN_MOMENTUM] [--bn-eps BN_EPS] [--sync-bn] [--dist-bn DIST_BN]
[--split-bn] [--model-ema] [--model-ema-force-cpu] [--model-ema-decay MODEL_EMA_DECAY] [--seed S] [--worker-seeding WORKER_SEEDING] [--log-interval N] [--recovery-interval N]
[--checkpoint-hist N] [-j N] [--save-images] [--amp] [--amp-dtype AMP_DTYPE] [--amp-impl AMP_IMPL] [--no-ddp-bb] [--pin-mem] [--no-prefetcher] [--output PATH] [--experiment NAME]
[--eval-metric EVAL_METRIC] [--tta N] [--local_rank LOCAL_RANK] [--use-multi-epochs-loader] [--log-wandb]
DIR
train.py: error: unrecognized arguments: --local-rank=2
import local models
usage: train.py [-h] [--dataset NAME] [--train-split NAME] [--val-split NAME] [--dataset-download] [--class-map FILENAME] [--model MODEL] [--pretrained] [--initial-checkpoint PATH] [--resume PATH]
[--no-resume-opt] [--num-classes N] [--gp POOL] [--img-size N] [--in-chans N] [--input-size N N N N N N N N N] [--crop-pct N] [--mean MEAN [MEAN ...]] [--std STD [STD ...]]
[--interpolation NAME] [-b N] [-vb N] [--channels-last] [--torchscript | --aot-autograd] [--fuser FUSER] [--fast-norm] [--grad-checkpointing] [--opt OPTIMIZER] [--opt-eps EPSILON]
[--opt-betas BETA [BETA ...]] [--momentum M] [--weight-decay WEIGHT_DECAY] [--clip-grad NORM] [--clip-mode CLIP_MODE] [--layer-decay LAYER_DECAY] [--sched SCHEDULER] [--sched-on-updates]
[--lr LR] [--lr-base LR] [--lr-base-size DIV] [--lr-base-scale SCALE] [--lr-noise pct, pct [pct, pct ...]] [--lr-noise-pct PERCENT] [--lr-noise-std STDDEV] [--lr-cycle-mul MULT]
[--lr-cycle-decay MULT] [--lr-cycle-limit N] [--lr-k-decay LR_K_DECAY] [--warmup-lr LR] [--min-lr LR] [--epochs N] [--epoch-repeats N] [--start-epoch N]
[--decay-milestones MILESTONES [MILESTONES ...]] [--decay-epochs N] [--warmup-epochs N] [--warmup-prefix] [--cooldown-epochs N] [--patience-epochs N] [--decay-rate RATE] [--no-aug]
[--scale PCT [PCT ...]] [--ratio RATIO [RATIO ...]] [--hflip HFLIP] [--vflip VFLIP] [--color-jitter PCT] [--aa NAME] [--aug-repeats AUG_REPEATS] [--aug-splits AUG_SPLITS] [--jsd-loss]
[--bce-loss] [--bce-target-thresh BCE_TARGET_THRESH] [--reprob PCT] [--remode REMODE] [--recount RECOUNT] [--resplit] [--mixup MIXUP] [--cutmix CUTMIX]
[--cutmix-minmax CUTMIX_MINMAX [CUTMIX_MINMAX ...]] [--mixup-prob MIXUP_PROB] [--mixup-switch-prob MIXUP_SWITCH_PROB] [--mixup-mode MIXUP_MODE] [--mixup-off-epoch N] [--smoothing SMOOTHING]
[--train-interpolation TRAIN_INTERPOLATION] [--drop PCT] [--drop-connect PCT] [--drop-path PCT] [--drop-block PCT] [--bn-momentum BN_MOMENTUM] [--bn-eps BN_EPS] [--sync-bn] [--dist-bn DIST_BN]
[--split-bn] [--model-ema] [--model-ema-force-cpu] [--model-ema-decay MODEL_EMA_DECAY] [--seed S] [--worker-seeding WORKER_SEEDING] [--log-interval N] [--recovery-interval N]
[--checkpoint-hist N] [-j N] [--save-images] [--amp] [--amp-dtype AMP_DTYPE] [--amp-impl AMP_IMPL] [--no-ddp-bb] [--pin-mem] [--no-prefetcher] [--output PATH] [--experiment NAME]
[--eval-metric EVAL_METRIC] [--tta N] [--local_rank LOCAL_RANK] [--use-multi-epochs-loader] [--log-wandb]
DIR
train.py: error: unrecognized arguments: --local-rank=1
[2024-06-13 17:39:24,518] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 2) local_rank: 0 (pid: 2465952) of binary: /home/gtzelepis/miniconda3/envs/dformer/bin/python3
Traceback (most recent call last):
File "/home/gtzelepis/miniconda3/envs/dformer/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/gtzelepis/miniconda3/envs/dformer/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/gtzelepis/miniconda3/envs/dformer/lib/python3.10/site-packages/torch/distributed/launch.py", line 196, in <module>
main()
File "/home/gtzelepis/miniconda3/envs/dformer/lib/python3.10/site-packages/torch/distributed/launch.py", line 192, in main
launch(args)
File "/home/gtzelepis/miniconda3/envs/dformer/lib/python3.10/site-packages/torch/distributed/launch.py", line 177, in launch
run(args)
File "/home/gtzelepis/miniconda3/envs/dformer/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/home/gtzelepis/miniconda3/envs/dformer/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/gtzelepis/miniconda3/envs/dformer/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
train.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2024-06-13_17:39:24
host : pmlabgpus
rank : 1 (local_rank: 1)
exitcode : 2 (pid: 2465953)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2024-06-13_17:39:24
host : pmlabgpus
rank : 2 (local_rank: 2)
exitcode : 2 (pid: 2465954)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2024-06-13_17:39:24
host : pmlabgpus
rank : 3 (local_rank: 3)
exitcode : 2 (pid: 2465955)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-06-13_17:39:24
host : pmlabgpus
rank : 0 (local_rank: 0)
exitcode : 2 (pid: 2465952)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Any help is welcome
Hello,
I've been trying to run train the model on 4 GPUs but I am having some problems with the rank of the GPUs. In fact it cannot see the arguement. More specifically:
Any help is welcome