Multi source input + multi-dist training

[taeil]

We were using old version of torchvision.transform which only support PIL image.

https://pytorch.org/docs/1.6.0/torchvision/transforms.html#transforms-on-torch-tensor

either change to PIL image when loading the image or do our own transformation (which is probably bad idea).

Need to see if we can only upgrade torchvision only.

If try to transformToPIL, getting this error https://discuss.pytorch.org/t/error-while-using-transforms-topilimage-and-randomresizedcrop/12861

Colorado's change with custom data loader:
Berkeley-Data/OpenSelfSup#3

[taeil]

Tried to upgrade torchvision to the latest but it automatically upgrade torch to the matching version. So we won't be able to upgrade to newer torchvision on colorado's machine.

(hp121) ➜  OpenSelfSup git:(taeil) pip install torchvision==0.8.0
Collecting torchvision==0.8.0
  Downloading torchvision-0.8.0-cp37-cp37m-manylinux1_x86_64.whl (11.8 MB)
     |████████████████████████████████| 11.8 MB 8.0 MB/s
Collecting torch==1.7.0
  Downloading torch-1.7.0-cp37-cp37m-manylinux1_x86_64.whl (776.7 MB)
     |████████████████████████████████| 776.7 MB 2.4 kB/s
Requirement already satisfied: numpy in /scratch/crguest/miniconda3/envs/hp121/lib/python3.7/site-packages (from torchvision==0.8.0) (1.19.2)
Requirement already satisfied: pillow>=4.1.1 in /scratch/crguest/miniconda3/envs/hp121/lib/python3.7/site-packages (from torchvision==0.8.0) (8.1.0)
Requirement already satisfied: future in /scratch/crguest/miniconda3/envs/hp121/lib/python3.7/site-packages (from torch==1.7.0->torchvision==0.8.0) (0.18.2)
Requirement already satisfied: typing-extensions in /scratch/crguest/miniconda3/envs/hp121/lib/python3.7/site-packages (from torch==1.7.0->torchvision==0.8.0) (3.7.4.3)
Collecting dataclasses
  Downloading dataclasses-0.6-py3-none-any.whl (14 kB)
Installing collected packages: dataclasses, torch, torchvision
  Attempting uninstall: torch
    Found existing installation: torch 1.2.0
    Uninstalling torch-1.2.0:
      Successfully uninstalled torch-1.2.0
  Attempting uninstall: torchvision
    Found existing installation: torchvision 0.4.0a0+6b959ee
    Uninstalling torchvision-0.4.0a0+6b959ee:
      Successfully uninstalled torchvision-0.4.0a0+6b959ee
Successfully installed dataclasses-0.6 torch-1.7.0 torchvision-0.8.0

[taeil]

tried on AWS using the latest torch with the latest torchvision. Getting error.

/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py:168: UserWarning: Runner was deprecated, please use EpochBasedRunner instead
  'Runner was deprecated, please use EpochBasedRunner instead')
2021-03-12 05:28:37,613 - openselfsup - INFO - Start running, host: ubuntu@ip-172-31-5-178, work_dir: /home/ubuntu/OpenSelfSup/work_dirs/selfsup/moco/r50_v2_sen12ms_in_basetrain_20ep
2021-03-12 05:28:37,613 - openselfsup - INFO - workflow: [('train', 1)], max: 20 epochs
Traceback (most recent call last):
  File "tools/train.py", line 163, in <module>
    main()
  File "tools/train.py", line 159, in main
    debug=args.debug)
  File "/home/ubuntu/OpenSelfSup/openselfsup/apis/train.py", line 110, in train_model
    model, dataset, cfg, logger=logger, timestamp=timestamp, meta=meta)
  File "/home/ubuntu/OpenSelfSup/openselfsup/apis/train.py", line 335, in _non_dist_train
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 122, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 27, in train
    for i, data_batch in enumerate(data_loader):
  File "/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/ubuntu/OpenSelfSup/openselfsup/datasets/contrastive.py", line 52, in __getitem__
    s1_img = self.pipeline(s1_img)
  File "/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 67, in __call__
    img = t(img)
  File "/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 1392, in forward
    num_output_channels = F._get_image_num_channels(img)
  File "/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/torchvision/transforms/functional.py", line 39, in _get_image_num_channels
    return F_pil._get_image_num_channels(img)
  File "/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/torchvision/transforms/functional_pil.py", line 33, in _get_image_num_channels
    raise TypeError("Unexpected type {}".format(type(img)))
TypeError: Unexpected type <class 'numpy.ndarray'>

[taeil]

Changed to Tensor and tried pipeline.. now error saying it must be 3 channels :(

  File "/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/torchvision/transforms/functional_tensor.py", line 215, in rgb_to_grayscale
    raise TypeError("Input image tensor should 3 channels, but found {}".format(c))
TypeError: Input image tensor should 3 channels, but found 2

[taeil]

Changed to Tensor and tried pipeline.. now error saying it must be 3 channels :(

  File "/home/ubuntu/anaconda3/envs/hpt/lib/python3.7/site-packages/torchvision/transforms/functional_tensor.py", line 215, in rgb_to_grayscale
    raise TypeError("Input image tensor should 3 channels, but found {}".format(c))
TypeError: Input image tensor should 3 channels, but found 2

disabled gray channel augmentation but other channels seems to be fine.

[taeil]

result looks strange...
https://wandb.ai/cal-capstone/hpt/runs/2p9a1ijm?workspace=user-taeil

[taeil]

https://wandb.ai/cal-capstone/hpt2/runs/tg4g765y?workspace=user-taeil

seems to be better with including the fix from Colorado.

[taeil]

running on full dataset to get feeling around the difference.