pre-training evaluation

[taeil]

Trying to evaluate on RGB

1. Zero pad on top of RGB channels to match 11 channel or Repeat channel
2. Freeze backbone
3. Test time learning? Universal representation
4. domain generalization

Other evaluation

• knn
• Linear evaluation on openselfsup
• T-sne (visualization)

Please refer to Colorado's ideas and confirm here.

[suryagutta]

Using BigEarthNet data
One option is to download BigEarthNet S1 and S2 and create fusion data using tools from BigEarthNet-MM model https://git.tu-berlin.de/rsim/BigEarthNet-MM_tools
" This script reads GeoTIFF files for each of the spectral and radar bands of a pair of Sentinel-1 and Sentinel-2 image patches in the corresponding BigEarthNet archives. The script is capable of reading either all spectral radar bands of one patch pair (-p1 option for Sentinel-1 and -r2 for BigEarthNet-S2 root folder to lookup the corresponding patch from) or all bands for all Sentinel-1 and corresponding Sentinel-2 patches (-r1 and -r2 option). After reading the files, the values of the Sentinel-1 and Sentinel-2 image patches can be used as numpy arrays for further purposes. "
https://git.tu-berlin.de/rsim/BigEarthNet-MM_19-classes_models

[taeil]

4/25

• supervised result seems to be the biggest issues (no evidence of improvement)
  • potential explanation: 1K dataset is big enough to maximize the performance (possible)
  • but, there are a couple of papers that uses 1K dataset but hosws difference
    • In-domain representation learning for remote sensing
      • BE dataset: mAP, 43 class => rerunning the experiment on complex classess to match the in-domain representation paper.
      • should check mAP to match the results (or top 1 accuracy)
    • Self-Supervised Pretraining Improves Self-Supervised Pretraining - need to understand what accurcy was used for evaluation
• evaluation parameters can be an issue
• number of iterations: 500 iterations per epoch => 100K iterations is equivalent to

Questions:

• indomain?
• different metrics
• pretraining on sen12ms and finetune on BE datasets. (indomain paper table-2)

Possible Priority

Method 1: Evaluation (BigEarthNet)
try to replicate in-domain accuracy with supervised training (mAP, 43 classes)

Method 2: pretraining (increase epochs to 500)

• pleasant-moon-88
• and see if the performance improves

Method 3: Evaluation
To reproduce evaluation (confirm evaluation is correct)

• just train with generic moco code and pretrain (in case pretraining is wrong).
• using only RGB data from sen12ms (this should work well).
• (maybe) and then, use resisc 45 RGB to see same results on sen12ms evaluation

Method 4: Evaluation
To reproduce the bigearthnet numbers, we could start with using an imagenet pretrained model, since the "in domain representatino learning" include these number -- just to verify the evaluation part of our code is working.

[taeil]

Larger number of classes evaluation

• Redone supervised training with larger number classes. See some more drop in the performance for sen12ms (4~12% drop). For BE dataset, almost no drop (0.5). Redoing just to make sure to cover all parameters.
• Rerunning evaluation on (partial fusion) decent-bird-80 on larger number classes and (full fusion 200 epocs) volcanic-disco-84.

[taeil]

500/900 epochs pretrained model (full fusion 500 epochs) pleasant-moon-88

• ran evaluation on complex (need to check the results)
• running evaluation on simple
• running 900 pretrained model evaluations

[taeil]

In domain paper evaluation

We report performance results using accuracy metrics commonly used in computer vision tasks: for multi-class problems we use the Top-1 global accuracy metric, which denotes the percentage of correctly labeled samples. For multi-label problems we use the mean average precision (mAP) metric, which denotes the mean over the average precision (AV) values (AV is the integral over the precision-recall curve) of the individual labels.

[taeil]

sen12ms just released paper

[taeil]

3/28

Surya (& Ernesto)

• bigeartnet data

• input module weights are shared with query and key encoder (?)

• not reliable evaluation but we want to get signal

• no baseline to compare to

• signal 1: supervised training on 1k should be good benchmark without loading input module

• signal 2: supervised training on 1k should be good benchmark with loading input module

• signal 3: 1K epoch pretrained module should be better

• augmentation

  • randomresizecrop is the most important

• pretraining

• single source / cross source (35:40)

• random finetuning evaluation (all combination)

  • s1, s2, s1/s2

• pretraining configuration (s1, s2, s1/s2)

• s1 positive example, s2 positive example

[taeil]

3/26

1. Random fine tuning evaluation -- using all the combinations that we have that we are reporting, _s1, _s2, _s1s2
2. Load input module 
3. Choose augmentations to use - (1) random resize crop/ (2) random crop and random scale
4. Flag - configuration for pre-training on s1, pretrianing on s2 -- (1)  we are only pre-training on s1 positive examples with the same s1 positive example (s2 pos with the same s2 pos). (2) - - previous thing + allow s1 positive examples to align with s2 positive examples

S1 vs s1 , s2 vs s2

1. Evaluate on rgb data 
2. 200 epochs vs 1000 epochs

[taeil]

3/8

• augmentation is not going to fix model convergence.
• prioritize visualization (sometimes obvious issue) after pipeline
• s1, s2 std looks high
• initial hump on training is not uncommon with aerial images
• adjust learning rate
• batch size is problematic: 1gpu 64. 0.003 256 / 4

[IDEA / Discussion] Colorado mentioned that learning rate rule of thumb is to divide the learning rate by same ratio for batch size change. Maybe 0.001 => 0.00001 may be too big ratio. Either we increase the batch size based on the original pretraining parameters.

4 GPU (64 * 4) 0.03 => 1 GPU (64) 0.03 / 4