Issue in reproducing results on Semantic 3D dataset

[saba155]

I tried several times to reproduce results as published in paper for Semantic 3d but could not able to produce same. My results are far lagging behind the original results. I have followed all the steps mentioned in readme and uploaded the labels on semantic 3d dataset site.

[loicland]

Hi,

Are you using our trained model or training from scratch? What results are you obtaining? On the test set or the validation set?

[saba155]

I am training from scratch and using full test set.

[loicland]

So you mean that the server-side evaluation gives you abd results? Can you share them here?

What happens when you use the validation set for testing, do you get similar results?

[saba155]

Thank you for prompt reply. . I am training from scratch and using full test set. I achieved 13% overall accuracy which is too low. I have followed following steps, kindly tell me if I am missing some thins:

• CUDA_VISIBLE_DEVICES=0 python learning/main.py --dataset sema3d --SEMA3D_PATH $SEMA3D_DIR --db_test_name testfull --db_train_name trainval --epochs -1 --lr_steps '[350, 400, 450]' --test_nth_epoch 100 --model_config 'gru_10,f_8' --ptn_nfeat_stn 11 --nworkers 2 --odir "results/sema3d/trainval_best"

• python partition/write_Semantic3d.py --SEMA3D_PATH $SEMA3D_DIR --odir "results/sema3d/trainval_best" --db_test_name testred

• uploaded above generated labels on Semantic 3D evaluation tool.

[loicland]

Can you try to visualize the results to see what's wrong? For example with:

python partition/visualize.py --dataset sema3d --ROOT_PATH $SEMA3D_DIR --res_file 'results/sema3d/trainval_best/prediction_testred' --file_path 'test_reduced/MarketplaceFeldkirch_Station4' --output_type ifprs

and then load the results .ply files in meshlab or cloud compare.

[saba155]

Visualization seems fine. I am uploading snapshot of bird fountain station1 point cloud here.

[loicland]

Ok that's weird, that is certainly not a 13%. Try to check the consistency between the size of the data files and the number of columns in the .labels files. Check the labels as well to make sure they are consistent with the typology of semantic 3D. In particular, the labels should start at 1 and not 0.

Are you talking about the reduced test set by the way?

Also try to run the code with trained models and compare directly the .labels files.

[saba155]

I am using full test set and my labels starts with different values like 4, 6 etc and have only one column. Total number of lines are same.

[loicland]

Would you please post here the first 50 or so columns of a .labels file of your choice (along with its name).

I will compare it to mine (but no before Monday I'm afraid).

[saba155]

thanks alot no problem take your time. I am uploading first 200 lines of sg27_station3.
sg27_3.labels.txt

[saba155]

Did u manage to compare .labels?

[loicland]

Hi,
I did not have time yet as the upsampling of labels takes a long time.
In case you have it handy, could you post labels from castleblatten_station1 as this one has finished for me?

Have you tried using our pretrained model by the way to compare the labels?

[saba155]

No I do not tried on pre-trained model actually I want to reproduce results from scratch. Here I am uploading the first 200 lines of castleblatten_station1. castleblatten_station1.labels.txt

[loicland]

Hi,

So two things:

• it is not normal to have 0 in your predictions because this is reserved for unannotated datas, and normally never predicted. Something is wrong. Can you post a link (google drive for example) to your predictions_testfull.h5?

• another thing I just noticed is that the command line you use seems wrong:
  CUDA_VISIBLE_DEVICES=0 python learning/main.py --dataset sema3d --SEMA3D_PATH $SEMA3D_DIR --db_test_name testfull --db_train_name trainval --epochs -1 --lr_steps '[350, 400, 450]' --test_nth_epoch 100 --model_config 'gru_10,f_8' --ptn_nfeat_stn 11 --nworkers 2 --odir "results/sema3d/trainval_best"

if --epochs is set to -1 (inference only) then you should add --resume RESUME in order for the trained model to be loaded. Else it would use random weights, and, yes, have about 13% (1/8) accuracy.
If this is your main training loop then --epochs should be set to at least 200.

[hagianga21]

Hi, I also have that problems. Although I run this line of code:

CUDA_VISIBLE_DEVICES=2 python learning/main.py --dataset sema3d --SEMA3D_PATH 'datasets/semantic3d' --db_test_name testfull --db_train_name trainval --epochs -1 --lr_steps '[350, 400, 450]' --test_nth_epoch 100 --model_config 'gru_10,f_8' --ptn_nfeat_stn 11 --nworkers 2 --odir "results/bremen/standard" --resume RESUME

There is some 0 labelled points in label files