Issue on training

[Freja71122]

Hi There,

I followed the training guide in README，and I met this problem: TypeError: unsupported operand type(s) for /: 'Nonetype' and 'int'
Do you have any ideas how to fix this problem?

My training script is like this:

[lalalune]

That looks like the error where it's not actually finding the training data. It's expecting an int num items but it's null.

I suspect that this is some weirdness with the CLI input and your path -- maybe you need ./ori_data/ ?

Otherwise could be a version mismatch in dependencies but I've only gotten this issue from training path mismatch

IMO add something to the script that checks if the path exists and errors out if it does not

[Freja71122]

Thank you for your reply.
I have solved it. The path is correct, and the reason is there are past pkl files in the cache, so the correct data is not loaded.

[Freja71122]

In addition, it takes about a whole day for each syn_id to render, and it will take about a month for all data to be rendered. Is it normal to be so slow?

[SteveJunGao]

Hi @Freja71122

Yes, the rendering is slow, I didn't optimize that part, you can take a look at this MR to see whether it can help you

[Freja71122]

It hangs after rendering one syn_id 02958343. Is it normal?

My script is python render_all.py --save_folder PATH_TO_SAVE_IMAGE --dataset_folder PATH_TO_3D_OBJ --blender_root PATH_TO_BLENDER

[SteveJunGao]

Emmm,

this is not normal, it shouldn't hang there.. do you have any other error messages?

[Freja71122]

No any error message... I press ctrl+c twice, and it continue rendering...

[Siziff]

I have the same error when preparing data. How long did it take you to prepare the data? I have been processing data for more than 24 hours and I haven't even prepared half of it. I work on a server with one V100 graphics card. How long does it take?

[Freja71122]

@Siziff Hi, I work on 8 V100 graphics card, and it takes more than 24 hours for one synset dataset. There are three synset, so I guess it will take another 2 days.
(I don't know if it actually uses 8 cards, since I use shared gpus...

Synset list can refer to: https://github.com/nv-tlabs/GET3D/blob/master/render_shapenet_data/render_all.py#L28

@SteveJunGao also took 2-3 days, which was mentioned in #6 (comment)

[SteveJunGao]

Yes, the code I wrote for blender is not optimized, so it can take 2-3 days to render one synset.

You can check this MR to see whether it can speed things up

[Freja71122]

Thank you! @SteveJunGao

I close this issue, since I have got enough help and information about initial question.

[efyphil]

@Freja71122 Hi, I have the same error as you. Can you tell me exactly where to delete the pkl file? I already deleted it in the cache folder