Questions about evaluation on redwood dataset

[DHuiTnut]

Hi, thanks for your excellent work! With the help of your tool, I have done most of the performance evaluation of the model on the REAL dataset.
But when I do the performance evaluation on the RedWood dataset, the obtained metrics look very different from those in the paper (take dpdn as an example, see the snapshot below).

I visualized the input of the dataset and found that the angle of the generated mask is not consistent with the angle in the RGB image,. I am not sure if this is the reason for this problem. Could you please tell me how to solve this problem?

[roym899]

I can't really reproduce this issue. The mask is auto-generated from the depth image and the ground-truth mesh, so my best guess is that something with the dataset is not correct.

Could you try to redownload it by deleting the redwood and redwood75 directories and rerun the script with this fix: 8f32355?

[DHuiTnut]

I can't really reproduce this issue. The mask is auto-generated from the depth image and the ground-truth mesh, so my best guess is that something with the dataset is not correct.

Could you try to redownload it by deleting the redwood and redwood75 directories and rerun the script with this fix: 8f32355?

Sorry for the late reply to your comment. I redownloaded the redwood dataset following your latest code. Unfortunately, the generated mask still does not match the picture.
Is it convenient for you to provide the image or npy file of the mask you generated? Thanks for your help!

[roym899]

I have created a file containing the masks: redwood75_masks.zip I think it should be relatively straightforward to modify redwood_dataset.py to use these instead of the generated ones.

Could you maybe share the output of pip freeze? This might help to debug this issue. I suspect that open3d behaves differently on your platform when generating the depth image.

[DHuiTnut]

Thanks for your sharing. I have done the evaluation for my model on redwood75 dataset by the masks you provided.

And here is the output of pip freeze:

absl-py @ file:///C:/b/abs_d3cv5rzljl/croot/absl-py_1686852506854/work
aiohttp @ file:///C:/b/abs_bc6tmjiy12/croot/aiohttp_1701112585940/work
aiosignal @ file:///tmp/build/80754af9/aiosignal_1637843061372/work
ansi2html==1.8.0
asttokens==2.4.1
async-timeout @ file:///C:/b/abs_43ozhz2a8g/croots/recipe/async-timeout_1664876362767/work
attrs @ file:///C:/b/abs_35n0jusce8/croot/attrs_1695717880170/work
Babel==2.14.0
backcall==0.2.0
beautifulsoup4==4.12.3
black==24.1.1
blinker @ file:///C:/b/abs_d9y2dm7cw2/croot/blinker_1696539752170/work
Brotli @ file:///C:/Windows/Temp/abs_63l7912z0e/croots/recipe/brotli-split_1659616056886/work
cachetools @ file:///tmp/build/80754af9/cachetools_1619597386817/work
certifi @ file:///C:/b/abs_91u83siphd/croot/certifi_1700501720658/work/certifi
cffi @ file:///C:/b/abs_924gv1kxzj/croot/cffi_1700254355075/work
chardet==5.2.0
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
click @ file:///C:/b/abs_f9ihnt72pu/croot/click_1698129847492/work
cloudpickle @ file:///C:/b/abs_3796yxesic/croot/cloudpickle_1683040098851/work
colorama @ file:///C:/b/abs_a9ozq0l032/croot/colorama_1672387194846/work
comm==0.2.0
ConfigArgParse==1.7
contourpy @ file:///C:/b/abs_d5rpy288vc/croots/recipe/contourpy_1663827418189/work
coverage==7.4.1
-e git+https://github.com/roym899/pose_and_shape_evaluation.git@0632e3214274439a63cf8b4ae027c05ede49e330#egg=cpas_toolbox
cryptography @ file:///C:/b/abs_f4do8t8jfs/croot/cryptography_1694444424531/work
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
cytoolz @ file:///C:/b/abs_d43s8lnb60/croot/cytoolz_1701723636699/work
dash==2.14.1
dash-core-components==2.0.0
dash-html-components==2.0.0
dash-table==5.0.0
dask @ file:///C:/b/abs_59fkcimley/croot/dask-core_1683065223377/work
decorator==5.1.1
easydict==1.11
exceptiongroup==1.2.0
executing==2.0.1
fastjsonschema==2.18.1
filelock==3.13.1
filterpy==1.4.5
flake8==7.0.0
flake8-annotations==3.0.1
flake8-docstrings==1.7.0
Flask==3.0.0
fonttools==4.25.0
frozenlist @ file:///C:/b/abs_d8e__s1ys3/croot/frozenlist_1698702612014/work
fsspec @ file:///C:/b/abs_97mpfsesn0/croot/fsspec_1701286534629/work
gdown==5.1.0
ghp-import==2.1.0
google-auth @ file:///C:/b/abs_defnokp9xd/croot/google-auth_1694152741394/work
google-auth-oauthlib @ file:///tmp/build/80754af9/google-auth-oauthlib_1617120569401/work
griffe==0.40.0
grpcio @ file:///C:/b/abs_654jr7av_l/croot/grpc-suite_1698063171901/work
idna @ file:///C:/b/abs_bdhbebrioa/croot/idna_1666125572046/work
imagecodecs @ file:///C:/b/abs_e2g5zbs1q0/croot/imagecodecs_1695065012000/work
imageio @ file:///C:/b/abs_3eijmwdodc/croot/imageio_1695996500830/work
importlib-metadata @ file:///C:/b/abs_20ndzb2j6v/croot/importlib-metadata_1678997085534/work
importlib-resources @ file:///C:/b/abs_9d6rmtszbl/croot/importlib_resources_1698254653090/work
iniconfig==2.0.0
ipython==8.12.3
ipywidgets==8.1.1
isort==5.13.2
itsdangerous==2.1.2
jedi==0.19.1
Jinja2==3.1.2
joblib==1.3.2
jsonschema==4.19.2
jsonschema-specifications==2023.7.1
jupyter_core==5.5.0
jupyterlab-widgets==3.0.9
kiwisolver @ file:///C:/b/abs_88mdhvtahm/croot/kiwisolver_1672387921783/work
llvmlite==0.41.1
locket @ file:///C:/ci/locket_1652904093019/work
Markdown==3.5.1
MarkupSafe @ file:///C:/ci/markupsafe_1654489871526/work
matplotlib==3.5.3
matplotlib-inline==0.1.6
mccabe==0.7.0
mergedeep==1.3.4
mkdocs==1.5.3
mkdocs-autorefs==0.5.0
mkdocs-material==9.5.7
mkdocs-material-extensions==1.3.1
mkdocstrings==0.24.0
mkdocstrings-python==1.8.0
mkl-fft @ file:///C:/b/abs_19i1y8ykas/croot/mkl_fft_1695058226480/work
mkl-random @ file:///C:/b/abs_edwkj1_o69/croot/mkl_random_1695059866750/work
mkl-service==2.4.0
multidict @ file:///C:/b/abs_44ido987fv/croot/multidict_1701097803486/work
munkres==1.1.4
mypy-extensions==1.0.0
nbformat==5.7.0
nest-asyncio==1.5.8
networkx @ file:///C:/b/abs_e6gi1go5op/croot/networkx_1690562046966/work
ninja==1.11.1.1
numba==0.58.1
numpy @ file:///C:/Users/dev-admin/mkl/numpy_and_numpy_base_1682982345978/work
oauthlib @ file:///C:/b/abs_44riqpe5gg/croot/oauthlib_1679489644705/work
open3d==0.17.0
opencv-python==4.8.1.78
opencv-python-headless==4.9.0.80
packaging @ file:///C:/b/abs_28t5mcoltc/croot/packaging_1693575224052/work
paginate==0.5.6
parso==0.8.3
partd @ file:///C:/b/abs_46awex0fd7/croot/partd_1698702622970/work
pathspec==0.12.1
pickleshare==0.7.5
Pillow @ file:///C:/b/abs_20ztcm8lgk/croot/pillow_1696580089746/work
pkgutil_resolve_name==1.3.10
platformdirs @ file:///C:/b/abs_b6z_yqw_ii/croot/platformdirs_1692205479426/work
plotly==5.18.0
pluggy==1.4.0
ply==3.11
plyfile==1.0.3
pooch @ file:///C:/b/abs_a8nmng7d_x/croot/pooch_1695850149827/work
prompt-toolkit==3.0.39
protobuf==3.20.3
pure-eval==0.2.2
pyasn1 @ file:///Users/ktietz/demo/mc3/conda-bld/pyasn1_1629708007385/work
pyasn1-modules==0.2.8
pybind11==2.11.1
pybind11-global==2.11.1
pycodestyle==2.11.1
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pydocstyle==6.3.0
pyflakes==3.2.0
Pygments==2.16.1
PyJWT @ file:///C:/ci/pyjwt_1657529430378/work
pymdown-extensions==10.7
pyOpenSSL @ file:///C:/b/abs_08f38zyck4/croot/pyopenssl_1690225407403/work
pyparsing @ file:///C:/Users/BUILDE~1/AppData/Local/Temp/abs_7f_7lba6rl/croots/recipe/pyparsing_1661452540662/work
PyQt5==5.15.10
PyQt5-sip @ file:///C:/b/abs_c0pi2mimq3/croot/pyqt-split_1698769125270/work/pyqt_sip
PySocks @ file:///C:/ci/pysocks_1605287845585/work
pytest==8.0.0
pytest-cov==4.1.0
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pytz==2024.1
PyWavelets @ file:///C:/b/abs_a8r4b1511a/croot/pywavelets_1670425185881/work
pywin32==306
PyYAML @ file:///C:/b/abs_782o3mbw7z/croot/pyyaml_1698096085010/work
pyyaml_env_tag==0.1
referencing==0.30.2
regex==2023.12.25
requests @ file:///C:/b/abs_316c2inijk/croot/requests_1690400295842/work
requests-oauthlib==1.3.0
retrying==1.3.4
rpds-py==0.12.0
rsa @ file:///tmp/build/80754af9/rsa_1614366226499/work
ruamel.yaml==0.18.5
ruamel.yaml.clib==0.2.8
scikit-image @ file:///C:/b/abs_63r0vmx78u/croot/scikit-image_1669241746873/work
scikit-learn==1.3.2
scipy==1.10.1
sip @ file:///C:/b/abs_edevan3fce/croot/sip_1698675983372/work
six @ file:///tmp/build/80754af9/six_1644875935023/work
snowballstemmer==2.2.0
soupsieve==2.5
stack-data==0.6.3
tenacity==8.2.3
tensorboard @ file:///C:/Users/builder/adipietro/mc3/tf210/conda-bld/tensorboard_1669760968711/work/tensorboard-2.10.0-py3-none-any.whl
tensorboard-data-server @ file:///C:/b/abs_2fhvpo862s/croot/tensorboard-data-server_1670853600144/work/tensorboard_data_server-0.6.1-py3-none-any.whl
tensorboard-plugin-wit @ file:///C:/tf/b/tensorboard-plugin-wit_1660162132996/work/tensorboard_plugin_wit-1.8.1-py3-none-any.whl
tensorboardX==2.6.2.2
threadpoolctl==3.2.0
tifffile @ file:///C:/b/abs_45o5chuqwt/croot/tifffile_1695107511025/work
tikzplotlib==0.10.1
tomli @ file:///C:/Windows/TEMP/abs_ac109f85-a7b3-4b4d-bcfd-52622eceddf0hy332ojo/croots/recipe/tomli_1657175513137/work
toolz @ file:///C:/b/abs_cfvk6rc40d/croot/toolz_1667464080130/work
torch==1.12.1
torchaudio==0.12.1
torchvision==0.13.1
tornado @ file:///C:/b/abs_0cbrstidzg/croot/tornado_1696937003724/work
tqdm @ file:///C:/b/abs_f76j9hg7pv/croot/tqdm_1679561871187/work
traitlets==5.13.0
transforms3d==0.4.1
trimesh==4.1.3
typing_extensions @ file:///C:/b/abs_213vg2cd1l/croot/typing_extensions_1690297804941/work
urllib3 @ file:///C:/b/abs_9cmlsrm3ys/croot/urllib3_1698257595508/work
watchdog==3.0.0
wcwidth==0.2.9
webcolors==1.13
Werkzeug==3.0.1
widgetsnbextension==4.0.9
win-inet-pton @ file:///C:/ci/win_inet_pton_1605306167264/work
yarl @ file:///C:/b/abs_8bxwdyhjvp/croot/yarl_1701105248152/work
yoco==1.1.1
zipp @ file:///C:/b/abs_b9jfdr908q/croot/zipp_1672387552360/work