dnnf installation errors

[shenw000]

I am installing dnnf on my GPU workstation with OS Ubuntu 20.04 by Source Install. The cuda version is 11.3.
After following your installation instructions on https://github.com/dlshriver/dnnf, I typed in the following command
./install.sh --include-cleverhans --include-foolbox --include-tensorfuzz

The execution of the above command leads to the following errors (part of the errors), where the bold line indicates the starting of errors. Can you point me to how to fix the issue? Thanks in advance!

Due to the large errors list, I included the complete error list in the attachment.

................

Building wheels for collected packages: absl-py, pyflann, gast, grpcio, termcolor, html5lib
Building wheel for absl-py (setup.py) ... done
Created wheel for absl-py: filename=absl_py-0.11.0-py2-none-any.whl size=127443 sha256=a6e6a8ece588e5c948f0a8011eda593302785611d94750c2399e094199083c70
Stored in directory: /home/shenw/.cache/pip/wheels/be/12/8b/0c41b135a6383624dedb48fa1c3a4c99f7961edafcb42e8189
Building wheel for pyflann (setup.py) ... done
Created wheel for pyflann: filename=pyflann-1.6.14-py2-none-any.whl size=7187134 sha256=e20ed18fc315fb585acbdfb1a653a4c9faa78ab44d06099caf18422d286524b6
Stored in directory: /home/shenw/.cache/pip/wheels/17/76/ba/e154e4f689dde77a3432fa0d4e031eb130ae12c67fbc5b883d
Building wheel for gast (setup.py) ... done
Created wheel for gast: filename=gast-0.5.3-py2-none-any.whl size=19444 sha256=908adba73077d7f7f5b05ae1dfab1be6c3551139066ed7e10908427972b06684
Stored in directory: /home/shenw/.cache/pip/wheels/de/fe/24/e89a0314a054e758bd6c765ce25d81337da83ccd207f9a9603
Building wheel for grpcio (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /u02/shenw/sandbox/nn_assurance/dnnf/.venv/tensorfuzz/.venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vUYCgP/grpcio/setup.py'"'"'; file='"'"'/tmp/pip-install-vUYCgP/grpcio/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-wkGcs6
cwd: /tmp/pip-install-vUYCgP/grpcio/
Complete output (599 lines):
Found cython-generated files...
running bdist_wheel
running build
running build_py
running build_project_metadata
creating python_build
creating python_build/lib.linux-x86_64-2.7
creating python_build/lib.linux-x86_64-2.7/grpc
copying src/python/grpcio/grpc/_compression.py -> python_build/lib.linux-x86_64-2.7/grpc
copying src/python/grpcio/grpc/_auth.py -> python_build/lib.linux-x86_64-2.7/grpc
copying src/python/grpcio/grpc/_runtime_protos.py -> python_build/lib.linux-x86_64-2.7/grpc
copying src/python/grpcio/grpc/_channel.py -> python_build/lib.linux-x86_64-2.7/grpc
copying src/python/grpcio/grpc/init.py -> python_build/lib.linux-x86_64-2.7/grpc
copying src/python/grpcio/grpc/_grpcio_metadata.py -> python_build/lib.linux-x86_64-2.7/grpc
copying src/python/grpcio/grpc/_simple_stubs.py -> python_build/lib.linux-x86_64-2.7/grpc
copying src/python/grpcio/grpc/_interceptor.py -> python_build/lib.linux-x86_64-2.7/grpc
copying src/python/grpcio/grpc/_common.py -> python_build/lib.linux-x86_64-2.7/grpc
copying src/python/grpcio/grpc/_utilities.py -> python_build/lib.linux-x86_64-2.7/grpc
copying src/python/grpcio/grpc/_plugin_wrapping.py -> python_build/lib.linux-x86_64-2.7/grpc
copying src/python/grpcio/grpc/_server.py -> python_build/lib.linux-x86_64-2.7/grpc
creating python_build/lib.linux-x86_64-2.7/grpc/aio
copying src/python/grpcio/grpc/aio/_call.py -> python_build/lib.linux-x86_64-2.7/grpc/aio
copying src/python/grpcio/grpc/aio/_base_server.py -> python_build/lib.linux-x86_64-2.7/grpc/aio
copying src/python/grpcio/grpc/aio/_channel.py -> python_build/lib.linux-x86_64-2.7/grpc/aio
copying src/python/grpcio/grpc/aio/init.py -> python_build/lib.linux-x86_64-2.7/grpc/aio
copying src/python/grpcio/grpc/aio/_typing.py -> python_build/lib.linux-x86_64-2.7/grpc/aio
copying src/python/grpcio/grpc/aio/_base_call.py -> python_build/lib.linux-x86_64-2.7/grpc/aio
copying src/python/grpcio/grpc/aio/_utils.py -> python_build/lib.linux-x86_64-2.7/grpc/aio
copying src/python/grpcio/grpc/aio/_interceptor.py -> python_build/lib.linux-x86_64-2.7/grpc/aio
copying src/python/grpcio/grpc/aio/_metadata.py -> python_build/lib.linux-x86_64-2.7/grpc/aio
copying src/python/grpcio/grpc/aio/_base_channel.py -> python_build/lib.linux-x86_64-2.7/grpc/aio
copying src/python/grpcio/grpc/aio/_server.py -> python_build/lib.linux-x86_64-2.7/grpc/aio
creating python_build/lib.linux-x86_64-2.7/grpc/_cython
copying src/python/grpcio/grpc/_cython/init.py -> python_build/lib.linux-x86_64-2.7/grpc/_cython
creating python_build/lib.linux-x86_64-2.7/grpc/experimental
copying src/python/grpcio/grpc/experimental/session_cache.py -> python_build/lib.linux-x86_64-2.7/grpc/experimental
copying src/python/grpcio/grpc/experimental/init.py -> python_build/lib.linux-x86_64-2.7/grpc/experimental
copying src/python/grpcio/grpc/experimental/gevent.py -> python_build/lib.linux-x86_64-2.7/grpc/experimental
creating python_build/lib.linux-x86_64-2.7/grpc/beta
copying src/python/grpcio/grpc/beta/_client_adaptations.py -> python_build/lib.linux-x86_64-2.7/grpc/beta
copying src/python/grpcio/grpc/beta/_server_adaptations.py -> python_build/lib.linux-x86_64-2.7/grpc/beta
copying src/python/grpcio/grpc/beta/init.py -> python_build/lib.linux-x86_64-2.7/grpc/beta
copying src/python/grpcio/grpc/beta/utilities.py -> python_build/lib.linux-x86_64-2.7/grpc/beta
copying src/python/grpcio/grpc/beta/_metadata.py -> python_build/lib.linux-x86_64-2.7/grpc/beta
copying src/python/grpcio/grpc/beta/implementations.py -> python_build/lib.linux-x86_64-2.7/grpc/beta
copying src/python/grpcio/grpc/beta/interfaces.py -> python_build/lib.linux-x86_64-2.7/grpc/beta
creating python_build/lib.linux-x86_64-2.7/grpc/framework
copying src/python/grpcio/grpc/framework/init.py -> python_build/lib.linux-x86_64-2.7/grpc/framework
creating python_build/lib.linux-x86_64-2.7/grpc/_cython/_cygrpc
copying src/python/grpcio/grpc/_cython/_cygrpc/init.py -> python_build/lib.linux-x86_64-2.7/grpc/_cython/_cygrpc
creating python_build/lib.linux-x86_64-2.7/grpc/experimental/aio
copying src/python/grpcio/grpc/experimental/aio/init.py -> python_build/lib.linux-x86_64-2.7/grpc/experimental/aio
creating python_build/lib.linux-x86_64-2.7/grpc/framework/foundation
copying src/python/grpcio/grpc/framework/foundation/logging_pool.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/foundation
copying src/python/grpcio/grpc/framework/foundation/stream_util.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/foundation
copying src/python/grpcio/grpc/framework/foundation/abandonment.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/foundation
copying src/python/grpcio/grpc/framework/foundation/stream.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/foundation
copying src/python/grpcio/grpc/framework/foundation/init.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/foundation
copying src/python/grpcio/grpc/framework/foundation/future.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/foundation
copying src/python/grpcio/grpc/framework/foundation/callable_util.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/foundation
creating python_build/lib.linux-x86_64-2.7/grpc/framework/interfaces
copying src/python/grpcio/grpc/framework/interfaces/init.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/interfaces
creating python_build/lib.linux-x86_64-2.7/grpc/framework/common
copying src/python/grpcio/grpc/framework/common/style.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/common
copying src/python/grpcio/grpc/framework/common/cardinality.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/common
copying src/python/grpcio/grpc/framework/common/init.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/common
creating python_build/lib.linux-x86_64-2.7/grpc/framework/interfaces/face
copying src/python/grpcio/grpc/framework/interfaces/face/face.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/interfaces/face
copying src/python/grpcio/grpc/framework/interfaces/face/init.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/interfaces/face
copying src/python/grpcio/grpc/framework/interfaces/face/utilities.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/interfaces/face
creating python_build/lib.linux-x86_64-2.7/grpc/framework/interfaces/base
copying src/python/grpcio/grpc/framework/interfaces/base/init.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/interfaces/base
copying src/python/grpcio/grpc/framework/interfaces/base/utilities.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/interfaces/base
copying src/python/grpcio/grpc/framework/interfaces/base/base.py -> python_build/lib.linux-x86_64-2.7/grpc/framework/interfaces/base
creating python_build/lib.linux-x86_64-2.7/grpc/_cython/_credentials
copying src/python/grpcio/grpc/_cython/_credentials/roots.pem -> python_build/lib.linux-x86_64-2.7/grpc/_cython/_credentials
running build_ext
Found cython-generated files...
building 'grpc._cython.cygrpc' extension
creating python_build/temp.linux-x86_64-2.7
creating python_build/temp.linux-x86_64-2.7/src
creating python_build/temp.linux-x86_64-2.7/src/python
creating python_build/temp.linux-x86_64-2.7/src/python/grpcio
creating python_build/temp.linux-x86_64-2.7/src/python/grpcio/grpc
creating python_build/temp.linux-x86_64-2.7/src/python/grpcio/grpc/_cython
creating python_build/temp.linux-x86_64-2.7/src/core
creating python_build/temp.linux-x86_64-2.7/src/core/ext
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/census
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/health
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/lb_policy
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/lb_policy/grpclb
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/lb_policy/pick_first
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/lb_policy/priority
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/lb_policy/ring_hash
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/lb_policy/round_robin
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/lb_policy/weighted_target
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/lb_policy/xds
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/resolver
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/resolver/dns
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/resolver/dns/c_ares
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/resolver/dns/native
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/resolver/fake
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/resolver/google_c2p
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/resolver/sockaddr
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_channel/resolver/xds
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/client_idle
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/deadline
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/fault_injection
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/http
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/http/client
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/http/message_compress
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/http/server
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/max_age
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/message_size
creating python_build/temp.linux-x86_64-2.7/src/core/ext/filters/workarounds
creating python_build/temp.linux-x86_64-2.7/src/core/ext/transport
creating python_build/temp.linux-x86_64-2.7/src/core/ext/transport/chttp2
creating python_build/temp.linux-x86_64-2.7/src/core/ext/transport/chttp2/alpn
creating python_build/temp.linux-x86_64-2.7/src/core/ext/transport/chttp2/client

..........................

dnnf_installation_errors.txt

[shenw000]

OK, I removed the last option in the dnnf install.sh (--include-tensorfuzz), and the dnnf installation is successful. So I guess that we can do the install without the --include-tensorfuzz option. So just do the dnnf install as below.
./install.sh --include-cleverhans --include-foolbox

Once I successfully installed the dnnf, i performed a test run using the command
dnnf properties/dronet_property_0.py --network N onnx/dronet.onnx --backend cleverhans.projected_gradient_descent

and it produced the following output with errors. I thought the above command doesn't use the GPU (as the default option is no cuda). But the following errors appear to contain cuda operations on the RTX 8000 GPU. Could you point me to the potential source of the errors. Thank you in advance!

Falsifying: Forall(x, (((0 <= x) & (x <= 1) & (N(slice(2, -3, None), 1) <= -2.1972245773362196)) ==> ((-0.08726646259971647 <= N(slice(2, -1, None), 0)) & (N(slice(2, -1, None), 0) <= 0.08726646259971647))))

Network N:
Input_0 : Input([ -1 200 200 1], dtype=float32)
Transpose_0 : Transpose(Input_0, permutation=[0 3 1 2])
Conv_0 : Conv(Transpose_0, out_channels=32, kernel_shape=[5, 5], strides=[2, 2], pads=[1, 1, 2, 2], group=1, dilations=[1, 1])
MaxPool_0 : MaxPool(Conv_0)
Identity_0 : Identity(MaxPool_0)
Conv_1 : Conv(Identity_0, out_channels=32, kernel_shape=[1, 1], strides=[2, 2], pads=[0, 0, 0, 0], group=1, dilations=[1, 1])
BatchNormalization_0 : BatchNormalization(Identity_0)
Relu_0 : Relu(BatchNormalization_0)
Conv_2 : Conv(Relu_0, out_channels=32, kernel_shape=[3, 3], strides=[2, 2], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
BatchNormalization_1 : BatchNormalization(Conv_2)
Relu_1 : Relu(BatchNormalization_1)
Conv_3 : Conv(Relu_1, out_channels=32, kernel_shape=[3, 3], strides=[1, 1], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
Add_0 : Add(Conv_1, Conv_3)
Conv_4 : Conv(Add_0, out_channels=64, kernel_shape=[1, 1], strides=[2, 2], pads=[0, 0, 0, 0], group=1, dilations=[1, 1])
BatchNormalization_2 : BatchNormalization(Add_0)
Relu_2 : Relu(BatchNormalization_2)
Conv_5 : Conv(Relu_2, out_channels=64, kernel_shape=[3, 3], strides=[2, 2], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
BatchNormalization_3 : BatchNormalization(Conv_5)
Relu_3 : Relu(BatchNormalization_3)
Conv_6 : Conv(Relu_3, out_channels=64, kernel_shape=[3, 3], strides=[1, 1], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
Add_1 : Add(Conv_4, Conv_6)
Conv_7 : Conv(Add_1, out_channels=128, kernel_shape=[1, 1], strides=[2, 2], pads=[0, 0, 0, 0], group=1, dilations=[1, 1])
BatchNormalization_4 : BatchNormalization(Add_1)
Relu_4 : Relu(BatchNormalization_4)
Conv_8 : Conv(Relu_4, out_channels=128, kernel_shape=[3, 3], strides=[2, 2], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
BatchNormalization_5 : BatchNormalization(Conv_8)
Relu_5 : Relu(BatchNormalization_5)
Conv_9 : Conv(Relu_5, out_channels=128, kernel_shape=[3, 3], strides=[1, 1], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
Add_2 : Add(Conv_7, Conv_9)
Transpose_1 : Transpose(Add_2, permutation=[0 2 3 1])
Flatten_0 : Flatten(Transpose_1, axis=1)
Relu_6 : Relu(Flatten_0)
Identity_1 : Identity(Relu_6)
MatMul_0 : MatMul(Identity_1, ndarray(shape=(6272, 1)))
Add_3 : Add(MatMul_0, [0.01033337])
Identity_2 : Identity(Add_3)
MatMul_1 : MatMul(Identity_1, ndarray(shape=(6272, 1)))
Add_4 : Add(MatMul_1, [1.1624445])
Sigmoid_0 : Sigmoid(Add_4)
Identity_3 : Identity(Sigmoid_0)
Concat_0 : Concat(['Identity_2', 'Identity_3'])

2022-09-27 15:15:40.163850: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-09-27 15:15:41.687585: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 29819 MB memory: -> device: 0, name: Quadro RTX 8000, pci bus id: 0000:19:00.0, compute capability: 7.5
2022-09-27 15:15:41.688906: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 41378 MB memory: -> device: 1, name: Quadro RTX 8000, pci bus id: 0000:1a:00.0, compute capability: 7.5
2022-09-27 15:15:41.690076: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 41432 MB memory: -> device: 2, name: Quadro RTX 8000, pci bus id: 0000:67:00.0, compute capability: 7.5
2022-09-27 15:15:41.691208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 41425 MB memory: -> device: 3, name: Quadro RTX 8000, pci bus id: 0000:68:00.0, compute capability: 7.5
2022-09-27 15:15:42.363408: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8204
2022-09-27 15:15:43.706544: E tensorflow/stream_executor/cuda/cuda_blas.cc:197] failed to set new cublas math mode: CUBLAS_STATUS_INVALID_VALUE
2022-09-27 15:15:43.706581: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:442 : INTERNAL: Failed initializing math mode
Traceback (most recent call last):
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/converters/tensorflow.py", line 58, in wrapped_func
self._cache[func] = func(*args, **kwargs)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/converters/tensorflow.py", line 588, in matmul_func
result = tf.matmul(a, b)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 7107, in raise_from_not_ok_status
raise core._status_to_exception(e) from None # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InternalError: Failed initializing math mode [Op:MatMul]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/shenw/workspace/anaconda3/envs/dnnf/bin/dnnf", line 8, in
sys.exit(main())
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnf/main.py", line 66, in main
return main(**vars(args), extra_args=extra_args)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnf/main.py", line 44, in main
result = falsify(phi, **kwargs)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnf/falsifier.py", line 75, in falsify
FalsificationModel(prop),
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnf/model.py", line 14, in init
self.op_graph = prop.suffixed_op_graph()
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnf/reduction.py", line 256, in suffixed_op_graph
OpGraphMerger().merge([OperationGraph([new_output_op])]).simplify()
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/graph.py", line 36, in simplify
return simplify(self.copy())
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/transformers/simplifiers/init.py", line 69, in simplify
simplified_graph = OperationGraph(dnn.walk(simplifier))
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/graph.py", line 24, in walk
result.append(visitor.visit(output_operation))
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/transformers/simplifiers/base.py", line 45, in visit
operation = simplifier.visit(operation)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/transformers/simplifiers/base.py", line 25, in visit
operation = super().generic_visit(operation)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/transformers/base.py", line 26, in generic_visit
new_value = self.visit(value)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/transformers/simplifiers/base.py", line 25, in visit
operation = super().generic_visit(operation)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/transformers/base.py", line 26, in generic_visit
new_value = self.visit(value)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/transformers/simplifiers/base.py", line 25, in visit
operation = super().generic_visit(operation)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/transformers/base.py", line 26, in generic_visit
new_value = self.visit(value)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/transformers/simplifiers/base.py", line 26, in visit
result = super().visit(operation)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/transformers/base.py", line 16, in visit
result = visitor(operation)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/transformers/simplifiers/convert_add.py", line 26, in visit_Add
input_details = OperationGraph([input_op]).output_details[0]
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/graph.py", line 60, in output_details
output = self(
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/graph.py", line 135, in call
output = tf_func(*x, **kwargs)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/converters/tensorflow.py", line 25, in func
output = output_func(*inputs)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/converters/tensorflow.py", line 67, in wrapped_func
raise TensorflowConverterError(
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/converters/tensorflow.py", line 58, in wrapped_func
self._cache[func] = func(*args, **kwargs)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/dnnv/nn/converters/tensorflow.py", line 588, in matmul_func
result = tf.matmul(a, b)
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/shenw/workspace/anaconda3/envs/dnnf/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 7107, in raise_from_not_ok_status
raise core._status_to_exception(e) from None # pylint: disable=protected-access
dnnv.nn.converters.tensorflow.TensorflowConverterError: InternalError:

[dlshriver]

From what you've posted, I'm not sure why the tensorfuzz installation failed. If you need to use tensorfuzz, I highly recommend using the provided Docker image, which should have everything pre-installed.

Also, I generally recommend installing dnnf with pip rather than ./install.sh. The install script is only tested on a couple different machines and will eventually be deprecated in favor of the pip install. Using pip, dnnf can be installed with pip install dnnf[foolbox,cleverhans]. If you want to be able to make changes to the source, you can clone the repo and from the root directory run pip install -e .[foolbox,cleverhans].

DNNF shouldn't run on the GPU without the --cuda flag. In this case it looks like TensorFloww, through DNNV (which DNNF depends on) sees a GPU and is trying to use it but fails. You should be able to avoid this by setting the environment variable CUDA_VISIBLE_DEVICES="" before running DNNF. It's not clear to me why it is failing here rather than just using the CPU, but I'll look into it a bit more.

[shenw000]

Thank you, Dave! Yes, I am able to successfully pip install the dnnf.

But when I ran the following command to perform benchmark:
dnnf properties/dronet_property_0.py --network N onnx/dronet.onnx --backend cleverhans.projected_gradient_descent --cuda
errors occurred. The errors are in the attachment.

dnnf_benchmark_errors.txt

Any ideas what went wrong?

Additionally, I found some related issues at
tensorflow/tensorflow#57359

co1lin commented on Aug 25
......

tf.config.experimental.enable_tensor_float_32_execution(False)
for gpu in tf.config.experimental.list_physical_devices('GPU'):
tf.config.experimental.set_memory_growth(gpu, True)

......

This solution disables TF32 related optimizations. But I don't know where in your code we could insert these lines, as I didn't find the location of your code using tensorflow. Please advise.

Thanks!

[dlshriver]

Sorry for the delay, I'm currently busy finishing up my thesis and preparing to defend and haven't been able to do much on this.

Is this the same error you report in #14?