[MACE v1.0.0] rpcmem_init failed on GPU

[gasgallo]

Before you open an issue, please make sure you have tried the following steps:

1. Make sure your environment is the same with (https://mace.readthedocs.io/en/latest/installation/env_requirement.html).
2. Have you ever read the document for your usage?
3. Check if your issue appears in HOW-TO-DEBUG or FAQ.
4. The form below must be filled.

---

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
• NDK version(e.g., 15c): 19c
• GCC version(if compiling for host, e.g., 5.4.0): 5.4.0
• MACE version (Use the command: git describe --long --tags): 1.0.0
• Python version(2.7): 3.6
• Bazel version (e.g., 0.13.0): 0.16.0
• CMake version: 3.16.0

Model deploy file (*.yml)

library_name: model
target_abis: [arm64-v8a, armeabi-v7a]
model_graph_format: code
model_data_format: code

models:
  model:
    platform: onnx
    model_file_path: "/models/cb487a80da4a4431a846be8a0d73e5b5.onnx"
    model_sha256_checksum: ba2fa1b0d4850a1831327c98c56c97acc5bce949fd2a344b2713d4ee1e35a742
    subgraphs:
      - input_tensors:
          - data
        input_shapes:
          - 1,3,192,192
        input_data_formats:
          - NCHW
        output_tensors:
          - mobilepose0_conv1_fwd
        output_shapes:
          - 1,128,6,6
    obfuscate: 0
    runtime: cpu+gpu
    winograd: 2

Describe the problem

Model fails to run on GPU with openCL with below error.

To Reproduce

Steps to reproduce the problem:

1. cd /path/to/mace
2. python3 tools/converter.py convert --config=/path/to/yml
3. python3 tools/converter.py run --config=/path/to/yml

Error information / logs

Please include the full log and/or traceback here.

WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading: 
Loading: 0 packages loaded
WARNING: The major revision of the Android NDK referenced by android_ndk_repository rule 'androidndk' is 19. The major revisions supported by Bazel are [10, 11, 12, 13, 14, 15, 16]. Bazel will attempt to treat the NDK as if it was r16. This may cause compilation and linkage problems. Please download a supported NDK version.
Analyzing: target //mace/tools:mace_run_static (2 packages loaded)
Analyzing: target //mace/tools:mace_run_static (11 packages loaded)
INFO: Analysed target //mace/tools:mace_run_static (33 packages loaded).
INFO: Found 1 target...
[0 / 13] [-----] BazelWorkspaceStatusAction stable-status.txt
[29 / 205] no action
[191 / 237] [-----] Creating source manifest for //mace/tools:mace_run_static
[207 / 718] Compiling mace/ops/reverse.cc; 0s local ... (6 actions running)
[212 / 718] Compiling mace/ops/group_norm.cc; 2s local ... (6 actions running)
[242 / 718] Compiling mace/ops/ref/deconv_2d.cc; 1s local ... (3 actions, 2 running)
[573 / 718] Compiling mace/ops/opencl/buffer/softmax.cc; 1s local ... (5 actions running)
[581 / 718] Compiling mace/ops/opencl/image/resize_bicubic.cc; 2s local ... (5 actions running)
[591 / 718] Compiling mace/ops/opencl/image/extract_image_patches.cc; 2s local ... (3 actions running)
[598 / 718] Compiling mace/ops/opencl/image/eltwise.cc; 2s local ... (6 actions running)
[610 / 718] Compiling mace/ops/opencl/buffer/conv_2d_general.cc; 1s local ... (6 actions running)
[629 / 718] Compiling mace/ops/opencl/image/depthwise_deconv2d.cc; 1s local ... (5 actions running)
[650 / 718] Compiling mace/ops/arm/base/gemm.cc; 1s local ... (5 actions running)
[709 / 721] Compiling mace/libmace/mace.cc; 1s local ... (6 actions, 5 running)
[729 / 740] Compiling mace/ops/addn.cc; 2s local ... (5 actions running)
[754 / 762] Compiling mace/ops/eltwise.cc; 4s local ... (5 actions running)
[780 / 781] [-----] Linking mace/tools/mace_run_static
Target //mace/tools:mace_run_static up-to-date:
  bazel-bin/mace/tools/mace_run_static
INFO: Elapsed time: 76.118s, Critical Path: 6.67s
INFO: 191 processes: 191 local.
INFO: Build completed successfully, 194 total actions
INFO: Build completed successfully, 194 total actions
CMD> bazel version
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Build label: 0.16.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jul 31 17:01:24 2018 (1533056484)
Build timestamp: 1533056484
Build timestamp as int: 1533056484
CMD> bazel build //mace/proto:mace_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/proto:mace_py (12 packages loaded)
INFO: Analysed target //mace/proto:mace_py (17 packages loaded).
INFO: Found 1 target...
[2 / 9] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/proto:mace_py up-to-date:
bazel-genfiles/mace/proto/mace_pb2.py
INFO: Elapsed time: 1.681s, Critical Path: 0.03s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
CMD> cp -f bazel-genfiles/mace/proto/mace_pb2.py tools/python/py_proto
CMD> bazel build //mace/proto:micro_mem_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/proto:micro_mem_py (12 packages loaded)
INFO: Analysed target //mace/proto:micro_mem_py (17 packages loaded).
INFO: Found 1 target...
[0 / 4] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/proto:micro_mem_py up-to-date:
bazel-genfiles/mace/proto/micro_mem_pb2.py
INFO: Elapsed time: 1.648s, Critical Path: 0.05s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
CMD> cp -f bazel-genfiles/mace/proto/micro_mem_pb2.py tools/python/py_proto
CMD> bazel build //third_party/caffe:caffe_py
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //third_party/caffe:caffe_py (12 packages loaded)
INFO: Analysed target //third_party/caffe:caffe_py (17 packages loaded).
INFO: Found 1 target...
[0 / 5] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //third_party/caffe:caffe_py up-to-date:
bazel-genfiles/third_party/caffe/caffe_pb2.py
INFO: Elapsed time: 1.709s, Critical Path: 0.05s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
CMD> cp -f bazel-genfiles/third_party/caffe/caffe_pb2.py tools/python/py_proto
tools/converter.py:319: Downloading model, please wait ...
tools/converter.py:289: 
Downloaded successfully.
* Build //mace/tools:mace_run_static with ABI arm64-v8a
('build', '//mace/tools:mace_run_static', '--config', 'android', '--cpu=arm64-v8a', '--define', 'neon=true', '--define', 'opencl=true', '--define', 'quantize=false', '--define', 'bfloat16=false', '--define', 'fp16=false', '--define', 'rpcmem=true', '--define', 'hexagon=false', '--define', 'hta=false', '--define', 'apu=false', '--config', 'optimization', '--config', 'symbol_hidden', '--per_file_copt=mace/tools/mace_run.cc@-DMODEL_GRAPH_FORMAT_CODE')
('build', '//mace/tools:mace_run_static', '--config', 'android', '--cpu=arm64-v8a', '--define', 'neon=true', '--define', 'opencl=true', '--define', 'quantize=false', '--define', 'bfloat16=false', '--define', 'fp16=false', '--define', 'rpcmem=true', '--define', 'hexagon=false', '--define', 'hta=false', '--define', 'apu=false', '--config', 'optimization', '--config', 'symbol_hidden', '--per_file_copt=mace/tools/mace_run.cc@-DMODEL_GRAPH_FORMAT_CODE')
Build done!
******************************************************
          Run model model on TPS980P          
******************************************************
Generate input file:  build/model/_tmp/model/52ccb254544535bc260dea8cc3fcac30/TPS980P_rk3399/arm64-v8a/model_input_data
Generate input file done.
* Run 'model' with round=100, restart_round=1, tuning=False, out_of_range_check=False, num_threads=(-1,), cpu_affinity_policy=(1,), gpu_perf_hint=(3,), gpu_priority_hint=(3,)
Push build/model/_tmp/model/52ccb254544535bc260dea8cc3fcac30/TPS980P_rk3399/arm64-v8a/model_input_data to /data/local/tmp/mace_run
Push third_party/nnlib/arm64-v8a/libhexagon_controller.so to /data/local/tmp/mace_run
Push build/model/_tmp/arm64-v8a/mace_run_static to /data/local/tmp/mace_run
Push /tmp/cmd_file-model-1610601805.2930682 to /data/local/tmp/mace_run/cmd_file-model-1610601805.2930682
I mace/tools/mace_run.cc:544] model name: model
I mace/tools/mace_run.cc:545] mace version: v1.0.0-0-g0945e49
I mace/tools/mace_run.cc:546] input node: data
I mace/tools/mace_run.cc:547] input shape: 1,3,192,192
I mace/tools/mace_run.cc:548] output node: mobilepose0_conv1_fwd
I mace/tools/mace_run.cc:549] output shape: 1,128,6,6
I mace/tools/mace_run.cc:550] input_file: /data/local/tmp/mace_run/model_input
I mace/tools/mace_run.cc:551] output_file: /data/local/tmp/mace_run/model_out
I mace/tools/mace_run.cc:552] input dir:
I mace/tools/mace_run.cc:553] output dir:
I mace/tools/mace_run.cc:554] model_data_file:
I mace/tools/mace_run.cc:555] model_file:
I mace/tools/mace_run.cc:556] apu_cache_policy: 0
I mace/tools/mace_run.cc:557] apu_binary_file:
I mace/tools/mace_run.cc:558] apu_storage_file:
I mace/tools/mace_run.cc:559] device: CPU
I mace/tools/mace_run.cc:560] round: 100
I mace/tools/mace_run.cc:561] restart_round: 1
I mace/tools/mace_run.cc:562] gpu_perf_hint: 3
I mace/tools/mace_run.cc:563] gpu_priority_hint: 3
I mace/tools/mace_run.cc:564] num_threads: -1
I mace/tools/mace_run.cc:565] cpu_affinity_policy: 1
I mace/tools/mace_run.cc:568] limit_opencl_kernel_time: 0
I mace/tools/mace_run.cc:573] opencl_queue_window_size: 0
I mace/libmace/mace.cc:558] Creating MaceEngine, MACE version: v1.0.0-0-g0945e49
I mace/libmace/mace.cc:628] Initializing MaceEngine
I mace/libmace/mace.cc:790] Destroying MaceEngine
I mace/tools/mace_run.cc:616] restart round 0
I mace/libmace/mace.cc:558] Creating MaceEngine, MACE version: v1.0.0-0-g0945e49
I mace/libmace/mace.cc:628] Initializing MaceEngine
I mace/tools/mace_run.cc:286] Create Mace Engine latency: 49.954 ms
I mace/tools/mace_run.cc:293] Total init latency: 50.098 ms
I mace/tools/mace_run.cc:387] Warm up run
I mace/tools/mace_run.cc:423] 1st warm up run latency: 106.728 ms
I mace/tools/mace_run.cc:431] Run model
I mace/tools/mace_run.cc:493] Average latency: 65.5742 ms
I mace/tools/mace_run.cc:508] Write output file /data/local/tmp/mace_run/model_out_mobilepose0_conv1_fwd with size 18432 done.
========================================================
     capability(CPU)        init      warmup     run_avg
========================================================
time          52.470      50.098     106.728      65.574
I mace/libmace/mace.cc:790] Destroying MaceEngine
Running finished!
Dana service is not available.
* Run 'model' with round=100, restart_round=1, tuning=False, out_of_range_check=False, num_threads=(-1,), cpu_affinity_policy=(1,), gpu_perf_hint=(3,), gpu_priority_hint=(3,)
Push build/landmark/_tmp/model/52ccb254544535bc260dea8cc3fcac30/TPS980P_rk3399/arm64-v8a/model_input_data to /data/local/tmp/mace_run
Push third_party/nnlib/arm64-v8a/libhexagon_controller.so to /data/local/tmp/mace_run
Push build/landmark/_tmp/arm64-v8a/mace_run_static to /data/local/tmp/mace_run
Push /tmp/cmd_file-model-1610601817.087663 to /data/local/tmp/mace_run/cmd_file-model-1610601817.087663
I mace/tools/mace_run.cc:544] model name: model
I mace/tools/mace_run.cc:545] mace version: v1.0.0-0-g0945e49
I mace/tools/mace_run.cc:546] input node: data
I mace/tools/mace_run.cc:547] input shape: 1,3,192,192
I mace/tools/mace_run.cc:548] output node: mobilepose0_conv1_fwd
I mace/tools/mace_run.cc:549] output shape: 1,128,6,6
I mace/tools/mace_run.cc:550] input_file: /data/local/tmp/mace_run/model_input
I mace/tools/mace_run.cc:551] output_file: /data/local/tmp/mace_run/model_out
I mace/tools/mace_run.cc:552] input dir:
I mace/tools/mace_run.cc:553] output dir:
I mace/tools/mace_run.cc:554] model_data_file:
I mace/tools/mace_run.cc:555] model_file:
I mace/tools/mace_run.cc:556] apu_cache_policy: 0
I mace/tools/mace_run.cc:557] apu_binary_file:
I mace/tools/mace_run.cc:558] apu_storage_file:
I mace/tools/mace_run.cc:559] device: GPU
I mace/tools/mace_run.cc:560] round: 100
I mace/tools/mace_run.cc:561] restart_round: 1
I mace/tools/mace_run.cc:562] gpu_perf_hint: 3
I mace/tools/mace_run.cc:563] gpu_priority_hint: 3
I mace/tools/mace_run.cc:564] num_threads: -1
I mace/tools/mace_run.cc:565] cpu_affinity_policy: 1
I mace/tools/mace_run.cc:568] limit_opencl_kernel_time: 0
I mace/tools/mace_run.cc:573] opencl_queue_window_size: 0
I mace/libmace/mace.cc:558] Creating MaceEngine, MACE version: v1.0.0-0-g0945e49
I mace/libmace/mace.cc:628] Initializing MaceEngine
I mace/libmace/mace.cc:790] Destroying MaceEngine
I mace/tools/mace_run.cc:616] restart round 0
W ./mace/utils/tuner.h:201] Failed to read tuned param file: /data/local/tmp/mace_run/model_tuned_opencl_parameter.device.rk3399.bin
I mace/libmace/mace.cc:558] Creating MaceEngine, MACE version: v1.0.0-0-g0945e49
W mace/core/kv_storage.cc:109] Failed to read kv store file: /data/local/tmp/mace_run/interior//mace_cl_compiled_program.bin
W mace/core/runtime/opencl/opencl_runtime.cc:442] Load OpenCL cached compiled kernel file failed. Please make sure the storage directory exist and you have Write&Read permission
F mace/core/rpcmem.cc:23] Check failed: rm.flag == 0 rpcmem_init failed!
F mace/core/rpcmem.cc:23] backtrace:
F mace/core/rpcmem.cc:23]  pc 0x58cdeee2c4
F mace/core/rpcmem.cc:23]  pc 0x58cdef03a0
F mace/core/rpcmem.cc:23]  pc 0x58cdef0354
F mace/core/rpcmem.cc:23]  pc 0x58cdef0550
F mace/core/rpcmem.cc:23]  pc 0x58cdef05ac
F mace/core/rpcmem.cc:23]  pc 0x58cdee1148
F mace/core/rpcmem.cc:23]  pc 0x58cded0164
F mace/core/rpcmem.cc:23]  pc 0x58cdecff30
F mace/core/rpcmem.cc:23]  pc 0x58cde0ab68
F mace/core/rpcmem.cc:23]  pc 0x58cde0edec
F mace/core/rpcmem.cc:23]  pc 0x58cde0edb8
F mace/core/rpcmem.cc:23]  pc 0x58cddd10e0
F mace/core/rpcmem.cc:23]  pc 0x58cddd1a04
F mace/core/rpcmem.cc:23]  pc 0x58cddd422c
F mace/core/rpcmem.cc:23]  pc 0x58cddd43c4
F mace/core/rpcmem.cc:23]  pc 0x7d26d04590 __libc_init
F mace/core/rpcmem.cc:23]  pc 0x58cddd0f40
Aborted

Additional context

Add any other context about the problem here, e.g., what you have modified about the code.

[lee-bin]

@gasgallo rpcmem is a qualcomm library that is enabled by default, which should not happen on other soc. You can disable it by modifying this line https://github.com/XiaoMi/mace/blob/v1.0.0/tools/sh_commands.py#L275

BTW, if the model's output_data_formats is NCHW, it should be specified in the yml.

[gasgallo]

@lee-bin thanks for the hint! So enable_rpcmem=True should default to False instead? Is this a bug?

[lee-bin]

Yes, it should be False by default to run on all the platforms. You can consider it a bug, we seldom test android devices other than Qualcomm or MTK.

[gasgallo]

@lee-bin can you check #710?

[lu229]

@gasgallo We have reconstructed the MACE framework and resolve this bug, thanks for your contribution!