Error compiling internal package libwally

[sosaucily]

Overview

Receiving an assembly compilation error when trying to build cfd-js. Shown below. The error has to do with libwally, but I can successfully build libwally directly with the same environment. Any ideas?

FAILED: libwally-core/build/CMakeFiles/wally.dir/src/secp256k1/src/secp256k1.c.o 
/usr/local/bin/ccache /usr/local/bin/ccache /usr/local/bin/ccache /usr/local/opt/ccache/libexec/cc -DBUILD_ELEMENTS -DHAVE_CONFIG_H -DSECP256K1_BUILD -DWALLY_CORE_BUILD -Dwally_EXPORTS -I/Users/user/Proj/dlc/cfd-js/external/libwally-core/include -I/Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/include -I/Users/user/Proj/dlc/cfd-js/build/libwally-core/build -I/Users/user/Proj/dlc/cfd-js/external/libwally-core/. -I/Users/user/Proj/dlc/cfd-js/external/libwally-core/src -I/Users/user/Proj/dlc/cfd-js/external/libwally-core/src/ccan -I/Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1 -I/Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src -O3 -DNDEBUG -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -fPIC -Wall -Wextra -Wno-unused-function -MD -MT libwally-core/build/CMakeFiles/wally.dir/src/secp256k1/src/secp256k1.c.o -MF libwally-core/build/CMakeFiles/wally.dir/src/secp256k1/src/secp256k1.c.o.d -o libwally-core/build/CMakeFiles/wally.dir/src/secp256k1/src/secp256k1.c.o -c /Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src/secp256k1.c
/Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src/secp256k1.c:7:9: warning: 'SECP256K1_BUILD' macro redefined [-Wmacro-redefined]
#define SECP256K1_BUILD
        ^
<command line>:3:9: note: previous definition is here
#define SECP256K1_BUILD 1
        ^
In file included from /Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src/secp256k1.c:14:
In file included from /Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src/field_impl.h:17:
In file included from /Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src/field_5x52_impl.h:19:
/Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src/field_5x52_asm_impl.h:282:3: error: invalid input constraint 'b' in asm
: "b"(b), "D"(r)
  ^
/Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src/field_5x52_asm_impl.h:497:3: error: invalid input constraint 'D' in asm
: "D"(r)
  ^
In file included from /Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src/secp256k1.c:15:
In file included from /Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src/scalar_impl.h:24:
/Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src/scalar_4x64_impl.h:370:7: error: unknown register name 'rax' in asm
    : "rax", "rdx", "r8", "r9", "r10", "r11", "r12", "r13", "r14", "cc");
      ^
/Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src/scalar_4x64_impl.h:449:7: error: unknown register name 'rax' in asm
    : "rax", "rdx", "r8", "r9", "r10", "r11", "r12", "r13", "cc");
      ^
/Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src/scalar_4x64_impl.h:494:52: error: invalid input constraint 'D' in asm
    : "g"(p0), "g"(p1), "g"(p2), "g"(p3), "g"(p4), "D"(r), "i"(SECP256K1_N_C_0), "i"(SECP256K1_N_C_1)
                                                   ^
/Users/user/Proj/dlc/cfd-js/external/libwally-core/src/secp256k1/src/scalar_4x64_impl.h:699:7: error: invalid output constraint '+d' in asm
    : "+d"(pb)
      ^
1 warning and 6 errors generated.
[48/212] Building CXX object googletest/build/googlemock/gtest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
ninja: build stopped: subcommand failed.
ERR! OMG Process terminated: 1
ERROR: "cmake_compile" exited with 1.
ERROR: "install_cmake" exited with 1.
npm ERR! code 1
npm ERR! path /Users/user/Proj/dlc/cfd-js
npm ERR! command failed
npm ERR! command sh -c prebuild-install --runtime napi --verbose || run-s install_cmake

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/user/.npm/_logs/2022-07-25T13_32_31_534Z-debug-0.log

System Details

OS: OSX
Apple clang version 13.1.6 (clang-1316.0.21.2.5)
Target: x86_64-apple-darwin21.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
LLVM Version: 13.0.0

[k-matsuzawa]

Thanks!　I'll look into it!

[k-matsuzawa]

@sosaucily
Sorry, could you give me some more information?

• target cfd-js version (v0.4.6 ?)
• cmake version (result of cmake --version )

[k-matsuzawa]

I checked the log above carefully and for some reason the architecture is wrong.

`-O3 -DNDEBUG -arch arm64 `
  - arm64
Target: x86_64-apple-darwin21.5.0
  - x64

What is the version of cmake?

Also, could you please set up the following and then run npm install and see what happens?

export CFDJS_UNUSE_ASSET=1

[k-matsuzawa]

Understood. You are using Rosetta2.

When calling cmake directly, you can switch by setting CMAKE_OSX_ARCHITECTURES. However, I have not yet confirmed that this works well when using npm, so I will check it now!

[k-matsuzawa]

@sosaucily
Hi!
I have done the build on x64 using Rosetta2 and everything runs fine (without any modifications).

One possibility is that the node.js in your environment is using arm64 instead of x64. Could you please unify the node.js architecture and see if you can run it?

checking architecture:

node -p process.arch

[sosaucily]

node -p process.arch -> arm64
arch -> i386

Incredible! great sleuthing, thanks for pointing that out. Yes, let me unify that and get back to you. 👍

[sosaucily]

Updates: I'm importing cfd-js in package.json as an npm import. Now that I switched to using an x64 compatible installation of node, strangely it doesn't seem like cfd-js is doing a full compilation after running npm install on the parent project. The installation of cfd-js happens too fast for it to be running make, and I don't see much under node_modules/cfd-js/build.

And I'm getting an error in my chrome-extension which imports from cfd-js about missing modules.

So to test, if I go into node_modules/cfd-js and type npm install that works, but npm run cmake_call fails.

However, both npm install and npm run cmake_all works on the cfd-js project alone, when I clone and pull that down.

1. Do you have any tips on why my project isn't calling cmake_all on cfd-js after running npm install?

2. Or why it's not compiling when i go directly into the node_modules/cfd-js folder (I'm not sure that could ever work but I wanted to try)? When I do go into the node_modules/cfd-js and try to compile, here's the error I get.

cfd-js git:(master) ✗ npm run cmake_all

> cfd-js@0.4.6 cmake_all
> run-s cmake_configure cmake_build_parallel


> cfd-js@0.4.6 cmake_configure
> run-script-os


> cfd-js@0.4.6 cmake_configure:darwin
> cmake-js configure -O build --CDENABLE_SHARED=on --CDENABLE_CAPI=off --CDENABLE_JSONAPI=off --CDTARGET_RPATH="@executable_path;./build/Release" -C

[
  '/Users/user/.nvm/versions/node/v16.16.0/bin/node',
  '/Users/user/Proj/dlc/wallet/node_modules/cfd-js/node_modules/.bin/cmake-js',
  'configure',
  '-O',
  'build',
  '--CDENABLE_SHARED=on',
  '--CDENABLE_CAPI=off',
  '--CDENABLE_JSONAPI=off',
  '--CDTARGET_RPATH=@executable_path;./build/Release',
  '-C'
]
info TOOL Using clang++ compiler, because preferClang option is set, and clang++ is available.
info TOOL Using Ninja generator, because ninja is available.
info CMD CONFIGURE
info RUN [
info RUN   'cmake',
info RUN   '/Users/user/Proj/dlc/wallet/node_modules/cfd-js',
info RUN   '--no-warn-unused-cli',
info RUN   '-G',
info RUN   'Ninja',
info RUN   '-DCMAKE_JS_VERSION=6.3.2',
info RUN   '-DCMAKE_BUILD_TYPE=Release',
info RUN   '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/Users/user/Proj/dlc/wallet/node_modules/cfd-js/build/Release',
info RUN   '-DCMAKE_JS_INC=/Users/user/.cmake-js/node-x64/v16.16.0/include/node;/Users/user/Proj/dlc/wallet/node_modules/nan',
info RUN   '-DCMAKE_JS_SRC=',
info RUN   '-DNODE_RUNTIME=node',
info RUN   '-DNODE_RUNTIMEVERSION=16.16.0',
info RUN   '-DNODE_ARCH=x64',
info RUN   '-DCMAKE_OSX_ARCHITECTURES=x86_64',
info RUN   '-DENABLE_SHARED=on',
info RUN   '-DENABLE_CAPI=off',
info RUN   '-DENABLE_JSONAPI=off',
info RUN   '-DTARGET_RPATH=@executable_path;./build/Release',
info RUN   '-DCMAKE_CXX_COMPILER=clang++',
info RUN   '-DCMAKE_C_COMPILER=clang',
info RUN   '-DCMAKE_CXX_FLAGS=-D_DARWIN_USE_64_BIT_INODE=1 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DBUILDING_NODE_EXTENSION',
info RUN   '-DCMAKE_SHARED_LINKER_FLAGS=-undefined dynamic_lookup'
info RUN ]
Not searching for unused variables given on the command line.
-- Enable ccache
-- [CFDJS_API] use installed library.
-- Enable ccache
-- [CMAKE_JS_INC]=/Users/user/.cmake-js/node-x64/v16.16.0/include/node;/Users/user/Proj/dlc/wallet/node_modules/nan
-- [cfdjs_api_FOUND]=TRUE
-- [cfdjs_api_DIR]=/Users/user/Proj/dlc/wallet/node_modules/cfd-js/addon/../wrap_js/dl_lib/cmake
-- [find LIBRARY_DIRS]=/Users/user/Proj/dlc/wallet/node_modules/cfd-js/addon/../wrap_js/dl_lib/cmake/../lib
-- [find INCLUDE_FOLDER]=/Users/user/Proj/dlc/wallet/node_modules/cfd-js/addon/../wrap_js/dl_lib/cmake/../include
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/user/Proj/dlc/wallet/node_modules/cfd-js/build

> cfd-js@0.4.6 cmake_build_parallel
> cmake --build build --parallel 2 --config Release

[4/4] Linking CXX shared library Release/cfd_js.node
FAILED: Release/cfd_js.node 
: && /usr/local/opt/ccache/libexec/clang++ -D_DARWIN_USE_64_BIT_INODE=1 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DBUILDING_NODE_EXTENSION -O3 -DNDEBUG -arch x86_64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -o Release/cfd_js.node -install_name @rpath/cfd_js.node addon/CMakeFiles/cfd_js.dir/cfdjs_node_addon_init.cpp.o -L/Users/user/Proj/dlc/wallet/node_modules/cfd-js/addon/.   -L/Users/user/Proj/dlc/wallet/node_modules/cfd-js/addon/../wrap_js/dl_lib/cmake/../lib -Wl,-rpath,@executable_path -Wl,-rpath,./build/Release -Wl,-rpath,./ -Wl,-rpath,@rpath  -lpthread  Release/libcfdjs.dylib  ../addon/../wrap_js/dl_lib/cmake/../lib/libcfdjs_api.a  ../addon/../wrap_js/dl_lib/cmake/../lib/libcfd.a  ../addon/../wrap_js/dl_lib/cmake/../lib/libcfdcore.a  ../addon/../wrap_js/dl_lib/cmake/../lib/libunivalue.a  ../addon/../wrap_js/dl_lib/cmake/../lib/libwally.a && cd /Users/user/Proj/dlc/wallet/node_modules/cfd-js/build/addon && /usr/local/Cellar/cmake/3.23.0/bin/cmake -E copy /Users/user/Proj/dlc/wallet/node_modules/cfd-js/addon/../wrap_js/dl_lib/cmake/../lib/libwally.dylib /Users/user/Proj/dlc/wallet/node_modules/cfd-js/build/Release && /usr/local/Cellar/cmake/3.23.0/bin/cmake -E copy /Users/user/Proj/dlc/wallet/node_modules/cfd-js/addon/../wrap_js/dl_lib/cmake/../lib/libunivalue.dylib /Users/user/Proj/dlc/wallet/node_modules/cfd-js/build/Release && /usr/local/Cellar/cmake/3.23.0/bin/cmake -E copy /Users/user/Proj/dlc/wallet/node_modules/cfd-js/addon/../wrap_js/dl_lib/cmake/../lib/libcfdcore.dylib /Users/user/Proj/dlc/wallet/node_modules/cfd-js/build/Release && /usr/local/Cellar/cmake/3.23.0/bin/cmake -E copy /Users/user/Proj/dlc/wallet/node_modules/cfd-js/addon/../wrap_js/dl_lib/cmake/../lib/libcfd.dylib /Users/user/Proj/dlc/wallet/node_modules/cfd-js/build/Release && /usr/local/Cellar/cmake/3.23.0/bin/cmake -E copy /Users/user/Proj/dlc/wallet/node_modules/cfd-js/addon/../wrap_js/dl_lib/cmake/../lib/libcfdjs_api.dylib /Users/user/Proj/dlc/wallet/node_modules/cfd-js/build/Release
Error copying file "/Users/user/Proj/dlc/wallet/node_modules/cfd-js/addon/../wrap_js/dl_lib/cmake/../lib/libwally.dylib" to "/Users/user/Proj/dlc/wallet/node_modules/cfd-js/build/Release".
ninja: build stopped: subcommand failed.
ERROR: "cmake_build_parallel" exited with 1.
➜  cfd-js git:(master) ✗ node -p process.arch
x64
➜  cfd-js git:(master) ✗ arch
i386

[k-matsuzawa]

Do you have any tips on why my project isn't calling cmake_all on cfd-js after running npm install?

This is because it uses prebuild-install. x64 has binaries, so prebuild-install installs binaries directly. There are also places where cfd-js keeps binaries on its own.
Try setting the following and then running npm install.

export npm_config_build_from_source=true
export CFDJS_UNUSE_ASSET=1

see:

• https://www.npmjs.com/package/prebuild-install
• https://github.com/cryptogarageinc/cfd-js#for-installed-fail

---

Or why it's not compiling when i go directly into the node_modules/cfd-js folder (I'm not sure that could ever work but I wanted to try)? When I do go into the node_modules/cfd-js and try to compile, here's the error I get.

Oh, sorry. I tried it and it didn't work with osx. Could you please use the following command?

npm run cmake_release_parallel

I will modify the README.

[sosaucily]

Thanks! Updates:

With export npm_config_build_from_source=true & export CFDJS_UNUSE_ASSET=1 then running npm install it does run a full compilation step. However if as you said the prebuilt bins were there anyway, I guess this wasn't the problem. However, it's nice to see them built locally.

Unfortunately I'm still getting my error in the browser. I also tried running npm run cmake_release_parallel from within node_modules/cfd-js which succeeds, but does not help the import issue.

Uncaught TypeError: Cannot read properties of undefined (reading 'modules')
    at Object.eval (bindings.js:29:24)
    at eval (bindings.js:223:30)
    at ./node_modules/bindings/bindings.js (chunk-vendors.js:35485:1)
    at __webpack_require__ (popup.js:1430:30)
    at eval (index.js:1:15)
    at ./node_modules/cfd-js/index.js (chunk-vendors.js:37651:1)
    at __webpack_require__ (popup.js:1430:30)
    at eval (Oracle.js:3:64)
    at ./src/models/Oracle.js (chunk-common.js:8945:1)
    at __webpack_require__ (popup.js:1430:30)
    at eval (DLCAccept.vue?./node_modules/cache-loader/dist/cjs.js??ref--13-0!./node_modules/babel-loader/lib!./node_modules/cache-loader/dist/cjs.js??ref--1-0!./node_modules/@vue/cli-service/node_modules/vue-loader/lib??vue-loader-options:41:73)
    at ./node_modules/cache-loader/dist/cjs.js?!./node_modules/babel-loader/lib/index.js!./node_modules/cache-loader/dist/cjs.js?!./node_modules/@vue/cli-service/node_modules/vue-loader/lib/index.js?!./src/views/DLC/DLCAccept.vue?vue&type=script&lang=js& (chunk-common.js:620:1)
    at __webpack_require__ (popup.js:1430:30)
    at eval (DLCAccept.vue:2:314)
    at ./src/views/DLC/DLCAccept.vue?vue&type=script&lang=js& (chunk-common.js:9929:1)
    at __webpack_require__ (popup.js:1430:30)

Do you have any tips for this part? It seems your package does build well, but perhaps isn't linking into node very well. I'll keep playing with is as well. You've been a great help so far.

[sosaucily]

Update, I can get the cfd-js package working through the node REPL, so I guess everything is set up right, just something weird about my imports in the project. Will be looking further.

[k-matsuzawa]

Since you are using webpack, you may (one possibility) need to copy the native library to the proper location.
The following is part of the webpack.config for a private project.
I had to copy the binaries because webpack changes the paths that are used as a base for search by bindings.
(From the logs, the externals seem to be OK, try to copy them in plugins).

  externals: {
    bindings: 'commonjs bindings',
    'cfd-js': 'commonjs cfd-js',
    sqlite3: 'commonjs sqlite3',
    usb: 'commonjs usb',
    'usb-detection': 'commonjs usb-detection',
    grpc: 'require("grpc")'
  },

  plugins: [
    new CopyPlugin({
      patterns: [
        {
          from: path.join(__dirname, '../../../../', 'node_modules/cfd-js/build/Release/'),
          to: path.join(__dirname, '../../', 'build/Release'),
          context: path.join(__dirname, '../../', 'dist')
        },
        {
          from: path.join(__dirname, '../../../../', 'node_modules/usb/build/Release/usb_bindings.node'),
          to: path.join(__dirname, '../../', 'build/Release'),
          context: path.join(__dirname, '../../', 'dist')
        },
        {
          from: path.join(__dirname, '../../../../', 'node_modules/node-hid/build/Release/HID.node'),
          to: path.join(__dirname, '../../', 'build/Release'),
          context: path.join(__dirname, '../../', 'dist')
        }
      ]
    }),

[sosaucily]

Thank you, you've been very helpful. I tried the above but am not having much luck. I think I might be taking on too big of a task for myself. I was trying to get this cfd-js package to run in the Liquality wallet, which is a big/ mature chrome extension / vue app and it's very tricky. Lots of new tech for me.

If you have time and you're interested in helping get it running, we could work on it together. My goal was to get DLC wallet functionality into an open-source wallet. Our startup could really use a DLC powered wallet. I've definitely made some progress. But otherwise I might have to give up on this and start over with a smaller project.

Thanks a lot for your time! I'm sure we can close this issue for now. Let me know if you would be interested in setting up a call to discuss more!

[k-matsuzawa]

I tried the above but am not having much luck.

Oh...

@sosaucily
I am not sure if this is appropriate for your application, but if your implementation does not need to call C++ functions, it may be useful to use cfd-js-wasm.
https://github.com/cryptogarageinc/cfd-js-wasm

(There is a repository that is forked by DLC, so you may want to take a look at that.)