Can we resurrect node-packer to get it working with latest Node.js versions?

[saurabh-deep]

Hey @pmq20! Hope you are doing well.

Is there a possibility of resurrecting this project? It has been a very popular project with some forks even managing to get this working till Node10. But it seems we are at the end of the road as I am not aware if anyone has had any success with Node12.

There are not many other options available to create Node.js binary and all of them have issues with the OS X signing/notarization and the binaries are too big.

I understand you being busy with your current job and projects. Still wanted to check if it was a possibility to bring this project back to life.

Many thanks! Cheers!

[saurabh-deep]

@pmq20 - I thought more about it and I feel that it can also be a viable option to pass on the knowledge to the community so that someone from the community can carry forward this great tool. Thoughts?

[saurabh-deep]

Hello @pmq20! I remember you mentioning on the Node 10 PR about working on upgrade to Node 12. Just wanted to follow up on the same. Do you have an approx time frame in mind for that? Would very much appreciate your response on this. :-)

[saurabh-deep]

Hey @pmq20! I had been looking forward to getting the Node12 support after your comment on the Node10 PR. But I just noticed in the about section that this repository has stopped maintenance.

Does that mean this project is officially dead? I hope NOT. If yes, it is very unfortunate. There are no active forks also.

There are a few options available to generate Node.js binary like pkg, nexe etc. But none of them close to the quality of the binary created by node-packer, without any custom coding being required.

Will it be possible for you to create a how to doc for me to try and take this forward? This will be a massive help.

Eagerly looking forward to your response.

[pmq20]

@saurabh-deep Thank you for the feedback! Your feedback brings a lot of encouragement to me. I really appreciate it. I previously put up the stop-maintenance notice because there seems to be no time to maintain it for me personally due to my weekday work. But it does appear that the project is still relevant and many people want to use it, so I should continue maintaining it. Hence I have deleted that notice.

I also do agree that community is the key to keep the project alive, because one person's time is too limited. Please feel free to ask me any technical questions about its implementation. Everybody's participation is greatly welcome and appreciated!

Let me start resurrecting it this weekend (because of work I can only work on it in the weekends). I think the first thing to do is to upgrade it to node.js v14. I'll let you know what I accomplished after this weekend.

[saurabh-deep]

@pmq20 - Best news I have had in a while. 😄

v14 is great! We will also need v12 - but surely, that can come later. Please do let me know if I can help in any way.

These are some the node-packer positives which aren't to be found in any other packaging solution -

1. No custom coding and adjustments needed. All the node code and npm dependencies work as they are (at least for me).
2. Fairly small size. Actually less than half of what pkg and nexe create.
3. Best of the lot - Finally binary can be successfully signed for Mac OS and Windows both. I haven't seen any other tool which creates correctly formatted Mac OS binaries.

[pmq20]

@saurabh-deep Thanks for the compliments. I am really glad that people actually love this tool. I tried upgrading to 14.8.0 this weekend and it was easier than I thought. I have reviewed all the hacks, and only kept the essential ones. They were gathered in this commit: a4ab86d

You can see that there's not too many essential hacks, so it is promising that we can maintain the project in the long run without too much burden. Of course the heavy lifting still happens in libsquash, but that part is easier to maintain (just copy and paste). But I have not actually tested if the old hacks still work for the new Node.js. Let me do that next week with the goal of setting up a CI to check the quality of the tool. Also I plan to make CI build and release nodec itself automatically, just like what I did in Ruby Packer.

[saurabh-deep]

This is wonderful @pmq20. I will give it a spin once the latest nodec with Node v14.8.0. executable is ready. Or do you think I can try directly with the source like we have been doing for a while since nodec was still at Node v8.x? Something like the below. Or do you think I should wait a little since your current changes are a WIP?

ruby ~/build/node-packer/bin/nodec --tmpdir=~/build/compile --output=~/build/a.out bin/index.js --skip-npm-install

Second, lets say I want to build nodec for the current Node.js LTE version (v12.18.3). What should be my steps?

• Download Node.js v12.18.3 from https://github.com/nodejs/node/tree/v12.18.3
• Put the Node.js downloaded code in the node directory of node-packer
• Apply Enclose.IO hacks (As per commit a4ab86d)
• Any other steps?
• Compile all the new stuff into a new version of nodec - how to do this?

Also, I wanted to point out that Python 2 is End of Life now. Can we upgrade to Python 3 now somehow or do you think it will cause an issue?

I also noticed that you added cross-compilation to the README.md. That is a very welcome change. I haven't tried it yet but will certainly do once you think it is ready for a test drive.

Once again, many thanks for hard work and bringing this project to life! 🙇‍♂️

I will let others in the community know about this progress so that more people can banefit from your hard work and stop compromising on features (like Mac OS code-signing) which do not work correctly using any other tool.

Please let me know about my questions on building a new nodec for Node.js v12.18.3 and I will try and contribute on that as soon as my schedule allows.

[pmq20]

@saurabh-deep Yeah, as a user one should probably wait a little since the current changes are WIP; unless one wish to try out the bleeding edge and contribute to the WIP development, which is greatly welcome.

Yeah that is pretty much the steps. To compile all the new stuff into a new version of nodec (standalone binary), you need to use Ruby Packer. Otherwise you can just use the ruby command you mentioned above on the checked-out code.

Thanks for the info on Python 3. I imagine it might already works, as I cannot recall any special usage of python 2 specifically.

To support both Node 12 and Node 14, I plan to maintain two node folders (https://github.com/pmq20/node-packer/tree/master/node). Perhaps we can name them lts and current.

I wish I had more time to work on it. Let me keep working on it the next weekend.

[saurabh-deep]

@pmq20 - I will be compiling my binary with Node 12 this week. For that, I will try to use the lts folder and use ruby command for the compilation since nodec is not packed yet. I will let you know how that goes. Keeping the fingers crossed. Thank you very much for bringing it this far.

[pmq20]

@saurabh-deep Last time I tried, There was some errors when running that needs to be fixed. I plan to use some coming weekends to fix them. Also, if possible, could you share the project that you were trying to pack, so that I can use it as a reference to test the packing process? Thank you again for your feedback!

[saurabh-deep]

@pmq20 - My project is in a private (company-owned) GitLab repository, which I am not authorised to share, unfortunately. Apologies for this! But I will setup a similar public project on GitHub that you can use as a reference. I should be able to set this up over the weekend.

For the errors that you mentioned, please let me know whenever you get a chance to work on them. I will be more than happy to test and share feedback with you. One request though. Can we please start with lts version (12.x) first and look at the current version (14.x) after lte is finalised? Though I think, if both of them have similar problems, fixing one should help with the other as well.

To tell you the truth, we are currently running Node 10.15.3 using one of the node-packer forks. But we have already started to face issues where some npm dependencies require a higher Node version. So we are very much looking forward to this lts version build.

And yes, just so you know, once we have finalised this first new build, I will be sharing this important update with the rest of the community who have to use other inferior packing tools today because there is no option. I know for a fact that those people will love to use node-packer, as I do. 🥰

[pmq20]

@saurabh-deep Thanks for the reply. No problems. I'll work on lts first this weekend. Looking forward to your public repo.

[YYago]

Windows 10

• Branch: master;
• Env: Windows 10 Pro 64; Ruby-2.6.3

Bat:

call bin/nodec.bat D:/devWorkSpace/AdRanking/win/index.js --lts --tmpdir=D:/devWorkSpace/nodec/temp --output=D:/devWorkSpace/nodec/a.exe --dest-os=win --clean-tmpdir > log.txt

Log:

Node.js Compiler (nodec) v140800.121803.dev
- entrance: D:/devWorkSpace/AdRanking/win/index.js
- options: {
    :lts=>true, 
    :tmpdir=>"D:/devWorkSpace/nodec/temp", 
    :output=>"D:/devWorkSpace/nodec/a.exe", 
    :os=>"win", 
    :clean_tmpdir=>true, 
    :npm=>"npm", 
    :make_args=>"-j4", 
    :vcbuild_args=>"x64"}

-> rm -rf D:/devWorkSpace/nodec/temp
-> mkdir -p D:/devWorkSpace/nodec/temp
-> cp -r "D:/devWorkSpace/nodec/node-packer/lts" "D:/devWorkSpace/nodec/temp/node-140800.121803.dev-lts"
-> rm -rf D:/devWorkSpace/nodec/temp/__work_dir__
-> mkdir -p D:/devWorkSpace/nodec/temp/__work_dir__
-> cp -r "D:/devWorkSpace/AdRanking/win" "D:/devWorkSpace/nodec/temp/__work_dir__/__enclose_io_memfs__"
........
........
6.14.5
audited 162 packages in 2.63s

8 packages are looking for funding
  run `npm fund` for details

found 2 vulnerabilities (1 low, 1 high)
  run `npm audit fix` to fix them, or `npm audit` for details
mksquashfs version 4.3 (2014/05/12)
copyright (C) 2014 Phillip Lougher <phillip@squashfs.org.uk>

[----------------------------------------------------------]
Compiled by Sebastiaan Ebeltjes @ 2015-06-08 (Domoticx.nl)
Compiled with Cygwin on Windows 8.1, compiler: GCC v4.9.2

This version comes with added support for:
- LZMA v4.65
- LZMA2 (XZ) v5.5.0
- LZO v2.09
- LZ4 r130

Enjoy!

[----------------------------------------------------------]

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2,
or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.
Parallel mksquashfs: Using 8 processors
Creating 4.0 filesystem on deps/libsquash/sample/enclose_io_memfs.squashfs, block size 131072.

[=============================================================/] 7470/7470 100%

Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments, compressed xattrs
	duplicates are removed
Filesystem size 6283.46 Kbytes (6.14 Mbytes)
	18.99% of uncompressed filesystem size (33083.26 Kbytes)
Inode table size 67746 bytes (66.16 Kbytes)
	26.82% of uncompressed inode table size (252581 bytes)
Directory table size 71139 bytes (69.47 Kbytes)
	42.44% of uncompressed directory table size (167603 bytes)
Number of duplicate files found 808
Number of inodes 7873
Number of files 7383
Number of fragments 152
Number of symbolic links  0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 490
Number of ids (unique uids + gids) 2
Number of uids 1
	omg (197609)
Number of gids 1
	None (197121)
Looking for Python
Python 2 found in C:\Python27\\python.exe
Looking for NASM
Looking for Visual Studio 2017
calling: "D:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\\Auxiliary\Build\vcvarsall.bat" amd64
**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.22
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
Found MSVS version 15.0
configure  --dest-cpu=x64
Warning: Missing input files:
tools\msvs\genfiles\node_etw_provider.rc
tools\msvs\genfiles\node_etw_provider.h
tools\v8_gypfiles\..\..\deps\v8\src\regexp\regexp-special-case.h
INFO: configure completed successfully
Project files generated.
......
......
    正在创建库 out\Release\node_mksnapshot.lib 和对象 out\Release\node_mksnapshot.exp
    正在创建库 out\Release\mkcodecache.lib 和对象 out\Release\mkcodecache.exp
enclose_io_libsquash.lib(enclose_io_win32.obj) : error LNK2019: 
    无法解析的外部符号 __imp_PathMatchSpecA，该符号在函数 EncloseIOFindFirstFileHelper 中被引用 [D:\devWorkSpace\nodec\temp\node-140800.121803.dev-lts\mkcodecache.vcxproj]

enclose_io_libsquash.lib(extract.obj) : error LNK2019: 
    无法解析的外部符号 __imp_PathFileExistsW，该符号在函数 squash_tmpf 中被引用 [D:\devWorkSpace\nodec\temp\node-140800.121803.dev-lts\mkcodecache.vcxproj]

enclose_io_libsquash.lib(enclose_io_win32.obj) : error LNK2019: 
    无法解析的外部符号 __imp_PathMatchSpecA，该符号在函数 EncloseIOFindFirstFileHelper 中被引用 [D:\devWorkSpace\nodec\temp\node-140800.121803.dev-lts\node_mksnapshot.vcxproj]

enclose_io_libsquash.lib(extract.obj) : error LNK2019: 
    无法解析的外部符号 __imp_PathFileExistsW，该符号在函数 squash_tmpf 中被引用 [D:\devWorkSpace\nodec\temp\node-140800.121803.dev-lts\node_mksnapshot.vcxproj]

out\Release\mkcodecache.exe : fatal error LNK1120: 
    2 个无法解析的外部命令 [D:\devWorkSpace\nodec\temp\node-140800.121803.dev-lts\mkcodecache.vcxproj]

out\Release\node_mksnapshot.exe : fatal error LNK1120: 
    2 个无法解析的外部命令 [D:\devWorkSpace\nodec\temp\node-140800.121803.dev-lts\node_mksnapshot.vcxproj]

[saurabh-deep]

Hi @pmq20 - Apologies for the delay due to some health issues in the past couple of weeks. I have now set up this test CLI project which you can use to test binary compilation. I haven't been able to set up GitHub Action for this as of yet, but I tried the latest node-packer code to build the binary on my local Mac and Windows, and in an ubuntu docker container for Linux - all 3 using the Ruby command. Till now, all 3 compilations are failing. Linux and Mac have the same error, while Windows shows a different one.

Please note that the project is fully compatible with Node12 (lts) and may not be fully compatible with Node14(current), but it should still compile. Also, Yarn is used with the project instead of NPM. I hope it doesn't cause any issues for you. I will try to set up GH Actions for the test Node project ASAP, but since I don't have much experience on GH Actions - so it will take time. If you want to give it a go, please let me know. I will very glad to share the access for that project with you.

Now the errors that I have been facing so far -

First of all, the compilation doesn't start and fail if the current working directory is set to the test Node project (node-cli-test). To be able to start the compilation, I am forced to execute the Ruby command from the node-packer directory, which is going to be a problem in CI. Earlier version of node-packer allowed us to execute the ruby command from the node project itself. It is important for the Ruby command to be executable from the node project itself. As of now, I receive the following error when I execute the following command from the test Node project -

Working directory: ~/Code/DX/node-cli-test
Command: ruby ~/Code/DX/node-packer/bin/nodec --tmpdir=~/adx-build/compile12 --output=~/adx-build/a.out --npm=yarn --lts bin/cli.js
Error: /Users/sdeep3/Code/DX/node-packer/bin/nodec:11:in <main>': private method require' called for Bundler:Module (NoMethodError)

Alternatively, when I execute the Ruby command from the node-packer directory, it starts the compilation but fails with the following error, both for Mac (OSX 10.14.6) and Linux (Ubuntu). Same errors are encountered if the pre-compiled nodec binary is used for compilation. Can I also take this opportunity to request you to downgrade the MacOS version from macos-10.15 to macos-10.14 in the GH Action workflow? The nodec binary created from macos-10.15 is not compatible with 10.14 and I am not in a position to upgrade as of this moment. Please let me know if you have any concerns related to this.

Working directory: ~/Code/DX/node-packer
Command: ruby bin/nodec --tmpdir=~/adx-build/compile12 --output=~/adx-build/a.out --npm=yarn --lts ~/Code/DX/node-cli-test/bin/cli.js
Error: cc -o /Users/sdeep3/adx-build/compile12/node-140800.121803-lts/out/Release/obj.target/uvwasi/deps/uvwasi/src/fd_table.o ../deps/uvwasi/src/fd_table.c '-DV8_DEPRECATION_WARNINGS' '-DV8_IMMINENT_DEPRECATION_WARNINGS' '-D_DARWIN_USE_64_BIT_INODE=1' '-DOPENSSL_NO_PINSHARED' '-DOPENSSL_THREADS' '-D_LARGEFILE_SOURCE' '-D_FILE_OFFSET_BITS=64' -I../deps/uvwasi/include -I../deps/uv/include -O3 -gdwarf-2 -fvisibility=hidden -mmacosx-version-min=10.10 -arch x86_64 -Wall -Wendif-labels -W -Wno-unused-parameter -fno-strict-aliasing -MMD -MF /Users/sdeep3/adx-build/compile12/node-140800.121803-lts/out/Release/.deps//Users/sdeep3/adx-build/compile12/node-140800.121803-lts/out/Release/obj.target/uvwasi/deps/uvwasi/src/fd_table.o.d.raw -c
In file included from ../deps/uvwasi/src/clocks.c:8:
../deps/uv/include/uv.h:37:10: fatal error: 'enclose_io.h' file not found
In file included from ../deps/uvwasi/src/fd_table.c:9:
../deps/uv/include/uv.h:37:10: fatal error: #include "enclose_io.h"
^~~~~~~~~~~~~~
'enclose_io.h' file not found
#include "enclose_io.h"
^~~~~~~~~~~~~~
1 error generated.
1 error generated.
make[1]: *** [/Users/sdeep3/adx-build/compile12/node-140800.121803-lts/out/Release/obj.target/uvwasi/deps/uvwasi/src/fd_table.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [/Users/sdeep3/adx-build/compile12/node-140800.121803-lts/out/Release/obj.target/uvwasi/deps/uvwasi/src/clocks.o] Error 1
rm ae1ad5e88e93fc855faf392957725a5689c3fc98.intermediate
make: *** [node] Error 2

[saurabh-deep]

Hello @pmq20 - I forgot to include the Windows error that I am receiving when building from node-packer source using Ruby command. For Linux and Mac OS, I shared the same in my previous message.

Windows error:

Creating library out\Release\mkcodecache.lib and object out\Release\mkcodecache.exp
enclose_io_libsquash.lib(enclose_io_win32.obj) : error LNK2019: unresolved external symbol __imp_PathMatchSpecA referen
ced in function EncloseIOFindFirstFileHelper [C:\adx-build\compile\node-140800.121803-lts\node_mksnapshot.vcxproj]
enclose_io_libsquash.lib(extract.obj) : error LNK2019: unresolved external symbol __imp_PathFileExistsW referenced in f
unction squash_tmpf [C:\adx-build\compile\node-140800.121803-lts\node_mksnapshot.vcxproj]
out\Release\node_mksnapshot.exe : fatal error LNK1120: 2 unresolved externals [C:\adx-build\compile\node-140800.121803-
lts\node_mksnapshot.vcxproj]
enclose_io_libsquash.lib(enclose_io_win32.obj) : error LNK2019: unresolved external symbol __imp_PathMatchSpecA referen
ced in function EncloseIOFindFirstFileHelper [C:\adx-build\compile\node-140800.121803-lts\mkcodecache.vcxproj]
enclose_io_libsquash.lib(extract.obj) : error LNK2019: unresolved external symbol __imp_PathFileExistsW referenced in f
unction squash_tmpf [C:\adx-build\compile\node-140800.121803-lts\mkcodecache.vcxproj]
out\Release\mkcodecache.exe : fatal error LNK1120: 2 unresolved externals [C:\adx-build\compile\node-140800.121803-lts
mkcodecache.vcxproj]