The issue of inference time for prediction

[xiaoxinxing12]

Thank you for open-sourcing this project. I tested the .onnx file with the provided image in the demo, and it took 287 milliseconds for inference. What could be the reason for this? Is it related to the environment?
i7-12700H 2.30 GHz

[ukicomputers]

It is because of first time of inference, because model is yet loading in memory. I just did not do warmup model (pass up some random array to detect, no preprocessing...). This is just to not allocate memory immediatly, and to first inference time be little be faster (not warmup time + inferece time of picture). Inferences after that should be faster.

[ukicomputers]

It is maybe good point to do warmup model just when calling YoloNAS class (and not confuse others about inference time). I will stay updated.

[xiaoxinxing12]

The second inference was faster, but also took 200ms，The main time spent in a function
“ net.forward(outDet, net.getUnconnectedOutLayersNames());”

[ukicomputers]

Ok, I'm fixing bug, keep updated.

[ukicomputers]

Ok, I was able to get inference time down to 926ms from ~1600ms (my CPU is Intel Pentium G640, so don't worry why is that big). I am pushing new commit, and also testing video inference.

[xiaoxinxing12]

I tested with new code, which required 200ms, but I used python's yolo-nas, which required only 50ms！

[ukicomputers]

What version of OpenCV are you using? Try maybe running code out of IDE (I had that issues in the past). The may issue is located at OpenCV's implementation of ONNX Runtime (that line of code, net.forward()), something is happening there wrong (that I can't change it, also I will take a look at source of dnn, because machine is responsible there for getting tensor output and same forward processing), or I made wrong something at preprocessing. I will keep updated.

[xiaoxinxing12]

The code runs without errors ,What I use is
win11 vs2019+opencv4.9+opencv_world490.lib Release x64

[xiaoxinxing12]

Give me an email address and I'll send you the demo

[ukicomputers]

Ok, nice, that is latest release of OpenCV (and I also have 4.9.0). I compiled code with G++ (GNU SDK) on Ubuntu 23.04 (codename mantic). As you can see, all my install/build scripts are written for Linux/UNIX OS'es. Maybe, that is making some problems (to cross platform compatibility), but I wonder why? Also, I am utilizing CMake, not sure if you do, but that isn't case of problem.

[ukicomputers]

Send me an email and I'll send you the Demo

Here you go: yolonascpp.ukicomputers.github@proton.me

[xiaoxinxing12]

好的，很好，这是OpenCV的最新版本（我也有）。我在 Ubuntu 23.04（代号）上使用 G++ （GNU SDK） 编译了代码。正如你所看到的，我所有的安装/构建脚本都是为Linux/UNIX操作系统编写的。也许，这会带来一些问题（跨平台兼容性），但我想知道为什么？另外，我正在使用 CMake，不确定您是否这样做，但这不是问题。4.9.0``mantic

我使用的是Visual Studio 2019 Microsoft Visual C++编译器（MSVC）

[ukicomputers]

Firstly, change from debug to release model in IDE, and compile it (I'm not sure because in project ZIP that you sent me, active mode is Debug, but on image release). Execute it only from File Explorer (go to it's Release folder, double click on .exe file, maybe also via CMD line). Also, you can try executing program as admin or setting it's priority to Very High in task manager. Btw, do you have or use GPU?