What does this script.py refer to?

[qwe962]

'''https://github.com/eth-easl/orion/blob/main/PROFILE.md#notes'''
“This guide assumes the user has setup a script.py to profile"
What does this script.py refer to?

This file is one of the files here?'''https://github.com/eth-easl/orion/tree/main/profiling/postprocessing'''

Thanks!

[fotstrt]

Hi! Thank you for your comment!

The script.py can be any Python script containing the model you want to profile.
For example: https://github.com/eth-easl/orion/blob/main/profiling/benchmarks/vision_models.py

Let me know if that helps, or you need further clarification.

[qwe962]

I ran the script using the file you specified, but when it reached:

python profiling/postprocessing/process_ncu.py --results_dir <path to profiling files directory>

I encountered an error:

ValueError: invalid literal for int() with base 10: '5,376'

It seems that the numbers in the ncu-csv file may use commas as separators. Can I remove the commas and continue running the script?

Why does this error occur? Is there a script.py file that can generate the figures in your paper?

Additionally, the two commands for nsys analysis:

Use NSYS for profiling:
nsys profile -w true -t cuda,nvtx,osrt,cudnn,cublas -s none -o output_nsys --cudabacktrace=true --capture-range=cudaProfilerApi --stop-on-range-end=true -f true -x true python3 script.py

Convert NSYS output to CSV:
nsys stats --report gputrace --format csv,column --output .,- output_nsys.qdrep

seem to have become outdated. Are there newer equivalent commands?

Thanks!

[fotstrt]

Hello,

the output format and commands depend on the versions of NCU and NSYS that you are using.
Can you please tell me the versions?

[qwe962]

nuc Version 2024.3.2.0
NVIDIA Nsight Systems version 2024.5.1.113-245134619542v0

[qwe962]

The version problem has been solved. I will ask you if I have any questions later. Thank you.

[qwe962]

Hello！
I encounter this error:
script.py:26: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 36: API call is not supported in the installed CUDA driver (Triggered internally at /opt/conda/conda-bld/pytorch_1724789116784/work/c10/cuda/CUDAFunctions.cpp:108.)

CUDA Version: 12.6
Can this version of CUDA be used successfully?

[fotstrt]

Hi! I'm glad the previous issue was resolved.
Can you please explain the setup that you get this error? what is script.py, and what exact command are you running?