Skip to content

[202605][NVIDIA]: Add SAI/SDK Debian packaging for debugging/profiling builds - #29489

Open
nazariig wants to merge 2 commits into
sonic-net:202605from
nazariig:202605-dbg-build-nvda
Open

nazariig wants to merge 2 commits into
sonic-net:202605from
nazariig:202605-dbg-build-nvda

Conversation

@nazariig

Copy link
Copy Markdown
Collaborator

Signed-off-by: Nazarii Hnydyn nazariig@nvidia.com

Why I did it

Make the NVIDIA SAI/SDK packages honor SONiC's debug/profiling build switches and
ship correct Debian metadata, and stop registering build artifacts that the
recipes cannot produce.

Five problems are addressed.

1. SONIC_DEBUGGING_ON / SONIC_PROFILING_ON had no effect on the SDK

slave.mk exports DEB_BUILD_OPTIONS=nostrip (debugging) or nostrip noopt
(profiling). Those are debhelper options. sys-sdk is packaged by CMake/CPack,
which never reads DEB_BUILD_OPTIONS, so the SDK was built and packaged exactly
the same way in every mode:

  • libraries always stripped, DWARF always moved into a separate .ddeb
  • always compiled optimized

Source-level debugging of the SDK inside syncd was therefore impossible even
with profiling enabled: gdb could resolve function names, but not file:line.

2. sys-sdk packages had an empty Depends: field

CPack does not run dpkg-shlibdeps and does not convert component dependencies
into Debian dependencies unless explicitly told to. Both generated packages
shipped with no Depends: at all, so dpkg could not enforce anything:
sys-sdk-dev did not require its runtime package, and the runtime package did
not require libc6 / libnl / libxml2. Correct install order happened only as
a side effect of the Make dependency graph.

3. mlnx-sai did not depend on the SDK

sys-sdk published no shlibs control file, so when SAI ran dh_shlibdeps there
was no SONAME-to-package mapping for the SDK shared libraries. SAI's
debian/rules runs dh_shlibdeps --dpkg-shlibdeps-params=--ignore-missing-info,
so the missing information was not an error — the build simply produced an
mlnx-sai package with no dependency on the SDK it links against, and installing
it on a system without the SDK would only fail at runtime.

4. dbgsym packages were demanded in cases where nothing produces them

The dbgsym artifacts were declared without regard to whether the selected build
mode actually emits them:

  • MLNX_SDK_DBG_DEBS always listed sys-sdk-*-dbgsym.ddeb, and
    docker-syncd-mlnx.mk always appended it to the -dbg image, while the
    package itself was registered only for source builds — and, once nostrip is
    in DEB_BUILD_OPTIONS, is not produced by the source build either
  • mlnx-sai-dbgsym was registered unconditionally as a derived package of the
    runtime deb, so on a nostrip build the recipe was expected to emit a file
    that dh_strip never creates, and on the download path the asset was fetched on
    every build even when no -dbg image was requested

A dbgsym that nothing can produce shows up as a missing prerequisite:

make: *** No rule to make target 'target/debs/trixie/sys-sdk_1.mlnx.4.10.1106_amd64-dbgsym.ddeb',
needed by 'target/docker-syncd-mlnx-dbg.gz'

5. _DEPENDS mixed compile-time headers with runtime install ordering

$(MLNX_SAI)_DEPENDS unconditionally listed MLNX_SDK_DEBS (sys-sdk-dev).
Installing SAI in the slave therefore always installed SDK headers, including on
the default path where SAI is downloaded as a prebuilt asset and nothing compiles
against the SDK.

Work item tracking

  • N/A

How I did it

slave.mk — introduce SPLIT_DBGSYM

Moved the SONIC_DEBUGGING_ON / SONIC_PROFILING_ON handling next to the other
debug switches and made it set a single variable that package rules can read:

SPLIT_DBGSYM = y
ifeq ($(SONIC_DEBUGGING_ON),y)
DEB_BUILD_OPTIONS_GENERIC := nostrip
SPLIT_DBGSYM = n
endif
ifeq ($(SONIC_PROFILING_ON),y)
DEB_BUILD_OPTIONS_GENERIC := nostrip noopt
SPLIT_DBGSYM = n
endif

SPLIT_DBGSYM=y (default, production): runtime deb is stripped and a dbgsym
package exists. SPLIT_DBGSYM=n: DWARF stays in the runtime deb and no dbgsym
package is produced, so no rule may register one. This is independent of
INSTALL_DEBUG_TOOLS, which only decides whether debug images ship in the
installer.

platform/mellanox/sdk-src/sys-sdk/Makefile — map DEB_BUILD_OPTIONS onto CMake/CPack

Collected the flags into SDK_CMAKE_ARGS / SDK_CPACK_ARGS and translated the
debhelper options that CPack cannot see:

  • noopt -> -DCMAKE_BUILD_TYPE=Debug (configure time)
  • nostrip -> -D CPACK_DEBIAN_DEBUGINFO_PACKAGE=OFF (package time)

and added the missing packaging metadata:

  • -D CPACK_DEBIAN_ENABLE_COMPONENT_DEPENDS=ON and
    -D CPACK_COMPONENT_DEV_DEPENDS=main -- -dev now depends on the runtime
    package. The component ids come from the SDK CMake project (dev / main);
    CPack resolves them to the generated package names, so nothing here hardcodes
    the runtime package name.
  • -D CPACK_DEBIAN_PACKAGE_SHLIBDEPS=ON -- runtime deb gets its own
    Depends: generated from the libraries it links against.
  • -D CPACK_DEBIAN_PACKAGE_GENERATE_SHLIBS=ON -- the runtime deb ships a
    shlibs file so other packages (SAI) can resolve SDK SONAMEs.

platform/mellanox/sdk.mk / mlnx-sai.mk — declare only what each path delivers

Both files now split their declarations by acquisition method, and _DEPENDS is
scoped to what that path actually needs:

built from source downloaded
SAI _DEPENDS sys-sdk-dev, libnl-route-3-dev sys-sdk, libnl-route-3-200
SDK _DEPENDS libnl-3-dev, libnl-genl-3-dev libnl-3-200, libnl-genl-3-200
dbgsym derived package, only when SPLIT_DBGSYM=y own online deb + own URL (lazy)
SDK -dev derived package own online deb + _DEPENDS on runtime

_RDEPENDS is unchanged, so what lands in the docker images is identical.

platform/mellanox/mlnx-sai/Makefile / sdk-src/sys-sdk/Makefile — conditional derived targets

DERIVED_TARGETS now includes the dbgsym target only when nostrip is absent, so
the recipe never tries to mv a file that was not generated.

platform/mellanox/docker-syncd-mlnx.mk — consumer side

_DBG_DEPENDS asks for a dbgsym package only in the combinations where one
exists: source builds only when SPLIT_DBGSYM=y, downloaded packages always. On
nostrip builds the symbols are already inside the runtime debs the base image
installs, so nothing extra is added.

How to verify it

Production build (default, SPLIT_DBGSYM=y)

  1. Build SDK and SAI with no debug flags.
  2. Confirm dbgsym packages are produced and runtime debs are stripped:
    ls target/debs/trixie/sys-sdk_*-dbgsym.ddeb
    ls target/debs/trixie/mlnx-sai-dbgsym_*.deb
    
  3. Confirm the metadata:
    dpkg-deb -f target/debs/trixie/sys-sdk_*_amd64.deb Depends
    dpkg-deb -f target/debs/trixie/sys-sdk_*-dev.deb Depends
    dpkg-deb -f target/debs/trixie/mlnx-sai_*.deb Depends
    dpkg-deb -I target/debs/trixie/sys-sdk_*_amd64.deb shlibs
    

Profiling build (SONIC_PROFILING_ON=y, SPLIT_DBGSYM=n)

  1. Rebuild SDK and SAI.
  2. Confirm no dbgsym package is emitted:
    ls target/debs/trixie/sys-sdk_*-dbgsym.ddeb   # must not exist
    ls target/debs/trixie/mlnx-sai-dbgsym_*.deb   # must not exist
    
  3. Confirm DWARF is inside the runtime debs and code is unoptimized:
    dpkg-deb -x target/debs/trixie/sys-sdk_*_amd64.deb /tmp/sdk
    readelf -SW /tmp/sdk/usr/lib/<sdk-lib>.so | grep debug_info
    readelf --debug-dump=info /tmp/sdk/usr/lib/<sdk-lib>.so | grep -m1 DW_AT_producer
    
  4. Build the -dbg syncd image and confirm it still builds with no missing
    dbgsym prerequisite.
  5. Attach gdb to syncd and confirm file:line breakpoints in SDK sources now
    resolve.

Dependency laziness

  1. Default path (SAI and SDK downloaded): build syncd and confirm
    sys-sdk-dev is not installed in the slave.
  2. SAI built from source: confirm sys-sdk-dev is installed before SAI compiles.

Previous command output (if the output of a command-line utility has changed)

# Profiling build: dbgsym still produced, libraries stripped, code optimized
$ ls target/debs/trixie/sys-sdk_1.mlnx.4.10.1106_amd64-dbgsym.ddeb
target/debs/trixie/sys-sdk_1.mlnx.4.10.1106_amd64-dbgsym.ddeb

$ readelf -SW /tmp/sdk/usr/lib/<sdk-lib>.so | grep -c debug_info
0

# No Depends generated for the SDK at all
$ dpkg-deb -f target/debs/trixie/sys-sdk_1.mlnx.4.10.1106_amd64.deb Depends
$ dpkg-deb -f target/debs/trixie/sys-sdk_1.mlnx.4.10.1106_amd64-dev.deb Depends

# No shlibs, so SAI does not depend on the SDK it links against
$ dpkg-deb -I target/debs/trixie/sys-sdk_1.mlnx.4.10.1106_amd64.deb shlibs
dpkg-deb: error: control file 'shlibs' not found

$ dpkg-deb -f target/debs/trixie/mlnx-sai_1.mlnx.SAIBuild2605.37.0.31_amd64.deb Depends
libc6 (>= 2.34), libnl-3-200 (>= 3.7.0-0.2+b1sonic1), libnl-genl-3-200 (>= 3.7.0-0.2+b1sonic1),
libnl-route-3-200 (>= 3.7.0-0.2+b1sonic1), libxml2 (>= 2.7.4)

# nostrip build of a package whose dbgsym is registered but never produced
make: *** No rule to make target 'target/debs/trixie/sys-sdk_1.mlnx.4.10.1106_amd64-dbgsym.ddeb',
needed by 'target/docker-syncd-mlnx-dbg.gz'

New command output (if the output of a command-line utility has changed)

# Profiling build: no dbgsym, DWARF inside the runtime deb, no optimization
$ ls target/debs/trixie/sys-sdk_1.mlnx.4.10.1106_amd64-dbgsym.ddeb
ls: cannot access '...-dbgsym.ddeb': No such file or directory

$ readelf -SW /tmp/sdk/usr/lib/<sdk-lib>.so | grep -c debug_info
1

$ readelf --debug-dump=info /tmp/sdk/usr/lib/<sdk-lib>.so | grep -m1 DW_AT_producer
DW_AT_producer : GNU C17 ... -ggdb -O0 ...

# Depends now generated for both components
$ dpkg-deb -f target/debs/trixie/sys-sdk_1.mlnx.4.10.1106_amd64.deb Depends
libc6 (>= 2.38), libelf1t64 (>= 0.131), libexpat1 (>= 2.0.1), libnl-3-200 (>= 3.7.0-0.2+b1sonic1),
libpcap0.8t64 (>= 0.9.8), libsystemd0, libxml2 (>= 2.7.4), zlib1g (>= 1:1.1.4)

$ dpkg-deb -f target/debs/trixie/sys-sdk_1.mlnx.4.10.1106_amd64-dev.deb Depends
sys-sdk-main (= 1.mlnx.4.10.1106)

# shlibs published, so SAI resolves the SDK automatically
$ dpkg-deb -I target/debs/trixie/sys-sdk_1.mlnx.4.10.1106_amd64.deb shlibs
<sdk-soname> 1 sys-sdk-main (= 1.mlnx.4.10.1106)
...

$ dpkg-deb -f target/debs/trixie/mlnx-sai_1.mlnx.SAIBuild2605.37.0.31_amd64.deb Depends
libc6 (>= 2.34), libnl-3-200 (>= 3.7.0-0.2+b1sonic1), libnl-genl-3-200 (>= 3.7.0-0.2+b1sonic1),
libnl-route-3-200 (>= 3.7.0-0.2+b1sonic1), libxml2 (>= 2.7.4), sys-sdk-main (= 1.mlnx.4.10.1106)

# Production (profiling off): dbgsym split, runtime stripped, code optimized
$ ls target/debs/trixie/sys-sdk_*_amd64-dbgsym.ddeb
target/debs/trixie/sys-sdk_1.mlnx.4.10.1106_amd64-dbgsym.ddeb

$ ls target/debs/trixie/mlnx-sai-dbgsym_*.deb
target/debs/trixie/mlnx-sai-dbgsym_1.mlnx.SAIBuild2605.37.0.31_amd64.deb

$ readelf -SW /tmp/sdk/usr/lib/<sdk-lib>.so | grep -c debug_info
0

$ readelf --debug-dump=info <dbgsym> | grep -m1 DW_AT_producer
DW_AT_producer : GNU C17 ... -O3 ...

Which release branch to backport (provide reason below if selected)

  • 202305
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511
  • 202512
  • 202605
  • 202608

Tracking issue/work item for backport/cherry-pick request (GitHub issue or Microsoft ADO): N/A
Failure type: N/A

Tested branch

  • master
  • 202305
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511
  • 202512
  • 202605
  • 202608
  • N/A

Test result

Verified on trixie / amd64 with sys-sdk 1.mlnx.4.10.1106 and
mlnx-sai 1.mlnx.SAIBuild2605.37.0.31, both built from source:

  • default / profiling off (SPLIT_DBGSYM=y) -- dbgsym packages emitted for both
    components, runtime debs stripped (no .debug_info, .gnu_debuglink present),
    DWARF lives in the dbgsym packages, and DW_AT_producer reports an optimized
    compile (-O3 / -O2, not -O0)
  • SONIC_PROFILING_ON=y (SPLIT_DBGSYM=n) -- no dbgsym package emitted for
    either component, .debug_info present in the SDK and SAI shared libraries,
    and DW_AT_producer reports -ggdb -O0 for the compile units
  • in both modes: sys-sdk Depends: populated from dpkg-shlibdeps;
    sys-sdk-dev depends on sys-sdk-main (= 1.mlnx.4.10.1106); sys-sdk ships
    a shlibs file, and mlnx-sai Depends: resolves
    sys-sdk-main (= 1.mlnx.4.10.1106) through it
  • -dbg syncd docker image builds with no missing dbgsym prerequisite
  • with profiling on, gdb on syncd resolves file:line breakpoints in SDK
    sources

Description for the changelog

  • N/A

Link to config_db schema for YANG module changes

  • N/A

Details if related

A picture of a cute animal (not mandatory but encouraged)

      .---.        .-----------
     /     \  __  /    ------
    / /     \(  )/    -----
   //////   ' \/ `   ---
  //// / // :    : ---
 // /   /  /`    '--
//          //..\\
       ====UU====UU====
           '//||\\`
             ''``

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@abelamit abelamit added the Tested for 202605 branch Tested for 202605 branch label Sep 14, 2026
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Signed-off-by: Nazarii Hnydyn <nazariig@nvidia.com>
Signed-off-by: Nazarii Hnydyn <nazariig@nvidia.com>
@nazariig
nazariig force-pushed the 202605-dbg-build-nvda branch from c72e4ab to 00b50ec Compare September 18, 2026 08:08
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants