Supporting custom section from config.json by HimanshuChoudhary-Xilinx · Pull Request #191 · Xilinx/aiebu

HimanshuChoudhary-Xilinx · 2025-12-09T16:13:22Z

Problem solved by the commit

Supporting custom section from config.json + testcase

sample json: test/aie2ps-ctrlcode/eff_net_coal/config.json in PR
Output: test/aie2ps-ctrlcode/eff_net_coal/eff_net_coal.gold in PR

Bug / issue (if any) fixed, which PR introduced the bug, how it was discovered

How problem was solved, alternative solutions (if any) and why they were rejected

Risks (if any) associated the changes in the commit

What has been tested and how, request additional testing if necessary

Documentation impact (if any)

All custom section have section type "SHT_LOUSER+1"
customer_section->section_name is the section name in elf

Config.json

{
    "xrt-kernels": [
      {
        "name" : "DPU",
       "arguments" : [
        {
          "name" : "arg0",
          "type" : "char *",
          "offset" : "0x00"
        },
        {
          "name" : "arg1",
          "type" : "char *",
          "offset" : "0x08"
        },
        {
          "name" : "arg2",
          "type" : "char *",
          "offset" : "0x10"
        },
        {
          "name" : "arg3",
          "type" : "char *",
          "offset" : "0x18"
        }
        ],
        "instance" : [
            {
              "id" : "vadd",
              "ctrl_code_file" : "./ml_asm/merged_control.asm"
            },
            {
              "id" : "va123",
              "ctrl_code_file" : "./ml_asm/merged_control.asm"
            }
        ]
      },
      {
       "name" : "RESNET",
       "arguments" : [
        {
          "name" : "arg0",
          "type" : "char *",
          "offset" : "0x00"
        },
        {
          "name" : "arg1",
          "type" : "char *",
          "offset" : "0x08"
        },
        {
          "name" : "arg2",
          "type" : "char *",
          "offset" : "0x10"
        },
        {
          "name" : "arg3",
          "type" : "char *",
          "offset" : "0x18"
        }
        ],
        "instance" : [
            {
              "id" : "rescc",
              "ctrl_code_file" : "./ml_asm/merged_control.asm"
            },
            {
              "id" : "res123",
              "ctrl_code_file" : "./ml_asm/merged_control.asm"

            }
        ]
      }
    ],
    "customer_section": [
      {
       "section_name": "ALL",
       "path": "./asm/pdi.asm"
      }
    ]
}

Global level

Its not part of any group
type will be SHT_LOUSER+1

Section Headers:
[Nr] Name                 Type         Addr     Off    Size   ES Flags Lk Inf Al
 [ 1] .shstrtab            STRTAB       00000000 311d5a 002480  0        0   0  1
 [ 2] .strtab              STRTAB       00000000 3141da 00003a  0        0   0  0
 [ 3] .symtab              SYMTAB       00000000 314214 000070 16        2   1  4
 [ 4] ALL                  SHT_LOUSER+1 00000000 004580 00007a  0 WA     0   0 16

Copilot

Pull request overview

This PR adds support for custom sections from config.json at three hierarchical levels: global, kernel, and instance. Custom sections are defined in JSON configuration files and loaded from external files, allowing additional data to be embedded in the final ELF output.

Key Changes:

Added parsing logic for "customer_section" arrays from JSON at global, kernel, and instance levels
Implemented data structures to track custom sections throughout the preprocessing and encoding pipeline
Integrated custom section writers into the ELF generation process

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/cpp/preprocessor/preprocessor_input.h	Added custom section storage and accessor methods to base preprocessor input class
src/cpp/preprocessor/aie2ps/aie2ps_preprocessor_input.h	Implemented custom section parsing and storage at all three levels for AIE2PS
src/cpp/preprocessor/aie2ps/aie2ps_preprocessor.h	Propagated custom sections from input to output during preprocessing
src/cpp/preprocessor/aie2ps/aie2ps_preprocessed_output.h	Added custom section storage and accessors to preprocessed output classes
src/cpp/preprocessor/aie2/aie2_blob_preprocessor_input.h	Added custom section data structures and parsing helper for AIE2
src/cpp/preprocessor/aie2/aie2_blob_preprocessor_input.cpp	Implemented custom section parsing from JSON for AIE2 at all levels
src/cpp/preprocessor/aie2/aie2_blob_preprocessor.h	Propagated custom sections from input to output in AIE2 preprocessor
src/cpp/preprocessor/aie2/aie2_blob_preprocessed_output.h	Added custom section storage structures to AIE2 output classes
src/cpp/encoder/aie2ps/aie2ps_encoder.h	Integrated custom sections into section writers during encoding
src/cpp/encoder/aie2/aie2_blob_encoder.h	Added custom section writers to AIE2 encoder output
src/cpp/elf/aie2ps/aie2ps_elfwriter.h	Included custom sections in ELF generation for AIE2PS
src/cpp/elf/aie2/aie2_blob_elfwriter.h	Included custom sections in ELF generation for AIE2
src/cpp/common/writer.h	Added custom section storage and accessor methods to writer classes

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/cpp/preprocessor/aie2ps/aie2ps_preprocessor_input.h

Copilot · 2025-12-09T16:14:43Z

src/cpp/preprocessor/aie2ps/aie2ps_preprocessor_input.h

+    const auto& pt_customer_sections = pt.get_child_optional("customer_section");
+    if (pt_customer_sections) {
+      for (const auto& [sec_unused, section] : pt_customer_sections.get()) {


Corrected spelling of 'pt_customer_sections' variable name to 'pt_custom_sections' for consistency.

Suggested change

const auto& pt_customer_sections = pt.get_child_optional("customer_section");

if (pt_customer_sections) {

for (const auto& [sec_unused, section] : pt_customer_sections.get()) {

const auto& pt_custom_sections = pt.get_child_optional("customer_section");

if (pt_custom_sections) {

for (const auto& [sec_unused, section] : pt_custom_sections.get()) {

Copilot · 2025-12-09T16:14:44Z

src/cpp/preprocessor/aie2/aie2_blob_preprocessor_input.h

+  // Helper to parse customer_section array from JSON
+  std::map<std::string, std::vector<uint8_t>> parse_customer_sections(
+      const boost::property_tree::ptree& pt,
+      const std::vector<std::string>& paths)
+  {
+    std::map<std::string, std::vector<uint8_t>> custom_sections;
+    const auto& pt_customer_sections = pt.get_child_optional("customer_section");
+    if (pt_customer_sections) {
+      for (const auto& [sec_unused, section] : pt_customer_sections.get()) {


Corrected spelling of 'customer_sections' to 'custom_sections'.

Suggested change

// Helper to parse customer_section array from JSON

std::map<std::string, std::vector<uint8_t>> parse_customer_sections(

const boost::property_tree::ptree& pt,

const std::vector<std::string>& paths)

{

std::map<std::string, std::vector<uint8_t>> custom_sections;

const auto& pt_customer_sections = pt.get_child_optional("customer_section");

if (pt_customer_sections) {

for (const auto& [sec_unused, section] : pt_customer_sections.get()) {

// Helper to parse custom_section array from JSON

std::map<std::string, std::vector<uint8_t>> parse_custom_sections(

const boost::property_tree::ptree& pt,

const std::vector<std::string>& paths)

{

std::map<std::string, std::vector<uint8_t>> custom_sections;

const auto& pt_custom_sections = pt.get_child_optional("custom_section");

if (pt_custom_sections) {

for (const auto& [sec_unused, section] : pt_custom_sections.get()) {

Copilot · 2025-12-09T16:14:44Z

src/cpp/preprocessor/aie2/aie2_blob_preprocessor_input.h

+    const auto& pt_customer_sections = pt.get_child_optional("customer_section");
+    if (pt_customer_sections) {
+      for (const auto& [sec_unused, section] : pt_customer_sections.get()) {


Corrected spelling of 'pt_customer_sections' variable name to 'pt_custom_sections' for consistency.

Suggested change

const auto& pt_customer_sections = pt.get_child_optional("customer_section");

if (pt_customer_sections) {

for (const auto& [sec_unused, section] : pt_customer_sections.get()) {

const auto& pt_custom_sections = pt.get_child_optional("customer_section");

if (pt_custom_sections) {

for (const auto& [sec_unused, section] : pt_custom_sections.get()) {

Copilot · 2025-12-09T16:14:45Z

src/cpp/preprocessor/aie2/aie2_blob_preprocessor_input.cpp

+      const auto& pt_customer_sections = pic.get_child_optional("customer_section");
+      if (pt_customer_sections) {
+        for (const auto& [sec_unused, section] : pt_customer_sections.get()) {


Corrected spelling of 'pt_customer_sections' variable name to 'pt_custom_sections' for consistency.

Suggested change

const auto& pt_customer_sections = pic.get_child_optional("customer_section");

if (pt_customer_sections) {

for (const auto& [sec_unused, section] : pt_customer_sections.get()) {

const auto& pt_custom_sections = pic.get_child_optional("customer_section");

if (pt_custom_sections) {

for (const auto& [sec_unused, section] : pt_custom_sections.get()) {

github-actions

clang-tidy made some suggestions

github-actions · 2025-12-09T16:18:37Z

src/cpp/preprocessor/aie2ps/aie2ps_preprocessor_input.h

 protected: // NOLINT
  std::map<std::string, std::map<std::string, std::shared_ptr<asm_preprocessor_input>>> m_preprocessor_input;
+  // Global level custom sections
+  std::map<std::string, std::vector<uint8_t>> m_global_custom_sections;


warning: member variable 'm_global_custom_sections' has protected visibility [cppcoreguidelines-non-private-member-variables-in-classes]

std::map<std::string, std::vector<uint8_t>> m_global_custom_sections; ^

github-actions · 2025-12-09T16:18:37Z

src/cpp/preprocessor/aie2ps/aie2ps_preprocessor_input.h

+  // Global level custom sections
+  std::map<std::string, std::vector<uint8_t>> m_global_custom_sections;
+  // Kernel level custom sections: kernel_name -> section_name -> data
+  std::map<std::string, std::map<std::string, std::vector<uint8_t>>> m_kernel_custom_sections;


warning: member variable 'm_kernel_custom_sections' has protected visibility [cppcoreguidelines-non-private-member-variables-in-classes]

std::map<std::string, std::map<std::string, std::vector<uint8_t>>> m_kernel_custom_sections; ^

github-actions · 2025-12-09T16:18:37Z

src/cpp/preprocessor/aie2ps/aie2ps_preprocessor_input.h

+  // Kernel level custom sections: kernel_name -> section_name -> data
+  std::map<std::string, std::map<std::string, std::vector<uint8_t>>> m_kernel_custom_sections;
+  // Instance level custom sections: kernel_name -> instance_name -> section_name -> data
+  std::map<std::string, std::map<std::string, std::map<std::string, std::vector<uint8_t>>>> m_instance_custom_sections;


warning: member variable 'm_instance_custom_sections' has protected visibility [cppcoreguidelines-non-private-member-variables-in-classes]

std::map<std::string, std::map<std::string, std::map<std::string, std::vector<uint8_t>>>> m_instance_custom_sections; ^

github-actions · 2025-12-09T16:18:37Z

src/cpp/preprocessor/preprocessor_input.h

 protected:
  std::map<std::string, std::vector<char>> m_data;
  std::map<std::string, std::vector<char>> m_controlpkt;
+  std::map<std::string, std::vector<uint8_t>> m_custom_data;


warning: member variable 'm_custom_data' has protected visibility [cppcoreguidelines-non-private-member-variables-in-classes]

std::map<std::string, std::vector<uint8_t>> m_custom_data; ^

github-actions

clang-tidy made some suggestions

github-actions · 2025-12-11T10:55:00Z

src/cpp/preprocessor/aie2ps/aie2ps_preprocessor_input.h

 {
 protected: // NOLINT
  std::map<std::string, std::map<std::string, std::shared_ptr<asm_preprocessor_input>>> m_preprocessor_input;
+  // Global level custom sections


warning: member variable 'm_preprocessor_input' has protected visibility [cppcoreguidelines-non-private-member-variables-in-classes]

std::map<std::string, std::map<std::string, std::shared_ptr<asm_preprocessor_input>>> m_preprocessor_input; ^

github-actions · 2025-12-11T10:55:00Z

src/cpp/preprocessor/aie2ps/aie2ps_preprocessor_input.h

  std::map<std::string, std::map<std::string, std::shared_ptr<asm_preprocessor_input>>> m_preprocessor_input;
+  // Global level custom sections
+  std::map<std::string, std::vector<uint8_t>> m_global_custom_sections;
+  // Kernel level custom sections: kernel_name -> section_name -> data


warning: member variable 'm_global_custom_sections' has protected visibility [cppcoreguidelines-non-private-member-variables-in-classes]

std::map<std::string, std::vector<uint8_t>> m_global_custom_sections; ^

github-actions · 2025-12-11T10:55:00Z

src/cpp/preprocessor/aie2ps/aie2ps_preprocessor_input.h

+  std::map<std::string, std::vector<uint8_t>> m_global_custom_sections;
+  // Kernel level custom sections: kernel_name -> section_name -> data
+  std::map<std::string, std::map<std::string, std::vector<uint8_t>>> m_kernel_custom_sections;
+  // Instance level custom sections: kernel_name -> instance_name -> section_name -> data


warning: member variable 'm_kernel_custom_sections' has protected visibility [cppcoreguidelines-non-private-member-variables-in-classes]

std::map<std::string, std::map<std::string, std::vector<uint8_t>>> m_kernel_custom_sections; ^

github-actions · 2025-12-11T10:55:00Z

src/cpp/preprocessor/aie2ps/aie2ps_preprocessor_input.h

+  std::map<std::string, std::map<std::string, std::vector<uint8_t>>> m_kernel_custom_sections;
+  // Instance level custom sections: kernel_name -> instance_name -> section_name -> data
+  std::map<std::string, std::map<std::string, std::map<std::string, std::vector<uint8_t>>>> m_instance_custom_sections;
+


warning: member variable 'm_instance_custom_sections' has protected visibility [cppcoreguidelines-non-private-member-variables-in-classes]

std::map<std::string, std::map<std::string, std::map<std::string, std::vector<uint8_t>>>> m_instance_custom_sections; ^

sonals

This should not be so complicated. Can we not just add key/value pair API where the key is section name and value is the content? The client should deal with "Added parsing logic for "customer_section" arrays from JSON at global, kernel, and instance levels"

larry9523 · 2025-12-17T23:27:39Z

This should not be so complicated. Can we not just add key/value pair API where the key is section name and value is the content? The client should deal with "Added parsing logic for "customer_section" arrays from JSON at global, kernel, and instance levels"

I am thinking the similar approach. We can just have the key(customer section name)/value(customer section blob) as a section in ELF. We can make use of the .group feature.

if it does not point to any group symbol, it is the global section.
if it points to a group symbol, it belongs to a kernel
if it points to a instance symbol, it belongs to a instance
The parsing logic should be in xrt::elf APIs

github-actions

clang-tidy made some suggestions

github-actions · 2026-03-10T11:22:11Z

src/cpp/elf/elfwriter.cpp

-    sec_data.set_type(ELFIO::SHT_PROGBITS);
+    if (buffer->get_type() == code_section::custom) {
+      sec_data.set_type(SHT_CUSTOM_SECTION);
+      sec_data.set_info(info_index);


warning: narrowing conversion from 'ELFIO::Elf_Word' (aka 'unsigned int') to signed type 'int' is implementation-defined [bugprone-narrowing-conversions]

sec_data.set_info(info_index); ^

github-actions · 2026-03-10T11:22:11Z

src/cpp/elf/elfwriter.h

 constexpr int phdr_align = 8;
 constexpr int program_header_static_count = 2;
 constexpr int program_header_dynamic_count = 3;
+constexpr int SHT_CUSTOM_SECTION = ELFIO::SHT_LOUSER + 1;


warning: narrowing conversion from constant value 2147483649 (0x80000001) of type 'Elf_Word' (aka 'unsigned int') to signed type 'int' is implementation-defined [bugprone-narrowing-conversions]

constexpr int SHT_CUSTOM_SECTION = ELFIO::SHT_LOUSER + 1; ^

larry9523 · 2026-03-10T17:26:44Z

@HimanshuChoudhary-Xilinx this looks much cleaner. Thanks！
Can we update the document in a bit detail?

from user point of view, what is the mapping between the name in config.json and section name in elf?
A bit detail (ELF spec) between the sh_info and entries in symbol table.
Should we combine sh_link as well to make it more informative, where it should point to symbol table section while sh_info points to kernel or instance?

Signed-off-by: Himanshu Choudhary <Himanshu.Choudhary@amd.com>

HimanshuChoudhary-Xilinx · 2026-03-11T18:38:50Z

@HimanshuChoudhary-Xilinx this looks much cleaner. Thanks！ Can we update the document in a bit detail?

from user point of view, what is the mapping between the name in config.json and section name in elf?

A bit detail (ELF spec) between the sh_info and entries in symbol table.

Should we combine sh_link as well to make it more informative, where it should point to symbol table section while sh_info points to kernel or instance?

Update

larry9523 · 2026-03-11T21:53:03Z

@HimanshuChoudhary-Xilinx this looks much cleaner. Thanks！ Can we update the document in a bit detail?

from user point of view, what is the mapping between the name in config.json and section name in elf?

A bit detail (ELF spec) between the sh_info and entries in symbol table.

Should we combine sh_link as well to make it more informative, where it should point to symbol table section while sh_info points to kernel or instance?

Update

Looks good.

One more question. Do we require customized sections in all level should have unique name? If so, we may want to put them into the ELF spec too.

sonals · 2026-03-12T22:09:42Z

I still feel this is more specialized than it should be. Can we not have a simple key/value pair design where the key is defined by the user? The user can define the naming schema, etc. So essentially this reduces to supporting only the Global level.

Signed-off-by: Himanshu Choudhary <Himanshu.Choudhary@amd.com>

sonals

There seems to be a lot of code duplication where we re-implement a logic for all the supported platform types. Can we not move the repeated methods to base classes?

Signed-off-by: Himanshu Choudhary <Himanshu.Choudhary@amd.com>

HimanshuChoudhary-Xilinx · 2026-03-24T10:34:35Z

There seems to be a lot of code duplication where we re-implement a logic for all the supported platform types. Can we not move the repeated methods to base classes?

moved common code to common area

github-actions

clang-tidy made some suggestions

github-actions · 2026-03-24T10:36:18Z

src/cpp/common/writer.h

+class config_writer_base : public writer
+{
+protected:
+  std::vector<std::shared_ptr<writer>> m_global_custom_sections;


warning: member variable 'm_global_custom_sections' has protected visibility [cppcoreguidelines-non-private-member-variables-in-classes]

std::vector<std::shared_ptr<writer>> m_global_custom_sections; ^

github-actions · 2026-03-24T10:36:18Z

src/cpp/preprocessor/aie2ps/aie2ps_preprocessor_input.h

  const file_artifact* m_artifacts = nullptr;
 protected: // NOLINT
  std::map<std::string, std::map<std::string, std::shared_ptr<asm_preprocessor_input>>> m_preprocessor_input;
+  global_custom_section_storage m_global_custom_sections;


warning: member variable 'm_global_custom_sections' has protected visibility [cppcoreguidelines-non-private-member-variables-in-classes]

global_custom_section_storage m_global_custom_sections; ^

Copilot AI review requested due to automatic review settings December 9, 2025 16:13

Copilot AI reviewed Dec 9, 2025

View reviewed changes

github-actions bot reviewed Dec 9, 2025

View reviewed changes

HimanshuChoudhary-Xilinx requested a review from larry9523 December 11, 2025 10:50

github-actions bot reviewed Dec 11, 2025

View reviewed changes

larry9523 requested review from donwalkarsoham and sayyanna December 11, 2025 23:59

sonals self-requested a review December 17, 2025 00:46

sonals requested changes Dec 17, 2025

View reviewed changes

HimanshuChoudhary-Xilinx force-pushed the custom_section branch from 94bcd42 to 595576e Compare March 10, 2026 11:15

github-actions bot reviewed Mar 10, 2026

View reviewed changes

Supporting custom section from config.json

48ef999

Signed-off-by: Himanshu Choudhary <Himanshu.Choudhary@amd.com>

HimanshuChoudhary-Xilinx force-pushed the custom_section branch from 7a5f5ca to 48ef999 Compare March 11, 2026 18:21

larry9523 approved these changes Mar 11, 2026

View reviewed changes

HimanshuChoudhary-Xilinx added 4 commits March 17, 2026 23:01

only global level custom_section

3174da4

Signed-off-by: Himanshu Choudhary <Himanshu.Choudhary@amd.com>

fix

25049e7

Signed-off-by: Himanshu Choudhary <Himanshu.Choudhary@amd.com>

fix

d5604c0

Signed-off-by: Himanshu Choudhary <Himanshu.Choudhary@amd.com>

build fix

9ce78ea

Signed-off-by: Himanshu Choudhary <Himanshu.Choudhary@amd.com>

larry9523 requested a review from sonals March 23, 2026 21:05

sonals reviewed Mar 23, 2026

View reviewed changes

restructuring

95a9a61

Signed-off-by: Himanshu Choudhary <Himanshu.Choudhary@amd.com>

github-actions bot reviewed Mar 24, 2026

View reviewed changes

Conversation

HimanshuChoudhary-Xilinx commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem solved by the commit

Bug / issue (if any) fixed, which PR introduced the bug, how it was discovered

How problem was solved, alternative solutions (if any) and why they were rejected

Risks (if any) associated the changes in the commit

What has been tested and how, request additional testing if necessary

Documentation impact (if any)

Config.json

Global level

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

sonals left a comment

Choose a reason for hiding this comment

Uh oh!

larry9523 commented Dec 17, 2025

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Mar 10, 2026

Choose a reason for hiding this comment

Uh oh!

larry9523 commented Mar 10, 2026

Uh oh!

HimanshuChoudhary-Xilinx commented Mar 11, 2026

Uh oh!

larry9523 commented Mar 11, 2026

Uh oh!

HimanshuChoudhary-Xilinx commented Dec 9, 2025 •

edited

Loading

sonals commented Mar 12, 2026 •

edited

Loading