[Bug]: HETERO PLUGIN Not splitting cyclic dependencies properly

[danielliuce]

OpenVINO Version

Master Branch

Operating System

Other (Please specify in description)

Device used for inference

HETERO

Framework

ONNX

Model used

YOLOv26

Issue description

OS: SUSE15
There is a bug with the hetero plugin splitting cyclic dependencies. The bug occurs with a shared constant into two different subgraphs.

Then, the collect_subgraphs_ids function merges them incorrectly causing a cycle to occur.

This is the hetero graph that splits

Step-by-step reproduction

This is the graph.

yolo26s.xml

Any weights is possible, accuracy is not important, this is just the hetero plugin not splitting properly.
Run the graph through openvino's hetero plugin with two devices and assign the affinities of the graph nodes accordingly.

Relevant log output

Issue submission checklist

• I'm reporting an issue. It's not a question.
• I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
• There is reproducer code and related data files such as images, videos, models, etc.

[mlukasze]

Ref. CVS-186543

[wgzintel]

@danielliuce How can I reproduce this issue? Can you provide test cases to reproduce this issue? How do you set the affinities for YOLOv26?

[danielliuce]

@danielliuce How can I reproduce this issue? Can you provide test cases to reproduce this issue? How do you set the affinities for YOLOv26?

Sorry, I missed this but I see you have it open in #36024 so I'm very grateful, thank you so much for that. I'm gonna port it over and test and see if that fixes this issue

[danielliuce]

Can you provide test cases to reproduce this issue? How do you set the affinities for YOLOv26?

It looks like the issue is still there unfortunately :(
I'm working on the affinities and getting something to you to reproduce

[danielliuce]

        auto promote_boundaries_for_node = [&](size_t node_idx) {
            const auto& cyc_dep = node_subgraph_cyclic_input_dependencies[node_idx];
            if (!bit_any(cyc_dep))
                return;
            // Collect all subgraph inputs that the cyclic feedback transitively depends on.
            Bits cyclic_inputs_dependencies(W, 0ULL);
            for (size_t w = 0; w < W; ++w) {
                uint64_t bits = cyc_dep[w];
                while (bits) {
                    const size_t b = (w << 6) + ctz64(bits);
                    bits &= bits - 1;
                    const auto& cyclic_input = bit_to_input[b];
                    const auto cyclic_input_idx = get_index_by_node(cyclic_input.get_source_output().get_node());
                    bit_or(cyclic_inputs_dependencies, node_subgraph_input_dependencies[cyclic_input_idx]);
                }
            }
            const bool has_cyclic_deps = bit_any(cyclic_inputs_dependencies);
            for (const auto& input : ordered_inputs[node_idx]) {
                const auto input_source_idx = get_index_by_node(input.get_source_output().get_node());
                const auto& src_cyc_dep = node_subgraph_cyclic_input_dependencies[input_source_idx];
                const auto& src_sg_dep = node_subgraph_input_dependencies[input_source_idx];
                if (bit_intersects(cyc_dep, src_cyc_dep)) 
                    continue;
                if (!has_cyclic_deps) 
                    continue;
                if (bit_intersects(cyclic_inputs_dependencies, src_sg_dep)) {
                    _subgraph_inputs.insert(input);
                }
                // HACK
                else if (ov::op::util::is_constant(input.get_source_output().get_node()) &&
                         !input.get_source_output().get_element_type().is_real() &&
                         input.get_source_output().get_target_inputs().size() > 1 &&
                         _affinities.count(_ordered_ops[node_idx]) &&
                         _affinities.at(_ordered_ops[node_idx]) == "FPGA") {
                    _subgraph_inputs.insert(input);
                }
            }
        };
        for (size_t node_idx = 0; node_idx < nodes_count; ++node_idx) {
            promote_boundaries_for_node(node_idx);
        }

This is the HACK/Patch we put in place that fixed it, could the fix have this included in it maybe?

[danielliuce]

I've tried to create a minimal reproducible example, but it seems like it's extremely non-trivial to reproduce it on a minimal example.
Could it be possible to assign the affinities based on that BIG picture I have in the description with the graph.xml?
HETERO could be one device affinity
FPGA could be one device affinity
and assigned accordingly, and then I think you'll trigger a bad sort issue.

[wgzintel]

@danielliuce Can this issue be reproduced in windows? Is the error you're seeing "Cannot resolve cycles during submodels split!"? When a cycle occurs. Could you share the test cases and test environment used to reproduce the problem?

[peterchen-intel]

@danielliuce Can you reproduce it with a sub_graph and GPU + CPU devices? If it is yes, then it will be easier for us to fix.
Or can you create a PR which can fix this issue directly? We can update and review it, if it doesn't impact others, then we can merge it.
#36024 may not fix this issue, but it is still a valid fixing for certain scenario (as it is in test case).

[danielliuce]

@danielliuce Can this issue be reproduced in windows? Is the error you're seeing "Cannot resolve cycles during submodels split!"? When a cycle occurs. Could you share the test cases and test environment used to reproduce the problem?

It's actually this assertion
OPENVINO_ASSERT(subgraph_topo_sorts_step < subgraphs.size(), "Cannot sort subgraphs!");

[danielliuce]

@danielliuce Can you reproduce it with a sub_graph and GPU + CPU devices? If it is yes, then it will be easier for us to fix. Or can you create a PR which can fix this issue directly? We can update and review it, if it doesn't impact others, then we can merge it. #36024 may not fix this issue, but it is still a valid fixing for certain scenario (as it is in test case).

I've fixed it with that one line hack above, but it's very hardcoded and I don't like it.

[danielliuce]

I will look into how to reproduce the problem, I agree that assigning the affinities manually on the big yolov26 graph is not a good solution

[danielliuce]

OK. I have the following scripts that you can use to reproduce this problem.
The python script sets the affinities based on our in house implementation of query_model.
You will see the assertion getting triggered

repro_yolo26_hetero_cycle.py

yolo26seg_affinities.json

python3 repro_yolo26_hetero_cycle.py --graph yolo26s.xml --aff yolo26seg_affinities.json

You need the yolo26s.xml that I put in the description with some random .bin weights and it will go through the hetero plugin and set the affinities from the json file and then have the assertion get triggered. But it should resolve the cycle and put a boundary node correctly right?

[wgzintel]

@danielliuce Can you share your yolo26s.bin which can reproduce the cyclic issue? I don't know how to produce the weights data in bin file .