DOC OFED 3.1.0 and 3.2.1 Kernel Oops when tracing

[whi71800]

Debian 12, kernel 6.1.162, trace from OFED 3.1.0:

Feb 19eb 19 17:41:46 cnivc001690953 kern.alert kernel: [ 99.264621] #PF: supervisor read access in kernel mode
Feb 19 17:41:46 cnivc001690953 kern.alert kernel: [ 99.269760] #PF: error_code(0x0000) - not-present page
Feb 19 17:41:46 cnivc001690953 kern.info kernel: [ 99.274897] PGD 20a9d06067 P4D 20a9d06067 PUD 208a295067 PMD 0
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.280819] Oops: 0000 [#1] PREEMPT SMP NOPTI
[snip]
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.301696] RIP: 0010:mlx5_fc_id+0x5/0x10 [mlx5_core]
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.306800] Code: 48 8b 87 d0 07 00 00 c3 cc cc cc cc 48 8b 87 d8 0b 00 00 48 8b 80 d0 07 00 00 c3 cc cc cc cc 66 0f 1f 44 00 00 0f 1f 44 00 00 <8b> 07 c3 cc cc cc cc 0f 1f 40 00 0f 1f 44 00 00 48 63 47 6c 48 89
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.325538] RSP: 0018:ffffcfa552de7658 EFLAGS: 00010206
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.330763] RAX: 0000000000000000 RBX: ffff8c2137645260 RCX: 0000000000000000
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.337887] RDX: 0000000000000000 RSI: ffff8c213764525c RDI: 0000000000000054
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.345012] RBP: ffff8c29c1632b00 R08: 0000000000000001 R09: 00000000001f6c37
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.352137] R10: ffff8c408821ae50 R11: 0000000000000000 R12: ffff8c408821ae50
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.359262] R13: ffff8c212d5f4cc8 R14: 0000000000000001 R15: 0000000000000000
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.366393] FS: 00007ffff5515200(0000) GS:ffff8c3ffd200000(0000) knlGS:0000000000000000
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.374480] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.380226] CR2: 0000000000000054 CR3: 0000002724874003 CR4: 0000000000770ef0
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.387357] PKRU: 55555554
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.390063] Call Trace:
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.392518]
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.394622] trace_event_raw_event_mlx5_fs_add_rule+0xd2/0x100 [mlx5_core]
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.401541] add_rule_fg+0x453/0x620 [mlx5_core]
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.406203] _mlx5_add_flow_rules+0x132b/0x15e0 [mlx5_core]
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.411818] ? trace_event_buffer_commit+0x59/0x250
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.416697] mlx5_ib_handler_MLX5_IB_METHOD_CREATE_FLOW+0x92a/0xc90 [mlx5_ib]
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.423837] ? xa_load+0x86/0xe0
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.427071] ib_uverbs_cmd_verbs+0xb5c/0xbf0 [ib_uverbs]
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.432383] ? parse_flow_flow_action+0x80/0x80 [mlx5_ib]
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.437791] ? trace_buffer_unlock_commit_regs+0x38/0x1f0
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.443190] ? trace_event_buffer_commit+0x59/0x250
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.448068] ? ib_uverbs_cmd_verbs+0x782/0xbf0 [ib_uverbs]
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.453556] ? avc_has_extended_perms+0x177/0x490
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.458262] ? __rb_reserve_next.constprop.0+0x1d1/0x410
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.463572] ? __rb_reserve_next.constprop.0+0x1d1/0x410
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.468877] ? ring_buffer_lock_reserve+0x112/0x380
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.473757] ? ring_buffer_unlock_commit+0x35/0x160
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.478638] ? trace_buffer_unlock_commit_regs+0x38/0x1f0
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.484038] ? ioctl_has_perm.constprop.0.isra.0+0xbd/0x120
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.489608] ib_uverbs_ioctl+0xa4/0x110 [ib_uverbs]
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.494489] __x64_sys_ioctl+0x8d/0xd0
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.498243] do_syscall_64+0x55/0xb0
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.501821] ? ib_uverbs_cmd_verbs+0x782/0xbf0 [ib_uverbs]
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.507306] ? trace_event_raw_event_rseq_update+0x48/0x80
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.512791] ? mlx5_ib_handler_MLX5_IB_METHOD_FLOW_ACTION_CREATE_PACKET_REFORMAT+0x2d0/0x2d0 [mlx5_ib]
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.522092] ? __rseq_handle_notify_resume+0x380/0x480
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.527232] ? avc_has_extended_perms+0x177/0x490
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.531937] ? __rb_reserve_next.constprop.0+0x1d1/0x410
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.537250] ? __rb_reserve_next.constprop.0+0x1d1/0x410
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.542564] ? ring_buffer_lock_reserve+0x112/0x380
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.547442] ? ring_buffer_unlock_commit+0x35/0x160
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.552323] ? trace_buffer_unlock_commit_regs+0x38/0x1f0
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.557721] ? trace_event_buffer_commit+0x59/0x250
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.562601] ? ftrace_syscall_exit+0xbc/0x110
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.566961] ? arch_exit_to_user_mode_prepare.constprop.0+0x6d/0xa0
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.573225] ? syscall_exit_to_user_mode+0x2f/0x60
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.578019] ? do_syscall_64+0x61/0xb0
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.581770] ? do_syscall_64+0x61/0xb0
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.585515] ? arch_exit_to_user_mode_prepare.constprop.0+0x6d/0xa0
Feb 19 17:41:46 cnivc001690953 kern.warn kernel: [ 99.591780] entry_SYSCALL_64_after_hwframe+0x6e/0xd8

drivers/net/ethernet/mellanox/mlx5/core/diag/fs_tracepoint.h

TRACE_EVENT(mlx5_fs_add_rule,
[snip]
__entry->counter_id =
mlx5_fc_id(rule->dest_attr.counter);

counter has been changed from a value, called counter_id, to a pointer called counter in the newer DOCA OFED drivers, which mlx5_fc_id will dereference to get the id member. Presumably the pointer is invalid and cannot be dereferenced for the code path shown.

Regards,
Simon

[whi71800]

counter only valid for MLX5_FLOW_DESTINATION_TYPE_COUNTER. For:

TRACE_EVENT(mlx5_fs_add_rule,
[snip]
if (rule->dest_attr.type &
MLX5_FLOW_DESTINATION_TYPE_COUNTER)

dest_attr.type is mlx5_flow_destination_type and not a bit field. As a result that code matches other flows where counter pointer has not been assigned. This appears to want to be == MLX5_FLOW_DESTINATION_TYPE_COUNTER.

Evidence from fs.c:

func: raw_fs_rule_add

if (flow_act->action & MLX5_FLOW_CONTEXT_ACTION_COUNT) {
[snip]
dst[dst_num].type = MLX5_FLOW_DESTINATION_TYPE_COUNTER;
dst[dst_num].counter = counter;
dst_num++;
}

That is the only code path to assign counter.

[knneth]

Thank you for providing a detailed report. Please note that this repository is not maintained by or affiliated with NVIDIA. Still, the suggested fix has been submitted upstream to the Linux kernel, which, pending approval, may later be incorporated into their drivers by NVIDIA. We'll also have a fix included in upcoming btech driver builds.