B200 Cross Entropy Backward Performance

[Jokeren]

Seems like it's slower than torch.compile generated code. Is it expected? I'm using quack/main and triton 3.6.0

=== Cross Entropy Backward Pass Benchmark ===
Tensor dimensions: [8192, 16384]
Input Data type: BFloat16
Input tensor shapes:
x: torch.Size([8192, 16384]), dtype: torch.bfloat16
target: torch.Size([8192]), dtype: torch.int64
Kernel execution time: 0.4894 ms
Mem throughput: 1097.00 GB/s
Ref kernel execution time: 0.2265 ms
Ref mem throughput: 2371.00 GB/s

cc @tridao @lezcano

[tridao]

Hmm strange, i'm getting different results. Sometimes I've seen results being off because the cpu (not gpu) was in low-power state, running it a few times generally will fix things.

python benchmarks/benchmark_cross_entropy.py --M 8192 --N 16384 --backward

=== Cross Entropy Backward Pass Benchmark ===
Tensor dimensions: [8192, 16384]
Input Data type: BFloat16
Input tensor shapes:
x: torch.Size([8192, 16384]), dtype: torch.bfloat16
target: torch.Size([8192]), dtype: torch.int64
Kernel execution time: 0.1072 ms
Mem throughput: 5010.00 GB/s
Ref kernel execution time: 0.2241 ms
Ref mem throughput: 2396.00 GB/s

[lezcano]

Could be that you were not locking the GPU clocks and there was some throttling Keren?
I will try to repro sometime today.

[Jokeren]

Could be that you were not locking the GPU clocks and there was some throttling Keren?

Yeah that could be