Minimal FlashAttention in CUDA C++/CuTe: readable WMMA/CuTe kernels, no NxN workspace, up to 4.5x faster than naive PyTorch

FastCuda is a handwritten CUDA operator library featuring progressive GEMM and Reduce kernels, cuBLAS benchmarking, and C/C++/Python interfaces for learning, profiling, and performance optimization.

Bilingual CUDA SGEMM optimization tutorial and reference implementation, from naive kernels to Tensor Core WMMA | 双语 CUDA SGEMM 优化教程与参考实现，从朴素内核到 Tensor Core WMMA

Backward pass CUDA kernels for fused GEMM+Bias+GeLU on SM75 (Turing). Float4 vectorized WMMA kernels validated against PyTorch autograd. 3.1x faster than autograd at M=1024. 24/24 tests passing.

Compiler MVP that detects Transformer fusion patterns, generates optimized CUDA kernels with WMMA Tensor Cores, and executes them on real GPU hardware — 10.5 TFLOPs on RTX 2070, correctness validated against PyTorch.

Hand-written CUDA linear algebra library. A GEMM driven from a naive baseline to a compute-bound tensor-core kernel at ~90% of cuBLAS on an RTX 5070, profiled with Nsight Compute, plus GEMV, TRSM, CSR SpMV, cuSOLVER, a measured roofline, and NVML telemetry.

FlashAttention v1 forward pass in CUDA for NVIDIA Turing (SM75)

WMMA FP16-->FP32 Tensor Core GEMM with shared-memory tiling and cp.async-style pipelining, benchmarked against cuBLAS.

Community port of NVlabs/tiny-cuda-nn for AMD RDNA4 / gfx1201, validated on Radeon AI PRO R9700 with ROCm 7.2.

10,000-image LeNet-5 forward pass in ~28 ms on a single A40 via fused convolution and Tensor Cores (TF32).