Fast attention kernels and operator fusion reorganize transformer computation to limit data movement and kernel launches on the accelerator. These optimizations in principle leave the model weights and its mathematical function unchanged, even if the order of operations and precision can produce small numerical differences. The tutorial published by Machine Learning Mastery details these techniques, with code examples.
A transformer layer relies on common operations: linear projections, attention, normalization, activation functions, and residual additions. A direct implementation launches a separate compute kernel for each operation and writes each intermediate tensor to accelerator memory. The computation remains correct, but execution time can be spent more on moving values and launching kernels than on useful calculations.
Operator fusion
The SwiGLU block, used by many decoder-only models, illustrates this problem. A direct implementation launches matrix multiplications for the projections, a kernel for the SiLU function, another for element-wise multiplication, then a final matrix multiplication. At each boundary, an intermediate tensor may be written to high-bandwidth memory (HBM) and then reread. Fusion involves combining these steps into a single kernel. It saves time when memory movement and launches are heavy, and much less when a large matrix multiplication already dominates execution. Kernel choice remains empirical: the entire workload must be profiled.
Other fusions are common in transformers, such as adding a residual connection followed by RMSNorm or LayerNorm normalization, or combining dequantization with a matrix multiplication. Fusion has its limits, however: operations with incompatible data layouts or a very large intermediate state combine poorly. An overly large kernel can increase register pressure, reduce occupancy, and become slower than several well-tuned kernels.
FlashAttention and paged attention
Classic attention computes a score matrix of size T² per head, where T is the sequence length. On a GPU, producing it, writing it to global memory, and then rereading it consumes a lot of bandwidth, which can become a bottleneck during long prefill phases. FlashAttention processes queries, keys, and values in blocks. For each, it computes a portion of the scores, updates the softmax statistics, and accumulates the corresponding output in on-chip memory, without storing the full matrix. PyTorch exposes this optimization via the scaled_dot_product_attention() function. The tutorial notes that hardcoding a backend can cause code to fail if no backend supports the inputs or hardware.
The KV cache, which stores already computed keys and values, grows one token at a time, for an unpredictable final length. A contiguous allocation per request therefore leaves reserved and unused space. The paged cache divides storage into fixed-size blocks and relies on a block table that maps logical positions to physical blocks. The attention kernel must then translate each position through this table. Paging does not reduce the number of logical elements, but it limits fragmentation and reserved capacity.
In the provided example, vLLM uses the Qwen/Qwen2.5-1.5B-Instruct model with blocks of 16 tokens (block_size=16) and the ‘TRITON_ATTN’ backend. According to the tutorial, vLLM checks backend compatibility with the model and hardware during initialization and reports an error in case of incompatibility.
Fusion and CUDA Graphs with torch.compile
The torch.compile() function captures a sequence of operations with TorchDynamo, then compiles it with TorchInductor, its default backend. TorchInductor can generate a single Triton kernel for compatible operations that the standard mode would launch separately. Graph breaks, caused for example by data-dependent flow control or conversions of tensors to Python scalars, limit the areas the compiler can optimize.
The ‘reduce-overhead’ mode instructs TorchInductor to use CUDA Graphs when the captured graph is eligible. A CUDA Graph records the sequence of GPU operations and replays it with a single CPU submission. It does not create additional fusion; it complements that of TorchInductor. It requires stable memory addresses and compatible shapes, and it can consume more memory. The tutorial recommends checking results with tolerances appropriate to the data type, profiling execution, and measuring end-to-end performance.
