Throughput-Oriented Computing: An Architectural Account of the Modern GPU
1Introduction
A processor can be optimized for one of two quantities: the latency of a single task, or the throughput of many tasks. A modern CPU core is a latency machine. It devotes most of its transistor budget to large caches, branch predictors, out-of-order scheduling windows and speculative execution—mechanisms whose sole purpose is to make one instruction stream finish sooner. A GPU makes the opposite bet. It assumes the workload consists of thousands to millions of independent, similar operations (pixels, matrix elements, particles, tokens) and spends its silicon on arithmetic units and on the on-chip state needed to keep a very large number of threads in flight [1, 4].
The consequence is that an individual GPU thread is slow: it runs at a modest clock, has no out-of-order execution and may wait hundreds of cycles for memory. Yet the device as a whole delivers one to two orders of magnitude more floating-point operations and memory bandwidth than a contemporary CPU socket. Understanding a GPU therefore means understanding how it converts abundant parallelism into sustained throughput. Figure 1 summarizes the difference in resource allocation.
2The Execution Model: Kernels, Grids and SIMT
A GPU program is organized around a kernel: a function written from the perspective of a single scalar thread, which the host launches over an index space of many threads. The index space is hierarchical (Figure 2). Threads are grouped into thread blocks (CUDA) or work-groups (OpenCL/SYCL); blocks are arranged into a grid. Two properties of this hierarchy carry the design:
(i) Blocks are independent. The programming model guarantees nothing about the order or concurrency in which blocks execute. This lets the hardware schedule blocks onto however many multiprocessors a given chip has, so the same binary scales transparently from a laptop GPU with a handful of SMs to a data-center part with more than a hundred [2].
(ii) Threads within a block can cooperate. They are guaranteed to be co-resident on one multiprocessor, can exchange data through fast on-chip shared memory, and can synchronize with a barrier (__syncthreads()).
2.1SIMT: scalar programs, vector hardware
Although the programmer writes scalar code, the hardware does not execute threads individually. Consecutive threads of a block are bundled into a warp (32 threads on NVIDIA hardware), and a warp issues one instruction at a time, executed in lock-step across its 32 lanes. This is the Single-Instruction, Multiple-Thread (SIMT) model introduced with the Tesla architecture [1]. SIMT is functionally similar to SIMD vector processing, with two differences that make it far easier to program: each lane has its own registers and program-visible thread index, and the hardware—not the programmer—handles lanes that take different branches (Section 6). Amortizing instruction fetch, decode and scheduling over 32 data elements is the first reason a GPU can afford so many arithmetic units.
3Microarchitecture of the Streaming Multiprocessor
The GPU chip is an array of streaming multiprocessors (SMs; AMD: compute units) sharing an L2 cache and a set of memory controllers. Each SM is a self-contained throughput engine (Figure 3). In recent NVIDIA designs an SM is split into four processing blocks, each containing:
- a warp scheduler and dispatch unit able to issue one instruction per clock from any of its resident, ready warps;
- a slice of the register file—on the order of 64 KB per block, 256 KB per SM, which is larger than the SM’s L1 cache;
- integer and FP32 lanes (marketed as “CUDA cores”), FP64 units on data-center parts, special-function units (SFUs) for transcendental functions, and load/store units;
- a Tensor Core for dense matrix multiply-accumulate (Section 7).
The four blocks share an L1 data cache that is configurable with the block-visible shared memory, plus texture units. The striking feature is the size of the register file: because every resident thread keeps its full architectural state on-chip permanently, switching between warps costs zero cycles. There is no save/restore, unlike an operating-system context switch on a CPU.
4Latency Hiding Through Multithreading
A load from off-chip DRAM takes on the order of several hundred core cycles [6]. A CPU attacks this latency by avoiding it (caches, prefetchers) and by overlapping it within one thread (out-of-order execution). A GPU instead tolerates it. Each SM keeps up to dozens of warps resident (e.g. 48–64, i.e. 1,536–2,048 threads). On every cycle, each scheduler picks some warp whose next instruction has its operands ready and issues it. When a warp stalls on memory, the scheduler simply issues from another. If enough warps are available, the arithmetic units never idle even though every individual warp spends most of its time waiting.
The required amount of parallelism follows from Little’s Law [7], which relates the mean number of items in a system L to the arrival rate λ and the mean residence time W:
Applied to a memory system, the concurrency that must be in flight equals bandwidth multiplied by latency. For a device with memory bandwidth B = 3 TB/s and latency W ≈ 600 ns, roughly 1.8 MB of outstanding requests are needed to saturate DRAM—tens of thousands of concurrent 32-byte sectors spread across the chip. No small set of threads could generate that; a GPU with on the order of 105 resident threads can. Volkov [5] showed that the same concurrency can also be obtained with fewer warps that each expose more instruction-level parallelism (several independent loads before the first use), which is why high “occupancy” is a means, not a goal.
__launch_bounds__ matter for performance.5The Memory Hierarchy
Unlike a CPU, whose hierarchy is almost entirely transparent, the GPU exposes several memory spaces with very different costs, and programmers are expected to place data deliberately. Table 1 lists representative figures.
| Level | Scope | Capacity | Approx. latency | Managed by |
|---|---|---|---|---|
| Registers | Thread | ~256 KB / SM | ~1 cycle | Compiler |
| Shared memory / L1 | Block / SM | ~100–250 KB / SM | ~20–35 cycles | Programmer / HW |
| L2 cache | Device | Tens of MB | ~200–300 cycles | Hardware |
| Global memory (HBM/GDDR) | Device + host-visible | Tens to hundreds of GB | ~400–800 cycles | Programmer |
5.1Coalescing
Global memory is accessed in aligned 32-byte sectors (grouped into 128-byte lines). When the 32 threads of a warp execute a load, the hardware merges their addresses and issues the minimum number of sector transactions. If thread k reads element a[base + k] of a 4-byte array, the warp touches exactly 128 contiguous bytes: four sectors, perfect efficiency. If instead thread k reads a[k * stride] with a large stride, each thread needs its own sector, and effective bandwidth drops by up to 8×. This is why GPU data layouts favour structure-of-arrays over array-of-structures.
5.2Shared memory and tiling
Shared memory is a software-managed scratchpad divided into 32 banks, each 4 bytes wide. Threads of a warp accessing 32 distinct banks are served in one cycle; multiple threads hitting different addresses in the same bank are serialized (a bank conflict). The canonical use is tiling: a block cooperatively loads a tile of the input from global memory once, synchronizes, and then every thread reuses that tile many times from shared memory. In dense matrix multiplication with T×T tiles, each element fetched from DRAM is reused T times, raising arithmetic intensity (Section 8) by the same factor.
6Control-Flow Divergence
Because a warp shares one instruction stream, a data-dependent branch on which lanes disagree cannot be executed in a single pass. The hardware executes each path in turn, disabling (masking) the lanes not on the current path, then reconverges. For an if/else in which half the lanes go each way, the warp spends the time of both paths and achieves at most 50% lane utilization. Since Volta, NVIDIA GPUs track a program counter per thread (independent thread scheduling), which permits fine-grained synchronization between diverged threads, but it does not remove the throughput cost of divergence [8]. Research on dynamic warp formation [3] and similar techniques explores regrouping lanes to recover efficiency. The practical lesson is to arrange work so that threads within a warp follow the same control path, for example by sorting or bucketing inputs.
7Tensor Cores and Reduced Precision
Deep-learning workloads are dominated by dense matrix multiplication. Beginning with Volta (2017), NVIDIA added Tensor Cores: units that compute a small matrix multiply-accumulate D = AB + C on a tile (for example 4×4×4 per unit per clock in the original design) as a single warp-wide operation. A scalar FP32 lane performs one fused multiply-add per clock; a Tensor Core performs dozens to hundreds, because one instruction triggers a whole tile of multiplications whose operands are read from registers once and reused across the tile. Throughput rises further as precision falls: FP16 and BF16 inputs with FP32 accumulation, then FP8, and in newer generations 4-bit formats with per-block scaling. Each halving of operand width roughly doubles arithmetic per unit area and per byte moved. AMD’s analogous units are Matrix Cores. For AI practitioners, this is why numeric format choices (mixed precision, quantization) translate almost directly into training and inference speed.
8A Quantitative Model: The Roofline
Whether a kernel is limited by arithmetic or by memory can be estimated with the roofline model [10]. Define the kernel’s arithmetic intensity I as floating-point operations performed per byte moved to or from DRAM. With peak compute throughput Π (FLOP/s) and peak memory bandwidth β (B/s), attainable performance is bounded by
The ridge point I* = Π/β separates the two regimes. Modern accelerators have very high ridge points: a device offering ~1,000 TFLOP/s of dense low-precision Tensor Core throughput and ~3 TB/s of HBM bandwidth has I* ≈ 330 FLOP/byte. A vector addition (I ≈ 1/12 FLOP/byte for FP32) is therefore deeply memory-bound; large matrix multiplications, whose intensity grows linearly with matrix dimension, can be compute-bound. This explains, for instance, why the token-by-token decode phase of LLM inference (matrix-vector products, low intensity) is bandwidth-bound, while the batched prefill phase (matrix-matrix products) is compute-bound.
9A Worked Example
The following CUDA kernel computes y ← a·x + y (SAXPY). It illustrates the concepts above: each thread derives a unique global index from its block and thread coordinates; consecutive threads touch consecutive addresses (coalesced); and the bounds check is the only branch, diverging in at most one warp.
__global__ void saxpy(int n, float a, const float* x, float* y) { int i = blockIdx.x * blockDim.x + threadIdx.x; // global thread index if (i < n) // guard the tail block y[i] = a * x[i] + y[i]; // 2 FLOP, 12 bytes moved } // host: one thread per element, 256 threads (8 warps) per block int threads = 256; int blocks = (n + threads - 1) / threads; saxpy<<<blocks, threads>>>(n, 2.0f, d_x, d_y);
With 2 FLOP per 12 bytes, I ≈ 0.17 FLOP/byte, far left of the ridge point. Its runtime is therefore predicted by bytes moved divided by bandwidth, and no amount of extra arithmetic hardware would speed it up. The only meaningful optimization is to move fewer bytes, for instance by fusing it with neighbouring operations so that x and y are read from DRAM once. This is exactly the motivation behind kernel-fusion compilers and fused attention kernels in modern ML frameworks.
10Beyond a Single Device
Large workloads span many GPUs. Within a node, high-bandwidth point-to-point links (e.g. NVLink, AMD Infinity Fabric) and switches provide hundreds of GB/s to over a TB/s per GPU, an order of magnitude beyond PCIe, enabling tensor-parallel decomposition of a single model layer. Across nodes, RDMA networks (InfiniBand, RoCE) carry collective operations such as all-reduce, typically implemented by libraries like NCCL/RCCL. The same throughput reasoning applies at this scale: communication must be overlapped with computation, and the ratio of FLOPs to bytes exchanged determines how well a parallelization strategy scales.
11Conclusion
A GPU is best understood not as a faster CPU but as a different answer to the question of how to spend transistors. By amortizing control across 32-wide warps, keeping tens of thousands of thread contexts resident in an enormous register file, and switching among them at zero cost, it converts parallelism into latency tolerance and hence into throughput. Its performance is governed by a small number of measurable quantities: occupancy and instruction-level parallelism (to hide latency), coalescing and bank-conflict freedom (to use bandwidth efficiently), warp-uniform control flow (to use lanes efficiently) and arithmetic intensity relative to the ridge point (to know which resource is the bottleneck). Specialized matrix units and reduced-precision formats have pushed peak compute far ahead of memory bandwidth, so that for many modern workloads—including large-language-model inference—the decisive engineering problem is moving data, not computing on it.
References
- E. Lindholm, J. Nickolls, S. Oberman, J. Montrym. “NVIDIA Tesla: A Unified Graphics and Computing Architecture.” IEEE Micro, 28(2):39–55, 2008.
- J. Nickolls, I. Buck, M. Garland, K. Skadron. “Scalable Parallel Programming with CUDA.” ACM Queue, 6(2):40–53, 2008.
- W. W. L. Fung, I. Sham, G. Yuan, T. M. Aamodt. “Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow.” Proc. 40th IEEE/ACM MICRO, 2007.
- J. L. Hennessy, D. A. Patterson. Computer Architecture: A Quantitative Approach, 6th ed., ch. 4 (“Data-Level Parallelism in Vector, SIMD, and GPU Architectures”). Morgan Kaufmann, 2017.
- V. Volkov. “Better Performance at Lower Occupancy.” NVIDIA GPU Technology Conference, 2010.
- Z. Jia, M. Maggioni, B. Staiger, D. P. Scarpazza. “Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking.” arXiv:1804.06826, 2018.
- J. D. C. Little. “A Proof for the Queuing Formula: L = λW.” Operations Research, 9(3):383–387, 1961.
- NVIDIA Corporation. CUDA C++ Programming Guide, sections on the SIMT architecture and independent thread scheduling.
- D. B. Kirk, W. W. Hwu, I. El Hajj. Programming Massively Parallel Processors: A Hands-on Approach, 4th ed. Morgan Kaufmann, 2022.
- S. Williams, A. Waterman, D. Patterson. “Roofline: An Insightful Visual Performance Model for Multicore Architectures.” Communications of the ACM, 52(4):65–76, 2009.