Technical Exposition · Computer Architecture · November 2026

Throughput-Oriented Computing: An Architectural Account of the Modern GPU

Prepared with Claude (Anthropic)
A tutorial review for practitioners in software, cloud and AI systems
Abstract. Graphics Processing Units (GPUs) have evolved from fixed-function rasterization pipelines into general-purpose, massively parallel processors that underpin scientific computing and modern machine learning. This paper explains why GPUs are fast by contrasting their design philosophy—maximizing aggregate throughput—with the latency-minimizing philosophy of conventional CPUs. We describe the Single-Instruction, Multiple-Thread (SIMT) execution model, the organization of the streaming multiprocessor (SM), the use of massive hardware multithreading to hide memory latency (formalized via Little’s Law), the explicitly managed memory hierarchy, the cost of control-flow divergence, and the role of matrix-multiply accelerators (Tensor Cores). We conclude with the roofline model as a quantitative tool for reasoning about whether a workload is compute- or bandwidth-bound. Throughout, examples use NVIDIA CUDA terminology; the principles transfer directly to AMD (CDNA/RDNA) and other throughput architectures.
Keywords — GPU architecture, SIMT, latency hiding, occupancy, memory coalescing, Tensor Cores, roofline model

1Introduction

A processor can be optimized for one of two quantities: the latency of a single task, or the throughput of many tasks. A modern CPU core is a latency machine. It devotes most of its transistor budget to large caches, branch predictors, out-of-order scheduling windows and speculative execution—mechanisms whose sole purpose is to make one instruction stream finish sooner. A GPU makes the opposite bet. It assumes the workload consists of thousands to millions of independent, similar operations (pixels, matrix elements, particles, tokens) and spends its silicon on arithmetic units and on the on-chip state needed to keep a very large number of threads in flight [1, 4].

The consequence is that an individual GPU thread is slow: it runs at a modest clock, has no out-of-order execution and may wait hundreds of cycles for memory. Yet the device as a whole delivers one to two orders of magnitude more floating-point operations and memory bandwidth than a contemporary CPU socket. Understanding a GPU therefore means understanding how it converts abundant parallelism into sustained throughput. Figure 1 summarizes the difference in resource allocation.

CPU: latency-optimized GPU: throughput-optimized Control / OoO ALU ALU Large caches (L1/L2/L3) DRAM interface Wide HBM / GDDR interface Thousands of simple ALUs
Figure 1. Schematic allocation of die area. The CPU spends most of its area on control logic and caches that reduce the latency of a few threads; the GPU spends most of its area on arithmetic lanes, with thin control (red strip) and comparatively small caches (blue strip), relying on parallelism rather than caching to tolerate memory latency. Not to scale.

2The Execution Model: Kernels, Grids and SIMT

A GPU program is organized around a kernel: a function written from the perspective of a single scalar thread, which the host launches over an index space of many threads. The index space is hierarchical (Figure 2). Threads are grouped into thread blocks (CUDA) or work-groups (OpenCL/SYCL); blocks are arranged into a grid. Two properties of this hierarchy carry the design:

(i) Blocks are independent. The programming model guarantees nothing about the order or concurrency in which blocks execute. This lets the hardware schedule blocks onto however many multiprocessors a given chip has, so the same binary scales transparently from a laptop GPU with a handful of SMs to a data-center part with more than a hundred [2].

(ii) Threads within a block can cooperate. They are guaranteed to be co-resident on one multiprocessor, can exchange data through fast on-chip shared memory, and can synchronize with a barrier (__syncthreads()).

Grid Block(0,0)Block(1,0)Block(2,0) Block(0,1)Block(1,1)Block(2,1) Block(0,2)Block(1,2)Block(2,2) Thread block (e.g. 256 threads) → 8 warps of 32 Warp 0   threads 0–31 Warp 1   threads 32–63 Warp 2   threads 64–95 … Warp 7   threads 224–255 Shared memory + barrier (visible only within this block)
Figure 2. The thread hierarchy. A kernel launch creates a grid of independent blocks. Each block is placed on one SM, where its threads are partitioned into warps of 32 consecutive threads that share an instruction stream. AMD hardware uses the analogous wavefront of 32 or 64 work-items.

2.1SIMT: scalar programs, vector hardware

Although the programmer writes scalar code, the hardware does not execute threads individually. Consecutive threads of a block are bundled into a warp (32 threads on NVIDIA hardware), and a warp issues one instruction at a time, executed in lock-step across its 32 lanes. This is the Single-Instruction, Multiple-Thread (SIMT) model introduced with the Tesla architecture [1]. SIMT is functionally similar to SIMD vector processing, with two differences that make it far easier to program: each lane has its own registers and program-visible thread index, and the hardware—not the programmer—handles lanes that take different branches (Section 6). Amortizing instruction fetch, decode and scheduling over 32 data elements is the first reason a GPU can afford so many arithmetic units.

3Microarchitecture of the Streaming Multiprocessor

The GPU chip is an array of streaming multiprocessors (SMs; AMD: compute units) sharing an L2 cache and a set of memory controllers. Each SM is a self-contained throughput engine (Figure 3). In recent NVIDIA designs an SM is split into four processing blocks, each containing:

The four blocks share an L1 data cache that is configurable with the block-visible shared memory, plus texture units. The striking feature is the size of the register file: because every resident thread keeps its full architectural state on-chip permanently, switching between warps costs zero cycles. There is no save/restore, unlike an operating-system context switch on a CPU.

Streaming Multiprocessor (SM) L0/L1 instruction cache Warp scheduler + dispatch Register file (~64 KB) FP32 / INT32 FP64 Tensor Core LD/ST SFU Warp scheduler + dispatch Register file (~64 KB) FP32 / INT32 FP64 Tensor Core LD/ST SFU Warp scheduler + dispatch Register file (~64 KB) FP32 / INT32 FP64 Tensor Core LD/ST SFU Warp scheduler + dispatch Register file (~64 KB) FP32 / INT32 FP64 Tensor Core LD/ST SFU Unified L1 data cache / shared memory (configurable split, order of 100–250 KB) Texture units · interface to on-chip network → L2 → HBM
Figure 3. Representative organization of an SM in recent NVIDIA architectures (Volta onward). Four processing blocks each have their own scheduler, register-file slice and execution units; they share L1/shared memory. Sizes are indicative and differ by generation.

4Latency Hiding Through Multithreading

A load from off-chip DRAM takes on the order of several hundred core cycles [6]. A CPU attacks this latency by avoiding it (caches, prefetchers) and by overlapping it within one thread (out-of-order execution). A GPU instead tolerates it. Each SM keeps up to dozens of warps resident (e.g. 48–64, i.e. 1,536–2,048 threads). On every cycle, each scheduler picks some warp whose next instruction has its operands ready and issues it. When a warp stalls on memory, the scheduler simply issues from another. If enough warps are available, the arithmetic units never idle even though every individual warp spends most of its time waiting.

The required amount of parallelism follows from Little’s Law [7], which relates the mean number of items in a system L to the arrival rate λ and the mean residence time W:

L = λ · W(1)

Applied to a memory system, the concurrency that must be in flight equals bandwidth multiplied by latency. For a device with memory bandwidth B = 3 TB/s and latency W ≈ 600 ns, roughly 1.8 MB of outstanding requests are needed to saturate DRAM—tens of thousands of concurrent 32-byte sectors spread across the chip. No small set of threads could generate that; a GPU with on the order of 105 resident threads can. Volkov [5] showed that the same concurrency can also be obtained with fewer warps that each expose more instruction-level parallelism (several independent loads before the first use), which is why high “occupancy” is a means, not a goal.

Occupancy. Occupancy is the ratio of resident warps to the SM’s maximum. It is bounded by whichever per-SM resource runs out first: registers per thread, shared memory per block, or the hard limits on blocks and warps per SM. A kernel using 128 registers per thread on an SM with 65,536 registers can hold at most 512 threads (16 warps), regardless of other limits. This is why compiler register allocation and flags such as __launch_bounds__ matter for performance.

5The Memory Hierarchy

Unlike a CPU, whose hierarchy is almost entirely transparent, the GPU exposes several memory spaces with very different costs, and programmers are expected to place data deliberately. Table 1 lists representative figures.

Table 1. Representative GPU memory hierarchy (orders of magnitude; values vary by generation [6, 9]).
LevelScopeCapacityApprox. latencyManaged by
RegistersThread~256 KB / SM~1 cycleCompiler
Shared memory / L1Block / SM~100–250 KB / SM~20–35 cyclesProgrammer / HW
L2 cacheDeviceTens of MB~200–300 cyclesHardware
Global memory (HBM/GDDR)Device + host-visibleTens to hundreds of GB~400–800 cyclesProgrammer

5.1Coalescing

Global memory is accessed in aligned 32-byte sectors (grouped into 128-byte lines). When the 32 threads of a warp execute a load, the hardware merges their addresses and issues the minimum number of sector transactions. If thread k reads element a[base + k] of a 4-byte array, the warp touches exactly 128 contiguous bytes: four sectors, perfect efficiency. If instead thread k reads a[k * stride] with a large stride, each thread needs its own sector, and effective bandwidth drops by up to 8×. This is why GPU data layouts favour structure-of-arrays over array-of-structures.

5.2Shared memory and tiling

Shared memory is a software-managed scratchpad divided into 32 banks, each 4 bytes wide. Threads of a warp accessing 32 distinct banks are served in one cycle; multiple threads hitting different addresses in the same bank are serialized (a bank conflict). The canonical use is tiling: a block cooperatively loads a tile of the input from global memory once, synchronizes, and then every thread reuses that tile many times from shared memory. In dense matrix multiplication with T×T tiles, each element fetched from DRAM is reused T times, raising arithmetic intensity (Section 8) by the same factor.

6Control-Flow Divergence

Because a warp shares one instruction stream, a data-dependent branch on which lanes disagree cannot be executed in a single pass. The hardware executes each path in turn, disabling (masking) the lanes not on the current path, then reconverges. For an if/else in which half the lanes go each way, the warp spends the time of both paths and achieves at most 50% lane utilization. Since Volta, NVIDIA GPUs track a program counter per thread (independent thread scheduling), which permits fine-grained synchronization between diverged threads, but it does not remove the throughput cost of divergence [8]. Research on dynamic warp formation [3] and similar techniques explores regrouping lanes to recover efficiency. The practical lesson is to arrange work so that threads within a warp follow the same control path, for example by sorting or bucketing inputs.

7Tensor Cores and Reduced Precision

Deep-learning workloads are dominated by dense matrix multiplication. Beginning with Volta (2017), NVIDIA added Tensor Cores: units that compute a small matrix multiply-accumulate D = AB + C on a tile (for example 4×4×4 per unit per clock in the original design) as a single warp-wide operation. A scalar FP32 lane performs one fused multiply-add per clock; a Tensor Core performs dozens to hundreds, because one instruction triggers a whole tile of multiplications whose operands are read from registers once and reused across the tile. Throughput rises further as precision falls: FP16 and BF16 inputs with FP32 accumulation, then FP8, and in newer generations 4-bit formats with per-block scaling. Each halving of operand width roughly doubles arithmetic per unit area and per byte moved. AMD’s analogous units are Matrix Cores. For AI practitioners, this is why numeric format choices (mixed precision, quantization) translate almost directly into training and inference speed.

8A Quantitative Model: The Roofline

Whether a kernel is limited by arithmetic or by memory can be estimated with the roofline model [10]. Define the kernel’s arithmetic intensity I as floating-point operations performed per byte moved to or from DRAM. With peak compute throughput Π (FLOP/s) and peak memory bandwidth β (B/s), attainable performance is bounded by

P(I) = min( Π , β · I )(2)

The ridge point I* = Π/β separates the two regimes. Modern accelerators have very high ridge points: a device offering ~1,000 TFLOP/s of dense low-precision Tensor Core throughput and ~3 TB/s of HBM bandwidth has I* ≈ 330 FLOP/byte. A vector addition (I ≈ 1/12 FLOP/byte for FP32) is therefore deeply memory-bound; large matrix multiplications, whose intensity grows linearly with matrix dimension, can be compute-bound. This explains, for instance, why the token-by-token decode phase of LLM inference (matrix-vector products, low intensity) is bandwidth-bound, while the batched prefill phase (matrix-matrix products) is compute-bound.

Arithmetic intensity I (FLOP/byte, log scale) Attainable FLOP/s (log scale) 0.11101001000 ridge point I* = Π/β compute-bound: P = Π memory-bound: P = β·I vector add LLM decode (GEMV) stencil / convolution large GEMM / prefill
Figure 4. The roofline model. Kernels left of the ridge point are limited by memory bandwidth and benefit from data reuse (tiling, fusion, lower precision); kernels to the right are limited by peak arithmetic and benefit from Tensor Core use and instruction efficiency. Placement of example kernels is qualitative.

9A Worked Example

The following CUDA kernel computes y ← a·x + y (SAXPY). It illustrates the concepts above: each thread derives a unique global index from its block and thread coordinates; consecutive threads touch consecutive addresses (coalesced); and the bounds check is the only branch, diverging in at most one warp.

__global__ void saxpy(int n, float a, const float* x, float* y)
{
    int i = blockIdx.x * blockDim.x + threadIdx.x;   // global thread index
    if (i < n)                                     // guard the tail block
        y[i] = a * x[i] + y[i];                       // 2 FLOP, 12 bytes moved
}

// host: one thread per element, 256 threads (8 warps) per block
int threads = 256;
int blocks  = (n + threads - 1) / threads;
saxpy<<<blocks, threads>>>(n, 2.0f, d_x, d_y);

With 2 FLOP per 12 bytes, I ≈ 0.17 FLOP/byte, far left of the ridge point. Its runtime is therefore predicted by bytes moved divided by bandwidth, and no amount of extra arithmetic hardware would speed it up. The only meaningful optimization is to move fewer bytes, for instance by fusing it with neighbouring operations so that x and y are read from DRAM once. This is exactly the motivation behind kernel-fusion compilers and fused attention kernels in modern ML frameworks.

10Beyond a Single Device

Large workloads span many GPUs. Within a node, high-bandwidth point-to-point links (e.g. NVLink, AMD Infinity Fabric) and switches provide hundreds of GB/s to over a TB/s per GPU, an order of magnitude beyond PCIe, enabling tensor-parallel decomposition of a single model layer. Across nodes, RDMA networks (InfiniBand, RoCE) carry collective operations such as all-reduce, typically implemented by libraries like NCCL/RCCL. The same throughput reasoning applies at this scale: communication must be overlapped with computation, and the ratio of FLOPs to bytes exchanged determines how well a parallelization strategy scales.

11Conclusion

A GPU is best understood not as a faster CPU but as a different answer to the question of how to spend transistors. By amortizing control across 32-wide warps, keeping tens of thousands of thread contexts resident in an enormous register file, and switching among them at zero cost, it converts parallelism into latency tolerance and hence into throughput. Its performance is governed by a small number of measurable quantities: occupancy and instruction-level parallelism (to hide latency), coalescing and bank-conflict freedom (to use bandwidth efficiently), warp-uniform control flow (to use lanes efficiently) and arithmetic intensity relative to the ridge point (to know which resource is the bottleneck). Specialized matrix units and reduced-precision formats have pushed peak compute far ahead of memory bandwidth, so that for many modern workloads—including large-language-model inference—the decisive engineering problem is moving data, not computing on it.

References

  1. E. Lindholm, J. Nickolls, S. Oberman, J. Montrym. “NVIDIA Tesla: A Unified Graphics and Computing Architecture.” IEEE Micro, 28(2):39–55, 2008.
  2. J. Nickolls, I. Buck, M. Garland, K. Skadron. “Scalable Parallel Programming with CUDA.” ACM Queue, 6(2):40–53, 2008.
  3. W. W. L. Fung, I. Sham, G. Yuan, T. M. Aamodt. “Dynamic Warp Formation and Scheduling for Efficient GPU Control Flow.” Proc. 40th IEEE/ACM MICRO, 2007.
  4. J. L. Hennessy, D. A. Patterson. Computer Architecture: A Quantitative Approach, 6th ed., ch. 4 (“Data-Level Parallelism in Vector, SIMD, and GPU Architectures”). Morgan Kaufmann, 2017.
  5. V. Volkov. “Better Performance at Lower Occupancy.” NVIDIA GPU Technology Conference, 2010.
  6. Z. Jia, M. Maggioni, B. Staiger, D. P. Scarpazza. “Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking.” arXiv:1804.06826, 2018.
  7. J. D. C. Little. “A Proof for the Queuing Formula: L = λW.” Operations Research, 9(3):383–387, 1961.
  8. NVIDIA Corporation. CUDA C++ Programming Guide, sections on the SIMT architecture and independent thread scheduling.
  9. D. B. Kirk, W. W. Hwu, I. El Hajj. Programming Massively Parallel Processors: A Hands-on Approach, 4th ed. Morgan Kaufmann, 2022.
  10. S. Williams, A. Waterman, D. Patterson. “Roofline: An Insightful Visual Performance Model for Multicore Architectures.” Communications of the ACM, 52(4):65–76, 2009.