Programming Massively Parallel Processors: A Hands-on Approach

This course provides a comprehensive introduction to GPU computing and parallel programming using the CUDA C environment. It covers GPU architectures, data parallelism, thread management, memory optimization, and advanced performance considerations, illustrated through real-world case studies like MRI reconstruction and molecular visualization.

Master the art of high-performance parallel computing with a practical, hands-on guide to CUDA and GPU architectures.

Author: David B. Kirk, Wen-mei W. Hwu

Acknowledgments: Ian Buck, John Nickolls, NVIDIA DevTech team, Jensen Huang, David Luebke, Bill Bean, Simon Green, Mark Harris, Manju Hedge, Nadeem Mohammad, Brent Oster, Peter Shirley, Eric Young, and Cyril Zeller.

1. Distinguish between the design philosophies and performance trajectories of multicore CPUs and many-core GPUs.
2. Identify the key components of a modern GPU architecture, including Streaming Multiprocessors (SMs) and memory structures.
3. Apply Amdahl's Law to calculate theoretical speedup and identify the impact of sequential bottlenecks.
4. Contrast the architectural differences between fixed-function pipelines and programmable unified processor arrays.
5. Explain the role of "GPGPU" as an intermediate step and the restrictions of early shader programming models.
6. Analyze how hardware features like atomic operations, barrier synchronization, and double-precision support enabled the transition to scalable general-purpose computing.
7. Identify and exploit data parallelism within matrix-matrix multiplication algorithms.
8. Implement device memory management including allocation, data transfer between host and device, and deallocation.
9. Construct and launch CUDA kernels using appropriate thread indexing and grid/block configurations.
10. Design multidimensional thread hierarchies (Grids and Blocks) to map complex data structures to GPU hardware.