Programming Massively Parallel Processors: A Hands-on Approach

I've been using this book to teach my sessions in the cuda mode discord community, it's been by far the best reference I've found on the market to learn CUDA.In particular chapters 1-6 will give you the core foundation to be able to start working on your own CUDA kernels and if you supplement those chapters with learning how to integrate your kernels in pytorch using features like load_inline but also the ncu profiler you'll be well on your way to writing real-world kernels that are performant. There' a long glossary of confusing concepts like grids, blocks, threads, warps which you won't remember if you're browsing the occasional medium blogpost or Wikipedia article, even ChatGPT often makes subtle mistakes. Learning CUDA or at least the basics of it is very much a open the textbook and do the problems sort of exercise.Starting chapter 7 the book goes into various case studies of popular algorithms and how to optimize them , the lessons are generically helpful even if you're not interested in those subproblems but my point is more that the book becomes significantly easier to understand after the initial struggle from chapter 1 through 6. This is also a natural point for you to experiment with your own CUDA kernels in maybe a workload you're trying to accelerate at work, whenever you get stuck you can browse the book for inspiration on common CUDA patterns that accelerate performance.Before this book I'd been stuck in tutorial hell with cuda for many years but this book gave me the right foundation to start using kernels at my day job and it's been a fantastic level up. Keep in mind that with tools like ChatGPT or code generators like torch.compile you can focus on only learning CUDA as opposed to also having to learn about makefiles and c++Granted the main gap the book seems to have is that it doesn't really cover CUDA C++ so reading codebases like CUTLASS will still be a struggle but more importantly the book doesn't cover how to program with tensor cores or have a treatment of lower precision dtypes and with modern ML workloads. CUDA streams are also briefly covered but spending a bit more on NCCL would be really nice to see in future editions.

Very clear and thoughtful, covers not only the programming abstractions needed to use CUDA to develop applications, but uses that context to explain the hardware differences and challenges. One of the best programming books I've ever read.

A great overview along with deep-dives. College & Masters level content.

Just finished reading the first chapter and I am already impressed.

Covers CUDA programming and then has several chapters discussing massively parallel algorithms.

This book is okay and serves as a pretty good overview of a bunch of techniques for parallel programming. But it's pretty dry and the explanations aren't great. I have worked in this area (CUDA, perf optimization, MPI) for about 15 years, so I just skipped over the long run-on paragraphs describing how things work. But if you were completely new to the topic, I feel like a lot of their explanations would not be very helpful! I would personally just read the code fragments and try to understand them without reading any of the associated prose before trying to think things through myself.There are a few things that would make this book way better.* There are tons of typos in the prose & code, which is just a matter of editing.* There's also bad typography (e.g. inconsistent code formatting), etc. that should be an easy editorial fix.* Instead of long run-on explanations in excruciating detail of code, it would be nice to have some more quantitative analysis of performance on different architectures. The book gives a lot of versions of something (e.g. lots of ways of writing a convolution), but it would be much more immediately useful as a reference if there were some charts to look at. Of course, the reader can micro-benchmark things themselves (which is a useful exercise), but it would be nice if the book used more empirical numbers instead of favoring theoretical perf numbers (like ops/byte.)* The "exercises" are usually pretty trivial calculations of theoretical perf numbers. I feel like this is an exciting topic so there should be more exciting problems!

It's good

One egregious typo is in section 7.1 with the formula for convolution. The displayed formula on page 152 y_i as a multiple of x_i . The example on p. 153 has undefined terms and is (to put it mildly) puzzling.