CUDA Is Fast Because the Compiler Knows the Chip

Every few years somebody announces that a new language has made CUDA obsolete, and the benchmark that follows shows the port running at half the speed of the original. The port was not badly written. CUDA is fast because the compiler is allowed to assume a specific machine — a specific register file, a specific warp width, a specific scratchpad — and every portable model has to give some of that away. What transfers is the reasoning about memory. What does not is almost everything you do with it.

The Programming Model Is the Product

A CUDA kernel is written against a hierarchy the hardware actually has. Threads are grouped into warps of 32, warps into blocks, blocks into a grid, and the block is the unit that gets scheduled onto a streaming multiprocessor. Inside that model sit three memory tiers with three different costs: registers, a software-managed scratchpad called shared memory that you address explicitly, and global memory behind a cache hierarchy you do not control. None of this is a language feature. It is a description of the machine, published because publishing it lets programmers write code the compiler can reason about.

That is the part that gets lost in the language arguments. The compiler knows how many registers exist per multiprocessor, how large shared memory is, and what the maximum resident thread count is, so it can make an occupancy decision and tell you the consequences. You can cap registers or set a launch bound and get a report back. This is an unusually tight feedback loop between programmer intent and machine reality, and it is why hand-tuned CUDA reads like code written by someone who could see the schedule.

What sits underneath is also worth naming. CUDA source compiles to PTX, a virtual instruction set, and the driver translates PTX to the real machine code for whatever chip is present at load time. That is a genuine forward-compatibility trick — an old binary runs on new silicon — but it also means the final scheduling decision happens after you stopped looking, on a chip you may not have targeted.

What Ports, and What Gets Re-Tuned

HIP is the closest thing to a drop-in: a C++ dialect with the same launch syntax and the same broad memory hierarchy, mapping onto AMD hardware that also has warps, blocks and a scratchpad — local data share instead of shared memory. Translating a kernel is mechanical enough that tools do most of it. Then you discover that AMD's wavefront has historically been 64 threads wide where NVIDIA's warp is 32, so a block size chosen to fill a scheduler cleanly on one machine halves or doubles the granularity on the other. A reduction written for 32 lanes now has an extra step, and a shuffle-based communication pattern has to be rewritten.

SYCL takes the standards route: single-source C++ over a runtime, with sub-groups standing in for warps. The standard is careful to make the sub-group size implementation-defined and queryable at runtime rather than fixed, which is the honest way to write a portable model and also the reason a kernel that queries it cannot be tuned at compile time. Vulkan compute goes the other way: pipelines, descriptors and barriers, with nothing hidden and nothing provided — no math library, no profiler worth the name. OpenCL is the cautionary case: the first genuinely portable compute standard, and the one whose vendors' compilers diverged enough that "write once, tune everywhere" became the joke.

So the honest inventory is this. What transfers is the algorithm, the decomposition, and the classification of the kernel as bandwidth-bound or compute-bound. What gets re-tuned by hand is tile size, vector width, register pressure and occupancy target, shared-memory layout against bank conflicts, the shuffle and broadcast vocabulary, and whichever asynchronous copy primitive the platform offers this year.

Bandwidth Is the Portable Truth

The model that survives every port is arithmetic intensity: floating-point operations per byte moved, plotted against the machine's achievable bandwidth and peak throughput. It says something true regardless of vendor — a kernel below the ridge point cannot be saved by a better instruction mix, because it is waiting on memory. That is the portability that matters, because it tells you before you write a line of code whether the GPU you are porting to can even help.

Where it stops helping is the second-order effects. Divergence costs more per branch when a wavefront is 64 threads than when it is 32, so a kernel with scattered branches degrades differently on the two machines. Register pressure cuts occupancy, but the granularity of the cut depends on how the register file is allocated. Cache and scratchpad sizes differ, so the tile that fit comfortably in one machine's on-chip memory spills on the other. These are the tuning parameters, and they are precisely the ones the portable abstractions cannot own for you.

Where the Portability Layers Actually Landed

The successful answer has not been a portable source language. It has been a compiler that knows the target and a tuning search that runs per target. Triton is the clearest example: you write tiles and the compiler does the scheduling, with autotuning configurations swept per platform. Kokkos and OpenMP's target offload work the same way at a coarser level — the abstraction carries the parallelism, and the backend carries the machine knowledge. You accept a ceiling in exchange for not rewriting the kernel.

Below all of it sits the thing that actually sets the performance floor: the vendor libraries. Nobody hand-writes a dense matrix multiply that beats the tuned one, so the practical question is not "does my kernel run on this GPU" but "does this GPU have an equivalent library, at the same maturity, for the shape I need." That is a supply-chain question dressed as a portability question.

What I Think

I think the "CUDA lock-in" argument is usually framed backwards. The lock-in is not the C++ dialect or the launch syntax — those are easy to translate, and people have. The lock-in is that a decade of tuning knowledge, profiling tools and library coverage has been accumulated against one machine model, and that accumulation is the moat. A competitor can match the instruction set and still be behind on the boring middle: the profiler that tells you why occupancy dropped, the library that handles your odd matrix shape.

The reasonable posture for anyone buying compute is to treat portable source as insurance rather than strategy. Write the kernel so the decomposition is separable, keep the tuning parameters in one place, and budget for a week of hand work per target — because a kernel that runs at forty percent of peak is not a portability win, it is a slower version of the same mistake.