[{"data":1,"prerenderedAt":35},["ShallowReactive",2],{"chapter:kernels\u002Forientation\u002Fwhat-a-kernel-is.json":3},{"project":4,"route":5,"title":6,"titleHtml":6,"navTitle":6,"part":7,"sourcePath":8,"editUrl":9,"html":10,"toc":11,"hasMermaid":28,"prev":29,"next":32},"kernels","\u002Fkernels\u002Forientation\u002Fwhat-a-kernel-is","What a Kernel Is","Orientation","orientation\u002Fwhat-a-kernel-is.md","https:\u002F\u002Fgithub.com\u002Fteenygrad\u002Fteenygrad\u002Fedit\u002Fmain\u002Fbooks\u002Fkernels\u002Fsrc\u002Forientation\u002Fwhat-a-kernel-is.md","\u003Cp>A kernel is a function that runs on a graphics card.\u003C\u002Fp>\n\u003Cp>That is the whole definition. What makes it worth a book is everything that\nfollows from \u003Cem>where\u003C\u002Fem> it runs.\u003C\u002Fp>\n\u003Ch2 id=\"why-a-gpu-is-shaped-the-way-it-is\">Why a GPU is shaped the way it is\u003C\u002Fh2>\n\u003Cp>Your CPU has a handful of cores. Each one is enormously clever: it predicts\nbranches, reorders instructions, and keeps several megabytes of cache close by\nso it rarely has to wait for memory. It is built to run one thread of control as\nfast as physically possible.\u003C\u002Fp>\n\u003Cp>A GPU gives up all of that. It has thousands of small, simple cores that run in\nlockstep, and comparatively little cache. Any single one of them is far slower\nthan a CPU core. There are just a great many of them.\u003C\u002Fp>\n\u003Cp>This is a good trade only when your work looks a particular way: the same\noperation, applied to a lot of data, with no need to know one result before\ncomputing the next. Adding two million-element vectors is the perfect shape.\nWalking a linked list is the worst possible shape.\u003C\u002Fp>\n\u003Cp>Machine learning is almost entirely the first shape, which is why it runs on\nGPUs.\u003C\u002Fp>\n\u003Ch2 id=\"what-kernel-means\">What “kernel” means\u003C\u002Fh2>\n\u003Cp>Suppose you want to add two vectors of a million elements. On a CPU you write a\nloop.\u003C\u002Fp>\n\u003Cp>On a GPU you do not write the loop. You write the \u003Cem>body\u003C\u002Fem> of the loop — the part\nthat handles one piece of the work — and then you ask the card to run a million\ncopies of it at once. That function is the kernel.\u003C\u002Fp>\n\u003Cp>The mental shift is that a kernel describes \u003Cstrong>one worker’s job\u003C\u002Fstrong>, not the whole\njob. Every copy runs the same instructions. The only thing that differs between\nthem is a number telling each copy which piece of the data is its own.\u003C\u002Fp>\n\u003Cp>Everything else in this book is a consequence of that one idea.\u003C\u002Fp>\n\u003Ch2 id=\"why-you-would-write-your-own\">Why you would write your own\u003C\u002Fh2>\n\u003Cp>Libraries already contain fast kernels for the common operations. Matrix\nmultiply, convolution, softmax — those are solved, and yours will probably be\nslower. So why write one?\u003C\u002Fp>\n\u003Cp>Because of memory.\u003C\u002Fp>\n\u003Cp>A GPU can do arithmetic far faster than it can fetch numbers to do arithmetic\non. On typical hardware the gap is more than an order of magnitude. That means\nfor a lot of real work, the arithmetic is free and the \u003Cem>loads and stores\u003C\u002Fem> are\nthe entire cost.\u003C\u002Fp>\n\u003Cp>Now look at a sequence you find everywhere in vision models — a convolution,\nthen a batch normalisation, then a SiLU activation:\u003C\u002Fp>\n\u003Cpre class=\"code-panel\" data-lang=\"text\">\u003Ccode>conv       read input, write result   ← memory traffic\nbatchnorm  read result, write result  ← memory traffic\nsilu       read result, write result  ← memory traffic\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>Three separate library kernels means three round trips to memory. But the\nbatchnorm and the SiLU are a few arithmetic operations each — nothing. If you\nwrite one kernel that reads the input, does all three steps while the numbers\nare already in registers, and writes once, you have removed two thirds of the\nmemory traffic and almost none of the work.\u003C\u002Fp>\n\u003Cp>That is called \u003Cstrong>fusion\u003C\u002Fstrong>, and it is the single most common reason to write a\nkernel. teenygrad ships exactly this kernel — \u003Ccode>conv2d_bn_silu\u003C\u002Fcode> — for exactly\nthis reason.\u003C\u002Fp>\n\u003Cp>The other reasons, in rough order of how often they come up:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>The operation does not exist.\u003C\u002Fstrong> A new attention variant, a custom loss, an\nop from a paper published last month.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>You know something the library cannot.\u003C\u002Fstrong> Your sequence length is always 512.\nYour weights are always sparse in a particular pattern. A general kernel\ncannot assume that; yours can.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>You need it on hardware the library ignores.\u003C\u002Fstrong> Small embedded GPUs are often\nan afterthought upstream.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2 id=\"the-two-costs-named\">The two costs, named\u003C\u002Fh2>\n\u003Cp>You will meet these terms constantly, so here they are once:\u003C\u002Fp>\n\u003Cp>A kernel is \u003Cstrong>memory-bound\u003C\u002Fstrong> when it spends most of its time waiting for data.\nAdding two vectors is memory-bound: three numbers moved for one addition. Making\nit faster means moving fewer numbers, or moving them in a better order.\u003C\u002Fp>\n\u003Cp>A kernel is \u003Cstrong>compute-bound\u003C\u002Fstrong> when it spends most of its time doing arithmetic.\nA large matrix multiply is compute-bound: every number loaded gets used many\ntimes over. Making it faster means using the hardware’s specialised arithmetic\nunits properly.\u003C\u002Fp>\n\u003Cp>The first question to ask about any kernel is which of the two it is, because\nthe answer decides what is worth optimising. Most kernels you write will be\nmemory-bound. Part 4 goes into this properly.\u003C\u002Fp>\n\u003Ch2 id=\"what-you-are-about-to-do\">What you are about to do\u003C\u002Fh2>\n\u003Cp>The next three chapters cover the programming model, what the toolchain does to\nyour code, and how to install it. Then you write a kernel and run it.\u003C\u002Fp>\n\u003Cp>If you would rather see the thing before reading about it, the whole of Chapter\n5 is one file, and on a machine with a GPU it runs now:\u003C\u002Fp>\n\u003Cpre data-lang=\"bash\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#7FB6D9\">cargo\u003C\u002Fspan>\u003Cspan style=\"color:#D8A76B\"> run\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\"> -p\u003C\u002Fspan>\u003Cspan style=\"color:#D8A76B\"> teeny-triton\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\"> --features\u003C\u002Fspan>\u003Cspan style=\"color:#D8A76B\"> cuda\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\"> --example\u003C\u002Fspan>\u003Cspan style=\"color:#D8A76B\"> vector_add\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n",[12,16,19,22,25],{"id":13,"text":14,"level":15},"why-a-gpu-is-shaped-the-way-it-is","Why a GPU is shaped the way it is",2,{"id":17,"text":18,"level":15},"what-kernel-means","What “kernel” means",{"id":20,"text":21,"level":15},"why-you-would-write-your-own","Why you would write your own",{"id":23,"text":24,"level":15},"the-two-costs-named","The two costs, named",{"id":26,"text":27,"level":15},"what-you-are-about-to-do","What you are about to do",false,{"title":30,"titleHtml":30,"route":31},"Introduction","\u002Fkernels",{"title":33,"titleHtml":33,"route":34},"You Program a Block, Not a Thread","\u002Fkernels\u002Forientation\u002Fblock-not-thread",1786271829619]