[{"data":1,"prerenderedAt":40},["ShallowReactive",2],{"chapter:kernels\u002Ffirst-kernel\u002Floads-stores-masks.json":3},{"project":4,"route":5,"title":6,"titleHtml":6,"navTitle":6,"part":7,"sourcePath":8,"editUrl":9,"html":10,"toc":11,"hasMermaid":32,"prev":33,"next":36},"kernels","\u002Fkernels\u002Ffirst-kernel\u002Floads-stores-masks","Loads, Stores, and Masks","Your First Kernel","first-kernel\u002Floads-stores-masks.md","https:\u002F\u002Fgithub.com\u002Fteenygrad\u002Fteenygrad\u002Fedit\u002Fmain\u002Fbooks\u002Fkernels\u002Fsrc\u002Ffirst-kernel\u002Floads-stores-masks.md","\u003Cp>A kernel that does not touch memory is not doing anything. This chapter is about\nthe two operations that do — and about the mask, which is the thing that stops\nthe last program in a launch from corrupting your program’s memory.\u003C\u002Fp>\n\u003Ch2 id=\"pointers-plus-offsets\">Pointers plus offsets\u003C\u002Fh2>\n\u003Cp>Memory in a kernel is addressed by pointer arithmetic, but on whole blocks at\nonce:\u003C\u002Fp>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">a_ptr\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">.\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">add_offsets\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\">offsets\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">)\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>\u003Ccode>a_ptr\u003C\u002Fcode> is a single pointer — the base of the array. \u003Ccode>offsets\u003C\u002Fcode> is a tensor of\n\u003Ccode>BLOCK_SIZE\u003C\u002Fcode> integers. \u003Ccode>add_offsets\u003C\u002Fcode> combines them into a \u003Cstrong>tensor of pointers\u003C\u002Fstrong>,\none per lane.\u003C\u002Fp>\n\u003Cp>That is the type to keep in your head. \u003Ccode>T::Pointer<D>\u003C\u002Fcode> is one address;\n\u003Ccode>T::Tensor<T::Pointer<D>>\u003C\u002Fcode> is a block of them, and that is what \u003Ccode>T::load\u003C\u002Fcode> and\n\u003Ccode>T::store\u003C\u002Fcode> take. The offsets are in \u003Cem>elements\u003C\u002Fem>, not bytes — the dtype is in the\ntype, so the scaling is done for you.\u003C\u002Fp>\n\u003Cp>\u003Ccode>add_offsets\u003C\u002Fcode> comes from the \u003Ccode>AddOffsets\u003C\u002Fcode> trait, which is why every kernel in\nthis book carries this line:\u003C\u002Fp>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#6FBF98\">T\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Pointer\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> AddOffsets\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">i32\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> 1\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> T\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">I32Tensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Output\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> =\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> T\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Tensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">T\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Pointer\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>>>,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>Read it as: “a pointer to \u003Ccode>D\u003C\u002Fcode>, offset by a rank-1 tensor of \u003Ccode>i32\u003C\u002Fcode>, gives a\ntensor of pointers to \u003Ccode>D\u003C\u002Fcode>”. It says exactly what the sentence above says, in\ntypes.\u003C\u002Fp>\n\u003Ch2 id=\"the-mask\">The mask\u003C\u002Fh2>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#7F877D;font-style:italic\">    \u002F\u002F The last program runs off the end of the vector. This says which of its\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#7F877D;font-style:italic\">    \u002F\u002F lanes are real.\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">    let\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> in_bounds \u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">=\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> offsets\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">.\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">lt\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\">n_elements\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">);\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>\u003Ccode>offsets.lt(n_elements)\u003C\u002Fcode> compares every lane against the length and produces a\n\u003Ccode>BoolTensor\u003C\u002Fcode> — one \u003Ccode>true\u003C\u002Fcode> or \u003Ccode>false\u003C\u002Fcode> per lane. For every program but the last,\nall 128 are \u003Ccode>true\u003C\u002Fcode>. For the last, some are \u003Ccode>false\u003C\u002Fcode>.\u003C\u002Fp>\n\u003Cp>Here is why that matters. The grid is \u003Ccode>1000 \u002F 128\u003C\u002Fcode> rounded up, which is 8. The\neighth program computes offsets 896 through 1023. But the array has 1000\nelements, so lanes for offsets 1000 through 1023 point past its end.\u003C\u002Fp>\n\u003Cp>Without a mask, that kernel reads 24 values that are not yours and writes 24\nvalues over memory that is not yours. On a GPU that is not a segfault. It is\nusually silence, sometimes a wrong number somewhere else in your program, and\noccasionally a crash much later in something unrelated.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>The mask is not an optimisation. It is the bounds check.\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>When a lane’s mask is \u003Ccode>false\u003C\u002Fcode>, \u003Ccode>T::load\u003C\u002Fcode> does not read it and \u003Ccode>T::store\u003C\u002Fcode> does not\nwrite it. Nothing else about the kernel changes.\u003C\u002Fp>\n\u003Ch3 id=\"choosing-the-fill-value\">Choosing the fill value\u003C\u002Fh3>\n\u003Cp>A masked-off lane still holds \u003Cem>something\u003C\u002Fem> after a load. By default that value is\nundefined, which is fine when you are about to mask the store as well — the\nvector-add kernel never uses those lanes, so it does not care.\u003C\u002Fp>\n\u003Cp>It is not fine when the lane feeds a reduction. A sum over a block where some\nlanes are garbage gives a garbage sum. Pass \u003Ccode>other\u003C\u002Fcode> and the masked lanes get a\nknown value instead:\u003C\u002Fp>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">let\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> zeros \u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">=\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> T\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">zeros\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::<\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>(&[\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\">BLOCK_A\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">]);\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">let\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> anchor_x \u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">=\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> T\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">load\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\">anchor_x_ptr\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">.\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">add_offsets\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\">a_offs\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">),\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Some\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\">mask\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">),\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Some\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\">zeros\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">),\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> &[],\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> None\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> None\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> None\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\"> false\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">);\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>Zero is the right identity for a sum. For a maximum you want negative infinity,\nvia \u003Ccode>T::full\u003C\u002Fcode>. Getting this wrong is a classic reduction bug and Chapter 10 hits\nit directly.\u003C\u002Fp>\n\u003Ch2 id=\"the-full-signatures\">The full signatures\u003C\u002Fh2>\n\u003Cp>\u003Ccode>T::load\u003C\u002Fcode> takes eight arguments and \u003Ccode>T::store\u003C\u002Fcode> takes six. Two of each matter\nnow; the rest have a sensible “no thanks” value that you will write a great many\ntimes.\u003C\u002Fp>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">fn\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> load\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Dtype\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> const\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> N\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> usize\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>(\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    ptr\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> Self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Tensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">Self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Pointer\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>>,\u003C\u002Fspan>\u003Cspan style=\"color:#7F877D;font-style:italic\">   \u002F\u002F where to read\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    mask\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Option\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">Self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">BoolTensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003Cspan style=\"color:#7F877D;font-style:italic\">        \u002F\u002F which lanes are real\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    other\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Option\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">Self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Tensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>>,\u003C\u002Fspan>\u003Cspan style=\"color:#7F877D;font-style:italic\">        \u002F\u002F what masked lanes get\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    boundary_check\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> &[\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">i32\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">;\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> N\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">],\u003C\u002Fspan>\u003Cspan style=\"color:#7F877D;font-style:italic\">             \u002F\u002F block-pointer mode only\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    padding_option\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Option\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">PaddingOption\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003Cspan style=\"color:#7F877D;font-style:italic\"> \u002F\u002F block-pointer mode only\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    cache_modifier\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Option\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">CacheModifier\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003Cspan style=\"color:#7F877D;font-style:italic\"> \u002F\u002F L1\u002FL2 behaviour\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    eviction_policy\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Option\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">EvictionPolicy\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    volatile\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> bool\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#8A9088\">)\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ->\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> Self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Tensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>;\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">fn\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> store\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Dtype\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> const\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> N\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> usize\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>(\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    dest\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> Self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Tensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">Self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Pointer\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>>,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    src\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> Self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Tensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    mask\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Option\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">Self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">BoolTensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    boundary_check\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> &[\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">i32\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">;\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> N\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">],\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    cache_modifier\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Option\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">CacheModifier\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    eviction_policy\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Option\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">EvictionPolicy\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#8A9088\">);\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>The last four on each are performance hints and an alternative addressing mode.\n\u003Ccode>boundary_check\u003C\u002Fcode> and \u003Ccode>padding_option\u003C\u002Fcode> do nothing unless you built the pointer\nwith \u003Ccode>T::make_block_ptr\u003C\u002Fcode>, which Chapter 17 covers. \u003Ccode>cache_modifier\u003C\u002Fcode> and\n\u003Ccode>eviction_policy\u003C\u002Fcode> tell the hardware how to treat the data in cache; leave them\n\u003Ccode>None\u003C\u002Fcode> until you are measuring.\u003C\u002Fp>\n\u003Cp>In Python Triton the same call is \u003Ccode>tl.load(a_ptr + offsets, mask=in_bounds)\u003C\u002Fcode>,\nbecause Python has keyword arguments with defaults and Rust does not. There is\nno way around it today, and it is the single largest source of noise in these\nkernels. It is recorded in\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fteenygrad\u002Fteenygrad\u002Fblob\u002Fmain\u002Fbooks\u002Fkernels\u002FAPI-FRICTION.md\" target=\"_blank\" rel=\"noopener noreferrer\">\u003Ccode>API-FRICTION.md\u003C\u002Fcode>\u003C\u002Fa>\nas the first item.\u003C\u002Fp>\n\u003Ch2 id=\"comparisons\">Comparisons\u003C\u002Fh2>\n\u003Cp>\u003Ccode>lt\u003C\u002Fcode> is one of six, and each has a scalar form:\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Method\u003C\u002Fth>\n\u003Cth>Meaning\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Ccode>lt\u003C\u002Fcode>, \u003Ccode>le\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>less than, less than or equal\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>gt\u003C\u002Fcode>, \u003Ccode>ge\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>greater than, greater than or equal\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>eq\u003C\u002Fcode>, \u003Ccode>ne\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>equal, not equal\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003Cp>Written \u003Ccode>offsets.lt(n_elements)\u003C\u002Fcode> as a method — that spelling comes from the\n\u003Ccode>Comparison\u003C\u002Fcode> trait, the second of the three \u003Ccode>where\u003C\u002Fcode> clauses. There is also\n\u003Ccode>T::lt(x, y)\u003C\u002Fcode> for two tensors and \u003Ccode>T::lt_scalar(x, y)\u003C\u002Fcode> for a tensor against a\nscalar. All three exist; the method form is what the kernels in this tree use.\u003C\u002Fp>\n\u003Cp>Masks combine with \u003Ccode>&\u003C\u002Fcode> and \u003Ccode>|\u003C\u002Fcode>, so a two-dimensional bounds check is one\nexpression:\u003C\u002Fp>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">let\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> in_bounds \u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">=\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> row_offsets\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">.\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">lt\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\">n_rows\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">)\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> &\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> col_offsets\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">.\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">lt\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\">n_cols\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">);\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>And \u003Ccode>T::where_(cond, x, y)\u003C\u002Fcode> selects between two tensors lane by lane — Triton’s\n\u003Ccode>tl.where\u003C\u002Fcode>, spelled with a trailing underscore because \u003Ccode>where\u003C\u002Fcode> is a Rust keyword.\u003C\u002Fp>\n\u003Ch2 id=\"the-rule\">The rule\u003C\u002Fh2>\n\u003Cp>Every kernel that indexes memory with a computed offset needs a mask, unless you\ncan prove the size divides the block exactly.\u003C\u002Fp>\n\u003Cp>Sometimes you can. The softmax kernel in Chapter 10 requires the caller to round\nthe row length up to a power of two and pass it as \u003Ccode>BLOCK_SIZE\u003C\u002Fcode>, so\n\u003Ccode>BLOCK_SIZE == n_cols\u003C\u002Fcode> holds and no mask is needed. That is a real constraint\npushed onto the caller in exchange for a simpler, faster kernel — a trade worth\nrecognising, and worth documenting loudly when you make it.\u003C\u002Fp>\n\u003Cp>Next: what the macro built out of all this.\u003C\u002Fp>\n",[12,16,19,23,26,29],{"id":13,"text":14,"level":15},"pointers-plus-offsets","Pointers plus offsets",2,{"id":17,"text":18,"level":15},"the-mask","The mask",{"id":20,"text":21,"level":22},"choosing-the-fill-value","Choosing the fill value",3,{"id":24,"text":25,"level":15},"the-full-signatures","The full signatures",{"id":27,"text":28,"level":15},"comparisons","Comparisons",{"id":30,"text":31,"level":15},"the-rule","The rule",false,{"title":34,"titleHtml":34,"route":35},"The Kernel Body","\u002Fkernels\u002Ffirst-kernel\u002Fkernel-body",{"title":37,"titleHtml":38,"route":39},"What #[kernel] Generates","What \u003Ccode>#[kernel]\u003C\u002Fcode> Generates","\u002Fkernels\u002Ffirst-kernel\u002Fkernel-macro",1786271829676]