[{"data":1,"prerenderedAt":60},["ShallowReactive",2],{"chapter:kernels\u002Ffast\u002Fnumerics.json":3},{"project":4,"route":5,"title":6,"titleHtml":6,"navTitle":6,"part":7,"sourcePath":8,"editUrl":9,"html":10,"toc":11,"hasMermaid":53,"prev":54,"next":57},"kernels","\u002Fkernels\u002Ffast\u002Fnumerics","Numerics","Making It Fast","fast\u002Fnumerics.md","https:\u002F\u002Fgithub.com\u002Fteenygrad\u002Fteenygrad\u002Fedit\u002Fmain\u002Fbooks\u002Fkernels\u002Fsrc\u002Ffast\u002Fnumerics.md","\u003Cp>A kernel that is fast and slightly wrong is worse than one that is slow and\nright, because the wrongness is invisible until it is expensive.\u003C\u002Fp>\n\u003Cp>This chapter is about choosing dtypes and accumulators deliberately.\u003C\u002Fp>\n\u003Ch2 id=\"the-type-hierarchy\">The type hierarchy\u003C\u002Fh2>\n\u003Cp>The dtype bounds you put on a kernel are a real constraint, checked at compile\ntime:\u003C\u002Fp>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">pub\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> trait\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Dtype\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Copy\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> +\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Clone\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {}\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">pub\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> trait\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Num\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Dtype\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> const\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\"> BITS\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> u8\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">;\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> }\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">pub\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> trait\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Float\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Num\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> const\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\"> ZERO\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> Self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">;\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> const\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\"> ONE\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> Self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">;\u003C\u002Fspan>\u003Cspan style=\"color:#7F877D;font-style:italic\"> \u002F* from_f64 *\u002F\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> }\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">pub\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> trait\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Int\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Num\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {}\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">pub\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> trait\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Bool\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Dtype\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> +\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Copy\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {}\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>Pick the tightest one that admits your operations:\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Bound\u003C\u002Fth>\n\u003Cth>Admits\u003C\u002Fth>\n\u003Cth>Use when\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Ccode>Float\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>\u003Ccode>f32\u003C\u002Fcode>, \u003Ccode>f64\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>You call \u003Ccode>exp\u003C\u002Fcode>, \u003Ccode>log\u003C\u002Fcode>, \u003Ccode>sqrt\u003C\u002Fcode>, \u003Ccode>sigmoid\u003C\u002Fcode>\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>Int\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>\u003Ccode>i8\u003C\u002Fcode>…\u003Ccode>i64\u003C\u002Fcode>, \u003Ccode>u8\u003C\u002Fcode>…\u003Ccode>u64\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Bitwise operations, integer atomics\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>Num\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>both\u003C\u002Ftd>\n\u003Ctd>Arithmetic only\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>Dtype\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>everything\u003C\u002Ftd>\n\u003Ctd>Pure data movement\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003Cp>\u003Ccode>Float\u003C\u002Fcode> also gives you \u003Ccode>D::from_f64(...)\u003C\u002Fcode>, which is how you get a constant of\nthe right type into a kernel — \u003Ccode>T::full(&[BLOCK], D::from_f64(0.5))\u003C\u002Fcode>. There is\nno way to write a float literal of a generic float type without it.\u003C\u002Fp>\n\u003Cp>Choosing the tightest bound is not pedantry. \u003Ccode>D: Num\u003C\u002Fcode> on a kernel that calls\n\u003Ccode>T::exp\u003C\u002Fcode> is a compile error, and that error is the one Python Triton would have\ngiven you on a GPU, at run time, in production.\u003C\u002Fp>\n\u003Ch2 id=\"what-is-actually-available\">What is actually available\u003C\u002Fh2>\n\u003Cp>\u003Ccode>DtypeRepr\u003C\u002Fcode> — the runtime, type-erased tag — has thirteen variants including\n\u003Ccode>F16\u003C\u002Fcode> and \u003Ccode>BF16\u003C\u002Fcode>. The macro’s dtype-set inference does not:\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Bound\u003C\u002Fth>\n\u003Cth>Dtypes you get\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Ccode>Float\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>\u003Ccode>f32\u003C\u002Fcode>, \u003Ccode>f64\u003C\u002Fcode>\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>Int\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>\u003Ccode>i8\u003C\u002Fcode>, \u003Ccode>i16\u003C\u002Fcode>, \u003Ccode>i32\u003C\u002Fcode>, \u003Ccode>i64\u003C\u002Fcode>, \u003Ccode>u8\u003C\u002Fcode>, \u003Ccode>u16\u003C\u002Fcode>, \u003Ccode>u32\u003C\u002Fcode>, \u003Ccode>u64\u003C\u002Fcode>\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>Num\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>all of the above\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>Bool\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>\u003Ccode>bool\u003C\u002Fcode>\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>Dtype\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>all of the above\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003Cp>\u003Ccode>f16\u003C\u002Fcode> and \u003Ccode>bf16\u003C\u002Fcode> are absent, deliberately: the macro’s own comment says they are\nmarker-only and cannot be monomorphized, because there is no concrete Rust\nimplementation to instantiate a kernel against.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>This is the largest gap in the book.\u003C\u002Fstrong> Half precision is a substantial part of\nwhy people write GPU kernels — it halves memory traffic and doubles Tensor Core\nthroughput — and today you cannot write a \u003Ccode>#[kernel]\u003C\u002Fcode> that dispatches to it. It\nis item 4 in\n\u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fteenygrad\u002Fteenygrad\u002Fblob\u002Fmain\u002Fbooks\u002Fkernels\u002FKNOWN-GAPS.md\" target=\"_blank\" rel=\"noopener noreferrer\">\u003Ccode>KNOWN-GAPS.md\u003C\u002Fcode>\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cp>What follows is therefore mostly about \u003Ccode>f32\u003C\u002Fcode>, which is what you can use.\u003C\u002Fp>\n\u003Ch2 id=\"the-formats-for-when-this-is-fixed\">The formats, for when this is fixed\u003C\u002Fh2>\n\u003Cp>Worth knowing, because the trade-offs are what motivate everything below.\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Format\u003C\u002Fth>\n\u003Cth>Bits\u003C\u002Fth>\n\u003Cth>Exponent\u003C\u002Fth>\n\u003Cth>Mantissa\u003C\u002Fth>\n\u003Cth>Character\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Ccode>f64\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>64\u003C\u002Ftd>\n\u003Ctd>11\u003C\u002Ftd>\n\u003Ctd>52\u003C\u002Ftd>\n\u003Ctd>Rarely worth it on a GPU\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>f32\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>32\u003C\u002Ftd>\n\u003Ctd>8\u003C\u002Ftd>\n\u003Ctd>23\u003C\u002Ftd>\n\u003Ctd>The default\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>TF32\u003C\u002Ftd>\n\u003Ctd>19 stored in 32\u003C\u002Ftd>\n\u003Ctd>8\u003C\u002Ftd>\n\u003Ctd>10\u003C\u002Ftd>\n\u003Ctd>\u003Ccode>f32\u003C\u002Fcode>’s range, less precision, Tensor Cores\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>bf16\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>16\u003C\u002Ftd>\n\u003Ctd>8\u003C\u002Ftd>\n\u003Ctd>7\u003C\u002Ftd>\n\u003Ctd>\u003Ccode>f32\u003C\u002Fcode>’s range, much less precision\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>f16\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>16\u003C\u002Ftd>\n\u003Ctd>5\u003C\u002Ftd>\n\u003Ctd>10\u003C\u002Ftd>\n\u003Ctd>More precision than \u003Ccode>bf16\u003C\u002Fcode>, far less range\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003Cp>The exponent column is the important one. \u003Ccode>bf16\u003C\u002Fcode> has the same range as \u003Ccode>f32\u003C\u002Fcode>, so\na value that fits in \u003Ccode>f32\u003C\u002Fcode> fits in \u003Ccode>bf16\u003C\u002Fcode> — you lose precision, not magnitude.\n\u003Ccode>f16\u003C\u002Fcode> has five exponent bits and overflows above about 65,504, which is why\ntraining in \u003Ccode>f16\u003C\u002Fcode> needs loss scaling and training in \u003Ccode>bf16\u003C\u002Fcode> mostly does not.\u003C\u002Fp>\n\u003Ch2 id=\"what-f64-actually-costs\">What \u003Ccode>f64\u003C\u002Fcode> actually costs\u003C\u002Fh2>\n\u003Cp>\u003Ccode>examples\u002Fnumerics.rs\u003C\u002Fcode> runs an elementwise add over 16M elements, once as \u003Ccode>f32\u003C\u002Fcode>\nand once as \u003Ccode>f64\u003C\u002Fcode>:\u003C\u002Fp>\n\u003Cpre data-lang=\"bash\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#7FB6D9\">cargo\u003C\u002Fspan>\u003Cspan style=\"color:#D8A76B\"> run\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\"> --release\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\"> -p\u003C\u002Fspan>\u003Cspan style=\"color:#D8A76B\"> teeny-triton\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\"> --features\u003C\u002Fspan>\u003Cspan style=\"color:#D8A76B\"> cuda\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\"> --example\u003C\u002Fspan>\u003Cspan style=\"color:#D8A76B\"> numerics\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>On an \u003Cstrong>RTX 5070 (sm_120), CUDA 13.3, driver 610.43.02\u003C\u002Fstrong>:\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>dtype\u003C\u002Fth>\n\u003Cth style=\"text-align:right\">time\u003C\u002Fth>\n\u003Cth style=\"text-align:right\">bandwidth\u003C\u002Fth>\n\u003Cth style=\"text-align:right\">moved\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Ccode>f32\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">339.5 µs\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">593.1 GB\u002Fs\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">201 MB\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>f64\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">676.7 µs\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">595.0 GB\u002Fs\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">403 MB\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003Cp>\u003Cstrong>\u003Ccode>f64\u003C\u002Fcode> costs 1.99× the time — and the same bandwidth.\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>That equality is the whole story. This kernel is memory-bound, so \u003Ccode>f64\u003C\u002Fcode> costs\nexactly what its extra bytes cost and not one thing more. Both runs saturate the\ncard at ~594 GB\u002Fs, the same ceiling Chapters 16 and 17 hit with different\nkernels.\u003C\u002Fp>\n\u003Cp>Which means the folk wisdom “\u003Ccode>f64\u003C\u002Fcode> is catastrophically slow on a consumer GPU”\nis, for this kernel, wrong. It is catastrophically slow when \u003Cem>arithmetic\u003C\u002Fem> is the\nbottleneck — consumer cards run \u003Ccode>f64\u003C\u002Fcode> arithmetic at a small fraction of their\n\u003Ccode>f32\u003C\u002Fcode> rate — and this kernel barely does any. Move a byte, add once, move a\nbyte.\u003C\u002Fp>\n\u003Cp>So the honest rule: \u003Cstrong>on a memory-bound kernel \u003Ccode>f64\u003C\u002Fcode> costs 2×; on a\ncompute-bound one it costs far more.\u003C\u002Fstrong> Chapter 1’s question — which kind is\nthis? — decides which number applies.\u003C\u002Fp>\n\u003Ch2 id=\"accumulators\">Accumulators\u003C\u002Fh2>\n\u003Cp>Here is the rule that matters most:\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cp>\u003Cstrong>Accumulate in \u003Ccode>f32\u003C\u002Fcode>, whatever you multiply in.\u003C\u002Fstrong>\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cp>\u003Ccode>T::dot\u003C\u002Fcode> has two type parameters precisely for this:\u003C\u002Fp>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">fn\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> dot\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Num\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> O\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Num\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>(\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\">a\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Tensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> b\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Tensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> acc\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Option\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Tensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">O\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>>,\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ...)\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ->\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Tensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">O\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>;\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>\u003Ccode>D\u003C\u002Fcode> is the inputs, \u003Ccode>O\u003C\u002Fcode> is the accumulator. Mixing them is not a compromise; it is\nthe correct arrangement, and the Tensor Cores are built for it.\u003C\u002Fp>\n\u003Cp>The reason is that error accumulates with the number of additions. Multiplying\ntwo \u003Ccode>f16\u003C\u002Fcode> values gives a result \u003Ccode>f16\u003C\u002Fcode> can represent fine. Adding a thousand of\nthem in \u003Ccode>f16\u003C\u002Fcode> does not — each addition rounds, and after \u003Ccode>K\u003C\u002Fcode> additions the error\nhas grown roughly with \u003Ccode>sqrt(K)\u003C\u002Fcode> at best. In \u003Ccode>f32\u003C\u002Fcode>, with sixteen more mantissa\nbits, the same sequence stays accurate.\u003C\u002Fp>\n\u003Cp>The same applies to hand-written reductions. Summing a long \u003Ccode>f16\u003C\u002Fcode> row into an\n\u003Ccode>f16\u003C\u002Fcode> total loses precision that summing into \u003Ccode>f32\u003C\u002Fcode> does not.\u003C\u002Fp>\n\u003Ch3 id=\"measured\">Measured\u003C\u002Fh3>\n\u003Cp>The same example sums 16M \u003Ccode>f32\u003C\u002Fcode> values four ways. The reference is the identical\ndata summed in \u003Ccode>f64\u003C\u002Fcode>, so the only thing varying is how the sum is accumulated:\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>how\u003C\u002Fth>\n\u003Cth style=\"text-align:right\">result\u003C\u002Fth>\n\u003Cth style=\"text-align:right\">relative error\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>exact (\u003Ccode>f64\u003C\u002Fcode> reference)\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">2181038.0593\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">—\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>GPU block reduction, \u003Ccode>f64\u003C\u002Fcode> accumulator\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">2181037.9700\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">\u003Cstrong>4.09e-8\u003C\u002Fstrong>\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>GPU block reduction, \u003Ccode>f32\u003C\u002Fcode> accumulator\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">2181055.7500\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">8.11e-6\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>sequential \u003Ccode>f32\u003C\u002Fcode> loop on the CPU\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">2158069.0000\u003C\u002Ftd>\n\u003Ctd style=\"text-align:right\">\u003Cstrong>1.05e-2\u003C\u002Fstrong>\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003Cp>Read the last row first. \u003Cstrong>A plain \u003Ccode>f32\u003C\u002Fcode> loop is wrong in the fifth\nsignificant figure\u003C\u002Fstrong> — a 1% error, from nothing but adding numbers up in order.\nOnce the running total reaches ~2 million, adding 0.1 to it barely moves it, and\n16 million such additions lose most of what they should have contributed.\u003C\u002Fp>\n\u003Cp>The GPU’s block reduction is \u003Cstrong>over a thousand times more accurate\u003C\u002Fstrong> than that\nloop, and it is not because the GPU is careful. It is because a tree adds\nnumbers of \u003Cem>similar magnitude\u003C\u002Fem> to each other: 16M values pair down through 24\nlevels, and no partial sum ever dwarfs what is being added to it. The parallel\nalgorithm is more accurate than the obvious sequential one, which is the\nopposite of what most people expect.\u003C\u002Fp>\n\u003Cp>Widening only the final accumulator — the host-side sum of the per-block\npartials — from \u003Ccode>f32\u003C\u002Fcode> to \u003Ccode>f64\u003C\u002Fcode> gains another \u003Cstrong>200×\u003C\u002Fstrong>. That is one cast, on\n16384 values, and it is free next to the kernel.\u003C\u002Fp>\n\u003Cp>So the practical shape of an accurate reduction is: reduce in blocks on the\ndevice, accumulate the partials in something wider. You get the tree for free\nand the wide accumulator for almost nothing.\u003C\u002Fp>\n\u003Ch3 id=\"and-it-is-not-reproducible\">And it is not reproducible\u003C\u002Fh3>\n\u003Cp>The same data through the same kernel at two different block sizes:\u003C\u002Fp>\n\u003Cpre class=\"code-panel\" data-lang=\"text\">\u003Ccode>BLOCK=256   → 2181037.969997\nBLOCK=1024  → 2181037.969986\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>A different block size is a different tree shape, and floating-point addition is\nnot associative, so the answers differ — here by 5.2e-12 relative. Tiny, real,\nand enough to break an exact-equality assertion. This is the concrete version of\nthe warning below.\u003C\u002Fp>\n\u003Ch2 id=\"inputprecision\">\u003Ccode>InputPrecision\u003C\u002Fcode>\u003C\u002Fh2>\n\u003Cp>For \u003Ccode>f32 × f32\u003C\u002Fcode>, \u003Ccode>T::dot\u003C\u002Fcode> lets you choose what the hardware actually does:\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Value\u003C\u002Fth>\n\u003Cth>Behaviour\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Ccode>TF32\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Round inputs to 19 bits, use Tensor Cores. Fastest. Default on capable hardware\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>TF32x3\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Three TF32 products combined to recover most of \u003Ccode>f32\u003C\u002Fcode>’s precision\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>IEEE\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>True \u003Ccode>f32\u003C\u002Fcode> arithmetic. Slowest, exact\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003Cp>\u003Ccode>TF32\u003C\u002Fcode> keeps \u003Ccode>f32\u003C\u002Fcode>’s range and drops thirteen mantissa bits. For neural network\ntraining that is almost always fine, which is why it is the default.\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cp>\u003Cstrong>\u003Ccode>IEEE\u003C\u002Fcode> turns off the Tensor Cores.\u003C\u002Fstrong> It is not a slightly slower, slightly\nmore accurate mode — Triton only routes a \u003Ccode>dot\u003C\u002Fcode> to the matrix hardware for the\nreduced-precision modes, so \u003Ccode>IEEE\u003C\u002Fcode> falls back to software\nfused-multiply-add. The fused conv kernel in this tree was written with\n\u003Ccode>IEEE\u003C\u002Fcode> to match cuDNN’s accumulation and silently lost its Tensor Cores;\nit now uses \u003Ccode>TF32\u003C\u002Fcode>.\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cp>So “it felt safer” is an expensive instinct here. Choose \u003Ccode>IEEE\u003C\u002Fcode> when you need\nexact \u003Ccode>f32\u003C\u002Fcode> more than you need the hardware, knowing that is the trade.\u003C\u002Fp>\n\u003Cp>\u003Ccode>TF32x3\u003C\u002Fcode> is the middle option — three TF32 products recovering most of \u003Ccode>f32\u003C\u002Fcode>’s\nprecision while staying on the Tensor Cores — and is worth knowing about because\nmost people do not know it exists.\u003C\u002Fp>\n\u003Ch2 id=\"casting\">Casting\u003C\u002Fh2>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">fn\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> cast\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Src\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Dtype\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Dst\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Dtype\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>(\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\">x\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Tensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Src\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> rounding\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Option\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">FpDowncastRounding\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> bitcast\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> bool\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">)\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ->\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Tensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Dst\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>;\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>Two arguments to be careful with.\u003C\u002Fp>\n\u003Cp>\u003Ccode>FpDowncastRounding\u003C\u002Fcode> applies when narrowing: \u003Ccode>Rtne\u003C\u002Fcode> rounds to nearest with ties\nto even, \u003Ccode>Rtz\u003C\u002Fcode> truncates toward zero. \u003Ccode>Rtne\u003C\u002Fcode> is what you want; \u003Ccode>Rtz\u003C\u002Fcode> biases every\nvalue toward zero, and a systematic bias through a training loop compounds in a\nway that random rounding error does not.\u003C\u002Fp>\n\u003Cp>\u003Ccode>bitcast: true\u003C\u002Fcode> reinterprets the bits rather than converting the value. It is\noccasionally what you want, and it is never what you want by accident.\u003C\u002Fp>\n\u003Ch2 id=\"where-precision-goes\">Where precision goes\u003C\u002Fh2>\n\u003Cp>\u003Cstrong>Catastrophic cancellation.\u003C\u002Fstrong> Subtracting two nearly equal large numbers leaves\na result dominated by their rounding error. \u003Ccode>sqrt(a² - b²)\u003C\u002Fcode> is the classic; use\n\u003Ccode>sqrt((a-b)*(a+b))\u003C\u002Fcode>.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Summing many values.\u003C\u002Fstrong> Error grows with the count. A long reduction in \u003Ccode>f32\u003C\u002Fcode>\nis fine; the same in \u003Ccode>f16\u003C\u002Fcode> is not.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Exponentials.\u003C\u002Fstrong> \u003Ccode>exp(x)\u003C\u002Fcode> overflows \u003Ccode>f32\u003C\u002Fcode> above about 88, which is why softmax\nsubtracts the maximum first — Chapter 10.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Division by something near zero.\u003C\u002Fstrong> Normalisation layers add an epsilon for\nthis reason.\u003C\u002Fp>\n\u003Ch2 id=\"determinism\">Determinism\u003C\u002Fh2>\n\u003Cp>Two things make GPU results non-reproducible, and both are expected rather than\nbroken:\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Floating-point addition is not associative.\u003C\u002Fstrong> A reduction combines in an\nunspecified order, so the last bits can differ between a GPU and a CPU\nreference, or between two block sizes — measured above at 5.2e-12 relative for\na block size change alone.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Atomics arrive in an unspecified order.\u003C\u002Fstrong> Chapter 14. So a backward pass using\nthem is not bit-reproducible run to run.\u003C\u002Fp>\n\u003Cp>Consequences for your tests: compare with a tolerance, never exact equality. The\ntests in this tree use \u003Ccode>1e-5\u003C\u002Fcode> for a forward pass and \u003Ccode>1e-6\u003C\u002Fcode> for a backward one.\nAnd when a test does fail, check whether the difference is at the tolerance\nboundary — a mismatch in the last two bits is arithmetic; a mismatch in the first\ndigit is a bug.\u003C\u002Fp>\n\u003Ch2 id=\"a-checklist\">A checklist\u003C\u002Fh2>\n\u003Col>\n\u003Cli>\u003Cstrong>Tightest bound that admits your operations.\u003C\u002Fstrong> \u003Ccode>Float\u003C\u002Fcode> if you use \u003Ccode>exp\u003C\u002Fcode>.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Accumulate wider than you store.\u003C\u002Fstrong> \u003Ccode>f32\u003C\u002Fcode> for \u003Ccode>f16\u003C\u002Fcode> inputs; \u003Ccode>f64\u003C\u002Fcode> for the\nhost-side sum of \u003Ccode>f32\u003C\u002Fcode> partials, which costs nothing and bought 200× above.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Reduce in a tree, not a loop.\u003C\u002Fstrong> You get this for free from \u003Ccode>T::sum\u003C\u002Fcode>, and it\nwas 1000× more accurate than the sequential version.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Leave \u003Ccode>InputPrecision\u003C\u002Fcode> at \u003Ccode>TF32\u003C\u002Fcode>\u003C\u002Fstrong> unless you can say why not — \u003Ccode>IEEE\u003C\u002Fcode>\nturns the Tensor Cores off.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>\u003Ccode>Rtne\u003C\u002Fcode> when narrowing\u003C\u002Fstrong>, not \u003Ccode>Rtz\u003C\u002Fcode>.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Subtract the max before exponentiating.\u003C\u002Fstrong>\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Compare with a tolerance\u003C\u002Fstrong> in tests, and know which tolerance and why.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>\u003Ccode>f64\u003C\u002Fcode> costs 2× on a memory-bound kernel\u003C\u002Fstrong> and far more on a compute-bound\none. Know which you have before ruling it out.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Half precision is not available yet.\u003C\u002Fstrong> Design for it; do not depend on it.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Ch2 id=\"end-of-part-4\">End of Part 4\u003C\u002Fh2>\n\u003Cp>You can choose a block size, reason about layout, measure honestly, and pick\ndtypes deliberately.\u003C\u002Fp>\n\u003Cp>Every number in this part was measured on one card, an RTX 5070, and each\nchapter names it. Three independent kernels — the block-size sweep, the\ncoalescing comparison, and the \u003Ccode>f32\u003C\u002Fcode>\u002F\u003Ccode>f64\u003C\u002Fcode> add — all plateau within 1% of\n594 GB\u002Fs, which is the most useful single fact in these four chapters: it is\nthis card’s ceiling, and it is what a new kernel should be judged against.\u003C\u002Fp>\n\u003Cp>Your card’s number will differ. The method will not.\u003C\u002Fp>\n",[12,16,19,22,25,28,32,35,38,41,44,47,50],{"id":13,"text":14,"level":15},"the-type-hierarchy","The type hierarchy",2,{"id":17,"text":18,"level":15},"what-is-actually-available","What is actually available",{"id":20,"text":21,"level":15},"the-formats-for-when-this-is-fixed","The formats, for when this is fixed",{"id":23,"text":24,"level":15},"what-f64-actually-costs","What f64 actually costs",{"id":26,"text":27,"level":15},"accumulators","Accumulators",{"id":29,"text":30,"level":31},"measured","Measured",3,{"id":33,"text":34,"level":31},"and-it-is-not-reproducible","And it is not reproducible",{"id":36,"text":37,"level":15},"inputprecision","InputPrecision",{"id":39,"text":40,"level":15},"casting","Casting",{"id":42,"text":43,"level":15},"where-precision-goes","Where precision goes",{"id":45,"text":46,"level":15},"determinism","Determinism",{"id":48,"text":49,"level":15},"a-checklist","A checklist",{"id":51,"text":52,"level":15},"end-of-part-4","End of Part 4",false,{"title":55,"titleHtml":55,"route":56},"Measuring","\u002Fkernels\u002Ffast\u002Fmeasuring",{"title":58,"titleHtml":58,"route":59},"Your Kernel as a Graph Op","\u002Fkernels\u002Fin-a-model\u002Fgraph-op",1786271830085]