[{"data":1,"prerenderedAt":41},["ShallowReactive",2],{"chapter:kernels\u002Fin-a-model\u002Fgraph-op.json":3},{"project":4,"route":5,"title":6,"titleHtml":6,"navTitle":6,"part":7,"sourcePath":8,"editUrl":9,"html":10,"toc":11,"hasMermaid":34,"prev":35,"next":38},"kernels","\u002Fkernels\u002Fin-a-model\u002Fgraph-op","Your Kernel as a Graph Op","Kernels in a Real Model","in-a-model\u002Fgraph-op.md","https:\u002F\u002Fgithub.com\u002Fteenygrad\u002Fteenygrad\u002Fedit\u002Fmain\u002Fbooks\u002Fkernels\u002Fsrc\u002Fin-a-model\u002Fgraph-op.md","\u003Cp>Every kernel so far has been launched by hand: you built it, compiled it,\nallocated buffers, and called \u003Ccode>launch\u003C\u002Fcode>. That is the whole mechanism, and for a\nstandalone kernel it is all you need.\u003C\u002Fp>\n\u003Cp>Models do not work that way. A model is a graph of operations, and the framework\ndecides what runs, in what order, with which buffers. To put your kernel in one,\nyou have to describe it in terms the graph understands.\u003C\u002Fp>\n\u003Cp>This part is about that. It is more machinery than the rest of the book, and\nnone of it is needed to make a kernel \u003Cem>work\u003C\u002Fem> — only to make it a citizen of a\nmodel.\u003C\u002Fp>\n\u003Ch2 id=\"the-two-halves\">The two halves\u003C\u002Fh2>\n\u003Cp>There are two separate jobs, and separating them is the thing to understand\nfirst.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>\u003Ccode>CustomOp\u003C\u002Fcode> is the graph-level description.\u003C\u002Fstrong> What the operation is called,\nwhat shape it produces, and how to get a kernel out of it. It is consulted while\nthe graph is being built and lowered, before anything runs.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>\u003Ccode>RuntimeOp\u003C\u002Fcode> is the launch-time description.\u003C\u002Fstrong> How many inputs it takes, what\nscratch buffers it needs, how to pack arguments, and how big a grid to launch.\nIt is consulted every time the op executes.\u003C\u002Fp>\n\u003Cp>One is about shapes and identity. The other is about pointers and grids. The\nnext chapter is \u003Ccode>RuntimeOp\u003C\u002Fcode>; this one is everything before it.\u003C\u002Fp>\n\u003Ch2 id=\"symbolic-tensors\">Symbolic tensors\u003C\u002Fh2>\n\u003Cp>A graph is built by recording. You start with a placeholder and every operation\non it appends a node rather than computing anything:\u003C\u002Fp>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">let\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> (\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\">x\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> graph\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">)\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> =\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> SymTensor\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">input\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">DtypeRepr\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">F32\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> vec!\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">[\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">None\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Some\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\">784\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">)]);\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>\u003Ccode>SymTensor\u003C\u002Fcode> is a handle: a node index, a dtype, a shape, and a shared reference\nto the graph. Cloning one is cheap; it shares the graph.\u003C\u002Fp>\n\u003Cp>The shape is a \u003Ccode>Vec<Option<usize>>\u003C\u002Fcode>, and the \u003Ccode>None\u003C\u002Fcode> is the point. It means “this\ndimension is not known yet” — almost always the batch axis. So \u003Ccode>vec![None, Some(784)]\u003C\u002Fcode> is “any number of rows of 784”. Concrete sizes arrive later, when\nthe model is loaded and given real inputs.\u003C\u002Fp>\n\u003Cp>This is why your op cannot simply be handed shapes. It has to be able to \u003Cem>infer\u003C\u002Fem>\nits output shape from symbolic inputs, before anyone knows the batch size.\u003C\u002Fp>\n\u003Ch2 id=\"recording-your-op\">Recording your op\u003C\u002Fh2>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">let\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> y \u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">=\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> x\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">.\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">record_custom\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">CustomData\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">new\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">MyOp\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">new\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\">block_size\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">)),\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> &[],\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> None\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">);\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>Three arguments:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>the op\u003C\u002Fstrong>, wrapped in \u003Ccode>CustomData\u003C\u002Fcode>, which is an \u003Ccode>Arc<dyn CustomOp>\u003C\u002Fcode> that also\nimplements \u003Ccode>Debug\u003C\u002Fcode>;\u003C\u002Fli>\n\u003Cli>\u003Cstrong>additional inputs\u003C\u002Fstrong>, since \u003Ccode>self\u003C\u002Fcode> is the first one — a two-input op passes\n\u003Ccode>&[&other]\u003C\u002Fcode>;\u003C\u002Fli>\n\u003Cli>\u003Cstrong>an output dtype\u003C\u002Fstrong>, or \u003Ccode>None\u003C\u002Fcode> to keep the primary input’s.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>It returns a new \u003Ccode>SymTensor\u003C\u002Fcode> pointing at the node it just added. From the\ngraph’s point of view your op is now indistinguishable from a built-in one.\u003C\u002Fp>\n\u003Ch2 id=\"implementing-customop\">Implementing \u003Ccode>CustomOp\u003C\u002Fcode>\u003C\u002Fh2>\n\u003Cp>Four methods matter:\u003C\u002Fp>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">pub\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> trait\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> CustomOp\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Any\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> +\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Send\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> +\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Sync\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">    fn\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> name\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(&\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">)\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ->\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> &\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">str\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">;\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">    fn\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> infer_output_shape\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(&\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> input_shapes\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> &[&\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Shape\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">])\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ->\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Shape\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">;\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">    fn\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> as_any\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(&\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">)\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ->\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> &\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">dyn\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Any\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">;\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">    fn\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> lower\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(&\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">)\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ->\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Option\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><(\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">String\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> String\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> String\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Arc\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">dyn\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> RuntimeOp\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>)>\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> None\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> }\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">    fn\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> lower_backward_source\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(&\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">)\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ->\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> String\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> String\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">new\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">()\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> }\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#8A9088\">}\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>\u003Cstrong>\u003Ccode>name\u003C\u002Fcode>\u003C\u002Fstrong> is used in errors and debug output. Namespace it — the vision-rs ops\nuse \u003Ccode>"yolo.detect_decode"\u003C\u002Fcode> — because a bare \u003Ccode>"decode"\u003C\u002Fcode> in a lowering failure\ntells nobody anything.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>\u003Ccode>infer_output_shape\u003C\u002Fcode>\u003C\u002Fstrong> is the one with real content. It gets every input’s\nsymbolic shape, in the order they were recorded, and returns the output’s.\nShape-preserving ops are one line:\u003C\u002Fp>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">fn\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> infer_output_shape\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(&\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> input_shapes\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> &[&\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Shape\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">])\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ->\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Shape\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    input_shapes\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">[\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\">0\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">].\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">clone\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">()\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#8A9088\">}\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>An op that changes rank does the arithmetic here, propagating \u003Ccode>None\u003C\u002Fcode> wherever a\ndimension stays dynamic. Getting this wrong does not fail here — it fails much\nlater, when a buffer is allocated at the wrong size.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>\u003Ccode>as_any\u003C\u002Fcode>\u003C\u002Fstrong> is boilerplate, always \u003Ccode>fn as_any(&self) -> &dyn Any { self }\u003C\u002Fcode>. It\nlets a lowering downcast back to your concrete type.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>\u003Ccode>lower\u003C\u002Fcode>\u003C\u002Fstrong> is how your op becomes a kernel, and is the subject of the next\nchapter.\u003C\u002Fp>\n\u003Ch2 id=\"a-real-one\">A real one\u003C\u002Fh2>\n\u003Cp>There is no \u003Ccode>CustomOp\u003C\u002Fcode> implementation in the teenygrad repository itself — the\nbuilt-in ops go through \u003Ccode>Op\u003C\u002Fcode> variants instead. The worked examples are in\nvision-rs, and \u003Ccode>DetectDecodeOp\u003C\u002Fcode> is the clearest:\u003C\u002Fp>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">pub\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> struct\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> DetectDecodeOp\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> FloatBytes\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> +\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Send\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> +\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Sync\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> +\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> '\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">static\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">    pub\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> anchor_x\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Vec\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">f32\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">    pub\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> anchor_y\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Vec\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">f32\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">    pub\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> strides\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Vec\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">f32\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">    pub\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> block_a\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> i32\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">    _phantom\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> PhantomData\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#8A9088\">}\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">impl\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> FloatBytes\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> +\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Send\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> +\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Sync\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> +\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> '\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">static\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> CustomOp\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> for\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> DetectDecodeOp\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"><\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">D\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">>\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">    fn\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> name\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(&\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">)\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ->\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> &\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">str\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> \"\u003C\u002Fspan>\u003Cspan style=\"color:#D8A76B\">yolo.detect_decode\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">\"\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> }\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">    fn\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> infer_output_shape\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(&\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> input_shapes\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">:\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> &[&\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Shape\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">])\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ->\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Shape\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#7F877D;font-style:italic\">        \u002F\u002F boxes [B, 4, A] → [B, 4, A]: shape-preserving\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">        input_shapes\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">[\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\">0\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">].\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">clone\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">()\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#8A9088\">    }\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">    fn\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\"> as_any\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(&\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">)\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ->\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> &\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\">dyn\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Any\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> self\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> }\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#7F877D;font-style:italic\">    \u002F\u002F lower() — next chapter\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#8A9088\">}\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>\u003Cem>From \u003Ccode>vision-rs\u002Fsrc\u002Fmodels\u002Fyolo\u002Fkernels\u002Fdetect_decode.rs\u003C\u002Fcode>.\u003C\u002Fem>\u003C\u002Fp>\n\u003Cp>Notice what the struct holds: not tensors, but the \u003Cem>configuration\u003C\u002Fem> the kernel\nneeds — a precomputed anchor grid and a block size. A \u003Ccode>CustomOp\u003C\u002Fcode> is built once,\nwhen the model is defined, and consulted many times. It should hold parameters,\nnever per-inference state.\u003C\u002Fp>\n\u003Ch2 id=\"what-the-lowering-does-with-it\">What the lowering does with it\u003C\u002Fh2>\n\u003Cp>When the graph is compiled, \u003Ccode>TritonLowering\u003C\u002Fcode> walks it and turns every node into\nsomething executable. Your node hits this arm:\u003C\u002Fp>\n\u003Cpre data-lang=\"rust\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#6FBF98\">Op\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">Custom\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> data \u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">}\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> =>\u003C\u002Fspan>\u003Cspan style=\"color:#FF5F9E\"> match\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> data\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">.\u003C\u002Fspan>\u003Cspan style=\"color:#B79AD4\">0\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">.\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">lower\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">()\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#6FBF98\">    Some\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">((\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\">name\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> kernel_source\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> entry_point\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> runtime_op\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">))\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> =>\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#6FBF98\">        Box\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">new\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">KernelExecutable\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> name\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> kernel_source\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#E6E8E3\"> entry_point\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> ...\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> })\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#8A9088\">    }\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#6FBF98\">    None\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> =>\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\"> {\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#FF5F9E\">        return\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\"> Err\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003Cspan style=\"color:#6FBF98\">anyhow\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">::\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">anyhow!\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">(\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#8A9088\">            \"\u003C\u002Fspan>\u003Cspan style=\"color:#D8A76B\">custom op '\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">{}\u003C\u002Fspan>\u003Cspan style=\"color:#D8A76B\">' is not handled — implement CustomOp::lower()\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">\"\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#E6E8E3\">            data\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">.\u003C\u002Fspan>\u003Cspan style=\"color:#7FB6D9\">name\u003C\u002Fspan>\u003Cspan style=\"color:#8A9088\">()\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#8A9088\">        ));\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#8A9088\">    }\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#8A9088\">},\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>\u003Cem>From \u003Ccode>kernels\u002Fteeny-kernels\u002Fsrc\u002Fgraph\u002Fmod.rs\u003C\u002Fcode>.\u003C\u002Fem>\u003C\u002Fp>\n\u003Cp>So \u003Ccode>lower\u003C\u002Fcode> returning \u003Ccode>None\u003C\u002Fcode> — the default — is a runtime error naming your op,\nnot a compile error. If you implement \u003Ccode>CustomOp\u003C\u002Fcode> and forget \u003Ccode>lower\u003C\u002Fcode>, this is the\nmessage you will get.\u003C\u002Fp>\n\u003Cp>The result is a \u003Ccode>KernelExecutable\u003C\u002Fcode>: kernel source, entry-point symbol, output\nshape and dtype, and the \u003Ccode>RuntimeOp\u003C\u002Fcode>. From there it is compiled to PTX exactly\nas in Chapter 9 — the graph path and the by-hand path converge on the same\ncompiler.\u003C\u002Fp>\n\u003Ch2 id=\"where-fusion-happens\">Where fusion happens\u003C\u002Fh2>\n\u003Cp>Chapter 12 fused operations by writing one kernel that did several things. The\ngraph has its own kind: the lowering can recognise a pattern of nodes and emit a\nsingle kernel for them, or split one node into several.\u003C\u002Fp>\n\u003Cp>Both happen in this tree. Conv2d-with-bias becomes two DAG nodes, which is why\n\u003Ccode>Lowering::extra_dag_names\u003C\u002Fcode> exists — the extra node needs a name so its weights\nload. And the fused conv kernels from Chapter 12 are selected by shape, in the\nlowering, from one graph node.\u003C\u002Fp>\n\u003Cp>Your custom op does not participate in this. It lowers to exactly the kernel you\ngive it, and the graph will not fuse it with its neighbours. If you want fusion,\nfuse it yourself, in the kernel.\u003C\u002Fp>\n\u003Cp>Next: the half that runs.\u003C\u002Fp>\n",[12,16,19,22,25,28,31],{"id":13,"text":14,"level":15},"the-two-halves","The two halves",2,{"id":17,"text":18,"level":15},"symbolic-tensors","Symbolic tensors",{"id":20,"text":21,"level":15},"recording-your-op","Recording your op",{"id":23,"text":24,"level":15},"implementing-customop","Implementing CustomOp",{"id":26,"text":27,"level":15},"a-real-one","A real one",{"id":29,"text":30,"level":15},"what-the-lowering-does-with-it","What the lowering does with it",{"id":32,"text":33,"level":15},"where-fusion-happens","Where fusion happens",false,{"title":36,"titleHtml":36,"route":37},"Numerics","\u002Fkernels\u002Ffast\u002Fnumerics",{"title":39,"titleHtml":39,"route":40},"Wiring the Runtime","\u002Fkernels\u002Fin-a-model\u002Fruntime-op",1786271830092]