[{"data":1,"prerenderedAt":22},["ShallowReactive",2],{"chapter:teenygrad\u002Fcli-and-aot\u002Fteeny-llm.json":3},{"project":4,"route":5,"title":6,"titleHtml":7,"navTitle":6,"part":8,"sourcePath":9,"editUrl":10,"html":11,"toc":12,"hasMermaid":13,"prev":14,"next":18},"teenygrad","\u002Fteenygrad\u002Fcli-and-aot\u002Fteeny-llm","teeny-llm","\u003Ccode>teeny-llm\u003C\u002Fcode>","CLI & AOT Compilation","cli-and-aot\u002Fteeny-llm.md","https:\u002F\u002Fgithub.com\u002Fteenygrad\u002Fteenygrad\u002Fedit\u002Fmain\u002Fbooks\u002Fteenygrad\u002Fsrc\u002Fcli-and-aot\u002Fteeny-llm.md","\u003Cp>\u003Ccode>teeny-llm\u003C\u002Fcode> is the workspace’s LLM inference application — vLLM-style serving, aimed at 2x the\nthroughput (see the \u003Ca href=\"\u002Fteenygrad\u002Fappendix\u002Ffaq-and-roadmap\">roadmap\u003C\u002Fa>).\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cp>⚠️ \u003Cstrong>Early scaffolding.\u003C\u002Fstrong> The \u003Ccode>teeny-llm\u003C\u002Fcode> binary currently only prints a placeholder banner.\nThe planned direction is a single binary offering both an interactive mode and an\nOpenAPI-compatible HTTP server mode — earlier separate \u003Ccode>agent\u003C\u002Fcode>\u002F\u003Ccode>console\u003C\u002Fcode> binaries were dropped\nin favor of this single-binary design.\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cpre data-lang=\"bash\" class=\"shiki teeny-datasheet\" style=\"background-color:#16181a;color:#e6e8e3\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#7FB6D9\">cargo\u003C\u002Fspan>\u003Cspan style=\"color:#D8A76B\"> install\u003C\u002Fspan>\u003Cspan style=\"color:#D8A76B\"> teeny-llm\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#7FB6D9\">teeny-llm\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cblockquote>\n\u003Cp>\u003Cstrong>TODO\u003C\u002Fstrong>: fill this chapter in as \u003Ccode>teeny-llm\u003C\u002Fcode> gains real functionality — request\u002Fresponse API\nshape, model loading, batching\u002Fscheduling behavior.\u003C\u002Fp>\n\u003C\u002Fblockquote>\n",[],false,{"title":15,"titleHtml":16,"route":17},"teeny-cli and Ahead-of-Time Compilation","\u003Ccode>teeny-cli\u003C\u002Fcode> and Ahead-of-Time Compilation","\u002Fteenygrad\u002Fcli-and-aot\u002Fteeny-cli",{"title":19,"titleHtml":20,"route":21},"teeny-quant and Model Quantization","\u003Ccode>teeny-quant\u003C\u002Fcode> and Model Quantization","\u002Fteenygrad\u002Fdeployment\u002Fteeny-quant",1786271829455]