Crate teeny_quant 

Quantization tooling for teenygrad: reads .safetensors model checkpoints and writes quantized .safetensors checkpoints following the compressed-tensors convention (see format). Initially validated against Ultralytics YOLO models.

Weight-only post-training quantization (INT8/INT4/FP8, quant) needs only the checkpoint itself and is what this crate supports today. Static activation quantization (calibrating from a forward pass over sample inputs) is tracked separately as teenygrad-303.10 since it depends on running the model’s ONNX export through teeny-onnx/teeny-compiler.

Re-exports

pub use error::Error;

pub use error::Result;

Modules

cli

The teeny-quant binary’s command-line interface.

error

Error types for teeny-quant.

format

Output convention: vllm-project/compressed-tensors layered on plain .safetensors, so quantized checkpoints stay loadable by existing HF/vLLM tooling for INT8/FP8. For a quantized weight tensor named foo.weight:

quant

Weight quantization primitives: INT8/INT4 affine quantization (affine, pack4) and F8_E4M3/F8_E5M2 (fp8), each parameterized by a Granularity.

read

Reading source .safetensors checkpoints. Opening/mmapping is delegated to teeny_data::safetensors::SafeTensors; this module only adds the “upcast whatever float dtype is on disk to f32 for quantization math” step.

validate

Tensor-level (not full-model) quantization error metrics: for each quantized tensor, dequantize and compare against the original f32 weights. No forward pass is needed – this is a pure diff between the original and quantized-then-reconstructed safetensors files.

write

Writing .safetensors output. teeny-data::safetensors only supports mmap’d reading, so this writes directly via the safetensors crate rather than extending that (leaf) crate – see teenygrad-303.2.