Expand description
Tokenizer trait and chat-message types.
Structs§
- Message
- A single chat-conversation turn: a
role(e.g."user","assistant","system") paired with itscontent.
Traits§
- Tokenizer
- Text tokenization and chat-templating, implemented per model/tokenizer family.