Trait Tokenizer 

pub trait Tokenizer {
    // Required methods
    fn apply_chat_template(
        &self,
        messages: &[Message],
        chat_template: &str,
        tokenize: bool,
        add_generation_prompt: bool,
        enable_thinking: bool,
    ) -> String;
    fn encode(&self, texts: &[String]) -> Vec<usize>;
    fn decode(&self, ids: &[usize]) -> String;
}

Text tokenization and chat-templating, implemented per model/tokenizer family.

Required Methods

fn apply_chat_template( &self, messages: &[Message], chat_template: &str, tokenize: bool, add_generation_prompt: bool, enable_thinking: bool, ) -> String

Renders messages through chat_template (a Jinja-style template string, per the Hugging Face chat-template convention), optionally tokenizing the result (tokenize), appending a generation prompt (add_generation_prompt), and enabling “thinking”/reasoning mode (enable_thinking) for models that support it.

fn encode(&self, texts: &[String]) -> Vec<usize>

Encodes texts into a flat sequence of token IDs.

fn decode(&self, ids: &[usize]) -> String

Decodes a sequence of token IDs back into text.

Implementors