Glossary
What is Tokenization?
Tokenization is the process of splitting text into the smaller units, called tokens, that a language model actually processes. A token is typically a word fragment rather than a whole word.
Related terms
Tokenization is the process of splitting text into the smaller units, called tokens, that a language model actually processes. A token is typically a word fragment rather than a whole word.