Glossary
What is Tokenization?
Tokenization is the process of splitting text into the smaller units, called tokens, that a language model actually processes. A token is typically a word fragment rather than a whole word.
Related terms
See how AI represents your brand today
Start measuring your first brand for free, or get a guided walkthrough with our team. Know where you stand before your competitors do.