The process of breaking text into individual words or subwords to input them into a language model. Example: Tokenizing a sentence “I am ChatGPT” into the words: “I,” “am,” “Chat,” “G,” and “PT.”
Tokenization is one of the first steps in how a large language model processes human language. Before a model like GPT, Claude, or Gemini can understand or generate text, the raw string has to be converted into tokens — the discrete units the model actually sees and predicts.
How tokenization works
Different models use different tokenizers. OpenAI’s GPT models use a Byte Pair Encoding (BPE) tokenizer, while Google’s Gemini and many open-source models use SentencePiece or WordPiece variants. The tokenizer looks at a chunk of text and splits it using a learned vocabulary. Common words usually become a single token, while rare words, names, or made-up terms get broken into subwords, or even individual characters.
Why tokens matter
Tokens are the unit of measurement for LLM pricing, context windows, and output limits. When a model is advertised with a 200K context window, that’s 200,000 tokens, not words or characters. For English, one token works out to roughly four characters, or about three-quarters of a word, so 100 tokens is around 75 words. Other languages, code, and non-Latin scripts can use significantly more tokens for the same amount of information.
You can see exactly how your own text gets tokenized using OpenAI’s free tokenizer tool, which visualizes the split in real time. For a deeper look at how different tokenizer algorithms compare, Hugging Face’s tokenizer summary is a good reference.
Related terms: Transformer, Natural Language Processing (NLP), ChatGPT.
Tokenization also shapes how models handle multilingual input. A short English prompt may cost 20 tokens, while the same meaning in Hindi, Arabic, or Mandarin can cost two to four times as much, because those scripts and words sit less neatly inside the model’s vocabulary. For teams building AI products, that difference affects latency, cost, and how much context you can realistically fit in a single call — making tokenization a quietly important part of real-world prompt design.
