Phase 2 — Working with LLMs · Lesson 19 · 20 XP
How LLMs work: tokens, context windows, sampling
A model doesn't see words or characters — it sees tokens, chunks of text a tokenizer maps to numbers. A short sentence in one language can use more tokens than a similar-length sentence in another, because tokenizers are trained on specific text distributions. Token count is what you pay for and what counts against the context window, not character count.
The context window is the maximum number of tokens the model can attend to in one request — input and output combined. Go over it and older content gets dropped or the request fails, depending on the API.
Generation is autoregressive: the model predicts one next token at a time, given everything before it, then repeats. Sampling controls how that prediction turns into an actual token — temperature 0 always picks the single most likely token (deterministic, repeatable), while a higher temperature allows less-likely tokens through for more varied, creative output.
Exercise
Use a tokenizer (e.g. tiktoken, or a provider's token-counting endpoint) to count tokens in a few prompts of different lengths and languages. Confirm token count doesn't track character or word count 1:1.
Check yourself
1. Why can a short sentence in one language use more tokens than a similar-length sentence in another?
2. What does setting temperature to 0 do to a model's output, and when would you want that?
Gradient descent and a neural net by hand
Answer the check-yourself questions to unlock this