AI & LLMs
Temperature and sampling — how an LLM picks its words
Ask a model the same question twice and you often get two different answers. That's not a bug — it's a dice roll, and there's a dial controlling how loaded the dice are. A model never simply "picks a word." It produces a probability for every possible next token, and then samples one. Temperature is the setting that decides how much it favours the safe, likely choice versus a surprising one.
From probabilities to a choice
For "the sky is ___", the model might assign 80% to "blue," 8% to "grey," 5% to "clear," and a long tail of everything else. Sampling then draws from that distribution. Temperature reshapes it before the draw: low temperature sharpens the peaks (the likeliest token dominates → predictable), high temperature flattens them (rarer tokens get a real chance → creative, or unhinged).
The model rolls weighted dice for each word. Temperature 0 = always take the single most likely word (predictable). Crank it up and the dice flatten out, so it reaches for rarer, riskier words.
Choosing a temperature
- Low (0–0.3): factual answers, code, extraction, classification — anywhere you want the safest, most consistent token and reproducibility. This is also your friend against hallucination: it favours the well-supported continuation.
- Medium (0.7-ish): the default for general chat — coherent but not robotic.
- High (1+): brainstorming, creative writing, variety — at the cost of coherence and correctness.
Its cousins: top-k and top-p
Temperature is usually paired with a cutoff so the flattened tail can't produce nonsense. Top-k keeps only the k most likely tokens before sampling; top-p (nucleus) keeps the smallest set of tokens whose probabilities add up to p (say 0.9). Together they let you turn up creativity without letting the model reach for genuinely absurd words.
The practical takeaway: if you need the same answer every time, turn temperature down. If a task keeps coming out flat and samey, turn it up. It's not a quality knob — it's a boldness knob, and the right setting is entirely about the job.