How LLMs choose words

A language model does not predict entire sentences directly, but calculates the probability of each possible next word (token) in the vocabulary step by step. Without guidance, the model simply always chooses the most likely word. Sampling parameters intervene in this probability distribution process.

The Most Important Parameters

1. Temperature

What it does: Temperature scales the probabilities of tokens before they are converted into likelihoods.

Low temperature (e.g., 0.0 - 0.2): Makes the probability distribution steeper. The most likely word receives almost all the preference, leading to predictable, deterministic, and factual answers.
High temperature (e.g., 0.8 - 1.2): Flattens the distribution. Less likely words suddenly stand a better chance, resulting in more creative, surprising output.

2. Top-p (Nucleus Sampling)

What it does: Top-p limits the choice of words to a cumulative probability threshold. With a setting of 0.9, the model only considers the set of words that together make up 90% of the cumulative probability. Words with a very low probability are immediately discarded, regardless of the temperature.

3. Frequency & Presence Penalty

What it does: These parameters discourage repetition.

Frequency Penalty: Lowers the probability of a word based on how often that word has already appeared in the generated text.
Presence Penalty: Encourages the model to talk about new topics by immediately penalizing already used words, regardless of their exact frequency.

When to use which value?

Use Case	Temperature	Top-p
Code generation / Data extraction	0.0 - 0.1	1.0
Business emails / Customer service	0.3 - 0.5	0.9
Creative writing / Brainstorming	0.8 - 1.0	0.95

Practical Exercise

You are building an application that analyzes legal contracts and creates bulleted summaries. Which settings do you choose for the API call to prevent the model from making up facts (hallucinating)?

Answer: Choose a low temperature (e.g., 0.1) to ensure maximum predictability and factual accuracy.