Determinism in the AI Tech Stack (LLMs): Temperature, Seeds, and Tools

Determinism in LLM outputs is achievable—but only in specific ways, and often only when randomness controls are paired with “deterministic” software scaffolding. A hands-on set of experiments shows that lowering temperature from 1 to 0 can dramatically stabilize an open-source model’s responses, yet it may still produce wrong answers. The most reliable path in practice comes from combining an LLM with conventional code execution: even when the open model struggles to solve a math equation directly, it can generate executable Python that then produces the correct result.

The tests begin with a baseline “software stack” approach: a Python program computes a target equation and returns a consistent numeric answer (11 16.95). Running the same task through an open-source LLM with temperature set to 1 produces wildly different outputs across repeated runs—72.34, 117.0, 147.450—making the result unusable for anything requiring repeatability. Switching temperature to 0 reduces variability sharply: repeated runs cluster around ~115.8–115.9, showing much stronger consistency even though the answer remains incorrect.

To interpret why, the transcript ties temperature to how token probabilities are sampled. At higher temperature, the model spreads probability mass across multiple next-token options, increasing diversity and creativity but also randomness. At temperature 0, sampling collapses toward the most likely token choices, yielding repeatable outputs—though potentially “boring” and still not guaranteed correct.

A key turning point arrives when a stronger foundation model (GPT-4) is used with temperature set to 0. Unlike the smaller open model, GPT-4 returns the correct equation result (11 16.95) consistently across repeated runs. That contrast frames the core tradeoff: temperature control improves stability, model capability determines correctness.

For the smaller open model (referred to as “f3”), the transcript demonstrates a hybrid strategy. Instead of asking the model to directly compute the equation, it prompts the model to generate Python code for the problem, then executes that code deterministically. With this tool-augmented setup, the system reaches the correct answer even when direct LLM-only attempts fail.

The final section shifts from temperature to OpenAI’s “seed” parameter in chat completions. In a beta feature, repeated requests using the same seed and parameters are intended to sample deterministically “with best effort,” while acknowledging determinism is not guaranteed and backend changes can affect results (tracked via a system fingerprint). In a simple prompt (“best pizza in New York” with a cheese constraint), seeded calls return the same answer repeatedly (attributed to “FAR Pizza in Brooklyn”), while unseeded calls vary.

Overall, the transcript lands on a practical conclusion: LLMs are inherently probabilistic and not designed to behave like traditional deterministic software. But stability improves with temperature=0, and repeatability can be further strengthened with seeds and—most effectively—by adding deterministic tools such as code execution and other grounding techniques to reduce hallucinations. The “hallucination” question is left open as a feature-versus-bug tension: creativity and novelty require some uncertainty, yet many real use cases demand repeatable, precise outputs.