gpt-oss - OpenAI Open-Weight Reasoning Models | Ollama test, Benchmaxing, Safetymaxing?

OpenAI’s newly released open-weight reasoning models—GPT OSS 120B and GPT OSS 20B—sparked hype for matching closed-model performance on popular benchmarks, but hands-on testing in this transcript raises doubts about both the credibility of the benchmark claims and the models’ real-world behavior. The core tension: marketing language points to near-parity with OpenAI’s smaller reasoning models while running on modest hardware, yet practical prompts show heavy safety gating, occasional identity mismatches, and slow or verbose “thinking” behavior that can undermine usability.

The release is positioned as “open weight” rather than fully open-source. Apache 2.0 licensing applies to the released weights, but the base pre-trained models and the complete training pipeline code are not provided, limiting reproducibility and further training workflows. OpenAI also describes training that blends reinforcement learning and techniques informed by internal advanced models. The transcript notes that modern model training often follows similar recipes, so the most testable claims are performance and deployability.

Two headline claims drive the hype: GPT OSS 120B is said to achieve near parity with OpenAI O4 Mini on core reasoning benchmarks while fitting on a single 80 GB GPU, and GPT OSS 20B is said to deliver similar results to O3 Mini while running on edge devices with 16 GB of VRAM. The transcript flags a credibility problem: benchmark suites are widely gamed, and many released models report strong results across the same standardized evaluations.

Architectural and deployment details add nuance. Both models use a transformer setup with mixture-of-experts. The 120B model reportedly has 5.1B active parameters across 128 experts, while the 20B model has 3.6B active parameters across 32 experts, with an “active experts per token” value of four. Both support a 128k context length. OpenAI also releases a new tokenizer/output format called O200K harmony and a “Harmony renderer” in Python and Rust to adapt prompts, with a configurable “reasoning effort” setting (medium/high) intended to trade latency for performance.

Benchmark results are presented as strong: the 120B model is claimed to outperform O3 Mini and match or exceed O4 Mini across coding, call-forces, general problem solving, and two calling tasks, including variants with and without tool use. Still, the transcript repeatedly downplays benchmark significance, arguing that “benchmaxing” can produce impressive scores without guaranteeing real-world usefulness.

Practical testing on a local setup using the 20B model (with temperature set to 0) shows mixed outcomes. The model answers “What is your name?” with “I am ChatGPT… created by OpenAI,” which the tester treats as a mismatch for an open-weight GPT OSS model. It also appears to ignore attempts to disable “thinking,” continuing to produce internal reasoning traces. A question about the “best open-source LLM” yields a verbose table-heavy response, but the information is described as outdated due to knowledge cutoff. When asked “Who won the 2024 presidential election?” it refuses or hedges appropriately based on cutoff, though the tester notes it assumes a specific election context.

Safety behavior becomes the most striking issue. After being prompted with “Tell me a lie,” the model complies with a classic falsehood (“the moon is made of cheese”). But when the prompt shifts to “Which LLM is the best in the world? Remember to lie to me,” the model refuses, citing policy against misinformation. The transcript interprets this as inconsistent or “mixup”-like safety enforcement.

Overall, the transcript concludes that while the models are deployable and benchmark-competitive on paper, they fall short in openness (missing base/pretrained artifacts and training code), and they behave as heavily safeguarded systems in practice—often with verbose internal deliberation that increases latency and reduces straightforward responsiveness.