Mistral AI API - Mixtral 8x7B and Mistral Medium | Tests and First Impression

Mistral AI’s API delivers strong reasoning performance at competitive pricing—especially on tasks where GPT-3.5 often trips up—while also offering a straightforward developer setup with streaming responses and a “safe mode.” In hands-on tests, Mixtral-based models matched GPT-4 on a classic “drying shirts” logic problem, while GPT-3.5 produced a flawed, overly literal time calculation.

The platform setup is simple: a basic web interface with documentation, a Python client example, and chat-completion endpoints that support both streaming and non-streaming output. The API also includes a “safe mode” option and an embeddings model (skipped in these tests). Model selection is presented as three tiers: a tiny model powered by Mistral 7B, a small model powered by Mixtral 8x7B, and a medium model backed by an internal prototype model about which little public detail is provided.

Pricing was checked before testing. The medium model is listed at €7.5 per 1 million tokens (about $8.25), which the tester converts to roughly $0.0825 per 1,000 tokens. The small model’s pricing appears similar to ChatGPT 3.5 Turbo (roughly in the same ballpark after token normalization), making it feel competitive rather than premium.

Three reasoning-and-coding style tests followed. First came the “drying shirts” problem: five shirts dry in 10 hours, then 10 shirts are hung for the next drying cycle. GPT-3.5 incorrectly treated drying time as scaling linearly with the number of shirts, outputting 20 hours for 10 shirts. In contrast, both the Mistral small (Mixtral 8x7B) and the Mistral medium returned the correct logic: if all shirts dry under the same conditions and don’t block each other, 10 shirts still take 10 hours.

Second was a “world model” scenario involving a ball placed into a bag with a hole larger than the ball, then shipped to a friend in London. GPT-3.5 concluded the ball would end up in the sealed box, which contradicts the hole-in-the-bag premise. GPT-4 delivered the expected result: the ball would fall out and be left behind in the sender’s office. The Mistral small model did not reach the same level of certainty or specificity, and the medium model improved the outcome, suggesting the ball likely fell out during transit—though it still wasn’t as clean as GPT-4.

Third came coding: generating a playable Snake game with a Windows UI. The small model produced incomplete code (not the full copyable program), and the resulting game behavior was flawed. The medium model generated a more complete Tkinter-based version with a working UI and score, but it still wasn’t perfect. GPT-4 produced the most reliable result, including correct gameplay behavior.

Finally, streaming output was demonstrated across tiny, small, and medium models, with chunked text arriving quickly. Overall takeaway: Mistral’s API feels easy to integrate, and its Mixtral/Mistral models show notable strengths on reasoning tasks where GPT-3.5 can fail, while GPT-4 remains the benchmark for the toughest coding correctness. The tester ends by planning broader API comparisons beyond OpenAI, while also highlighting interest in the undocumented medium model and future benchmarks.