Mixture of Models (MoM) - SHOCKING Results on Hard LLM Problems!

Mixture-of-Models (MoM) systems can outperform single-model prompting on hard reasoning tasks, but the gains depend heavily on how the models are combined. Using three architectures—“King” (one strong model synthesizes many advisers), “DuoPoly” (two strong models debate), and “Democracy” (equal-weight voting)—the results show that structured synthesis and adversarial discussion beat simple majority voting, especially on tricky logic and coding problems.

In the “King” setup, multiple smaller LLMs (“peasants”) answer the same prompt independently. Their responses are then fed into a single higher-capability model (“king”), identified as GPT-4 turbo, which produces the final solution using the advisers’ context. The transcript reports strong performance on a classic marble-in-a-cup logic puzzle: the system concluded the marble would fall out before the cup is placed into the microwave, aligning with gravity and the lack of containment once the cup is lifted. Several advisers contributed key physical reasoning, and GPT-4 turbo synthesized those inputs into a step-by-step plan and final answer.

The “King” architecture also handled a simple age reasoning question correctly: Lena ends up 27 years old, not half John’s current age, because the age gap remains three years. For a harder LeetCode-style coding task (sourced from “context 3 93,” described as not in training data), the system produced a solution that passed all tested cases after being run in the LeetCode environment. The only notable miss came from a constrained text task—writing 10 sentences ending with “apples”—where the system produced 9 correct sentences, falling short by one.

The “DuoPoly” architecture replaced single synthesis with debate. Two strong models—GPT-4 turbo and Claude 3 Opus—were prompted to challenge each other’s assumptions using the advisers’ outputs as shared context. This approach improved some areas (the age question stayed correct, and the coding task passed all cases after the solution was tested), but it failed the marble puzzle, landing on an incorrect physical conclusion. The constrained “apples” task also deteriorated, with multiple failures (2 wrong out of 10), suggesting that debate can amplify uncertainty when the underlying reasoning is brittle.

Finally, “Democracy” treated every model as equal and selected the most-voted answer. The transcript’s vote counts show why this can struggle: the marble puzzle’s winning vote still produced the wrong outcome, and the coding task’s most-voted solution came from a smaller model and only partially worked (one case passed, two failed). The age question was the clear bright spot—most models voted for 27 years old (6 votes). For the apples task, the most-voted answer was judged best among the three architectures, though it still wasn’t perfect.

Overall, the transcript frames MoM as a practical engineering pattern: more models help, but the aggregation method matters. Structured synthesis (“King”) delivered the most reliable results across the hardest mix of physics-like logic, arithmetic reasoning, and coding, while equal-weight voting (“Democracy”) and debate (“DuoPoly”) showed more failure modes.

The session also includes implementation details: looping over multiple model calls, collecting adviser outputs, and using system prompts to steer synthesis, debate, or voting. A small UI tracks progress and task timing, and the author shares that the code will be posted to a community GitHub for members, alongside a community Discord. The closing section highlights an unrelated open-source cybersecurity incident-response testing tool that can run on local models and uses LLMs to generate tailored scenarios.