When Does Mixing Models Pay? Open Fusion, a Bounded Mixture-of-Agents Runtime, and Registered Evidence from Local Coding Benchmarks

Abstract

Mixture-of-Agents (MoA) methods combine the outputs of several large language models (LLMs) at inference time and have reported gains on conversational benchmarks. Whether those gains survive deployment constraints — fixed call budgets, local hardware, objective grading and fair same-model baselines — is less clear. I present Open Fusion, an open-source, Open AI-compatible runtime that implements fourteen bounded multi-model strategies, from fallback and Self-MoA to semantic voting, rank-then-fuse, layered refinement and uncertainty cascades, under a hard per-request call ceiling. It ships with an evaluation laboratory that reports Wilson intervals, paired tests, order-balanced judging and panel-complementarity diagnostics.

Using the project's published artifacts, I report a series of local studies on two NVIDIA GB10 workstations with 4B-parameter open models. In a registered prospective study on 500 previously unused MBPP tasks, a fixed test-guided mixed-model workflow passed 78.1% of all-test evaluations versus 72.5% for repeated same-model repair (+5.6 points, 95% CI [3.6, 7.7]), with 17.2% lower mean latency and 29.2% fewer tokens. A confidence-gated escalation policy raised held-out accuracy on a 70-item programmatic benchmark from 75.7% to 91.4% while calling a stronger model on 40% of items.

Five later controls, however, did not show that model diversity itself causes the gains. Against fresh sampling from the same model, mixed panels tied or underperformed: −0.57 points on 1,500 tasks, and −5.0 points after a 32-policy development search. I conclude that, in this setting, the measurable benefit of MoA-style systems comes from verification, bounded retries and selective escalation rather than from mixing similarly sized models. I close with a fivestep selection rule and a reporting checklist for practitioners.