Same model, different Agent framework: task completion rate jumps from 30% to 81%

The IBM CUGA team published a comparative study on the Hugging Face Blog, exploring how the same model's task completion rate differs across Agent harnesses (frameworks). The tests were based on 90 tasks from the AppWorld benchmark (covering 30 scenarios), using the same open-weight model, GLM-5.3-flash, to compare three harnesses: IBM's own CUGA, Hermes, and LangChain's Deep Agents. The results showed task completion rates of 81.1%, 46.7%, and 30.0% respectively—a gap of more than 50 percentage points between highest and lowest; the fully- scenario pass rates (SGC) were 20, 8, and 2 respectively. This shows that without changing the underlying model, the choice of harness alone has an enormous impact on Agent task performance.

AI overview of reporting · Updated 2 h前

Not enough continuous observations to show a trend yet.