Study: Differences Among Coding Agent Frameworks Are Mostly Noise; Complex Frameworks Cost More Without Gains
Understand the story
On October 24, 2026, The Decoder reported on a study by Liu and Han. The study used 5 models and 3 harnesses (Claude Code, OpenCode, mini-SWE-agent), tested on the 45 hardest tasks in SWE-bench Verified and a pool of 447 tasks. Results showed that switching harnesses caused task-level result flips at a rate of about 13%, comparable to the flip rate (13%) produced by simply repeating runs with the same harness, indicating that pass-rate differences among coding agent frameworks mostly fall within measurement noise. Meanwhile, token consumption across harnesses could differ by two to three times, meaning more complex coding agent frameworks cost more but do not improve task pass rates. The findings call into question the value of coding agent frameworks, pointing out that their performance differences lack statistically meaningful support while the cost burden is real.
AI overview of reporting · Updated 3 h前
Reporting timeline
Explore different aspects of the story through its reporting.
- The DecoderFancy harnesses make coding agents more expensive but not better, study finds
A study by Liu and Han tested 5 models across 3 harnesses (Claude Code, OpenCode, mini-SWE-agent) on the 45 hardest tasks in SWE-bench Verified and a 447-task pool, finding that task-level result flips caused by switching harnesses (13%) were comparable to simply re-running (13%), with pass-rate differences mostly within measurement noise while token consumption could vary two- to threefold.
Story heat trend
Current heat9·Comparable peak10（10/24 19:00）·Comparable change over 24 hours –
The trend compares the same continuously observed sources, which may cover fewer sources than the current heat score. Hover or select the chart to inspect each hour; use the left and right arrow keys to navigate.