Vals AI tested GPT-6 Sol and Claude Opus 5.5 in single-agent versus team setups on Vibe Code Bench, finding teams cost 1.8x to 5.1x as much as single agents, and of four comparisons only GPT-6 Sol at the medium reasoning tier improved 7.3 points with statistical significance.
GitHub has released ReviewBench, an open offline benchmark for evaluating AI code review agents. The benchmark is modeled on the distribution of 103.9 million GitHub pull requests.
A benchmark tested whether a local Qwen3.8-27B-Q4_K_M.gguf model can produce exact results of positive integer addition using only English words, with 5,070 no-reasoning cases and 169 reasoning-enabled control cases. With reasoning off, numerical accuracy was only 23.57% — 97.04% for 1-3 digit problems but 6.44% for 10-13 digits, with 96.17% format compliance; with reasoning on, it answered 167 of 169 attempts correctly.
Microsoft and Hugging Face released the ThinkingBox agent sandbox and ThinkingBox-Bench benchmark, with 507 stateful business workflows and 20 runs per task, using executable checks to evaluate end-state database state and side effects, now open on Hugging Face via OpenEnv.
Why it matters: The article uses executable backend-state checks to distinguish surface success from real outcomes, and provides consistency and cost data for each model, a directly useful reference for evaluating agents.
Mercor's research had 12 licensed accountants with an average of five and a half years of experience complete simplified tasks from the APEX Accounting Benchmark; results showed today's AI models outperform accountants on speed, accuracy, and cost, whereas 18 months ago the best models scored about 37% below the accountants' average.
OpenAI has launched MentalHealthBench, an expert-informed benchmark for evaluating how effective and safe AI responses are in real-world mental health conversations.
The UK AI Security Institute (AISI) has begun using EvalEval's infrastructure to publicly share evaluation results, including verified results for five benchmarks—HealthBench, FrontierMath, Humanity's Last Exam, SWE-Bench Pro and Terminal-Bench 2.0—as well as two cyber evaluations, Cyber CTFs and The Last Ones.
Hugging Face has launched BenchMIRT, a multidimensional IRT-based method that audits what capabilities LLM benchmarks actually measure at the level of individual prompts.
Voice Arena and Hugging Face released four Monsoon evaluation sets for the Open ASR Leaderboard, covering Indian English and Hindi, with Hindi the first non-European language on the multilingual leaderboard.
Why it matters: The original details the Monsoon sets' split structure, speaker coverage and regional analysis examples, helping readers understand sources of model differences beyond aggregate WER.
DiG-bench (Discovery in Games) has been released, a benchmark of 70 games for testing AI's ability to autonomously discover hidden rules through exploration; 21 of the games are publicly available.
Microsoft Research introduced the MindTopo benchmark, evaluating multimodal large models' ability to reason about and plan with topological concepts such as connectivity, closure, ordering, separation, and knots. Testing shows models perform markedly better on static image recognition than on interactive planning tasks, and both fall far short of human level, with errors mostly involving lost structural relations during planning or physically infeasible actions. The benchmark aims to provide a controlled diagnostic tool for research on agents in robotics and interactive environments.
Hume released the Real World VoiceEQ benchmark to evaluate the human quality of voice AI interactions, covering ASR, TTS, S2S, and speech understanding, assessing over 40 leading closed- and open-source voice models across 15+ dimensions and 60+ metrics.
Hugging Face launched the open-source benchmark ScarfBench (Self-Contained Application Refactoring Benchmark), which evaluates AI agents on enterprise Java cross-framework migration tasks across the Spring, Jakarta EE and Quarkus ecosystems.
In the paper 'Forest vs Tree: The (N, K) Trade-off in Reproducible ML Evaluation,' Google Research studies the trade-off in AI evaluation between the number of annotations per item and the number of items annotated, and open-sources a simulator based on real datasets.