Ai2 releases LLM novelty benchmark Old Ideas, Novel Problems
Understand the story
On October 28, 2026, Ai2 (Allen Institute for AI) released the research and benchmark Old Ideas, Novel Problems on Hugging Face, aiming to test the reliability of large language models as judges of scientific idea novelty. This release is a first-hand source, including the accompanying benchmark, data, and code. The study tested the performance of six LLMs: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6. Currently, reporting only discloses the benchmark's release contents and the list of tested models; no specific evaluation results or conclusive data have been provided, and the story is still being followed.
AI overview of reporting · Updated 2 h前
Reporting timeline
Explore different aspects of the story through its reporting.
- Hugging Face BlogCan we trust LLMs to judge whether a research idea is novel?
Ai2 released the study Old Ideas, Novel Problems on Hugging Face, along with a companion benchmark, data, and code, testing the reliability of six LLMs (GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6) as novelty judges.
Story heat trend
Not enough continuous observations to show a trend yet.