🔍 Read the full analysis: Can AI Spot What’s New In A Research Proposal? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Ai2 researchers report that six large language models gave unstable judgments about whether scientific ideas were novel. In one pairwise test, changing a few prompt words shifted accuracy by more than 50 percentage points; retrieval and additional reasoning did not consistently improve results. The team released a benchmark, data, code and a pipeline, while warning that some novelty labels are imperfect.
Ai2 researchers report in the original analysis that six large language models gave sharply different judgments about the novelty of scientific ideas when researchers changed evaluation choices, including prompt wording. In one pairwise experiment, a few altered words shifted accuracy by more than 50 percentage points, raising questions about using model scores to screen research proposals. The team released a benchmark, data, code and an automated pipeline for further testing.
The study evaluated GPT-5.1, GPT-5.2 and GPT-5.4, along with Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. The models assessed ideas through both absolute novelty ratings and pairwise comparisons. The researchers say adding literature retrieval or allowing more reasoning did not consistently improve the judgments. The reported results point to sensitivity to evaluation design, rather than establishing that every model or setup will fail.
For one benchmark set, the team drew on ICLR 2026 peer reviews to identify ideas with explicit reviewer comments about originality. The high-novelty group comprised 154 ideas from accepted papers in the top-rated 10% of their research areas, selected when reviewers praised novelty and none disputed it. The lower-novelty group comprised 145 ideas from rejected papers in the bottom 10%, selected when reviewers raised originality concerns and none praised it.
An annotator agreed with the direction of the extracted novelty signal in 47 of 50 checked cases; the authors say none of the three disagreements changed a review’s classification. A separate test paired the 154 human-authored ideas with 154 proposals from a basic language-model generator. The researchers treated generated proposals as less novel, but describe those labels as weak and not individually verified. In an expert check of 30 pairs, selected heavily from cases the strongest judge got wrong, the expert preferred the human idea in every pair and judged none of the generated ideas novel. The authors also report that two purpose-built novelty systems performed worse than a simple prompted Claude Opus 4.6 baseline.
Why Proposal Screening Needs Stable Scores
Research groups may use language models to sort candidate ideas before spending time on literature reviews, experiments or expert assessment. If a small prompt change can substantially alter a verdict, a score may reflect the chosen evaluation setup as much as the idea’s originality. That makes rankings difficult to compare across research teams or projects unless the testing method is reported and shown to be reliable.
The findings also matter to efforts to automate parts of scientific discovery. Systems that generate hypotheses or synthesize papers need a credible way to judge whether a proposal adds something not already known. The study does not show that all AI novelty assessments are unusable, nor does it establish that models cannot help researchers. It does show why a strong result on one selected prompt or benchmark is not, by itself, evidence of dependable judgment.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
Logitech MX Master 4 for Mac, Ergonomic Bluetooth Mouse with Haptic Feedback, Ultra-Fast Scrolling, USB-C Charging, for MacOS, iPadOS, Windo
LG UltraGear 27-Inch QHD OLED Gaming Monitor, 240Hz, 0.03ms, G-SYNC Compatible, FreeSync Premium Pro, HDMI 2.1
As an affiliate, we earn on qualifying purchases.
How Ai2 Constructed Its Novelty Tests
A novelty judge is a model asked to assess whether a research idea is new relative to prior work, either by rating it alone or comparing it with another idea. The authors say there is no standard evaluation recipe: teams can choose different models, prompts, reasoning budgets, comparison formats and literature-retrieval systems. They argue that judges are often tested on human-written papers even when the intended use is to assess AI-generated proposals.
The benchmark separates ideas supported by peer-review comments about originality from a second setting that includes model-generated proposals. The paper cautions that acceptance or rejection alone does not establish novelty, since papers may succeed or fail for reasons other than originality. Reviewer statements provide a useful signal for constructing a test set, but do not amount to a definitive measure of whether an idea is new.
“Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.”
— Ai2 study authors
Limits of the Novelty Labels
The benchmark labels have limits. Reviewer comments and paper rankings are indirect evidence of relative novelty, not a conclusive standard. In the human-versus-generated set, the researchers assumed the basic model-generated proposals were less novel, but did not verify each proposal individually. The expert review covered 30 pairs, not the full benchmark, and its cases were selected heavily from mistakes made by the strongest judge.
The study description does not provide full results for every model, prompt, reasoning budget or evaluation format. It also does not establish whether the findings carry over to other research fields, different idea generators or systems with broader access to scientific literature. The source material says some conclusions rely on disagreement between evaluation setups rather than benchmark labels, but does not detail how broadly that result holds. No specific follow-up study or schedule is announced.
Further Tests Across Research Fields
Ai2 has made the paper, benchmark data and code available, along with a pipeline intended to help researchers build additional evaluation sets. Other teams can use those resources to test whether prompt sensitivity persists across disciplines, proposal sources and judge designs, and to compare approaches aimed at making results more consistent.
For now, the release offers a way to examine how evaluation choices affect novelty scores; it does not identify a proven combination of prompts, retrieval and model design that produces dependable judgments. The study materials provided do not name a next milestone or publication date. The practical question is whether independent tests can confirm the instability and find methods that keep assessments consistent without treating imperfect benchmark labels as ground truth.
Key Questions
What did the Ai2 study find?
Researchers report that six language models gave unstable novelty judgments across evaluation setups. In one pairwise test, changing a few prompt words shifted accuracy by more than 50 percentage points.
Which models were tested?
The study tested GPT-5.1, GPT-5.2 and GPT-5.4, as well as Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6.
Did literature retrieval or extra reasoning improve the scores?
The authors report that adding literature retrieval or giving models more reasoning did not consistently improve their novelty judgments. The supplied results do not provide a full breakdown of every tested configuration.
Are the benchmark labels definitive?
No. The human-idea set uses reviewer comments and paper rankings as signals, not a conclusive measure of originality. The researchers also describe the generated-idea labels as weak and not individually verified.
What did Ai2 release?
Ai2 released the study, benchmark data, code and an automated pipeline intended to support further evaluation and the creation of additional test sets.
Primary source: Hugging Face · via ThorstenMeyerAI.com
