dorsal/arxiv
View SchemaProof of Time: A Benchmark for Evaluating Scientific Idea Judgments
| Authors | Bingyang Ye, Shan Chen, Jingxuan Tu, Chen Liu, Zidi Xiong, Samuel Schmidgall, Danielle S. Bitterman |
|---|---|
| Categories | |
| ArXiv ID | 2601.07606vv1 |
| URL | https://arxiv.org/abs/2601.07606 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Large language models are increasingly being used to assess and forecast research ideas, yet we lack scalable ways to evaluate the quality of models' judgments about these scientific ideas. Towards this goal, we introduce PoT, a semi-verifiable benchmarking framework that links scientific idea judgments to downstream signals that become observable later (e.g., citations and shifts in researchers' agendas). PoT freezes a pre-cutoff snapshot of evidence in an offline sandbox and asks models to forecast post-cutoff outcomes, enabling verifiable evaluation when ground truth arrives, scalable benchmarking without exhaustive expert annotation, and analysis of human-model misalignment against signals such as peer-review awards. In addition, PoT provides a controlled testbed for agent-based research judgments that evaluate scientific ideas, comparing tool-using agents to non-agent baselines under prompt ablations and budget scaling. Across 30,000+ instances spanning four benchmark domains, we find that, compared with non-agent baselines, higher interaction budgets generally improve agent performance, while the benefit of tool use is strongly task-dependent. By combining time-partitioned, future-verifiable targets with an offline sandbox for tool use, PoT supports scalable evaluation of agents on future-facing scientific idea judgment tasks.
{
"annotation_id": "e46db89a-2f91-4b4a-94a6-b425f7d6e7d7",
"date_created": "2026-02-17T05:53:12.389000Z",
"date_modified": "2026-02-17T05:53:12.389000Z",
"file_hash": "0486f8159f3558b468a50833c0778d5060875dc25cd20491737038be5119b8b2",
"private": false,
"record": {
"abstract": "Large language models are increasingly being used to assess and forecast research ideas, yet we lack scalable ways to evaluate the quality of models\u0027 judgments about these scientific ideas. Towards this goal, we introduce PoT, a semi-verifiable benchmarking framework that links scientific idea judgments to downstream signals that become observable later (e.g., citations and shifts in researchers\u0027 agendas). PoT freezes a pre-cutoff snapshot of evidence in an offline sandbox and asks models to forecast post-cutoff outcomes, enabling verifiable evaluation when ground truth arrives, scalable benchmarking without exhaustive expert annotation, and analysis of human-model misalignment against signals such as peer-review awards. In addition, PoT provides a controlled testbed for agent-based research judgments that evaluate scientific ideas, comparing tool-using agents to non-agent baselines under prompt ablations and budget scaling. Across 30,000+ instances spanning four benchmark domains, we find that, compared with non-agent baselines, higher interaction budgets generally improve agent performance, while the benefit of tool use is strongly task-dependent. By combining time-partitioned, future-verifiable targets with an offline sandbox for tool use, PoT supports scalable evaluation of agents on future-facing scientific idea judgment tasks.",
"arxiv_id": "2601.07606",
"authors": [
"Bingyang Ye",
"Shan Chen",
"Jingxuan Tu",
"Chen Liu",
"Zidi Xiong",
"Samuel Schmidgall",
"Danielle S. Bitterman"
],
"categories": [
"cs.CL",
"cs.AI"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "Proof of Time: A Benchmark for Evaluating Scientific Idea Judgments",
"url": "https://arxiv.org/abs/2601.07606",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "2f668a07-968a-4170-8933-c4499e9cde81",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}