REFUTE
github.com/connerlambden/refute-inspectREFUTE is an open benchmark for scientific critique honesty and epistemic calibration on recent life-science and biomedical literature. It tests whether models keep claims inside what the evidence allows (overclaim / planted-flaw / falsifier selection) and whether stated confidence is calibrated, with judge-free MCQ axes plus open-ended critique scoring.
Sourced from
- GitHub — github.com/connerlambden/refute-inspect
- bio.tools — bgpt-refute
Related resources
A benchmarking platform for molecular generation models.
Large-scale benchmark suite for protein fitness prediction and design, aggregating 200+ deep mutational scanning assays and clinical variant datasets across diverse protein families and taxa, with standardized zero-shot and supervised leaderboards for variant effect prediction, mutation effect prediction, and protein language model evaluation (OATML & Marks Lab, NeurIPS 2023 Spotlight, Datasets & Benchmarks)
Scientific machine learning benchmarks & differential equation solvers
Unified benchmarking framework for protein representation learning, providing standardized interfaces for pre-training and diverse downstream tasks including structure prediction, fitness prediction, and property prediction across multiple protein datasets and model architectures (ICLR 2024, 273+ stars, MIT License)
Research coding benchmark curated by scientists with 338 subproblems across 16 subdomains (physics, math, materials, biology, chemistry), evaluating LLMs on realistic scientific programming tasks with gold-standard solutions (NeurIPS 2024)