RecursionBench
How good a model can you build, and at what cost?
The task
You have an environment with the usual stack (CUDA, drivers, common ML libraries), internet access, a data corpus, and 10,000 B200 GPUs. Your task is to create an AI model together with its full training and post-training pipeline, not necessarily a transformer. Use the provided corpus, download more, or generate data. The data and compute are enough for an ~8B-class model.
Your model is then run against 25 modern benchmarks, producing a performance line. Everything you spend is captured: GPU-hours to set up and to train, wall-clock time, amount of data, and more. Improve along one or more axes without decreasing on the others.
What is measured
- GPU-hours for setup, experimentation and architecture search
- GPU-hours to train and post-train the model
- GPU-hours to run the 25 benchmarks
- Wall-clock time to set up and produce the model
- Amount of data used (corpus plus anything downloaded)
- Percentage of training data that was generated
- The performance line: score on each of the 25 benchmarks
Full list: MEASURED.md
The 25 benchmarks
Modern public benchmarks, known in advance. What is fixed is the corpus and the compute, not the questions.
- MMLU-Pro
- MMLU
- GPQA Diamond
- Humanity's Last Exam
- BBH
- MuSR
- HellaSwag
- FrontierMath
- AIME 2026
- MATH-500
- GSM8K
- SWE-bench Verified
- Aider Polyglot
- LiveCodeBench
- SciCode
- HumanEval
- MBPP
- BFCL
- TAU-bench
- GAIA
- MMMU
- ARC-AGI-2
- DROP
- IFEval
- Chatbot Arena
Full list: BENCHMARKS.md
The performance line