Static benchmarks leak into training data within months, and leaderboards drift toward memorisation. A research group has launched a rolling benchmark that is regenerated every week from fresh source material, with full disclosure after the fact.
What the first weeks showed
Several models that topped public leaderboards fell by double digits on the rotating set. The gap between claimed and measured capability was largest on reasoning tasks and smallest on summarisation. Labs have started reporting both numbers.
Key takeaways
- Questions are regenerated weekly from new material
- Leaderboard leaders dropped sharply on fresh questions
- Labs now report static and rolling scores
This story is part of our ongoing coverage of ai. We will update it as more details emerge.
