June 3–9: 12 frontier papers out of 1,754 read.
Papers rated as moving the field
How we choose“On Lean-IMO-Bench, LEAP boosts the one-shot formal solve rate of general-purpose LLMs from below 10% to 70%, notably surpassing the 48% benchmark set by a specialized, gold-medal-caliber IMO system.”
The authors, in the abstract
Supercharging LLMs for Formal Mathematics with Agentic Frameworks
“Oracle-plan experiments further show that, when plan quality is controlled, the trained duplex model produces high-quality guidance and large gains on Out-of-Plan recovery.”
“Privileged vision+XML access improves frontier models by up to 26 percentage points, while smaller models do not benefit from added accessibility-tree input.”
“Across 61 checkpoint rows and 1920 row-level transitions, aggressive PPO has the highest localized reward-hacking rate (14.45%; bootstrap 95% CI: 10.16-18.75), while UP-PPO yields lower rates in the same aggressive regime (11.33-10.94%).”
“On the out-of-distribution MMEVOKE benchmark, CRANE reaches 87.0% under gold retrieval.”
“To study this phenomenon, we introduce SocioHack, a sandbox of 72 societal environments, and find that within these environments, reward hacking naturally emerges and leads to regulatory loophole discovery.”
“To evaluate reward-model robustness, we introduce RewardHackBench containing 13 reward-hacking patterns covering real life high-stakes domains and general settings, and we find severe failures on specific subcategories across eight reward models.”
“We find that these systems leave headroom for improved continual learning: agents frequently overfit to immediate observations or fail to reuse knowledge across instances, and dedicated memory systems do not fix this -- in fact, naive ICL outperforms systems dedicated to memory management.”
“First, we show that evaluation awareness persists even when we tell models they are deployed, which contradicts the scheming story which predicts less misalignment when the model perceives evaluation.”
“Our results demonstrate that self-sustaining AI-driven cyber-threats are no longer theoretical.”
Jun 3–9: 145 cleared, 12 frontier
The week’s papers have not been sorted into fields yet.
Papers this week that name the model
Papers this week with an author at the lab
133 papers