June 6–12: 9 frontier papers out of 1,915 read.
Papers rated as moving the field
How we choose“Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated.”
The authors, in the abstract
A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
“Privileged vision+XML access improves frontier models by up to 26 percentage points, while smaller models do not benefit from added accessibility-tree input.”
“We evaluate 19 open-weight and proprietary LLMs, and find that current models achieve penetration success rates ranging from 10.7% to 69.3%.”
“The headline contrast lives in the hard regime: on SWE-bench Lite, the firewall reduces fabrication from 33.7% (StateFlow) to 0.67%, a paired difference of -33.07 pp [95% CI -36.53, -29.73].”
“On the out-of-distribution MMEVOKE benchmark, CRANE reaches 87.0% under gold retrieval.”
“Our results reveal a critical vulnerability in post-training: alignment can be overridden by a single example.”
“In this paper, we demonstrate generalization hacking, in which a model collects reward during RL while preventing the rewarded behavior from generalizing.”
“First, we show that evaluation awareness persists even when we tell models they are deployed, which contradicts the scheming story which predicts less misalignment when the model perceives evaluation.”
Jun 6–12: 185 cleared, 9 frontier
The week’s papers have not been sorted into fields yet.
Papers this week that name the model
Papers this week with an author at the lab
176 papers