June 5–11: 8 frontier papers out of 1,890 read.
Papers rated as moving the field
How we choose“Across frontier model-runtime pairings, the best PassRate reaches only 41.2%, showing the benchmark remains far from saturated.”
The authors, in the abstract
A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
“Privileged vision+XML access improves frontier models by up to 26 percentage points, while smaller models do not benefit from added accessibility-tree input.”
“The headline contrast lives in the hard regime: on SWE-bench Lite, the firewall reduces fabrication from 33.7% (StateFlow) to 0.67%, a paired difference of -33.07 pp [95% CI -36.53, -29.73].”
“On the out-of-distribution MMEVOKE benchmark, CRANE reaches 87.0% under gold retrieval.”
“We find that these systems leave headroom for improved continual learning: agents frequently overfit to immediate observations or fail to reuse knowledge across instances, and dedicated memory systems do not fix this -- in fact, naive ICL outperforms systems dedicated to memory management.”
“In this paper, we demonstrate generalization hacking, in which a model collects reward during RL while preventing the rewarded behavior from generalizing.”
“First, we show that evaluation awareness persists even when we tell models they are deployed, which contradicts the scheming story which predicts less misalignment when the model perceives evaluation.”
Jun 5–11: 168 cleared, 8 frontier
The week’s papers have not been sorted into fields yet.
Papers this week that name the model
Papers this week with an author at the lab
160 papers