August 14–20: 9 frontier papers out of 1,876 read.
Papers rated as moving the field
How we choose“Our results reveal that many market-validated workflows remain beyond the reliable capabilities of current general-purpose agents, establishing StartupBench as an empirical measure of progress toward E2E completions of real-world user tasks.”
The authors, in the abstract
Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows
“We show that critique word limits (effective up to 150 words) successfully balance the game and avoid judge hacking, though this introduces a trade-off by restricting critic expressive clarity.”
“Evaluation of state-of-the-art open-source and closed-source MLLMs reveals that even the best-performing model, Gemini 2.5 Pro, achieves only 71.8% macro-average accuracy, remaining 22.4 percentage points below the corrected human baseline of 94.2%.”
“Across representative skill evolution methods, SafeEvolve reduces unsafe retrieval and fresh-session harm by 26.7 and 17.3 percentage points, respectively, while mean benign utility changes by only 0.4 points.”
“Extensive experiments show that Omni-LiveAvatar generates high-quality, synchronized minute-level avatars in real time.”
Aug 14–20: 161 cleared, 9 frontier
The week’s papers have not been sorted into fields yet.
Papers this week that name the model
Papers this week with an author at the lab
152 papers