June 27 – July 3: 14 frontier papers out of 2,161 read.
Papers rated as moving the field
How we choose“The strongest and the most cost effective agent, Codex GPT-5.5, achieves only approximately 42% success rate.”
The authors, in the abstract
A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents
“Even the strongest frontier system in its default reasoning mode reaches only about 59.3% on Domain Q increasing thinking depth within a tier does not move performance materially, while moving to a higher product tier yields modest gains.”
“Across five model pairs spanning four model families, we find that 91.3% of reasoning-mode gains are knowledge-dependent rather than structure-invariant (63/69 gains; Wilson 95% CI [82.3%, 96.0%]), directly challenging the assumption that chain-of-thought reasoning improves short-horizon procedural scientific problem-solving.”
“Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks.”
“All ASR claims are 3-judge-calibrated (cell rankings stable across judges; absolute levels \pm18pp; see App. B.13).”
“Human-as-Humanoid yields a 4.8--7.2x raw demonstration-throughput gain over humanoid teleoperation in our data-collection analysis, and on several downstream tasks, policies post-trained only with the converted human labels generalize to real-robot deployment without target-task robot demonstrations.”
“Our extensive online inference experiments reveal a stark reality: state-of-the-art MLLMs struggle significantly with concurrent streams, achieving only about 50% score and exhibiting poor proactive ability.”
“Extensive evaluations show that MMBench-Live preserves stable model rankings, maintains semantic alignment with the original benchmark, and exhibits weaker contamination-related memorization signals, suggesting a practical and scalable paradigm for sustainable multimodal benchmark evolution.”
“Experiments reveal a substantial reliability gap: agents that perform well with reliable tools often fail under recoverable hazards.”
“Experiments show the scalability of the proposed paradigm and verify that stronger world latent enables stronger downstream readouts.”
Jun 27 – Jul 3: 150 cleared, 14 frontier
The week’s papers have not been sorted into fields yet.
136 papers