July 2–8: 14 frontier papers out of 1,858 read.
Papers rated as moving the field
How we choose“Even the strongest frontier system in its default reasoning mode reaches only about 59.3% on Domain Q increasing thinking depth within a tier does not move performance materially, while moving to a higher product tier yields modest gains.”
The authors, in the abstract
“Across five model pairs spanning four model families, we find that 91.3% of reasoning-mode gains are knowledge-dependent rather than structure-invariant (63/69 gains; Wilson 95% CI [82.3%, 96.0%]), directly challenging the assumption that chain-of-thought reasoning improves short-horizon procedural scientific problem-solving.”
“The same stream also quantifies the failure modes: a loose guard silently mislabels 48.9% of compiled answers, and an unfaithful deopt reference causes the verification gate to refuse recompilation.”
“An adversarial evaluation of monitors of increasing semantic depth shows false-allows falling from 75% (name-based gating) to zero (dynamic effect tracing), and refusal history transfers compliance to held-out red-line families.”
“Our evaluation also shows that SENTINEL reduces FARMA's attack success rate to as low as 0% with no false positives observed across 326 benign agent traces.”
“All ASR claims are 3-judge-calibrated (cell rankings stable across judges; absolute levels \pm18pp; see App. B.13).”
“Extensive evaluations show that MMBench-Live preserves stable model rankings, maintains semantic alignment with the original benchmark, and exhibits weaker contamination-related memorization signals, suggesting a practical and scalable paradigm for sustainable multimodal benchmark evolution.”
“Extensive experimental analysis reveals substantial performance disparities across evaluation protocols, demonstrating that navigation success alone is an insufficient predictor of real-world application effectiveness.”
“We show it fails structurally: conditioned on a candidate, a judge scores plausibility, not correctness, leaving false-positive basins a policy learns to exploit.”
Jul 2–8: 124 cleared, 14 frontier
The week’s papers have not been sorted into fields yet.
Papers this week that name the model
Papers this week with an author at the lab
110 papers