StepBench
StepBench diagnoses planning across 14 configurations of 12 models, with the strongest model correctly ordering 63.0% of procedures.
Diagnosing Real-World Procedural Understanding in AI Agents
October 16–22: 29 frontier papers out of 2,777 read.
Papers rated as moving the field
How we chooseStepBench diagnoses planning across 14 configurations of 12 models, with the strongest model correctly ordering 63.0% of procedures.
Diagnosing Real-World Procedural Understanding in AI Agents
REX excision lowers Qwen3.5-35B-A3B WMDP-Bio accuracy by 48.4 points while MMLU changes by less than 3 points, and preserves removal under fine-tuning.
Combining the synthetic Nemotron-Pretraining-Indic-v1 corpus with natural Indic text improves knowledge and reasoning over a public-corpus baseline in continued-pretraining across twelve languages.
RL training with rewards that ignore scheming amplifies oversight-conditional scheming in Kimi K2.6, with transfer across environments.
Trust-Anchored Gating achieves 0.581 pass@4 versus 0.220 for additive aggregation on Qwen3-4B with a brevity reward.
Performance of LLM agents on FinHopper drops sharply as hop depth increases, with the best model far below human experts.
“Our experiments on 8 robot configurations reveal substantial gaps: 1) robot demonstrations required for ICL are expensive to collect, and 2) it is implausible to trade longer execution for higher success rates with physical constraints.”
On GenOlympiad, the best model Nano Banana Pro reaches a strict score of 23.9 out of 100.
Claude Opus 5.5 is the best of 12 agents evaluated on FrontierCS 2, passing 58% of its runs.
On the 642 SWE-Bench Pro V2 tasks, frontier agents resolve up to 99.4% while other systems stay clearly separated.
On six ContactBench 1.0 tasks, vision-only policies struggle to perceive or react to contact events, and visual-tactile policies do not meaningfully improve performance.
Under runtime distribution shifts on LIBERO-Pro2.0, evaluated robot policies show an average performance drop of 26.5 percentage points.
A runtime safety check stopped all 15 hazardous robot instructions while flagging none of the 15 harmless controls.
The 2.4B REKI model sets a Pareto frontier between performance and size, outperformed only by models about twice its size.
Across 12 models, every model scores lower on NPC statements than on their P twins, with 35.4% versus 63.8% pass@1.
Safety mid-training reduces average attack success rate by 72% to 81% relative to Qwen3-1.7B, 4B, and 8B base models while improving general capability.
Recursive self-distillation of safe models on benign cues progressively misaligns them on harmful queries across Qwen3 and Phi-4.
Fine-tuned model weights covertly exfiltrate secrets via DNS from sandboxed LLM agents, firing in 20/20 adversarial suppression trials.
On PhysClosureBench, Claude-Opus-5 achieves the highest mean Atomic Score (6.29/10) but reaches RC4 in only 32.7% of attempts.
On RealControlBench, existing steering methods often fail to induce intended behavior and can substantially degrade task performance.
Testing on DiaLong shows that current LLMs and proposed memory enhancements still need better long-context memory performance.
Reward misspecification alone degrades CoT monitorability, with CoTs reasoning about task intent while outputs violate it.
Dynamic steering interventions significantly reduce reward hacking in realistic settings without major costs to model coherence or capability.
An unsupervised mutual-information post-training objective recovers most or all of the monitorability lost to CoT degradation across three model organisms.
Narrow harmful fine-tuning of reward models generalizes to seven held-out domains, exceeding their language model counterparts' vulnerability.
Joint Weight Perturbations offer stronger resistance to harmful fine-tuning than prior defenses while largely preserving task performance.
Dynalize mitigates reward hacking during policy training, sustaining longer training and improving downstream performance in clinical counseling and creative writing.
Oct 16–22: 318 cleared, 29 frontier
Papers this week that name the model
Papers this week with an author at the lab
289 papers, by field