MKA
On the MKA benchmark, Claude Fable 5.2 achieves 56.6% Climb-WSR, with Gemini 4 Carbon close behind.
Evaluating Multimodal Knowledge Agents on end-to-end professional tasks
Friday, November 20: 5 frontier papers out of 449 read.
Papers rated as moving the field
How we chooseOn the MKA benchmark, Claude Fable 5.2 achieves 56.6% Climb-WSR, with Gemini 4 Carbon close behind.
Evaluating Multimodal Knowledge Agents on end-to-end professional tasks
Three vision-language-action models evaluated on PhyDEX all achieve success rates below 3%, with ground-truth physical properties roughly doubling success.
Keel's weight edits remove most loops on Qwen3.5-0.8B and Qwen3.5-2B across six image benchmarks and raise average accuracy.
Beam refuses or pivots on 88.9% of harmful requests while complying with 81.5% of harmless requests, averaged across conversation turns and harm categories.
Verifying AI agent safety is undecidable in general and PSPACE-complete even under finite constraints.
Nov 14–20: 339 cleared, 18 frontier
Papers this week that name the model
Papers this week with an author at the lab
78 papers, by field