“We tested several leading frontier models and find that the strongest models achieve a success rate of 44.5%, which leaves substantial room for future improvements.”
Benchmark
“To improve real-world applicability, MotuBrain introduces a unified multiview representation, explicit language-action coupling, and an efficient inference stack, achieving over 50x speedup for real-time deployment.”
“AI assistance also reduced median diagnostic time by 18.3%, increased diagnostic confidence by 9.0%, and improved inter-rater agreement from moderate (kappa = 0.55) to substantial (kappa = 0.76).”
“We demonstrate statistical significance of these improvements and robustness under up to 30% supervisory noise, albeit with a trade-off in peak observed reward compared to unconstrained baselines.”
“We find that current agents remain far from reliably surpassing human-designed methods, and that engineering-style tuning is easier for them than genuine method invention.”
Benchmark
Benchmark
“Experiments reveal false green verification, where malicious code modifications pass public tests while silently defeating security checks.”
“We show the latter.”