Reward Hacking Challenges Oversight of Autonomous Research Agents
“An LLM panel reviewing only submitted code and reported scores misses 33/505 confirmed hacks (6.5%).”
The authors, in the abstract
AI Research Radar
September 25 – October 1: 23 frontier papers out of 3,885 read.
Frontier
Papers rated as moving the field
How we chooseKilometerVision
“Instead of utilising true path integration or forming geometric survey knowledge, we find that VLMs rely almost entirely on 2D visual recognition and text-matching to bypass complex spatial reasoning.”
Turning Speech Language Models into Multilingual Listeners
“We compare the performance of a cascading system to open-weight and closed SLMs on MULTISPEECH-BENCH and find that the cascading system outperforms open-weight SLMs but not all closed SLMs.”
RMB
“Our extensive experiments demonstrate that RMB significantly improves reward accuracy on both in-distribution and out-of-distribution datasets, substantially mitigating the reward hacking issue and ultimately improving RLHF performance.”
Safety Reconstructed
Pistis Technical Report
“Experiments demonstrate that PAH enhances the model performance without updating the model parameters or increasing the interaction budget.”
Evaluating Whether GPT-6 Astra Performs Unsanctioned Supply-Chain Attacks
“Applying our methods to GPT-6 Astra and previous OpenAI models, with cyber safeguards disabled, we find that GPT-6 Astra attempts complete supply-chain attacks in simulation at a higher rate than GPT-5.6 Sol and GPT-5.5.”
E2E-SWE
“Evaluating 13 frontier models, we find substantial variation in end-to-end repository generation ability, with pass@1 ranging from 11.7% to 67.7%, providing strong model differentiation while leaving considerable headroom for future progress.”
AgentWorld
“Experiments with Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, and DeepSeek R1-70B show that even the best model achieves only 52.0% task success, with systematic failure modes including communication breakdowns, role confusion, and inability to maintain shared plans across rounds.”
EngiWorld
“Evaluation of seven frontier models reveals a substantial capability gap: the strongest model achieves an EngiScore of only 44.3, and just 3.6% of multi-software attempts succeed.”
RE-0
“Experiments across multiple Code-as-Policy embodied tasks show that RE-0 improves both teacher-assisted execution and the standalone student, and generalizes to novel robots and scenes.”
PROACT-Agent
“In AgentDojo, it reduces non-DoS targeted attack success from 20.82% to 0.40%.”
Instrumental Monitor Evasion Emerges Under Ordinary Task Pressure
“Across our evaluations, best-of-3 evasion attempt rates reach up to 98% and success rates up to 88%, with substantial variance across models.”
OSWorld-Science
“Our results show that current state-of-the-art VLMs with a strong harness still face challenges in addressing key questions in the scientific domains.”
CheatBench
Towards Mitigating Deceptive Safety Alignment in Large Reasoning Models
“Experiments show that SARA significantly mitigates deceptive safety alignment under both standard and adversarial settings while preserving helpfulness and utility.”
Magic-W0
Waypoint-1.5
Despite Instructions
“We find that model pairs can learn to communicate the secret using only one bit of feedback indicating whether the receiver inferred it correctly.”
Distillation Defenses Easily Break After Reinforcement Learning
“We show that simple attacks can steal reasoning capabilities from existing closed-source language models using data easily obtainable from current APIs, yielding reasoning improvements equivalent to more sophisticated attacks that extract the full hidden traces.”
InternW0-Δ
Codetta
LLM Agents Can Easily Tamper With Their Own Traces
“We show that local LLM agents such as Claude Code, Codex, Antigravity, Open Code and Grok Build fail to enforce this boundary.”
The week by field
Sep 25 – Oct 1: 366 cleared, 23 frontier
- Security146◆ 2
- Safety84◆ 10
- Agents38◆ 5
- Language23◆ 1
- Inference18
- Vision15◆ 2
- Robotics14◆ 2
- Society14
- 4 moreML, Science, Training, Reasoning14◆ 1
- Security146◆ 2
- Safety84◆ 10
- 8 moreInference, Vision, Robotics, Society, ML, Science, Training, Reasoning75◆ 5
- Agents38◆ 5
- Language23◆ 1
Size: cleared papers◆ and tint: frontier papers
Models under test
Papers this week that name the model
Labs publishing
Papers this week with an author at the lab
Also cleared
343 papers, by field
Security & privacy144
- Hiding in Plain Sight: Decoupling Pretext from Actuation for Skill Poisoning in LLM AgentsBaiduCode
- Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix itIBMCode
- Anchor-ECC: Local Integrity Checking for Watermarked LLM Outputs via Error-Correcting CodesGoogle
- Preemptive LLM Unlearning against Forbidden Capability Acquisition via Gradient SealingGoogle
- CounterSteer: Suppressing Indirect Prompt Injection with Activation SteeringMicrosoft
All 144 papers in Security & privacy
- Controlled Decoding Attacks on Black-Box LLMsAmazon
- TRAP: Understanding and Mitigating Privacy Memorization in Language ModelsNVIDIA
- On the Effectiveness of Kernel-Level Evidence for Agent SecurityAmazon
- SEW: Style-Encoded Watermarking of LLM-Generated CodeCode
- WARP: A Unified Benchmark for Invisible Image Watermarking -- Robustness and Protection Against AttacksCode
- Can Multimodal Large Language Models Generate and Detect Multimodal Social Media Fake News?Code
- MMSkillRisk: Can Agents Stay Safe When Multimodal Skills Become Traps?Code
- AutoMark: Enabling Autoresearch to Discover Better LLM WatermarksCode
- High-Capacity Robust Medical Image Exfiltration via Neural Network Weight ReplacementCode
- ORBIT: A Framework for Multi-Agent Safety and Security EvaluationsCode
- CertMark: Distortion-Free Multi-Bit Watermarking with Certified DecodingCode
- Climbing the Hill: Prompt Injection Red-Teaming Against Frontier Models with Curriculum Reinforcement LearningCode
- Integrity Detection and Characterization of Malicious Injections in RAVEN IICode
- Region-Level Black-Box Defense Against Stealthy Embedding-Space Backdoors in CLIPCode
- AEGIS: Audio Endogenous Guarding via Internal Signals Against Large Audio-Language Model JailbreaksCode
- T-Backdoor: Exploiting Temporal Redundancy in Neuromorphic Data for Spike-preserving Backdoor Attacks on SNNsCode
- Separation of Duties for Privileged LLM Agents: A Governed Execution Architecture with Measured Security-Utility Trade-offs
- Inference-Layer Security: Defending Against Adversarial Inference and Infrastructure Abuse
- ContractWarden: Kernel-Enforced Damage Boundaries for AI Agents via Human-Authorized Contracts
- VirusCascade: Hijacking Collaborative Reflection in LLM-Powered Recommender Agents
- Privacy in Personalized AI Is a System Property, Not Just a Model Property
- AI Agents are Vulnerable to Radicalization
- It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at ε ≤ 4/255
- Aegis: Generative Gradient Masking for Privacy-Preserving Medical Federated Learning
- PatchHolmes: Agentic Patch Retrieval via Listwise Selection
- SparLeak: Privacy Leakage from Sparse Attention in LLM Inference on Shared GPUs
- Forging LLM Authorship Fingerprints with Targeted Rewriting
- SceneJail: Exploiting Video Scenario Context to Jailbreak Multimodal LLMs
- APTInvestBench: Evaluating Autonomous APT Investigation under Varying Telemetry
- Approval Laundering: Systematizing Approval--Execution Binding Failures in AI Coding-Agent Harnesses
- Can Agents Trust Their Skills? Uncovering Unsafe Chains of Trust in Skill-Based LLM Agents
- SteerProbe: Learning to Bypass Safety Steering in Vision-Language Models
- Feature-Aware Token Attack for Compression-Triggered Stealthy Failures in Large Vision-Language Models
- MADBench: Benchmarking the Security of Multi-Agent Debate
- Exploiting Vulnerabilities: Universal Adversarial Attacks on Vision-Language-Action Models in Robotics
- Universal Cross-Prompt Adversarial Attacks on Promptable Concept Segmentation
- ActionGuard: Tool Call Authorization under Poisoned Skills
- Learning Normal Diffusion Dynamics for Backdoor Defense in Text-to-Image Models
- Speculative Safety Honeypot: Toward Proactive Defense Against Multi-turn Agent Attacks
- Pretext: Defeating Malicious Skill Detection Frameworks for AI Agents
- Typographic Attack Against VLM-based AI-generated Image Detection
- SEAR: Spoofing Evidence-Grounded Audio Reasoning Benchmark for Audio Language Models
- CodeMimicry: Exploiting Safety Generalization Lag in Large Language Models via Structured Code Completion
- TACTIC: Temporal and Context-Aware LLM Tactical Planning for Roadside LiDAR Attacks
- Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning
- Can We Still Trust Disaster Social Sensing? Empirical Evidence on Detecting AI-Generated Social Media Posts
- GenoTrace: Inheritable Watermarks for Genome Foundation Model Distillation
- GenomeOcean Anywhere: Private WebGPU Inference for Genome MoEs
- CipherGenome: Homomorphic Inference for Genomic Mixture-of-Experts
- Agentic Commerce Bench: Measuring Fraud Detection for Agents That Spend Money
- SINGED: Correct Outputs Do Not Certify Safe Execution in LLM Agents
- Similarity Is Not Validity: Defending LLM Semantic Caches Against Poisoning
- Cheap to Hypothesize, Costly to Verify: The Defense Surface of Agentic Vulnerability Discovery
- UNBIND: UNlearning By INference-time Directional Steering for Code LLMs
- Same Bytes, Different Authority: Reserved-Token Representations in Chat-Template Prompt Injection
- Render Before Reading: Visual Rendering as a Prompt Injection Defense
- Quantization Enables Private Dense Retrieval against Malicious Service Providers
- Eternal Sunshine of the Spotless Mind: Systematically Erasing LLM's Memories
- FinRT: Distilling Adaptive Red-Teaming Strategies into Reusable Adversarial Generators in Consumer Finance
- CyberPersistBench: Evaluating LLM-Based Cyber Attackers on Installation and Persistence
- Divide and Inject: Can Agents Reconstruct an Indirect Prompt Injection from Fragments?
- Self-Evolving Defense: Continual Security Policy Learning for LLM Agents
- Does the Unsafe Gradient Survive a Conversation? On the Fragility of Gradient-Based Jailbreak Detection in Multi-Turn Dialogue
- Safer Content or Firmer Refusals? A Hybrid Perturbation Defense for Alignment under Harmful Fine-tuning
- SKILLLITE: Evidence-Guided Malicious Skill Auditing with Compact LLMs
- Practical Secrets Extraction against Black-box LLMs
- ToolFence: Fine-Grained Authorization for Secure Tool-Using LLM Agents
- Backdoor Mitigation in Decentralized LLM Fine-Tuning
- Backdoor in the Loop: Compromising Agentic Search via Malicious Retrievers
- Concealing LLM-Based Multi-Agent Topology via Phantom Structure Injection
- Where Privacy Belongs: Placement Diagnosis and Certified Selection for Private Counterfactual Explanations on Graphs
- Where Do LLMs Decide to Break the Rules? Mechanistic Localization of Prompt Injection Compliance
- Selective Channel Restoration for Backdoored Vision-Language Models
- A Benchmark & Dataset for Detecting AI-Manipulated Visual Evidence in the Court System
- Dagger: Decoupling-based Model Stealing Attack against Graph Neural Networks
- Agents Can Use Base Models to Evade AI Detection
- A Large-Scale Benchmark and Risk Assessment of Traffic Analysis Attacks on Cloud LLM Services
- TokenScanner: Detecting Backdoors and Discovering Triggers in Text-to-Image LoRAs via Full Vocabulary Scanning
- Hardware-Rooted PUF Fingerprinting for Device-Level Traceability in Knowledge Distillation
- BMA: Backchain Memory Attacks Create Unauthorized Control Paths in LLM Agents
- Reading Is Not Leaking: Local, Auditable Measurement and Reduction of Inference Exposure from Public Footprints
- AnchorRep: Defending LLMs Against Cross-Model Adversarial Transfer via Representation Repulsion
- Trust the Brand, Lose Control: How Identity Hijacks LLM Agent Orchestration
- Ask Without Telling: Local SLMs Consult Cloud LLMs Without Revealing Task Intent
- Silent Failures in Agentic Security Evaluation: A Validated Harness for Tool-Call Mediation Under Indirect Prompt Injection
- Learning to Refer: Client-Resolved Generation for Privacy-Aware Language Models
- You Can't Spot a Deepfake?And Neither Can Your Brain Nor Eyes: A Neurophysiological Framework for Deepfake Exploitation of Cognitive Engagement and Implicit Visual Evaluation
- API Secrets Should Never Become Tokens in the LLM's Vocabulary: A Threat Analysis of API Credential Handling in LLM Agent Systems and an Empirical Evaluation of a Vault-Mediated Execution Boundary
- Concept Score Relearning: A Unified Cross-Architecture Attack on Concept Erasure
- Information Blackhole: Exploring Backdoor Mechanism in 3D Point Cloud Reconstruction
- RAISE: Reinforcing Access Control Policy Synthesis in LLMs via Symbolic Evaluation
- One Attack to Fool Them All: Highly Transferable Black-Box Adversarial Attacks on Frontier MLLMs
- Can Prompt Anonymity Protect Your Identity From LLM Providers?
- When Consent Outlives Context: Residual Authority Replay in Long-Lived Agents
- TrackFlood: Relocating Latency Attacks from NMS-Free Detectors to Real-Time Trackers
- The Privacy Fallacy of Crowdsourced Fine-Tuning: Extracting Proprietary Data via Topic-Based Poisoning
- DeMark: A Query-Free Black-Box Attack for Quality-Preserving Audio Watermark Removal
- LoRo-Mark:Provably Lossless and Robust Agent Watermarking
- From Attack Success to Attack Severity: Counterfactual Memory Attacks on LLM Agents
- Certified Multi-Source Integrity for Structured Agent Actions
- Making LLMs Truly Forget: Deep Unlearning by Searching, Selecting, and Severing Knowledge Paths
- Breaking Windows Malware Detection: A Comprehensive Evaluation of Problem-Space Adversarial Robustness
- CoDeL: Co-Evolutionary Defense against Indirect Prompt Injection in LLM-based Agents
- AuxMark: Defending Against Unauthorized Agent Distillation via Auxiliary Behavioral Watermarking
- Optimizing and Securing the Modern Watermarking Channel for Images
- DGF-Bench: A Benchmark for Simulating and Auditing Deception Against Multi-Agent Governance Boards
- RISE: Red-teaming via Iterative Strategy Evolution for Modern Text-to-Image Models
- When Valid Tool Calls Change Meaning: Formation-Consistent Dispatch for LLM Agents
- LENS: The Sum Is Worse Than the Parts for Set-Level Poisoning in Retrieval-Augmented Generation
- TANGO: Watermarking Masked Diffusion Language Models in Token Pairs
- Continuous Assurance of Agentic Security Auditors for Software Delivery Decision Gates
- "Nothing to See Here'': Unintended Disclosure through Revision Traces of LLM Deliverables
- Share-Borne AI Virus: Memory-Hopping Attacks Across LLM Agents
- Strategic Self-Consistency
- Stealth Apart, Harm Together: Skill Cascading Attacks on Skill-Based Agent Systems
- Containing Behavioral Cascades from Manipulated Claims in LLM-Powered Multi-Robot Systems
- Prompt Injection Detection for Email Agents Through Attack Chain Modeling
- AGATE: Provenance-Based Runtime Defense Against Compositional Attacks on LLM Agents
- FeatMark: Feature-level Watermark Protection against Mimicry Attacks with Diffusion Models
- MetaPermit: Scalable and Auditable Access Control for AI Agents via LLM-Inferred Meta-Attributes
- Cheap, open agents make LLM pollution harder to mitigate
- AuthGuard-R: Safety-Compliant Mission Hijacking and Dual-Gate Defense for LLM-Controlled Robots
- AgentXploit: Autonomous Repository-to-Runtime Red-Teaming for AI Agents
- FragToken: Amplifying LLM Inference Costs through Noncanonical Token Generation
- Privacy Leakage Through AI-mediated Analysis of Smartphone Data
- Who Is Behind the Harness? Fingerprinting LLMs through Agentic Behavior
- Persistent Billable State: Denial-of-Wallet Attacks and Defenses in Tool-Calling LLM Agents
- Progressive Skill Discovery as Access Control for Tool-Using LLM Agents: Structural Governance through Role-Scoped Capability Delivery
- DistillGuard: Malicious NPM Package Detection and API Attack Chain Analysis via Static Graph and LLM Distillation
- The Vulnerability of Neural Audio Watermarks under Speech Enhancement
- ClaimMirage: When Self-Claims in Domain Names Change LLM Threat Judgments
- Poster: FedWM-Guard: Thwarting Imagination Poisoning in Federated World Model-based Autonomous Driving
- TP-CRIV: A Framework for Third-Party Challenge-Response Identity Verification of AI Models
- AgentKernel: The Trust-Native Agentic Operating System
- OllamaDrama: Designing and Deploying a Honeypot to Measure Attacks on Exposed LLM Infrastructure
- Prefilling the Reasoning Channel: Output-Prefix Attacks on Reasoning LLMs
- Hard Stop: Kernel-Level Preemption and Containment for Rogue Agentic Execution
- PrivDrift: Auditing User-Secret Leakage Under Topic Drift in Active LLM Conversations
- Temporal Gradient Inversion for Private Trajectory Reconstruction in Embodied Reinforcement Learning
Safety & alignment74
- Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level RubricsByteDanceCode
- Covert Assistance: Helpful LLM Agents Evade Oversight in Multi-Agent SystemsNVIDIA
- ShieldCLIP: Selective Safety Alignment for Harmful Content Mitigation in Multimodal Foundation ModelsMeta
- HEAR: Real Voices, Real Bias: A Large-Scale Human-Recorded, Demographically Diverse Benchmark for Audio Language ModelsMeta
- Language Models Are "Insecure" ReportersGoogle
All 74 papers in Safety & alignment
- Constitutional adapters: Inference-time interventions for misalignment and misuseAnthropic
- Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge LeaderboardsGoogle
- CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RLCode
- ThinkingGuard: Decoding Implicit Hazards via Step-by-Step Risk Attribution in Multimodal Large Language ModelsCode
- Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and ConsistencyCode
- AdaGuard: An Adaptive Guard Model with User-defined PoliciesCode
- Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL PipelineCode
- Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment FailuresCode
- Framing the Narrative: Ideological Mimicry in Large Language Models
- When Does Randomized Oversight Align AI Agents That Can Conceal?
- Janus: Evidence-Before-Effect Sagas and Offline-Verifiable Provenance for Agentic LLMs
- Unlearning Deceptive Behaviors in LLMs with Contrastive Forget Sets
- Faithful Dual-constrained Erasure for Robust LLM Safety Alignment
- Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents
- Safety of Latent Communication in Multi-Agent Systems
- Learning Steganography Is Easy, Learning Steganographic Reasoning Is Hard
- LLM Persona Unlearning
- OpenAI-HuggingFace: A Reproduction & Lessons for Alignment Testing
- Tracing mechanisms of sycophantic agreement in language models
- Says Block, Still Acts: Why LLM Safety Judgments Fail to Govern Action in LLM Agents
- SameFact: The Same Safety Facts Lead to Different Responses Across Interfaces
- SaplingGuard: A Multidimensional-Profile-Aware Multi-Agent Guardrail for Developmentally Safe Adolescent-LLM Interaction
- Raising the Bar for Chinese Adolescent LLM Safety: A Culturally-Grounded, Fine-Grained Benchmark
- SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety
- CoRe: Co-Evolving Reward Models for Mitigating Latent Reward Hacking in Video Diffusion Models
- Training LLMs to Verbalize Evaluation Awareness
- LLMs Learn to Evade Latent Monitors from Prior Feedback Alone
- SafeCoEvo: Co-Evolving Safety Harnesses and Guards for LLM Agents at Test-Time
- Evaluating Bounded Autonomy in Regulated Agentic AI: A Diagnostic Harness with Constitutional Rewards, Escalation Labels, and Runtime Governance
- Weeding Out Bad Seeds: Initial-Noise-Robust Unlearning for Text-to-Image Diffusion Models
- Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable
- CompOrca: Corpus-Scale Compliance Labelling of Instruction-Tuning Data
- Can Vision-Language Models Stay Helpful When Facing Implicit Risks? Intent-Privilege OPSD for Efficient Safety-Helpfulness Alignment
- The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment
- Gender bias across LLMs is common and highly heterogenous
- Character Training for Risk-Averse Agents
- Contract monitoring: governing AI via separation of powers
- Reward Hacking and Agent Containment Failure: A Monte Carlo Study Based on the 2026 Hugging Face Incident
- "You're Right, Let Me Fix It": How LLM Agents Damage Correct Work When Falsely Accused
- Agent Safety From Within: Detecting Harmful Trajectories from LLM Internal States
- Reading Too Much into Context: Passive Exposure Can Steer LLM Decisions
- SafeMol: Dual-Modality Safety Alignment for Molecular Multimodal Models
- AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
- COGNIT-Guard: Calibrated Standalone Direct-Decision Guardrails with Heterogeneous CPU-NPU Confidence Cascading under Explicit Latency and False-Positive Constraints
- Learning Strategies to Break Judges
- Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation
- Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes
- How to Tame a Multi-Headed Hydra? Adaptive Multi-Category Safety Steering for Large Language Models
- Jailbreak Context Lingers: Divergent Safety Routing and Its Cross-Task Predictability in Tool Agents
- DeShortcut-Align: Decoupling Spurious Shortcuts for Robust Safety Alignment in Large Reasoning Models
- Tool Mediation Alters Refusal Mechanisms in Large Language Models
- Narrow Multimodal Fine-Tuning Can Induce Emergent Misalignment
- Epistemic Policy Divergence in Multi-Turn LLM Contamination: A Protocol-Gradient Investigation
- Don't Inoculate Everything: Stratified Inoculation Prompting Narrows Backdoor Triggers and Preserves Desired Traits
- Less Sycophancy, Stronger Refusal? Lessons for AI Safety from Mechanistic Interpretability
- SEABench: Benchmarking Endogenous Misalignment In Self-Evolving Agents
- Tracekit: Tamper-Evident Intent-Reasoning-Action Auditing for Autonomous Coding Agents
- Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control
- Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models
- Guarded Gradient-Based Activation Steering of Shutdown Responses in Qwen3.5-0.8B: A Minimum-Step Policy
- Auditing System-1 Models on Biosecurity-Relevant Benchmarks: Calibration, Selective Prediction, and Permutation Instability in a Non-Generative Model
- FARE: Forensic Acceptance Region Estimation for Catching Bait-and-Switch Image Generators
- Evaluating Sycophancy in Chinese Large Language Models on Factual Questions Derived from Online Search Queries
- Monitor Jailbreaking: Evading Chain-of-Thought Monitoring Without Encoded Reasoning
- RupeeBias: Auditing Demographic Bias in Indian Economic Guidance from Large Language Models
- The Domestic Unprotected Zone: Algorithmic Governance and the Reproduction of Perpetrator Discourse in Conversational AI
- Beyond Average Safety: Chance-Constrained LLM Fine-tuning
- Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases
- Style, Not Self: Surface Cues Explain Zero-Shot Code Attribution by Large Language Models
Agents33
- Traverse: Learning When to Remember, Reset, and Redirect for Long-Horizon Web SearchByteDance
- Context Language ModelsMeta
- EmbodiedSWE: Coding Agents for Long Horizon Dexterous RoboticsByteDance
- Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM AgentsMicrosoft
- RAPID: Robot Agentic Programming from DemonstrationsNVIDIA
All 33 papers in Agents
- AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop?Code
- AgBench: Agentic AI Benchmarks for Personal AI Devices
- DrivingBench: Can Vision-Language Models Drive a Toyota Corolla?
- RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce
- Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools
- Trustworthy Runtime Error Healing in Real-World Repositories: A Benchmark and Guardrail
- ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation
- LIBERO-Agent: Evaluating General-Purpose Agents for Direct Embodied Manipulation
- Prompted Identity Degrades Cooperation in Multi-Agent LLM Systems
- From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agents
- WEFT: Scaling Tool-Use Post-Training for General-Purpose Agents
- When Tools Silently Lie: Evaluating and Mitigating Blind Compliance in Tool-Augmented Data Agents
- Choir: An Open Protocol for Distributed Multi-Agent Autoformalization
- RepoMAS: Solving Progressively Specified Tasks with Issue-Driven Multi-Agent Systems
- Planner-as-Router: Joint Plan-Time Model Routing for Cost-Efficient Multi-Agent Workflows
- BIABench: Evaluating AI agents on real-world bioimage analysis tasks
- RSI-Router: Evolving Subtask-Level LLM Routing and Skills for Cost-Efficient Agents
- APEX-Voice: Can Voice Agents Complete Professional Workflows Through Full-Duplex Interaction
- MCP Error Messages Written for Developers Hurt the Most Capable Agents Most
- Epstein Files Engine: Agentic Search for Investigative Journalism
- A Safety-Bounded SDC-to-MCP Gateway for Medical AI Agents
- Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts
- PAWS: Policy-driven Agentic World Simulation
- Control the Harness, Control the Cost: Routing and Governing AI Coding Agents in the Enterprise
- IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking
- iCoder-27B: Recursive AI-Led Development of Frontier Industrial Coding Model
- Augur: A Synthetic Decision Lab for Rehearsing Reactions to Product and Policy Changes
- Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge
Language & speech22
- Open-Qwen-Music: An Auditable Framework for LLM-Based Music Composition and Diffusion RenderingTencentCode
- FFASR: Benchmarking Far-Field Automatic Speech Recognition using High-Fidelity Simulated RIRsHugging Face
- IndicFDB: Benchmarking Full-Duplex Voice Agents across Indian LanguagesNVIDIA
- Closing the Cross-Dialect Gap: Query Plans as a Portable Interface in Text-to-SQLIBM
- LLM unbranding: Erasing Commercial Identity while Preserving Generic UtilityCode
All 22 papers in Language & speech
- RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection
- Conversational Capture: A Trajectory-Level Framework for Evaluating Generative Engine Optimization in Multi-turn Human-Agent Interaction
- Index-Translate: A Multilingual Translation Model Family -- Text, Speech, Controlled Dubbing, and Long-Document Translation
- Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation
- RunyaNER: Auxiliary Language Selection for Runyankore NER
- DISCO: Distributed Long Context Scaling with Grounding-Reasoning Disaggregation
- MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models
- ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport
- Persona Following Is Not Selective Control: The Neutrality Gap in LLM User Simulation
- Muslim: A Deployed Arabic Voice AI Platform for Grounded Islamic Knowledge
- Temporal Taxation Compounds Under Post-Training Compression of Whisper Models
- COILD: An Indic-Centric Parallel Corpus and Benchmark for Machine Translation Across Indian Languages
- YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speech
- ModularSQL: A Runtime Guardrail for the Multiplicity Blind Spot in Text-to-SQL
- VietPrism: A large-scale Vietnamese speech and deepfake corpus with diverse dialects and code-switching
- Automated Regulatory Compliance Question Answering in Financial Services with Domain-Adapted Retrieval-Augmented Generation
- To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech
Inference & efficiency18
- Inference AuctionsGoogle
- Routing Should Pay for Itself: Sparse Supervision for Economical LLM RoutingCode
- Empowering Hybrid Attention Models on NPUsCode
- Characterizing High Bandwidth Flash for LLM Serving
- QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
All 18 papers in Inference & efficiency
- Beyond Accuracy: Prefix-Invariant Realizations of Low-Precision Fast Matrix Multiplication
- Koa-action: Fast and Consistent Structured Decision Making with Generative LLMs
- Calibrate the Decisions That Change the Future: On-Policy Post-Training Quantization for Multimodal Large Language Models
- IronLLM: Forging Compact Edge-Native Language Models for Real-Time Embodied Intelligence
- SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
- You Cannot Pick a Provider From the Price List: Market-Aware Routing for Open-Weight LLM Inference
- HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment
- Chameleon: Dynamic Format Adapter for Efficient Diffusion
- AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
- Beyond Energy: When Sustainability Dimensions Reshape LLM Serving Decisions
- HybridInfer: Thermal-Aware Reinforcement-Learning Tier Routing for On-Device, Edge, and Cloud LLM Inference
- ActKV: Efficient LLM Agents through Action-Guided KV Cache Management
- Can You Check That? The Checkability Boundary for Local LLM Network Automation
AI & society14
- What if automating AI R&D triggers an intelligence explosion?OpenAI, Microsoft, Anthropic
- Policy-Conditioned AI-Use Detection: An Evidentiary Framework for Academic Publishing
- Fairness Theatre: Evaluating Post-Hoc Fairness Interventions in Vendor-Controlled Early Warning Systems
- Persona and Persuasive Framing in AI Voice Agents: A 2×2 Field Experiment with Children
- Whose Voice Survives the Summary? A Voice-Retention Audit of LLM Employee Listening
All 14 papers in AI & society
- Breaking News Out of the Filter Bubble: Generative AI Search Diversifies Collective Attention and Raises Shared Information Consumption
- A Reusable Semantic Web Framework for Evidence-Grounded Fundamental Rights Impact Assessments under the EU AI Act
- An Empirical Study and Assessment of EU AI Act Compliance Checkers
- LLMAdBench: A Human Preference Benchmark for Advertising in LLM Responses
- Algorithmic Harms Associated with Generative Model-Augmented Recommendation Systems
- Beyond the Last Truffula Tree: SustainAI - A Water-Aware, Closed-Loop Framework for Environmentally Accountable AI
- Same Text, Different Numbers: The Divergence of LLM-Based Measures
- When the Model Retires: An Empirical Study of LLM Migration in Open-Source Applications
- RADAR: Readiness for AI Discovery and Agentic Reach
Vision & multimodal13
- SAGE: Subspace Alignment for Classifier-Free Guidance in Mixture-of-Experts Diffusion ModelsAlibaba
- GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives
- Seeing What Should Be Heard: Diagnosing and Repairing Cross-Modal Shortcuts in Omni-Modal LLMs
- PolyOCR-Venus: Unified OCR Foundation Models for Text-Centric Visual Intelligence
- Unified Trajectory Matching Policy Optimization: Diverse T2I Generation and VLA Generalization
All 13 papers in Vision & multimodal
- Privacy-Preserving Full-Body Meshing from mmWave Radar via Mesh Foundation Model Supervision
- When VLMs Trust Context: Evaluating Scene Text Recognition under Misleading Context
- Verifiable Visual Rewards Transfer from Synthetic Scenes to Natural Prompts
- Hard Vision, Easy Vision: What GPT-6 Astra Reveals Across Computer Vision
- TrafficImag: A Benchmark for Counterfactual Roadside Traffic Video Generation
- Open Vocabulary Domain Unlearning
- Where Hallucinations Live: A Cross-Architecture Circuit in VQ-Tokenized Vision-Language Models
- Reasoning Instructions Can Break Answer Decoding in Vision--Language Models
Robotics & embodied AI12
- RoboChrono: A Real Robot Benchmark for Streaming Task UnderstandingAlibaba
- Rho: A Foundation for Efficiently Adaptable VLA ModelsMicrosoft
- Counterfactual Video Generation Enables Scalable Humanoid Loco-ManipulationAmazon
- VLaRL: Augmenting Vision-Language-Action Models with Simulation-Trained Latent-Conditioned Residual RLMicrosoft
- Fiatlux: A Long-Horizon Benchmark for Humanoid Ladder Climbing and Light-Bulb Replacement
All 12 papers in Robotics & embodied AI
- Blackout vs. Freeze: Analyzing Physical Failure Modes of VLAs under Camera Faults
- Faster and Better? Benchmark Bugs and Design Limitations Distort the Evaluation of Vision-Language-Action Acceleration
- VehDyn: A Driving World Model Benchmark for Vehicle Dynamics
- ZeroBot: Learning from Scratch in Minutes with Generative Real2Sim
- When World Models Lie: Adaptive Safety Analysis Under Wrong Imaginations
- Self-Adaptive VLA for Robust Robot Deployment
- Training-free Behavior Cloning
Machine learning5
- TabFM: A Zero-Shot Foundation Model for Tabular DataGoogle
- Algorithmic Recourse Under Competition
- Support-Set Target Leakage in Relational Foundation Models during In-Context Learning: Impact, Detection, and Mitigation
- Large Language Models for Automated Cross-Domain Machine Learning Task Type Identification: A Benchmark Dataset and Evaluation
- Statistical attribute alignment for black-box generative AI via output post-processing
Science & health4
- A Surgical Foundation Model Reveals Task-Dependent Label EfficiencyCode
- Reliability Testing of Medical Model Performance under Distributed Deployment
- Multi-Site Real-World Performance of Commercial AI for Pulmonary and Incidental Pulmonary Embolism Detection
- BioEVAL: A global, multi-institutional benchmark of large language and multimodal models for bioengineering