HOLMES
“Experiments show that current LLMs still struggle on HOLMES, with an average accuracy of only 50.64% and the best model reaching 59.54%.”
The authors, in the abstract
Evaluating Higher-Order Logical Reasoning in LLMs
AI Research Radar
Tuesday, June 23: 5 frontier papers out of 971 read.
Frontier
Papers rated as moving the field
How we chooseOTTER
“Evaluated on 457 AdvBench prompts across four GPT models, OTTER raises average ASR from 7.0% to 84.0%.”
Safety in Self-Evolving LLM Agent Systems
“Comparative case studies of two open-source frameworks demonstrate that evolution-native design activates 3.5× more attack surface cells and achieves a 100% attack persistence rate (40/40 payloads across all CIA+Privacy categories), while co-located security scanners block only 2.5% of attacks.”
Vesta
BioMatrix
This week in research
Jun 17–23: 187 cleared, 14 frontier
The week’s papers have not been sorted into fields yet.
Models under test
Papers this week that name the model
Labs publishing
Papers this week with an author at the lab
Also cleared
86 papers
- Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent TeamsMicrosoftCode
- Tmax: A simple recipe for terminal agentsAllenAICode
- AgenticOS: An Intent-Oriented Secure Operating System Architecture for Autonomous AI AgentsTencent
- Warning labels shift perceptions of sycophantic AI, but not its influenceMicrosoft
- Investigating The Security of Modern AI and Cloud InfrastructureNVIDIA
- Understanding the (In)Security of Vibe-Coded ApplicationsMicrosoft
- Evaluation Awareness Is Not One Capability: Evidence from Open Language ModelsMicrosoft
- AI Exposure Scores: what they measure, what they miss, and what comes nextCohere
- LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action ModelsMicrosoft
- AutoDojo: Adaptive Black-Box Attacks Reveal the Limits of IPI Defenses and Task-Specification Effects in LLM AgentsCode
- Think Twice Before You Act: Protecting LLM Agents Against Tool Description Poisoning via Isolated PlanningCode
- Harness-MU: A Safe, Governed, and Effective Harness for Multi-User LLM AgentsCode
All 86 papers
- AgentLens: Interpretable Safety Steering via Mechanistic Subspaces for Multi-Turn Coding AgentCode
- From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents
- MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills
- TinyGiantALM: A Compact Audio-Language Model for Intent-Aware Reasoning under Resource Constraints
- What the Eyes See, the LLMs Miss: Exploiting Human Perception for Adversarial Text Attacks
- Mind your key: An Empirical Study of LLM API Credential Leakage in iOS Apps
- One Goal, Many Commands: Characterizing Denylist Fragility in AI Agents
- Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Staleness
- Ghost Vectors: Soft-Deleted Embeddings Remain Reconstructible in HNSW Vector Databases
- Infrastructure for the Agentic Web: Gap Analysis and Architecture from the Agentverse Platform
- Human-Less LLM Serving: Quantifying the Human Tax on Throughput
- Signals in the Noise: Open Source Intelligence (OSINT) for AI Loss of Control Detection
- BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems
- Agent Behavior Mining: Generative AI Agent Governance in Business Processes
- SafeDojo: Safe Reinforcement Learning for VLA via Interactive World Model
- FairTutor: Equity-Aware Pedagogical LLM Routing for Budget-Constrained AI Tutoring
- MIRAGE: Stealthy Visual Prompt Injection for Vulnerability Detection in Web Agents
- Mirage: a Clean-Label Backdoor against LiDAR 3D Object Detection
- Co-Construction Blindness and Asymmetric Epistemic Vulnerability in Human-LLM Interaction
- FirstPass: Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes
- The Substrate Collapse: AI Code Generation Invalidates Authorship-Based Knowledge Metrics
- Whose Agent Are You? Multi-Layer Fingerprinting and Attribution of Autonomous Web Agents
- Generative Responsible AI Data Evaluation Schema (GRAIDES) for AI Assurance in Local Government
- Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL
- Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layer
- MammoExpert: Benchmarking Chain-of-Thought Reasoning in Mammography Diagnosis
- Answer Engineering: Local Trajectory Editing for Protocol-Constrained Decision Making in Large Language Models
- PulseCX: Breaking the Closed-World Assumption in Real-Time CX
- ChronoLock: Protecting Videos from Unauthorized Text-to-Video Personalization
- AOR-Bench: Do Large Audio Language Models Over-Refuse Pseudo-Harmful Queries?
- BadDreamer: Transferable Backdoor Attacks against Video World Models for Autonomous Driving
- MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Models
- Recency/Frequency Adaptive KV Caching for Large Language Model Serving
- Human-AI Interaction Requirements in Public Sector Procurements
- "What Happens Locally, Leaks Globally": Detecting Privacy Leakage Risks in MCP Servers
- Don't Blindly Trust It: How Unreliable Feedback Breaks Tool-Using LLM Agents
- Per-Entity Bias Mapping for AI Visibility: Why Brand Mentions Require Entity-Specific Calibration
- Toward Open Weight Models Without Risks: Separating Public and Private Capabilities in LLMs
- Safe to Check, Unsafe to Use: Relinking at the Compression Boundary of LLM Agents
- Bridging the Age Gap: Towards Detecting Neural Audio Codec Synthesized Elderly Speech Deepfake
- Fixed RAG Compression Collapses Measured Reader Scaling
- Agent-Assisted Side-Channel Attacks on Non-Prefix KV Cache in RAG
- The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inference
- AgentRiskBOM: A Risk-Scoping Security Bill of Materials for Agentic AI Systems
- Old Fictions, New Skins: Evaluating the Manipulative Capabilities of LLMs in Healthcare
- L20-Edu-135M: An Auditable Single-GPU Study of Data-Efficient Small Language Modeling
- Revelio: Cost-Efficient Agentic Memory Safety Vulnerability Detection For Repository-Scale Codebases
- T-IMPACT: A Severity-Aware Benchmark for Contextual Image-Text Manipulation
- Formal-Method-Guided Vibe Coding: Closing the Verification Loop on AI-Generated Safety-Critical Software Through Model-Driven Engineering
- Music Playlist Captioning at Scale with Large Language Models
- RAVEN: Agentic RAG for Automated Vulnerability Repair
- Skin-Deep: A Geometric Diagnostic for Alignment Fragility in Large Language Model Representations
- Black-Box Forensics for Conversational LLM Agents
- Catching Lies Without Sending the Video: Privacy-Preserving Multimodal Deception Detection
- One-Prompt Censorship Evasion via Generative Diffusion Models
- Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code
- Language-Specific Sentiment Polarity Biases in Encoder and Large Language Model Classification of Product Reviews
- Towards Robust Personalized Federated Learning: Vulnerability Assessment and Defense Co-Design
- Cloak: Zero-Shot Cross-Embodiment Manipulation by Masking the End-Effector from the VLA
- RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving
- The Unseen Hand: Manipulating Model Fairness and SHAP with Targeted Identity Re-Association Attacks
- FedOT: Ownership Verification and Leakage Tracing via Watermarks for Federated LDMs
- Public Diffusion Models, Private Images: Key-Controlled Inversion for Conditional Reconstruction
- VCT: A Verifiable Transcript System for LLM Conversations
- Have You Ever Seen Them? Entity-level Membership Inference through Interrogating Large Language Models
- Who Owns the AI Recommendation? A Multi-Industry Empirical Map of Brand Category Ownership Across Large Language Models
- Cognitive Digital Twins: Ethical Risks and Governance for AI Systems That Model the Mind
- Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluation
- Unmasking LAION-5B: Age, Gender, Race, and Emotion Biases in Large-Scale Image Datasets
- TooBad: Backdoor Diffusion Models with Ultra-Low Poison Rate and Imperceptible Trigger
- HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models
- Detecting Malicious Agent Skills in the Wild using Attention
- AOHP: An Open-Source OS-Level Agent Harness for Personalized, Efficient and Secure Interaction
- A Watermark for Vision-Language-Action and World Action Models