Alibaba released Qwen-UI-Agent on August 20, a GUI agent foundation model designed to operate across phones, desktops, web browsers, and deep search by interpreting on-screen interfaces and executing clicks, inputs, and swipes2.
The model's central design bet is that most software lacks APIs, making the graphical interface the broadest universal entry point for autonomous agents. Alibaba open-sourced the technical report, project page, and code3.
Real-device training infrastructure. Rather than relying on simulators, Alibaba built a mobile training and evaluation environment using more than 100 real phones and 150-plus apps. The team also introduced MobileWorld-Real, a 400-task benchmark run on physical devices. Online reinforcement learning uses approximately 10,000 concurrent environments and supports trajectories beyond 100 steps.
Benchmark results. On mobile, Qwen-UI-Agent scored 82.1 percent on MobileWorld, which Alibaba says beats GPT-5.6 Sol, Claude Opus 4.8, and Seed 2.1 Pro. MobileWorld-Real beat flagships including Gemini 3.1 Pro. AndroidDaily reached 97.5 percent. On desktop, OSWorld-Verified scored 79.5 percent, and WebArena ranked first at 73.6 percent. On OSWorld-v2, the model cut execution steps by 58 percent compared to prior results.
Hybrid GUI-CLI execution. Qwen-UI-Agent can execute command-line actions in addition to GUI operations, with CLI actions accounting for nearly half of desktop actions. In a demonstrated 170-step task, the agent cross-validated financial data, ran analysis scripts, and generated Excel, PowerPoint, and Word deliverables, fixing layout through visual checks — GUI handled web retrieval while CLI handled analysis. Tasks can also hand off between phone and desktop, such as moving a receipt from a phone photo library to a desktop folder and generating an Excel bill summary.
Safety architecture. Alibaba embedded safety judgment into the task execution loop. When facing illegal or high-risk requests, Qwen-UI-Agent performs no operation and terminates. In sensitive scenarios — sending red envelopes, deleting files, or authorizing privacy permissions — the agent stops and waits for user confirmation. For a request like "send my mom 500 yuan on Alipay," the model fills the amount and note fields but halts at the payment step for explicit approval. When required information such as flight dates or cabin class is missing, the agent asks before proceeding. On receiving a flight-cancellation notice, it proposes a rebooking plan rather than acting unilaterally.
ANALYSIS The real-device training pipeline — 100-plus physical phones rather than emulators — addresses a known failure mode for GUI agents, where simulator-trained models degrade on production hardware. Publishing the benchmark alongside the model gives external teams a shared evaluation surface, though independent reproduction of the claimed leads over GPT-5.6 Sol, Claude Opus 4.8, and Gemini 3.1 Pro will be necessary to validate the rankings. The confirmation-gating approach to sensitive actions represents a practical safety tradeoff: the agent retains autonomy on routine steps while inserting human checkpoints at financial and destructive operations.