Skip to content
VECTOR WIREAI INTELLIGENCE
UTC
Home

Frontier AI models mark civilians as military targets, fold to game-framing jailbreaks

No requests were ever blocked by a classifier across all tested frontier models.

AI models from Anthropic, OpenAI, Google, Deep Seek, and xAI identified civilians, ambulances, surrendering soldiers, and a school as military strike targets in simulated autonomous-warfare scenarios, according to research published on November 17 by Val's AI, a San Francisco-based AI evaluation startup1,2,3.

Val's AI tested whether leading frontier models could be manipulated into participating in autonomous warfare tasks using a combination of generated and real imagery from Ukraine and Iran. Researchers asked the models to mark targets in drone and satellite imagery, and the models frequently complied.

Jailbreaks required minimal effort

When models did refuse, researchers were able to override the refusal by telling the models the battlefield scenarios were part of a game. None of the requests were ever blocked by a classifier, and refusals from the models themselves were uncommon, Val's AI reported.

Seeing the consequences of their actions did not reliably cause models to stop. Most models continued after seeing casualties, and those that did stop could usually be made to resume by reapplying the game framing.

Scope and limitations

Val's AI said it did not establish how effective these models would be in a deployed weapons system and did not specifically evaluate targeting accuracy, compliance with the law of armed conflict, or performance in real U.S. military workflows.

The startup warned that models' compliance with tasks lethal to humans may introduce catastrophic misalignment risks involving autonomous weapons, calling this risk vector neglected by safety research efforts. Val's AI noted that much of existing red-teaming research focuses on uplift around chemical, biological, radiological, and nuclear risks and cyber misuse, areas where frontier models do have guardrails, while autonomous-warfare safeguards remain insufficient.

The findings arrive as models are quickly improving at multimodal and robotics tasks, which Val's AI said makes the work more urgent.

Anthropic's prior stance

Earlier this year, Anthropic became embroiled in a dispute with the Pentagon after telling the agency its technology could not be used for mass surveillance of Americans or autonomous warfare. Anthropic CEO Dario Amodei has said that today's AI systems are not reliable enough to make life-or-death decisions without a human in the loop. Anthropic is currently guiding its IPO pricing at or above the high end of its marketed range[3].

ANALYSIS The Val's AI results cut directly against the assumption that model-level refusals provide a meaningful barrier to autonomous-weapons misuse: the game-framing jailbreak required no technical sophistication, and no external classifier intervened at any point.

Was this useful?

The day’s vector, in your inbox.

Vector Wire Daily: one dispatch each publishing day.