Amazon Web Services on November 11 published the Universal Agent Evaluation Framework (UAEF), an open-source system under Apache-2.0 that scores AI agents with 36 built-in metrics across seven dimensions, regardless of which framework built them2.
The tool addresses a concrete pain point: LangGraph, LangChain, Strands Agents, and Amazon Bedrock AgentCore each record messages, tool calls, and agent-to-agent handoffs in different formats1. Evaluation code written for one framework's trace structure must be rebuilt when teams switch or run agents on more than one, making it difficult to determine whether a change to a prompt, model, or tool actually improved the agent.
Canonical traces as the abstraction layer
UAEF's adapters normalize each framework's output into a canonical AgentTrace, the only representation the scoring engine consumes. The system can adapt an existing trace from a past run, log, or observability store, or invoke an agent through a runtime Amazon Resource Name or HTTP endpoint and convert the resulting execution. A generic JSON adapter extends the process to custom frameworks.
The seven evaluation dimensions span tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination, and reasoning. Ground-truth answers are optional overall, though reference-comparison metrics require them; when a metric lacks necessary input, UAEF returns no score with a warning rather than counting it as zero. Tool-calling evaluation can define expected tools, their order, and their arguments.
Comparison engine and persistence
For metrics shared by two runs, the comparison engine calculates absolute and percentage changes against a baseline, with a configurable threshold set to 5% by default separating improvements, regressions, and unchanged results. Regressions are graded minor, moderate, or severe and grouped by dimension. An optional statistical report covers score distributions, confidence intervals, outliers, metric correlations, and failed test cases without using an LLM. A separate Amazon Bedrock-generated analysis can propose root causes and prioritized next steps.
Amazon DynamoDB stores metadata for listing and comparison, including evaluation counts and average scores, while Amazon Simple Storage Service (Amazon S3) holds full JSON results with per-metric scores and judge reasoning. Individual runs can record the framework, model, tools, system prompt, parameters, configuration snapshot, and Git commit.
Security and deployment boundaries
UAEF treats evaluated-agent output as untrusted input to judge prompts, sending the rubric separately, escaping the content, strictly parsing judge responses, and requiring a verbatim evidence span before accepting a score. AWS states that judge scores should not gate security controls; runtime enforcement belongs in a separate Amazon Bedrock Guardrails invocation path. The repository describes UAEF as a reference architecture designed to run inside a platform that supplies user and tenant isolation; direct multi-tenant deployment without that host enforcement is unsupported.
AWS reports that customers across 12 organizations in telecom, healthcare, and financial services used UAEF to apply consistent metrics and compare agent versions before release. The awslabs/uaef repository contains the Python library, framework adapters, metrics, AWS deployment infrastructure, a web interface, and runnable notebooks. Installation requires Python 3.10 or later and the uv package manager. UAEF also supports custom metrics and integrations with ragas and DeepEval.
ANALYSIS By normalizing traces at the adapter layer rather than standardizing frameworks themselves, UAEF sidesteps the format fragmentation that currently forces teams to maintain parallel evaluation pipelines.