A technical evaluation published by Hamel Husain compares decision models from TypeSafe (Jev), Perplexity AI (Decider), Liquid (d1), Cloudflare (Clef), and OpenAI (Decisions API) as alternatives to LLM judges for automated evals1. The analysis, using 100 traces from an apartment leasing agent, examines cost, performance, and latency tradeoffs and finds that the default decision threshold of 0.50 often underperforms, with threshold tuning yielding improved results.
Decision Models Tested as Low-Cost Alternative to LLM Judges for Evals
New evaluation compares decision models from TypeSafe, Perplexity AI, Liquid, Cloudflare, and OpenAI against LLM judges, finding threshold tuning…