Guide

LLM as a Judge: Guide to LLM Evaluation Best Practices

Agenta · agenta.ai

A practical setup guide for automated judging, the technique that makes evals scale.

Open on agenta.ai ↗

The question it answers

  1. How do I test whether my AI feature actually works before launch (evals, in plain terms)?

Read with it

  1. The essay that made evals a founder-level topic; failed LLM products almost always skipped this. hamel.dev Article Your AI Product Needs Evals Hamel Husain
  2. Direct answers to the questions every team asks in week one, from sample sizes to judge prompts. hamel.dev Guide LLM Evals: Everything You Need to Know (FAQ) Hamel Husain
  3. A full walkthrough of creating an eval, playable inline; the fastest zero-to-competent listen. open.spotify.com Podcast Why AI evals are the hottest new skill for product builders Lenny's Podcast with Hamel Husain and Shreya Shankar
  4. The #1 course on the subject, taken by teams at OpenAI and Anthropic; the deep option. maven.com Course AI Evals For Engineers and PMs Hamel Husain and Shreya Shankar (Maven)
  5. A free written masterclass version if you want the method without the course fee. aakashg.com Guide How to Master AI Evals: Step-by-Step with Hamel and Shreya Aakash Gupta (Product Growth)
  6. Open-source, config-file evals that run in CI; used by OpenAI and Anthropic themselves. github.com Tool Promptfoo Promptfoo
eChai Partner Brands The partnership →