Hours of practitioners (Zapier, Braintrust, Arize) showing how evals work in real companies.
AI Engineer World's Fair 2025: Evals track
AI Engineer conference Jun 2025
Watch on YouTube youtube.com →An eval is just a repeatable test suite for your AI feature: 30-100 real example inputs, the output you expect, and a way to score each run. Start by reading your model's actual outputs and labelling errors by hand (error analysis); patterns you find become automated checks, either simple code assertions or an LLM grading another LLM. Teams that skip this iterate blind; teams that build even a scrappy eval loop ship better AI faster than competitors.
18 resources.
Hours of practitioners (Zapier, Braintrust, Arize) showing how evals work in real companies.
AI Engineer conference Jun 2025
Watch on YouTube youtube.com →A full walkthrough of creating an eval, playable inline; the fastest zero-to-competent listen.
Lenny's Podcast with Hamel Husain and Shreya Shankar Sep 2025
Listen on Spotify open.spotify.com →A live-demo episode: watch error analysis actually being done on screen.
How I AI (Claire Vo) 2025
Listen on Apple Podcasts podcasts.apple.com →The essay that made evals a founder-level topic; failed LLM products almost always skipped this.
Hamel Husain Mar 2024
Open hamel.dev →Direct answers to the questions every team asks in week one, from sample sizes to judge prompts.
Hamel Husain 2025
Open hamel.dev →The #1 course on the subject, taken by teams at OpenAI and Anthropic; the deep option.
Hamel Husain and Shreya Shankar (Maven) 2025
Open maven.com →A free written masterclass version if you want the method without the course fee.
Aakash Gupta (Product Growth) 2025
Open aakashg.com →Gets your first automated eval running in under an hour.
DataCamp 2025
Open datacamp.com →A current map of the eval tooling landscape so you pick once, correctly.
Braintrust 2025
Open braintrust.dev →A conversational companion to Hamel's essay with concrete team workflows.
Humanloop 2024
Open humanloop.com →Distils the conference's eval takeaways, including hunting implicit user feedback signals.
Vellum Jun 2025
Open vellum.ai →How to make your AI coding tools carry the eval workload for you.
Hamel Husain 2025
Open hamelhusain.substack.com →Turning evals into a deploy gate so a bad prompt change never reaches users.
Braintrust 2025
Open braintrust.dev →A vendor-neutral ranking to cross-check the tool vendors' own comparisons.
TECHSY 2026
Open techsy.io →A trusted independent voice explaining why iteration speed is the whole game.
Simon Willison Mar 2024
Open simonwillison.net →How to use one model to grade another without inheriting its biases.
Confident AI (DeepEval) 2026
Open deepeval.com →A practical setup guide for automated judging, the technique that makes evals scale.
Agenta 2025
Open agenta.ai →Open-source, config-file evals that run in CI; used by OpenAI and Anthropic themselves.
Promptfoo 2025
Open github.com →The same ground, over in Build the product, our Starting Up track.