Your AI Product Needs Evals
The essay that made evals a founder-level topic; failed LLM products almost always skipped this.
Open hamel.dev →An eval is just a repeatable test suite for your AI feature: 30-100 real example inputs, the output you expect, and a way to score each run. Start by reading your model's actual outputs and labelling errors by hand (error analysis); patterns you find become automated checks, either simple code assertions or an LLM grading another LLM. Teams that skip this iterate blind; teams that build even a scrappy eval loop ship better AI faster than competitors.
A quick orientation. The real value is below: resources worth your time, from people who've actually done it.
The essay that made evals a founder-level topic; failed LLM products almost always skipped this.
Open hamel.dev →Direct answers to the questions every team asks in week one, from sample sizes to judge prompts.
Open hamel.dev →A full walkthrough of creating an eval, playable inline; the fastest zero-to-competent listen.
Listen on Spotify open.spotify.com →The #1 course on the subject, taken by teams at OpenAI and Anthropic; the deep option.
Open maven.com →A free written masterclass version if you want the method without the course fee.
Open aakashg.com →Open-source, config-file evals that run in CI; used by OpenAI and Anthropic themselves.
Open github.com →Gets your first automated eval running in under an hour.
Open datacamp.com →A current map of the eval tooling landscape so you pick once, correctly.
Open braintrust.dev →Hours of practitioners (Zapier, Braintrust, Arize) showing how evals work in real companies.
Watch on YouTube youtube.com →A conversational companion to Hamel's essay with concrete team workflows.
Open humanloop.com →A live-demo episode: watch error analysis actually being done on screen.
Listen on Apple Podcasts podcasts.apple.com →Distils the conference's eval takeaways, including hunting implicit user feedback signals.
Open vellum.ai →How to make your AI coding tools carry the eval workload for you.
Open hamelhusain.substack.com →Turning evals into a deploy gate so a bad prompt change never reaches users.
Open braintrust.dev →A vendor-neutral ranking to cross-check the tool vendors' own comparisons.
Open techsy.io →A trusted independent voice explaining why iteration speed is the whole game.
Open simonwillison.net →How to use one model to grade another without inheriting its biases.
Open deepeval.com →A practical setup guide for automated judging, the technique that makes evals scale.
Open agenta.ai →