AI Engineer World's Fair 2025: Evals track
Hours of practitioners (Zapier, Braintrust, Arize) showing how evals work in real companies.
Watch on YouTube youtube.com →
Building & Product
Video
Podcast
Podcast
Article
Guide
Course
Guide
Guide
Article
Article
Article
Article
Article
Article
Article
Article
Guide
Tool
How do I test whether my AI feature actually works before launch (evals, in plain terms)?
An eval is just a repeatable test suite for your AI feature: 30-100 real example inputs, the output you expect, and a way to score each run. Start by reading your model's actual outputs and labelling errors by hand (error analysis); patterns you find become automated checks, either simple code assertions or an LLM grading another LLM. Teams that skip this iterate blind; teams that build even a scrappy eval loop ship better AI faster than competitors.
Go deeper
18 resources.
Why AI evals are the hottest new skill for product builders
A full walkthrough of creating an eval, playable inline; the fastest zero-to-competent listen.
Listen on Spotify open.spotify.com →Evals, error analysis, and better prompts | Hamel Husain
A live-demo episode: watch error analysis actually being done on screen.
Listen on Apple Podcasts podcasts.apple.com →Your AI Product Needs Evals
The essay that made evals a founder-level topic; failed LLM products almost always skipped this.
Open hamel.dev →LLM Evals: Everything You Need to Know (FAQ)
Direct answers to the questions every team asks in week one, from sample sizes to judge prompts.
Open hamel.dev →AI Evals For Engineers and PMs
The #1 course on the subject, taken by teams at OpenAI and Anthropic; the deep option.
Open maven.com →How to Master AI Evals: Step-by-Step with Hamel and Shreya
A free written masterclass version if you want the method without the course fee.
Open aakashg.com →Promptfoo Tutorial: A Hands-On Guide to LLM Evaluation
Gets your first automated eval running in under an hour.
Open datacamp.com →Best Prompt Evaluation Tools (Tested and Compared)
A current map of the eval tooling landscape so you pick once, correctly.
Open braintrust.dev →Why Your AI Product Needs Evals, with Hamel Husain
A conversational companion to Hamel's essay with concrete team workflows.
Open humanloop.com →Big Ideas from the AI Engineer World's Fair
Distils the conference's eval takeaways, including hunting implicit user feedback signals.
Open vellum.ai →Evals Skills for Coding Agents
How to make your AI coding tools carry the eval workload for you.
Open hamelhusain.substack.com →Best AI Eval Tools for CI/CD Pipelines
Turning evals into a deploy gate so a bad prompt change never reaches users.
Open braintrust.dev →8 LLM Eval Tools Ranked: No Product to Sell
A vendor-neutral ranking to cross-check the tool vendors' own comparisons.
Open techsy.io →Simon Willison on "Your AI Product Needs Evals"
A trusted independent voice explaining why iteration speed is the whole game.
Open simonwillison.net →LLM-as-a-Judge: Top evaluation techniques and best practices
How to use one model to grade another without inheriting its biases.
Open deepeval.com →LLM as a Judge: Guide to LLM Evaluation Best Practices
A practical setup guide for automated judging, the technique that makes evals scale.
Open agenta.ai →Promptfoo
Open-source, config-file evals that run in CI; used by OpenAI and Anthropic themselves.
Open github.com →More in AI in Your Product
- Should my startup add AI features, and which ones actually matter to users? 20 →
- How do founders ship their first AI feature (APIs, wrappers, RAG in plain terms)? 19 →
- How much does it cost to run AI features, and how do I keep costs sane? 19 →
- Which AI model should I build on (GPT, Claude, Gemini, open-source), and how do I avoid locking myself in? 17 →
- How do I stop my AI feature from making things up in front of customers? 17 →
- Prompt engineering, RAG, or fine-tuning: which one does my product actually need? 18 →
- How do I design AI features people trust (chat vs buttons, streaming, showing the work)? 18 →
- What is my moat if I'm building on someone else's model? 19 →
- Should I put an AI agent in my product, or is a simple workflow enough? 18 →
- Is it safe to send customer data to AI APIs, and what about privacy laws like India's DPDP Act? 18 →
- How should I price AI features: per seat, per use, or per outcome? 17 →
- My AI feature is live. How do I monitor it and make it better with real usage? 18 →
Also in Starting Up
The same ground, over in Build the product, our Starting Up track.
eChai Partner Brands
