Tool

Promptfoo

Promptfoo · github.com

Open-source, config-file evals that run in CI; used by OpenAI and Anthropic themselves.

Open on github.com ↗

The question it answers

  1. How do I test whether my AI feature actually works before launch (evals, in plain terms)?

Read with it

  1. The essay that made evals a founder-level topic; failed LLM products almost always skipped this. hamel.dev Article Your AI Product Needs Evals Hamel Husain
  2. Direct answers to the questions every team asks in week one, from sample sizes to judge prompts. hamel.dev Guide LLM Evals: Everything You Need to Know (FAQ) Hamel Husain
  3. A full walkthrough of creating an eval, playable inline; the fastest zero-to-competent listen. open.spotify.com Podcast Why AI evals are the hottest new skill for product builders Lenny's Podcast with Hamel Husain and Shreya Shankar
  4. The #1 course on the subject, taken by teams at OpenAI and Anthropic; the deep option. maven.com Course AI Evals For Engineers and PMs Hamel Husain and Shreya Shankar (Maven)
  5. A free written masterclass version if you want the method without the course fee. aakashg.com Guide How to Master AI Evals: Step-by-Step with Hamel and Shreya Aakash Gupta (Product Growth)
  6. Gets your first automated eval running in under an hour. datacamp.com Guide Promptfoo Tutorial: A Hands-On Guide to LLM Evaluation DataCamp
eChai Partner Brands The partnership →