Thread

Ethan Mollick: testing ChatGPT and Claude on a brutally hard Excel file

Ethan Mollick · x.com

Models the exact behavior you should copy: spot checks even when output looks perfect.

Open on x.com ↗

The question it answers

  1. How do I know the numbers AI gives me are actually right?

Read with it

  1. 2,000+ hours of testing distilled into verification techniques for ChatGPT, Claude, and Gemini. lennysnewsletter.com Article How to do AI analysis you can actually trust Caitlin Sullivan (Lenny's Newsletter)
  2. Video How to do AI analysis you can actually trust (audio) Lenny's Reads
  3. Spotify version for founders who live in podcasts. open.spotify.com Podcast How to do AI analysis you can actually trust (Spotify) Lenny's Reads
  4. Explains non-determinism: why even temperature zero runs differ, and what to do about it. chattermill.com Article Why ChatGPT and Claude Give You Different Answers Every Time You Analyze Feedback Chattermill
  5. Concrete failure cases like '40% growth' that was actually 14%, and the guardrails. databox.com Article How to Prevent AI Hallucinations: 3 Hidden Threats When AI Analyzes Your Data Databox
  6. A clear-eyed look at where LLM analysis of real datasets goes wrong. verbagpt.com Article On Accuracy When LLMs Analyze Real-World Data VerbaGPT
eChai Partner Brands The partnership →