How to do AI analysis you can actually trust
2,000+ hours of testing distilled into verification techniques for ChatGPT, Claude, and Gemini.
Open lennysnewsletter.com →
Research & Knowledge
Article
Video
Podcast
Article
Article
Article
Article
Article
Article
Guide
Article
Thread
Podcast
Article
Article
Guide
Article
Article
How do I know the numbers AI gives me are actually right?
The short answer
Treat AI analysis like a junior analyst's first draft: make it show the code or SQL it ran, re-run the key figures a second time, and check a few numbers you already know by heart. Tools that execute code on your actual file hallucinate far less than chat answers from memory, and asking twice catches most fabrications. A final human pass on anything going to investors is non-negotiable.
A quick orientation. The real value is below: resources worth your time, from people who've actually done it.
18 resources worth your time
How to do AI analysis you can actually trust (audio)
The same playbook in listenable form for a commute.
Watch on YouTube youtube.com →How to do AI analysis you can actually trust (Spotify)
Spotify version for founders who live in podcasts.
Listen on Spotify open.spotify.com →Why ChatGPT and Claude Give You Different Answers Every Time You Analyze Feedback
Explains non-determinism: why even temperature zero runs differ, and what to do about it.
Open chattermill.com →How to Prevent AI Hallucinations: 3 Hidden Threats When AI Analyzes Your Data
Concrete failure cases like '40% growth' that was actually 14%, and the guardrails.
Open databox.com →On Accuracy When LLMs Analyze Real-World Data
A clear-eyed look at where LLM analysis of real datasets goes wrong.
Open verbagpt.com →What Are AI Hallucinations?
The plain-language primer to share with your team on why AI invents things.
Open ibm.com →LLM Hallucination Statistics 2026
Aggregated hallucination-rate data so you calibrate trust per task type.
Open sqmagazine.co.uk →Which AI Hallucinates Least? Benchmark Rates Data
Current model-by-model hallucination benchmarks to inform your tool choice.
Open suprmind.ai →Beyond Fine-Tuning: Strategies for Mitigating Hallucinations in LLMs for Data Analytics
The research grounding for why code execution and grounding beat raw chat answers.
Open arxiv.org →Solving the Very-Real Problem of AI Hallucination
Practical mitigation patterns including cross-model disagreement checks.
Open knostic.ai →Ethan Mollick: testing ChatGPT and Claude on a brutally hard Excel file
Models the exact behavior you should copy: spot checks even when output looks perfect.
Open x.com →LLMs and Agents Are Overhyped, with Dr. Andriy Burkov (SDS 867)
A respected ML author's antidote to blind trust in AI analysis.
Open superdatascience.com →Julius AI Review: Is It Worth It for Data Analysts?
Documents where a dedicated AI analyst tool hallucinates plausible numbers on complex stats.
Open letdataspeak.com →"My AI is Lying to Me": user-reported LLM hallucinations
Peer-reviewed evidence of real users' hallucination encounters, patterns worth knowing.
Open ncbi.nlm.nih.gov →6 steps in fact-checking AI-generated content
A simple checklist you can turn into a team habit.
Open techtarget.com →Why Learning SQL Beats Relying on AI: Understanding Data Still Matters
The verification argument: if you can't verify generated analysis, you can't use it.
Open learnsql.com →LLM Hallucination: What It Is and How to Prevent It in Production
Production-grade prevention patterns once AI analysis becomes routine at your startup.
Open alicelabs.ai →More in Data Analysis
- Why let AI analyze my data instead of learning Excel formulas or SQL? 18 →
- How do founders get insights from CSVs, analytics, and surveys using AI? 18 →
- What can AI-in-spreadsheets (Claude in Excel, Gemini in Sheets) actually do? 20 →
- Which AI tool should I use for data analysis: ChatGPT, Claude, Gemini, or a dedicated tool? 18 →
- Can AI write SQL for me and query my database in plain English? 18 →
- Is it safe to put my company's data into AI tools? 18 →
- How do I build dashboards and charts with AI, without a BI tool or a data team? 18 →
- How do I analyze customer feedback, reviews, and support tickets at scale with AI? 18 →
- Which metrics should my startup track, and how can AI help me stay on top of them? 18 →
- Can AI clean up my messy data: duplicates, formats, and half-filled columns? 18 →
- Do I still need to hire a data analyst, or can AI cover it for now? 16 →
- How do founders use AI for financial modeling and forecasting? 18 →