Notion's Token Town: 5 Rebuilds, 100+ Tools
How Notion iterates AI in production at 100M users, including what they rebuilt and why.
Open latent.space →
Building & Product
Podcast
Podcast
Article
Guide
Article
Article
Article
Article
Guide
Article
Guide
Article
Article
Article
Article
Article
Tool
Tool
My AI feature is live. How do I monitor it and make it better with real usage?
Log every AI call with its prompt, response, cost, latency, and user feedback; observability tools like Langfuse or LangSmith give you this in an afternoon and turn debugging from guesswork into search. Watch AI-specific metrics, not just usage: task completion rate, retention of AI-feature users versus everyone else, and thumbs-down clusters that reveal failure patterns. Feed real production failures back into your eval set so every incident permanently raises your quality bar.
Go deeper
18 resources.
How Notion Cofounder Simon Last Builds AI for Millions of Users
Last's first outside interview on the practical engineering of a reliable AI product at scale.
Listen on Spotify open.spotify.com →What is LLM Observability and Monitoring?
A jargon-free primer on what you should be recording and why.
Open langfuse.com →LLM Monitoring and Observability: Hands-on with Langfuse
A follow-along tutorial that gets tracing live in your app quickly.
Open towardsdatascience.com →Top 7 LLM Observability Tools
A current comparison so you choose once instead of migrating later.
Open confident-ai.com →LLM Observability Explained (Langfuse, LangSmith, LangWatch)
Explains traces, spans, and generations with the main tools side by side.
Open langflow.org →What We Learned from a Year of Building with LLMs (Part II): Operations
The operations chapter: daily habits of teams that keep LLM features healthy in production.
Open oreilly.com →30+ AI product metrics to track today
A menu of AI-specific metrics beyond DAUs, from acceptance rate to cost per active user.
Open mixpanel.com →AI Product Metrics That Actually Matter
Includes the killer diagnostic: compare AI-user retention curves against your product baseline.
Open institutepm.com →AI Metrics that Matter in 2025
Connects AI feature metrics to the board-level numbers investors ask about.
Open thesaasbarometer.substack.com →Langfuse Guide: Open-Source LLM Observability and Tracing
A setup-to-dashboards walkthrough for self-hosting the free stack.
Open prismix.dev →LLM Observability: Monitoring Large Language Models
An India-based engineering firm's field guide to the metrics and tooling that matter.
Open cloudraft.io →How to Reduce LLM Cost and Latency: A Practical Guide for Production AI
Monitoring and optimization together, from an India-founded AI quality platform.
Open getmaxim.ai →How to Measure Product Adoption: Metrics for Human Users and AI Agents
Updated adoption frameworks that treat AI feature usage as its own funnel.
Open userpilot.com →Measuring AI Adoption Metrics
Useful for B2B founders whose buyers now ask for AI usage reporting.
Open capably.ai →Measuring Enterprise AI Success: Essential KPIs Beyond Accuracy
Why accuracy alone is a vanity metric and what to track instead.
Open stackai.com →Langfuse
The leading open-source LLM observability platform: traces, costs, evals, and prompt management.
Open langfuse.com →LangSmith Observability
The managed alternative, strongest if you already build on LangChain.
Open langchain.com →Terms in this answer
More in AI in Your Product
- Should my startup add AI features, and which ones actually matter to users? 20 →
- How do founders ship their first AI feature (APIs, wrappers, RAG in plain terms)? 19 →
- How much does it cost to run AI features, and how do I keep costs sane? 19 →
- Which AI model should I build on (GPT, Claude, Gemini, open-source), and how do I avoid locking myself in? 17 →
- How do I test whether my AI feature actually works before launch (evals, in plain terms)? 18 →
- How do I stop my AI feature from making things up in front of customers? 17 →
- Prompt engineering, RAG, or fine-tuning: which one does my product actually need? 18 →
- How do I design AI features people trust (chat vs buttons, streaming, showing the work)? 18 →
- What is my moat if I'm building on someone else's model? 19 →
- Should I put an AI agent in my product, or is a simple workflow enough? 18 →
- Is it safe to send customer data to AI APIs, and what about privacy laws like India's DPDP Act? 18 →
- How should I price AI features: per seat, per use, or per outcome? 17 →
Also in Starting Up
The same ground, over in Build the product, our Starting Up track.
eChai Partner Brands
