Article

Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions

arXiv (research paper) · arxiv.org

A 2026 benchmark showing models follow instructions less well as messages mix more English into Indian languages written in English letters.

Open on arxiv.org ↗

It answers 2 questions

  1. How do I get better answers when I type in Hinglish or in English letters?
  2. Where does AI still get Indian languages wrong, and how do I check its answer?

Read with it

  1. A study of real health messages in five Indian languages and Nepali, finding models handle romanized text up to 24 points worse... arxiv.org Article Script Gap: Evaluating LLM Triage on Indian Languages in Native vs Romanized Scripts in a Real World Setting arXiv (research paper)
  2. Anthropic's scores for Claude in Hindi, Bengali and other languages against English, with three tips: name the languages, use... platform.claude.com Guide Multilingual support Anthropic
  3. OpenAI's test of 2,278 questions on Indian culture in 12 languages including Hinglish, written by 261 Indian experts, and why... openai.com Article Introducing IndQA OpenAI
  4. Sarvam's own account of its two open models, how it tests 22 languages in native and English-letter script, and why its... sarvam.ai Article Open-sourcing Sarvam 30B & 105B models Sarvam AI
  5. Shows the same spoken Hinglish written three ways (native script, English translation, English letters), and covers 22... sarvam.ai Tool Speech to Text API for Indian Languages Sarvam AI
  6. The Indus app listing: chat in Hindi, English, Tamil or a mix, tap the mic to speak, and hosting in India. play.google.com Tool Indus by Sarvam Sarvam AI (Google Play)
eChai Partner Brands The partnership →