NVIDIA and AI4Bharat open a 2.8 trillion-token Indic pretraining corpus covering 22 languages, along with the pipeline that generated it
Not just the data but the recipe for making it. What's in the corpus and what the synthetic-data pipeline can be reused for.
// Key points
- 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted with the Gemma-4 tokenizer.
- It covers the 22 languages listed in India's Constitution, with romanized text for twelve, for 34 language-script configurations.
- The pipeline uses NeMo Data Designer and Gemma-4-31B-it to translate whole English documents, assess quality, repair low-scoring outputs and romanize a subset; prompts separate prose to translate from code, equations and URLs to preserve.
- Every output keeps its source identifier, so it can be traced back to the original document.
Builder's takeFor us the pipeline matters more than the Indic data itself: translate whole documents, score them, repair only the low scorers and preserve code and equations. That's a recipe I could borrow to localize the AI Interview question bank. Run it on 100 questions first and spot-check how much the repair step actually fixes.
// Background · from #Open models
Full timeline →- Oct 18 DeepSeek releases V4.1-Flash-1017 under MIT: DeepSWE v1.1 rises from 74.2 to 77.0 and Terminal-Bench 4.0 from 31.2 to 42.3
- Oct 18 Tencent Hunyuan open-sources Hy-Memory-30B-A3B under Apache 2.0: a roughly 3B-active memory writer averaging 55.49 across four memory frameworks, ahead of Claude Opus 5 at 54.62
- Oct 18 NVIDIA releases Dory1.0, a looped 4B model that trades compute for accuracy by loop count: SWE-bench Verified goes from 50.3 at one loop to 61.8 at four