Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
NVIDIA and IIT Madras's AI4Bharat released the Nemotron-pretraining-Indic-v1 pretraining corpus, roughly 2.8 trillion tokens covering all 22 scheduled languages of India, with 2,601.0B natural-script tokens and 224.6B romanized tokens (Gemma-4 tokenizer statistics); it will soon ship alongside an open-source, GPU-accelerated data generation pipeline.