NVIDIA and AI4Bharat release 2.8T-token Indic pretraining corpus

NVIDIA and AI4Bharat have released the Nemotron-pretraining-Indic-v1 corpus, an open pretraining collection containing approximately 2.8 trillion tokens ^[1]. Alongside the text repository, the team published an open, GPU-accelerated pipeline built specifically for data generation and automated translation ^[1].

The dataset covers native-script text across 22 Indian languages, while also incorporating romanized text for 12 languages ^[1]. This dual format provides researchers with linguistic coverage in both traditional alphabets and Latin-transliterated text. The release focuses on building out the linguistic foundations needed to train large-scale language systems in these specific vernaculars.

The initiative addresses persistent resource bottlenecks for regional computing. AI4Bharat head Mitesh Khapra noted the challenge confronting developers in the ecosystem, pointing out that "we do not have enough data for Indian languages" ^[1]. The new package provides both the core text material and the technical tooling required to manufacture and translate tokens on accelerated hardware ^[1].

By open-sourcing the underlying processing architecture in addition to the tokens, the collaborators are offering an end-to-end framework for model builders who work with non-English languages ^[1]. Teams can run the GPU pipeline to produce and translate training inputs rather than relying strictly on static web scrapes, supporting ongoing research and model development across all 22 targeted languages ^[1].