Nvidia and an Indian University Lab Just Open-Sourced the Pipeline for Training AI in 22 Languages — The Data Itself Is Still 'Coming Soon'

On October 16, Nvidia and AI4Bharat — a language-AI lab at IIT Madras — announced Nemotron-pretraining-Indic-v1, a synthetic training corpus built for the 22 languages named in India's Constitution. The headline number is 2.8 trillion tokens: 2,601.0 billion in native scripts and 224.6 billion more romanized, counted with Google's Gemma-4 tokenizer. What's actually available today, though, is narrower than that number suggests — the generation pipeline that produced the data is open now, but the corpus itself is still listed as releasing soon, with no license attached yet.

That distinction matters because the pipeline, not the dataset, is the part Nvidia is positioning as the durable contribution. Published as an open NeMo Data Designer plugin, it takes English source documents — in this case 15 subsets of Nvidia's existing Nemotron-CC-v2 web corpus — and runs them through machine translation and automated quality scoring to produce training text in languages that have almost none online in bulk.

Two pipelines, one big gap in coverage

The project splits into two tracks. Twelve languages, including Hindi, Tamil, Bengali, and Urdu, get both native-script and romanized versions: Google's Gemma-4-31B-it model translates full documents, then grades its own output against the English source using a rubric modeled on the Multidimensional Quality Metrics framework used in professional translation. Anything scoring below 75 out of 100 is sent back for automated repair and rescored, rather than discarded outright.

The other ten languages — Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi — get native-script data only, generated instead through Indic Translate, a model from Bodhan AI, a nonprofit incubated at IIT Madras. These are the languages with the thinnest digital footprint and the least mature machine-translation tooling, and it shows in the engineering: long Santali documents had to be translated in overlapping windows to fit model context limits, and Manipuri text is routed through Bengali script before being converted into its native Meitei Mayek, because direct translation tooling for that path doesn't really exist yet.

Nvidia tested the corpus by continuing pretraining on its Nemotron 3.5 Super VL model with 10 billion Indic tokens — a mix of the new synthetic data and some naturally written Indic text — and reported benchmark gains of roughly 5 to 10 points across Indic-language tasks. Nvidia's own writeup flags the obvious caveat: because natural and synthetic data were mixed together in that test, there's no way to isolate how much of the improvement the new corpus is actually responsible for.

The self-grading problem

The token total alone does not establish language-level quality or downstream model performance.

That line, from independent coverage of the release, points at the release's weakest spot. The quality-control loop for the twelve-language track uses Gemma-4-31B-it to translate and then uses the same model family to grade whether the translation is good — a closed loop that can miss systematic errors the model itself is prone to making, especially in lower-resource languages where ground-truth human evaluation is scarce to begin with. Nvidia acknowledges as much, noting the ten-language workflow in particular needs broader review from native speakers, and that no semantic deduplication was run across the corpus at all.

There's a second layer worth noting: a project framed around linguistic self-sufficiency for India's official languages is, under the hood, built on Google's Gemma models for translation and judging, running on Nvidia's commercial data-generation tooling. That's not unusual for how open-source AI infrastructure actually gets built — almost no one trains translation systems for Santali or Kashmiri from scratch — but it's a reminder that 'open' here means the recipe is published, not that the dependency on a handful of large labs' base models has gone away.

Why the languages chosen are the real story

India's 22 scheduled languages are, by the 2011 census Nvidia cites, the first languages of roughly 97% of the country's population, yet large language models have historically been trained overwhelmingly on English and Chinese text, with Hindi and a handful of others getting token counts in the low single-digit percentages at best. AI4Bharat, whose director Mitesh Khapra made Time's AI 100 list last year, has spent several years building smaller monolingual corpora for exactly this gap; this release is a scale-up of that effort paired with Nvidia's GPU-accelerated tooling, alongside a companion benchmark suite, Indic-LLM-Eval-Suite, built across eight capability categories and wired into Nvidia's NeMo-Gym evaluation framework.

The practical test will come once the corpus itself actually ships and outside teams — not just Nvidia's own benchmark run — can train on it and check whether translated text holds up against text written natively by Konkani or Bodo speakers rather than translated into those scripts by a model that was never fluent in them to begin with. Until then, what exists is a well-documented method for generating Indic training data at scale, not yet the data.