Tiny Aya - Cohere's Mini Multilingual Models

Choosing a language model for non-English languages is often a guessing game—especially for low-resource languages with limited internet data and tokenization schemes that break words into inefficient fragments. The core insight here is that Coher’s new “tiny” multilingual models are designed to close that gap by combining broad pretraining across 70+ languages with region-focused post-training, then packaging the result into small ~3.3B-parameter models that can be run and fine-tuned more easily than large multilingual systems.

The transcript lays out why recommendations are hard. Some languages may never have appeared in training, largely because low-resource languages lack enough web data—sometimes because communities don’t use Wikipedia extensively, leaving few pages to harvest. Even when training data exists, tokenization can quietly sabotage performance: older tokenizers (the example given is the Llama 2 tokenizer) can require many more tokens for scripts like Thai or Greek, forcing the model to learn meaning from near character-by-character pieces. That inefficiency can make multilingual learning harder, even if the model is otherwise capable.

Recent progress has improved multilingual coverage for major languages, helped by better tokenizers and multilingual datasets. Projects such as Translate Gemma (built on the Gemma 3 stack) and improvements in models like Llama 3 and Quen variants are credited with expanding language support—often by adding multilingual data and using larger tokenizers (the transcript cites 250k+ tokenizers for Translate Gemma). But smaller models still lag because they’re typically trained on fewer tokens and their post-training recipes may not prioritize multilingual behavior.

Coher’s response is a suite of tiny multilingual models released as both a research artifact and practical starting points. The base model is pre-trained on 70+ languages, explicitly including data from many low-resource languages. On top of that, four post-trained variants are built from the same base: a “tiny global” instruction-tuned model meant to cover most pretraining languages, plus three region-specialized models created by mixing languages grouped by geography and linguistic relatedness.

Those region models are “tiny earth” (West Asia, Africa, and some European languages—Arabic, Turkish, Hebrew, plus 10 African languages and 31 European languages), “tiny fire” (South Asia, emphasizing scripts that differ strongly—Hindi, Bengali, Tamil, Nepali, with English often appearing via code-switching), and “tiny water” (Asia-Pacific languages such as Tagalog, Bahasa, Vietnamese, Thai, Chinese, plus low-resource languages like Cam and Burmese, while also including some West Asia and European mixes). The post-training approach is described as region-specific SFT models that are then merged into the released variants.

A key technical detail is that Coher trained its own tokenizer rather than using an off-the-shelf one. Efficiency varies by language: the transcript notes Coher’s tokenizer can beat Gemma 3’s for some languages, while Gemma 3 may be slightly better for others—so results should be checked per target language. The suite also includes quantized versions intended for immediate use in lightweight runtimes (e.g., via Llama-style tooling). With ~3B models, the practical pitch is clear: these can run on phones and support mobile apps for countries and languages underserved by larger multilingual models.

Overall, the release positions Coher’s tiny models as a practical option when mainstream multilingual models underperform—especially for specific regions and low-resource languages—while also fitting the growing trend toward better multilingual tokenization and multilingual-focused training recipes in smaller architectures.