Bridging the Gap in Less-Resourced Languages: Building a Benchmark for Kyrgyz Language Models

The evaluation of Large Language Models (LLMs) across diverse languages is crucial for ensuring equitable technological progress. However, most multilingual benchmarks are created by translating existing English-centric datasets, a process that often introduces errors and fails to capture the unique linguistic and cultural nuances of the target language. This is especially problematic for lower-resourced languages like Kyrgyz, which lack dedicated, high-quality evaluation resources. To address this gap, we introduce KyrgyzLLM-Bench, the first comprehensive benchmark suite designed to measure deep language understanding in Kyrgyz. Our suite features two natively created components: a massive multitask multiple-choice dataset, KyrgyzMMLU, covering subjects from the Kyrgyz national curriculum, and a reading comprehension dataset, KyrgyzRC. These are supplemented by carefully translated versions of WinoGrande, HellaSwag, BoolQ, TruthfulQA and GSM8K to assess commonsense reasoning, reading comprehension and math. We provide an extensive evaluation of over X state-of-the-art open- and closed-source LLMs, analyzing their performance in zero-shot and few-shot settings. Our results provide the first in-depth look at the capabilities and limitations of modern LLMs on complex Kyrgyz tasks. We publicly release KyrgyzLLM-Bench to foster further research and development for the Kyrgyz language.