xCoT-Distill: Cross-Lingual Chain-of-Thought Distillation for Arabic Reasoning

This model is a QLoRA fine-tune of Qwen/Qwen3-8B trained to reason in English and answer in Arabic.

Method

xCoT-Distill generates (Arabic question, English CoT, Arabic answer) triples using a Qwen3-80B teacher model, then fine-tunes a student model using:

1. Cross-lingual SFT with Qwen3 native thinking format
2. Contrastive alignment loss on intermediate layers (10-18)
3. Curriculum training: binary → factual MCQ → reasoning MCQ

Training Data

~17K triples from:

• OALL/Arabic_MMLU (57 subject configs)
• OALL/Arabic_EXAMS (multi-subject Arabic exams)
• MBZUAI/ACVA (Arabic cultural value alignment, True/False)

Usage

Authors

Mark Kashirskiy, Artiom Lipinski, Ilya Makarov

Base model