Kabyle LoRA Adapter for AfriqueQwen3.5-4B

Fine-tuned LoRA adapter for Kabyle (kab) text generation, based on McGill-NLP/AfriqueQwen3.5-4B-ExtendedCM-ExtendedLangs.

This adapter eliminates the language confusion (Swahili, Somali, Igbo bleeding) present in the base model and produces coherent, grammatically correct Kabyle sentences.

---

Training Details

---

Dataset Filtering

• Source: Tatoeba sentences dump - 787,648 raw Kabyle sentences (lang == "kab")
• Quality filters applied:
  • Sentence length: 3-50 words
  • Minimum 2 Kabyle-specific characters: ɣ ṭ ḍ č ǧ ɛ ḥ ṛ ṣ ẓ
  • Contamination removal: excluded sentences containing Greek/Cyrillic look-alikes:
    • Greek epsilon ε (U+03B5)
    • Cyrillic epsilon Ԑ (U+0510), ԑ (U+0511)
    • Greek gamma γ (U+03B3), Γ (U+0393)
• Result: 326,147 clean, filtered Kabyle sentences

---

Results

Before vs After Fine-Tuning

Loss Progression

Key improvements:

• Zero language confusion - no more Swahili, Somali, or Igbo bleeding
• Grammatically correct - proper preverbs (ad, d-), clitic pronouns (t-, i), possessives (-iw)
• Semantically coherent - sentences make sense in context
• Natural endings - completes with periods and logical conclusions
• Cultural references - recognizes Kabyle names (Ḥmed) and places (Iɣil Azwaw)

---

Usage

Requirements

pip install transformers peft accelerate bitsandbytes torch

Load and Generate

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

BASE_MODEL = "McGill-NLP/AfriqueQwen3.5-4B-ExtendedCM-ExtendedLangs"
ADAPTER_MODEL = "boffire/AfriqueQwen3.5-4B-Kabyle-LoRA"

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
 load_in_4bit=True,
 bnb_4bit_compute_dtype=torch.float16,
 bnb_4bit_use_double_quant=True,
)

# Load tokenizer and base model
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
 BASE_MODEL,
 quantization_config=bnb_config,
 device_map="auto",
 trust_remote_code=True
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, ADAPTER_MODEL)

# Generate Kabyle text
prompt = "Taqbaylit d"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
 **inputs,
 max_new_tokens=25,
 do_sample=True,
 temperature=0.7,
 top_p=0.9,
 pad_token_id=tokenizer.eos_token_id
)
completion = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(completion)
# Output: Taqbaylit d tamaziɣt.

Example Prompts

test_prompts = [
 "Ur bɣiɣ ara",
 "Taqbaylit-iw d",
 "Anwa i",
 "Nekkni n",
 "Aselkim-iw ur",
 "Iḍelli tella",
 "Lemmer ad",
 "Aqcic i",
 "Ur tett ara",
 "Ass-a d",
]

---

Repository Contents

---

Base Model

This adapter is designed to be used with:

• Base: McGill-NLP/AfriqueQwen3.5-4B-ExtendedCM-ExtendedLangs
• Type: Causal Language Model (Base/Pre-trained)
• Parameters: 4B
• Context Length: 32,768 tokens
• African Languages: 50 (including Kabyle)

---

Citation

If you use this model, please cite:

AfriqueLLM paper:

@misc{yu2026afriquellmdatamixingmodel,
 title={AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages},
 author={Hao Yu and Tianyi Xu and Michael A. Hedderich and Wassim Hamidouche and Syed Waqas Zamir and David Ifeoluwa Adelani},
 year={2026},
 eprint={2601.06395},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2601.06395}
}

Tatoeba project:

• https://tatoeba.org

---

Limitations

Model Architecture

• Base model, not chat: This adapter performs text completion only. It does not follow instructions, answer questions, or engage in conversation. For chat capabilities, further supervised fine-tuning (SFT) with instruction-response pairs is required.
• No chat template: The model does not recognize system/user/assistant roles. Inputs are treated as raw text to be continued.
• Small LoRA rank (r=8): Only ~917K parameters are trainable (0.02% of total). Complex reasoning, multi-step logic, or rare morphological patterns may be beyond capacity.

Training and Data

• Short context window: Trained on max 64 tokens. Generation beyond ~64 tokens may degrade in quality or repeat phrases.
• Domain limited: Training data (Tatoeba) consists primarily of short, general-domain sentences. The model lacks exposure to technical, legal, medical, or academic Kabyle.
• No dialect awareness: Does not distinguish between Kabyle sub-dialects (At Mengellat, At Weɣlis, Tasaḥlit, etc.). Output may blend dialectal features.
• Script limited: Trained exclusively on Latin-script Kabyle. Tifinagh and Arabic-script Kabyle are not supported.

Semantic Coherence

• Grammatically correct but semantically inconsistent: The model reliably generates morphologically valid Kabyle (proper preverbs, clitics, possessives) but may produce sentences that are grammatically well-formed yet semantically odd or factually nonsensical. This is expected behavior for a base causal language model fine-tuned on text completion rather than instruction-following or reasoning tasks.
• Examples of semantic drift:
  • "Aselkim-iw ur yelli ara deg wexxam" (grammatical but semantically odd: "My computer is not at home")
  • "Taqbaylit-iw d taɛrabt i tt-yeččan" (contradiction: "My Kabyle was eaten by Arabic")
• Cause: The model predicts the most statistically likely next token based on training patterns. It does not understand meaning, fact-check, or reason about the real world. Lower sampling temperature or greedy decoding (do_sample=False) improves consistency at the cost of creativity.

Content and Safety

• No safety filtering: The model has no built-in guardrails. It may generate toxic, biased, harmful, or culturally inappropriate content if prompted.
• Hallucination risk: As a base language model, it has no factual grounding. It may invent false information about Kabyle history, people, places, or events.
• Gender bias: Training data may reflect historical gender stereotypes present in Tatoeba contributions.
• Temporal cutoff: Factual knowledge is limited to the base model's training cutoff. No awareness of events after approximately 2024.

Multilingual Behavior

• French interference: Despite fine-tuning, the base model's strong French knowledge may cause French words to appear in Kabyle completions, especially for modern/technical concepts.
• Code-switching untested: Natural Kabyle-French code-switching (common in Algeria) is not explicitly handled and may produce unpredictable results.
• No translation capability: The model was not trained on parallel data. English to Kabyle or Kabyle to French translation will be unreliable.

Compute and Deployment

• 4-bit quantization required: The adapter assumes 4-bit (NF4) base model loading. Running in full precision requires ~8GB VRAM; FP16 requires ~16GB.
• GPU recommended: CPU inference is possible but extremely slow (~10-30x slower than T4 GPU).

---

License

This model is released under the CC BY 4.0 License (same as the base model).

---

Acknowledgments

• McGill-NLP for the AfriqueLLM suite
• Tatoeba contributors for the Kabyle sentence corpus
• Qwen team for the base architecture
• Hugging Face for the PEFT and Transformers libraries

Base model

Qwen/Qwen3.5-4B-Base

McGill-NLP/AfriqueQwen3.5-4B-50Langs

(1)