Arabic AI TinyStories GPT (12M parameters)

A small decoder-only GPT trained from scratch on TinyStories. Built line-by-line in PyTorch as part of the Build a GPT From Scratch on a 6 GB GPU YouTube playlist series — no transformers model classes.

Model description

The model writes short children's stories in English. It was trained with a custom tokenizer (tinystories_tokenizer.json) and a hand-written GPT class in pure PyTorch.

Files in this repository

Usage

Install dependencies

pip install torch tokenizers huggingface_hub

Download and generate

import json
import torch
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

REPO_ID = "luayas1977/arabicai-tinystories-gpt"

# Download artifacts
ckpt_path = hf_hub_download(repo_id=REPO_ID, filename="model.pt")
tok_path = hf_hub_download(repo_id=REPO_ID, filename="tinystories_tokenizer.json")
model_py = hf_hub_download(repo_id=REPO_ID, filename="model.py")

# Load architecture (download model.py into cwd or add to path)
import importlib.util
spec = importlib.util.spec_from_file_location("gpt_model", model_py)
gpt_model = importlib.util.module_from_spec(spec)
spec.loader.exec_module(gpt_model)
GPT = gpt_model.GPT

device = "cuda" if torch.cuda.is_available() else "cpu"
checkpoint = torch.load(ckpt_path, map_location=device, weights_only=False)

model = GPT(checkpoint["config"]).to(device)
model.load_state_dict(checkpoint["model"])
model.eval()

tokenizer = Tokenizer.from_file(tok_path)
prompt = "Once upon a time"
prompt_ids = tokenizer.encode(prompt).ids
idx = torch.tensor([prompt_ids], dtype=torch.long, device=device)

eos_id = tokenizer.token_to_id("<|endoftext|>")
with torch.no_grad():
 out = model.generate(idx, max_new_tokens=200, temperature=0.8, top_k=40, eos_token_id=eos_id)

text = tokenizer.decode(out[0].tolist())
print(text)

Try it in the browser

Arabic AI TinyStories GPT Playground — interactive Gradio demo with temperature, top-k, and max-tokens controls.

Model repo: luayas1977/arabicai-tinystories-gpt

Training details

• Optimizer: AdamW (lr 3e-4, weight decay 0.1)
• Schedule: Linear warmup (500 steps) + cosine decay
• Batch size: 32 sequences × 256 tokens
• Hardware: Consumer GPU (~6 GB VRAM)
• Runtime: ~2 hours for 25,000 steps

Limitations

• Small model trained only on TinyStories — not suitable for general knowledge, code, or adult topics
• English only
• Context limited to 256 tokens
• May repeat story openers or run past a natural ending if EOS was not learned strongly (depends on data prep)

Citation / series

Part of Build a GPT From Scratch on a 6 GB GPU — YouTube playlist.