GPT-Neo 125M LoRA for TinyStories

This repo contains a LoRA adapter for the "EleutherAI/gpt-neo-125m" which has been trained on the karpathy/tinystories-gpt4-clean dataset for text generation.

Model Information

• Base model: EleutherAI/gpt-neo-125m
• Fine-tuned on: karpathy/tinystories-gpt4-clean
• Training Loop: Trained for 12,000 steps on a RTX 3050 (4GB VRAM)

Training Configuration

This adapter was trained with the following settings:

• LoRA rank: 16
• LoRA alpha: 32
• Target modules: c_attn, c_proj
• Dropout: 0.05
• Bias: none
• Task type: CAUSAL_LM
• Precision: fp16
• Gradient checkpointing: enabled
• Learning rate: 2e-5
• Batch size: 1
• Gradient accumulation: 4
• Save strategy: epoch

How to Use

Load the adapter from Hugging Face Hub

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model = "EleutherAI/gpt-neo-125m"
adapter_repo = "DireDreadlord/TinyStories-GPT-Neo-LoRA"

tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(base_model)
model = PeftModel.from_pretrained(model, adapter_repo)

Notes

• This repository stores only the LoRA adapter. The base model weights are not included here.
• To use the adapter, you must load the same base model (EleutherAI/gpt-neo-125m).

Example Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model = "EleutherAI/gpt-neo-125m"
adapter_path = "./gptn125-lora-tinystories"

tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(base_model)
model = PeftModel.from_pretrained(model, adapter_path)

text = "Once upon a time"
inputs = tokenizer(text, return_tensors="pt").input_ids
outputs = model.generate(inputs, max_length=150, do_sample=True, top_p=0.95, temperature=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Files Included

• adapter_model.safetensors — LoRA adapter weights
• adapter_config.json — LoRA configuration and metadata
• training_args.bin — training arguments saved by Hugging Face Trainer
• README.md — model card and usage instructions

Base model