Summary
Distilled with Distily library using teacher model gpt2 on dataset wikimedia/wikipedia.
Model Architecture:
- Architecture:
GPT2LMHeadModel - Total Parameters: 124,439,808
- Data Type (dtype): torch.bfloat16
- Model Size: 0.24 GB
Evaluation Metrics Comparison
| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
|---|---|---|---|---|---|---|---|---|---|
| teacher eval | 43.75 | 61.75 | 11.8125 | 19.125 | |||||
| 0 | 0 | 949187772416.0 | 76416058130432.0 | 21.75 | 0.1221 | 16.381 | 8.191 | 3556769792.0 | 13950053777408.0 |
| 20 | 1.0 | 13248.0 | 64000.0 | 5.6562 | 0.0646 | 30.969 | 15.485 | 7712.0 | 181248.0 |
Resource Usage Comparison
- VRAM Use: 7.9388 GB
`# Distillation (Teacher -> Student) Architecture Difference:
- Architecture:
GPT2LMHeadModel->GPT2LMHeadModel - Total Parameters: 124,439,808 -> 124,439,808
- Data Type (dtype): 124439808 -> torch.bfloat16
- Model Size: 0.16 GB -> 0.24 GB
Train Dataset
Trained on 149,632 tokens from the wikimedia/wikipedia dataset.
- Num Samples:
158 - Subset:
20231101.en - Split:
train
Training Objective
DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl))
Hyperparameters
The following hyperparameters were used during training:
Framework Versions
- Distily 0.2.0
- Transformers 4.44.0
- Pytorch 2.3.0
- Datasets 2.21.0
- Downloads last month
- 7
Safetensors
Model size
0.1B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for lapp0/distily_modelcard_try
Base model
openai-community/gpt2