gemma4-31b-it-glimmer-rp-r16a32
This model is a fine-tuned version of google/gemma-4-31B-it.
W&B run: https://wandb.ai/cooawoo-personal/Gemma4-31B/runs/nbmb3v4h
Training procedure
Hyperparameters
| Parameter | Value |
|---|---|
| Learning rate | 1e-05 |
| LR scheduler | rex (custom; max_lr=1e-5, min_lr=1e-6, warmup_ratio=0.05)_WITH_WARMUP |
| Per-device batch size | 1 |
| Gradient accumulation | 4 |
| Effective batch size | 4 |
| Epochs | 1 |
| Max sequence length | 6144 |
| Optimizer | OptimizerNames.PAGED_ADAMW_8BIT |
| Warmup ratio | 0.05 |
| Max gradient norm | 1.0 |
| Precision | bf16 |
| Loss type | nll |
| Assistant-only loss | yes |
| Chunked cross-entropy | yes |
LoRA configuration
| Parameter | Value |
|---|---|
| Rank (r) | 16 |
| Alpha | 32 |
| Target modules | .*language_model.layers.\d+.(self_attn.(q |
| Quantization | 4-bit (nf4) |
Dataset statistics
| Dataset | Samples | Total tokens | Trainable tokens |
|---|---|---|---|
| writing_critique.jsonl | 1,586 | 1,317,233 | 599,216 |
| instruct.jsonl | 962 | 933,867 | 838,340 |
| marvin_style_bible.jsonl | 2,549 | 11,096,548 | 10,492,459 |
| rp_generation_mistral.jsonl | 255 | 825,572 | 375,698 |
| rp_analysis.jsonl | 244 | 725,115 | 177,861 |
| rp_generation_final.jsonl | 129 | 513,974 | 240,019 |
| Total | 5,725 | 15,412,309 | 12,723,593 |
Framework versions
- PEFT 0.18.1
- Loft: 0.1.0
- Transformers: 5.5.4
- Pytorch: 2.6.0+cu124
- Datasets: 4.6.1
- Tokenizers: 0.22.2
- Downloads last month
- -
Safetensors
Model size
33B params
Tensor type
BF16
·
Model tree for BirdToast/Gemma-4-31B-glimmer-rp-v0.1
Base model
google/gemma-4-31B Finetuned
google/gemma-4-31B-it