intern-rp-lora
This model is a fine-tuned version of internlm/internlm3-8b-instruct on the ToastyPigeon/some-rp, the BeaverAI/cedo-unalignment, the BeaverAI/foundRP, the PocketDoc/Dans-Prosemaxx-Gutenberg, the ToastyPigeon/SpringDragon-Instruct, the allenai/tulu-3-sft-personas-instruction-following and the allura-org/fujin-cleaned-stage-2 datasets. It achieves the following results on the evaluation set:
- Loss: 1.7197
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 69
- distributed_type: multi-GPU
- num_devices: 4
- total_train_batch_size: 4
- total_eval_batch_size: 4
- optimizer: Use OptimizerNames.PAGED_ADEMAMIX_8BIT and the args are: No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 20
- num_epochs: 2
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 2.2794 | 0.0013 | 1 | 1.8317 |
| 1.6416 | 0.1 | 75 | 1.7826 |
| 2.3547 | 0.2 | 150 | 1.7643 |
| 1.9114 | 0.3 | 225 | 1.7546 |
| 2.0004 | 0.4 | 300 | 1.7474 |
| 2.2052 | 0.5 | 375 | 1.7428 |
| 1.9314 | 0.6 | 450 | 1.7377 |
| 2.202 | 0.7 | 525 | 1.7350 |
| 2.2453 | 0.8 | 600 | 1.7303 |
| 1.8392 | 0.9 | 675 | 1.7283 |
| 1.7018 | 1.0 | 750 | 1.7271 |
| 1.9736 | 1.0987 | 825 | 1.7264 |
| 2.0917 | 1.1987 | 900 | 1.7245 |
| 1.5679 | 1.2987 | 975 | 1.7239 |
| 2.0799 | 1.3987 | 1050 | 1.7225 |
| 1.8398 | 1.4987 | 1125 | 1.7220 |
| 1.9806 | 1.5987 | 1200 | 1.7211 |
| 1.7334 | 1.6987 | 1275 | 1.7209 |
| 2.1457 | 1.7987 | 1350 | 1.7205 |
| 1.7804 | 1.8987 | 1425 | 1.7202 |
| 2.1652 | 1.9987 | 1500 | 1.7197 |
Framework versions
- PEFT 0.14.0
- Transformers 4.47.1
- Pytorch 2.5.1+cu124
- Datasets 3.2.0
- Tokenizers 0.21.0
- Downloads last month
- 1
GGUF
Model size
9B params
Architecture
llama
Hardware compatibility
Log In to add your hardware
8-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for ToastyPigeon/intern-rp-lora
Base model
internlm/internlm3-8b-instruct