md7b-alpha
This model is a fine-tuned version of epfl-llm/meditron-7b on a set of datasets. It achieves the following results on the evaluation set:
- Loss: 1.0238
Evaluation
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 2000
- num_epochs: 4
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 2.1602 | 0.0 | 1 | 1.9066 |
| 1.1128 | 0.5 | 14744 | 1.1620 |
| 1.2463 | 1.0 | 29488 | 1.1288 |
| 0.8291 | 1.49 | 44232 | 1.1025 |
| 1.0524 | 1.99 | 58976 | 1.0771 |
| 1.0369 | 2.48 | 73720 | 1.0563 |
| 1.0402 | 2.98 | 88464 | 1.0299 |
| 0.943 | 3.47 | 103208 | 1.0271 |
| 1.0845 | 3.97 | 117952 | 1.0238 |
Framework versions
- Transformers 4.37.0.dev0
- Pytorch 2.0.1+cu118
- Datasets 2.16.1
- Tokenizers 0.15.0
Training procedure
The following bitsandbytes quantization config was used during training:
- quant_method: bitsandbytes
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16
- Downloads last month
- 3
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support