It's an SFT ontop of the largest Qwen which nobody seems to have done yet, Trained with a collection of normal Austral(Books, RP Logs, LNs, etc) datasets. I do not totally endorse the model yet and i think there's much work to be done in trying to make a decensored and well-writing finetune of this model but I just released this to give everyone a slight taste of a qwen3 finetune.
It was also a way for us to test out some Optims to actually get this model to train, Thanks to Intervitens <3 We used torchtune & an experimental hacky pytorch build: https://github.com/pytorch/pytorch/pull/156203 We trained this model over 24 Hours on 8xB200s. Graciously provided by Deepinfra & Cognitive Computations. Speeds were similar to a 70B trained with roughly the same data.
Prompting
Model has been tuned with the ChatML formatting. A typical input would look like this:
<|im_start|>system
system-prompt<|im_end|>
<|im_start|>user
user-prompt<|im_end|>
<|im_start|>assistant
<think>
</think>
assistant-prompt<|im_end|>
Torchtune config
Thank you so much for Intervitens for helping train this model:
Credits
Thank you to Lucy Knada, Auri, Intervitens, Deepinfra, Cognitive Computations and the rest of Anthracite &
Training
The training was done for 4 epochs. We used 8 x B200s GPUs graciously provided by Deepinfra for the full-parameter fine-tuning of the model, Tuning was done all thanks to Intervitens.
Safety
It's still aligned to the beliefs of the Chinese Communist Party:
👁 image/png
- Downloads last month
- 14