👁 image/png
This is the eighth in a series of models designed to replicate the prose quality of the Claude 3 models, specifically Sonnet and Opus. This model is fine-tuned on top of IntervitensInc/Llama-3.1-Minitron-4B-Width-Base-chatml.

Prompting

Model has been Instruct tuned with the ChatML formatting. A typical input would look like this:

"""<|im_start|>system
system prompt<|im_end|>
<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>
<|im_start|>assistant
"""

axolotl config

Credits

• anthracite-org/Stheno-Data-Filtered
• anthracite-org/kalo-opus-instruct-22k-no-refusal
• lodrick-the-lafted/NopmWritingStruct
• NewEden/Gryphe-3.5-16k-Subset
• Epiculous/Synthstruct-Gens-v1.1-Filtered-n-Cleaned
• Epiculous/SynthRP-Gens-v1.1-Filtered-n-Cleaned

This model has been a team effort, and the credits goes to all members of Anthracite.

Training

The training was done for 2 epochs. We used 2 x RTX 6000s GPUs graciously provided by Kubernetes_Bad for the full-parameter fine-tuning of the model.

👁 Built with Axolotl

Safety

...

Base model

nvidia/Llama-3.1-Minitron-4B-Width-Base

(8)

12 items • Updated Aug 23, 2024 • 7