Alpaca LoRa 7B

This repository contains a LLaMA-7B fine-tuned model on the Standford Alpaca cleaned version dataset.

⚠️ I used LLaMA-7B-hf as a base model, so this model is for Research purpose only (See the license)

Usage

Creating prompt

The model was trained on the following kind of prompt:

def generate_prompt(instruction: str, input_ctxt: str = None) -> str:
 if input_ctxt:
 return f"""Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input_ctxt}

### Response:"""
 else:
 return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:"""

Using the model

import torch
from transformers import GenerationConfig, LlamaTokenizer, LlamaForCausalLM

tokenizer = LlamaTokenizer.from_pretrained("chainyo/alpaca-lora-7b")
model = LlamaForCausalLM.from_pretrained(
 "chainyo/alpaca-lora-7b",
 load_in_8bit=True,
 torch_dtype=torch.float16,
 device_map="auto",
)
generation_config = GenerationConfig(
 temperature=0.2,
 top_p=0.75,
 top_k=40,
 num_beams=4,
 max_new_tokens=128,
)

model.eval()
if torch.__version__ >= "2":
 model = torch.compile(model)

instruction = "What is the meaning of life?"
input_ctxt = None # For some tasks, you can provide an input context to help the model generate a better response.

prompt = generate_prompt(instruction, input_ctxt)
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
input_ids = input_ids.to(model.device)

with torch.no_grad():
 outputs = model.generate(
 input_ids=input_ids,
 generation_config=generation_config,
 return_dict_in_generate=True,
 output_scores=True,
 )

response = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)
print(response)

>>> The meaning of life is to live a life of meaning.

All the Llama models I fine-tuned • 2 items • Updated Mar 2