nyrkr-joker-llama

New Yorker cartoon description and caption -> attempt at a joke explanation

Technical details:

• Based on LLaMa2-7b-hf (version 2, 7B params)
• Used QLoRA to fine-tune on 1.2k rows of New Yorker caption contest
• Merged LLaMa2 with the adapter weights (from checkpoint step=160, epoch=2.7)

Prompt options

The original paper, Figure 10 uses this format for joke explanations:

`In this task, you will see a description of an uncanny situation. Then, you will see a joke that was written about the situation. Explain how the joke relates to the situation and why it is funny. ###

{few-shot examples separated by ###, newline after "explanation of the caption:"} This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller. caption: Can I interest you in opening an offshore account? explanation of the caption: `

In training, I used just the individual example:

This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller. caption: Can I interest you in opening an offshore account? explanation of the caption:\n

In inference, I had some better results with a more natural prompt (no newline or space at end)

This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller. caption: Can I interest you in opening an offshore account? the caption is funny because

Training script

Trained on a V100

git clone https://github.com/artidoro/qlora
cd qlora

pip3 install -r requirements.txt --quiet

! cd qlora && python qlora.py \
 --model_name_or_path ../llama-2-7b-hf \
 --output_dir ../thatsthejoke \
 --logging_steps 20 \
 --save_strategy steps \
 --data_seed 42 \
 --save_steps 80 \
 --save_total_limit 10 \
 --evaluation_strategy steps \
 --max_new_tokens 64 \
 --dataloader_num_workers 1 \
 --group_by_length \
 --logging_strategy steps \
 --remove_unused_columns False \
 --do_train \
 --lora_r 64 \
 --lora_alpha 16 \
 --lora_modules all \
 --double_quant \
 --quant_type nf4 \
 --bits 4 \
 --warmup_ratio 0.03 \
 --lr_scheduler_type constant \
 --gradient_checkpointing \
 --dataset /content/nycaptions.jsonl \
 --dataset_format 'self-instruct' \
 --source_max_len 16 \
 --target_max_len 512 \
 --per_device_train_batch_size 1 \
 --gradient_accumulation_steps 16 \
 --max_steps 250 \
 --eval_steps 187 \
 --learning_rate 0.0002 \
 --adam_beta2 0.999 \
 --max_grad_norm 0.3 \
 --lora_dropout 0.1 \
 --weight_decay 0.0 \
 --seed 0