Model Description

The model is fine-tuned from openai's ViT-L-14 using PMC_OA_beta and roco's data sets, using the tool open_clip（https://github.com/mlfoundations/open_clip）.

Training

python -m training.main \
 --save-frequency 2 \
 --zeroshot-frequency 1 \
 --report-to tensorboard \
 --train-data="/home/data1/ryanyip/huggingface-models/pmc_oa_beta/train.csv" \
 --val-data="/home/data1/ryanyip/huggingface-models/pmc_oa_beta/sample_valid.csv" \
 --csv-separator "," \
 --csv-img-key image \
 --csv-caption-key caption \
 --warmup 10000 \
 --batch-size=128 \
 --lr=1e-5 \
 --wd=0.2 \
 --epochs=30 \
 --workers=8 \
 --model "ViT-L-14" \
 --name "pmc_vit_l_14" \
 --pretrained "ViT-L-14_state_dict.pt" \
 --save-most-recent

ViT-L-14_state_dict.pt is the pretrained weight from openai/ViT-L-14