DNA and Block Diffusion

Untrained architecture test using the Block Diffusion architecture and AgroNT's six-nucleotide-length tokens.

Loading model

from transformers import AutoModelForMaskedLM
m = AutoModelForMaskedLM.from_pretrained(
 "monsoon-nlp/dna-blockdiff",
 trust_remote_code=True,
)

Generating text

cd bd3lms
python -u main.py \
 loader.eval_batch_size=1 \
 model=small \
 algo=bd3lm \
 algo.T=900 \
 algo.backbone=hf_dit \
 algo.sampler=analytic \
 data=openwebtext-split \
 model.length=2048 \
 block_size=4 \
 wandb=null \
 mode=sample_eval \
 eval.checkpoint_path="monsoon-nlp/dna-blockdiff" \
 model.attn_backend=sdpa \
 sampling.nucleus_p=0.9 \
 sampling.kv_cache=true \
 sampling.logdir=$PWD/sample_logs/samples_genlen_bd3lm_blocksize4 \
 data.tokenizer_name_or_path="monsoon-nlp/dna-blockdiff"

Base model