Static Embeddings with ModernBERT tokenizer finetuned on various datasets

This is a static embedding sentence-transformers model trained on various english text retrieval datasets (see below for details). It maps sentences & paragraphs to a 1024-dimensional dense vector space with an intended use of english based semantic search. Notably, this model has 0 active parameters and was trained with Matryoshka Representation Learning, allowing for efficient and fast CPU/GPU based embedding operations.

This is based on prior work from Tom Aarsen, see blog post here.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: answerdotai/ModernBERT-base Tokenizer
• Maximum Sequence Length: inf tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Training Datasets:
  • gooaq
  • msmarco
  • squad
  • s2orc
  • allnli
  • paq
  • trivia_qa
  • msmarco_10m
  • swim_ir
  • pubmedqa
  • miracl
  • mldr
  • mr_tydi
• Languages: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
 (0): StaticEmbedding(
 (embedding): EmbeddingBag(50368, 1024, mode='mean')
 )
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("AdamLucek/static-retrieval-mrl-MBERT-base-en-v1")
# Run inference
sentences = [
 "What is the name of Nan's primary teacher?",
 "While still young, Nan left his military career so that he could commit himself fully to his study of Buddhism and to meditation. In 1942 at age 24, he went on a three-year meditation retreat in the Emei Mountains. It is said that it was there that he verified his experiences against the Chinese Buddhist canon. During this time, Nan's primary teacher was Yuan Huanxian (袁煥仙; 1887\xa0– 1966).",
 'Among other schemes that have been considered is a rapid transport link between Heathrow and Gatwick Airports, known as "Heathwick", which would allow the airports to operate jointly as an airline hub; In 2018, the Department for Transport began to invite proposals for privately funded rail links to Heathrow Airport. Projects being considered under this initiative include:The Mayor of London\'s office and Transport for London commissioned plans in the event of Heathrow\'s closure—to replace it by a large built-up area. Some of the plans seem to show terminal 5, or part of it, kept as a shopping centre.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Datasets: NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with InformationRetrievalEvaluator

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:

  {
   "dataset_names": [
   "climatefever",
   "dbpedia",
   "fever",
   "fiqa2018",
   "hotpotqa",
   "msmarco",
   "nfcorpus",
   "nq",
   "quoraretrieval",
   "scidocs",
   "arguana",
   "scifact",
   "touche2020"
   ]
  }
  

Training Details

Training Datasets

Evaluation Datasets

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 2048
• per_device_eval_batch_size: 2048
• learning_rate: 0.2
• num_train_epochs: 1
• warmup_ratio: 0.1
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 4.1.0
• Transformers: 4.51.3
• PyTorch: 2.8.0.dev20250319+cu128
• Accelerate: 1.6.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
 title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
 author = "Reimers, Nils and Gurevych, Iryna",
 booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
 month = "11",
 year = "2019",
 publisher = "Association for Computational Linguistics",
 url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
 title={Matryoshka Representation Learning},
 author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
 year={2024},
 eprint={2205.13147},
 archivePrefix={arXiv},
 primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
 title={Efficient Natural Language Response Suggestion for Smart Reply},
 author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
 year={2017},
 eprint={1705.00652},
 archivePrefix={arXiv},
 primaryClass={cs.CL}
}

Fine tuned sentence transformers for document retrieval • 4 items • Updated Nov 23, 2025