Asymmetric Static Embeddings with BERT uncased tokenizer finetuned on various datasets

This is a sentence-transformers model trained on the gooaq, msmarco, squad, s2orc, allnli, paq, trivia_qa, msmarco_10m, swim_ir, pubmedqa, miracl, mldr and mr_tydi datasets. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Maximum Sequence Length: inf tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Training Datasets:
  • gooaq
  • msmarco
  • squad
  • s2orc
  • allnli
  • paq
  • trivia_qa
  • msmarco_10m
  • swim_ir
  • pubmedqa
  • miracl
  • mldr
  • mr_tydi
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
 (0): Router(
 (sub_modules): ModuleDict(
 (query): Sequential(
 (0): StaticEmbedding(
 (embedding): EmbeddingBag(30522, 1024, mode='mean')
 )
 )
 (document): Sequential(
 (0): StaticEmbedding(
 (embedding): EmbeddingBag(30522, 1024, mode='mean')
 )
 )
 )
 )
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/asym-static-retrieval-mrl-en-v1")
# Run inference
queries = [
 "What is the name of Nan\u0027s primary teacher?",
]
documents = [
 "While still young, Nan left his military career so that he could commit himself fully to his study of Buddhism and to meditation. In 1942 at age 24, he went on a three-year meditation retreat in the Emei Mountains. It is said that it was there that he verified his experiences against the Chinese Buddhist canon. During this time, Nan's primary teacher was Yuan Huanxian (袁煥仙; 1887\xa0– 1966).",
 'Among other schemes that have been considered is a rapid transport link between Heathrow and Gatwick Airports, known as "Heathwick", which would allow the airports to operate jointly as an airline hub; In 2018, the Department for Transport began to invite proposals for privately funded rail links to Heathrow Airport. Projects being considered under this initiative include:The Mayor of London\'s office and Transport for London commissioned plans in the event of Heathrow\'s closure—to replace it by a large built-up area. Some of the plans seem to show terminal 5, or part of it, kept as a shopping centre.',
 'Formally, we represent each event in a distributed flow as a quadruple of the form (x,t,k,v), where "x" is the location (e.g., the network address of a physical node) at which the event occurs, "t" is the time at which this happens, "k" is a version, or a sequence number identifying the particular event, and "v" is a value that represents the event payload (e.g., all the arguments passed in a method call). Each distributed flow is a (possibly infinite) set of such quadruples that satisfies the following three formal properties. In addition to the above, flows can have a number of additional properties.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.4520, -0.0703, -0.0320]])

Evaluation

Metrics

Information Retrieval

• Datasets: NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with InformationRetrievalEvaluator

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:

  {
   "dataset_names": [
   "climatefever",
   "dbpedia",
   "fever",
   "fiqa2018",
   "hotpotqa",
   "msmarco",
   "nfcorpus",
   "nq",
   "quoraretrieval",
   "scidocs",
   "arguana",
   "scifact",
   "touche2020"
   ]
  }
  

Training Details

Training Datasets

Evaluation Datasets

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 2048
• per_device_eval_batch_size: 2048
• learning_rate: 0.2
• num_train_epochs: 1
• warmup_ratio: 0.1
• bf16: True
• batch_sampler: no_duplicates
• router_mapping: {'question': 'query', 'query': 'query', 'title': 'query', 'anchor': 'query', 'answer': 'document', 'positive': 'document', 'negative': 'document', 'negative1': 'document', 'negative2': 'document', 'negative3': 'document', 'negative4': 'document', 'negative5': 'document', 'negative6': 'document', 'negative7': 'document', 'negative8': 'document', 'negative9': 'document', 'negative10': 'document', 'negative11': 'document', 'negative12': 'document', 'negative13': 'document', 'negative14': 'document', 'negative15': 'document', 'negative16': 'document', 'negative17': 'document', 'negative18': 'document', 'negative19': 'document', 'negative20': 'document', 'abstract': 'document', 'text': 'document'}

All Hyperparameters

Training Logs

Environmental Impact

Carbon emissions were measured using CodeCarbon.

• Energy Consumed: 2.353 kWh
• Carbon Emitted: 0.630 kg of CO2
• Hours Used: 14.787 hours

Training Hardware

• On Cloud: No
• GPU Model: 1 x NVIDIA GeForce RTX 3090
• CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
• RAM Size: 31.78 GB

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 5.2.0.dev0
• Transformers: 4.55.4
• PyTorch: 2.7.1+cu126
• Accelerate: 1.6.0
• Datasets: 3.6.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
 title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
 author = "Reimers, Nils and Gurevych, Iryna",
 booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
 month = "11",
 year = "2019",
 publisher = "Association for Computational Linguistics",
 url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
 title={Matryoshka Representation Learning},
 author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
 year={2024},
 eprint={2205.13147},
 archivePrefix={arXiv},
 primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
 title={Efficient Natural Language Response Suggestion for Smart Reply},
 author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
 year={2017},
 eprint={1705.00652},
 archivePrefix={arXiv},
 primaryClass={cs.CL}
}