SentenceTransformer based on answerdotai/ModernBERT-base

This is a sentence-transformers model finetuned from answerdotai/ModernBERT-base on the msmarco, natural_questions, gooaq, ccnews and hotpotqa datasets. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Datasets:
  • msmarco
  • natural_questions
  • gooaq
  • ccnews
  • hotpotqa
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
 (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
 (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("hotchpotch/ModernBERT-embedding-CMNBRL")
# Run inference
queries = [
 "what is the best paying engineering job",
]
documents = [
 "The 20 highest-paying jobs for engineering majors. Engineering jobs pay well. To find out just how lucrative they really are, we turned to PayScale, the creator of the world's largest compensation database. To find the 20 highest-paying jobs for engineering majors, PayScale first identified the most common jobs for those with a bachelor's degree (and nothing more) who work full-time in the US. Chief architects and vice president's of business development topped the list, both earning an impressive $151,000 a year.",
 'Aviation is a combat arms branch which encompasses 80 percent of the commissioned officer operational flying positions within the Army (less those in Aviation Material Management and Medical Service Corps).',
 'Depending on the thickness and size of the chop, it can take anywhere from eight to 30 minutes. Hereâ\x80\x99s a helpful cooking chart and some tips to achieve delicious pork chops every time. Pork chops are a crowd pleaser, especially once you master your grilling technique. For safe consumption, itâ\x80\x99s recommended to cook pork until it reaches an internal temperature of 145°F or 65°C. Depending on the cut and thickness of your chop, the time it may take to reach this can vary. To make sure your chops are the right temperature, use a digital meat thermometer.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.9709, 0.7909, 0.6977]])

Evaluation

Metrics

Information Retrieval

• Datasets: NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with InformationRetrievalEvaluator

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator with these parameters:

  {
   "dataset_names": [
   "climatefever",
   "dbpedia",
   "fever",
   "fiqa2018",
   "hotpotqa",
   "msmarco",
   "nfcorpus",
   "nq",
   "quoraretrieval",
   "scidocs",
   "arguana",
   "scifact",
   "touche2020"
   ],
   "dataset_id": "sentence-transformers/NanoBEIR-en"
  }
  

Training Details

Training Datasets

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 8192
• per_device_eval_batch_size: 512
• learning_rate: 0.0001
• weight_decay: 0.01
• num_train_epochs: 1
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• seed: 12
• bf16: True
• dataloader_drop_last: True
• dataloader_num_workers: 12
• dataloader_prefetch_factor: 2
• remove_unused_columns: False
• optim: adamw_torch
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.11.14
• Sentence Transformers: 5.3.0.dev0
• Transformers: 4.57.1
• PyTorch: 2.8.0+cu129
• Accelerate: 1.12.0
• Datasets: 4.4.1
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
 title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
 author = "Reimers, Nils and Gurevych, Iryna",
 booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
 month = "11",
 year = "2019",
 publisher = "Association for Computational Linguistics",
 url = "https://arxiv.org/abs/1908.10084",
}

Base model

answerdotai/ModernBERT-base

(1335)