Ettin: an Open Suite of Paired Encoders and Decoders

👁 License: MIT
👁 Paper
👁 Models
👁 Data
👁 GitHub

🎯 TL;DR: State-of-the-art paired encoder and decoder models (17M-1B params) trained identically for fair comparison with open data. Encoders beat ModernBERT. Decoders beat Llama 3.2/SmolLM2.

📄 Paper (Coming Soon) | 🚀 GitHub Repository

This model is part of the Ettin suite - the first collection of paired encoder-only and decoder-only models trained with identical data, architecture, and training recipes. Ettin enables fair comparisons between encoder and decoder architectures across multiple scales, providing state-of-the-art performance for open-data models in their respective size categories.

Table of Contents

• Performance Highlights
• Quick Start
• Model Description
• Training Data
• Model Family
  • Encoder Models
  • Decoder Models
  • Cross-Objective Models
• Accessing Training Checkpoints
• Research Applications
• Training Details
• Model Architecture
• Usage Examples
• Fine-tuning Examples
• Citation

📊 Performance Highlights

Encoder Tasks (vs. ModernBERT)

• GLUE Average: 88.9 vs 88.4 (Base), 90.8 vs 90.4 (Large)
• MTEB v2 English Retrieval: 45.7 vs 43.9 (Base), 48.4 vs 47.0 (Large)
• Code Search and Long Context: Superior performance on CodeSearchNet and MLDR

Decoder Tasks (vs. SmolLM2 & Llama 3.2)

• Average Score: 46.2 vs 45.2 (SmolLM2-135M)
• 1B Model: 59.0 vs 56.6 (Llama 3.2-1B)
• Generative Tasks: Competitive across all model sizes

Key Finding

Architecture-specific advantages persist: A 400M encoder outperforms a 1B decoder on classification tasks, while a 400M decoder outperforms a 1B encoder on generation tasks.

🚀 Quick Start

Installation

pip install torch>=1.9.0
# until the new pip release, install from main to use decoders (transformers>=4.54.X will contain it)
# encoders work with transformers>=4.48.0
pip install git+https://github.com/huggingface/transformers.git

30-Second Examples

Encoder for Classification/Embeddings:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-encoder-150m")
model = AutoModel.from_pretrained("jhu-clsp/ettin-encoder-150m")

Decoder for Text Generation:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-decoder-150m")
model = AutoModelForCausalLM.from_pretrained("jhu-clsp/ettin-decoder-150m")

Model Description

Ettin models are designed to provide a foundation for comparing encoder-only and decoder-only architectures. Unlike previous comparisons that were limited by different training data, architectures, and recipes, Ettin models use:

1. Identical training data - Same high-quality mixture across all models
2. Open Training Data - Data is available now with batch-level training data for each of the 250+ checkpoints
3. Matched architectures - Only differing in attention patterns (bidirectional vs causal) and training objectives (MLM vs CLM)
4. Consistent training recipe - Three-phase training with 2T tokens
5. Multiple scales - From 17M to 1B parameters

This approach allows for true apples-to-apples comparisons between encoder and decoder models, revealing the inherent strengths of each architecture.

Training Data

The training data is publicly available and split across different phases:

• Pre-training Data: jhu-clsp/ettin-pretraining-data - 1.7T tokens of diverse data mixture
• Mid-training/Extension Data: jhu-clsp/ettin-extension-data - 250B tokens of higher-quality filtered data
• Decay Phase Data: jhu-clsp/ettin-decay-data - 100B tokens of premium data sources
• Training Data Order: jhu-clsp/ettin-data-order - Batch-level training order (columns: input_ids, step)

Model Family

Encoder Models

Decoder Models

Cross-Objective Models

These models demonstrate what happens when you continue training encoders as decoders (and vice versa). Important: Load these models using the architecture they were converted to, not their original architecture.

Encoders Trained from Decoders (Decoder → MLM)

Load as encoders using AutoModel or AutoModelForMaskedLM:

Decoders Trained from Encoders (Encoder → CLM)

Load as decoders using AutoModelForCausalLM:

Example Usage for Cross-Objective Models:

# Encoder-from-decoder: Load as encoder
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-encoder-from-decoder-150m")
model = AutoModel.from_pretrained("jhu-clsp/ettin-encoder-from-decoder-150m")

# Decoder-from-encoder: Load as decoder 
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-decoder-from-encoder-150m")
model = AutoModelForCausalLM.from_pretrained("jhu-clsp/ettin-decoder-from-encoder-150m")

Accessing Training Checkpoints

Beyond the final models listed above, we provide access to intermediate training checkpoints for research and analysis purposes. These checkpoints allow you to study model behavior and performance throughout the training process. You can get the checkpoints either in HF format or raw for continued pre-training (e.g. Composer format).

Raw Checkpoints

All raw training checkpoints are available in the jhu-clsp/ettin-checkpoints dataset.

HuggingFace Format Checkpoints

Each model repository contains multiple tagged versions representing different training stages:

• step{number} - Pretraining phase checkpoints (e.g., step599525, step596528)
• ext{number} - Extension/mid-training phase checkpoints (e.g., ext1000, ext2000)
• decay{number} - Decay phase checkpoints (e.g., decay100, decay500)

from transformers import AutoTokenizer, AutoModelForCausalLM

# Load a specific pretraining checkpoint
model = AutoModelForCausalLM.from_pretrained(
 "jhu-clsp/ettin-decoder-400m", 
 revision="step590532" # Specific checkpoint tag
)

# Load an extension phase checkpoint
model = AutoModelForCausalLM.from_pretrained(
 "jhu-clsp/ettin-decoder-400m", 
 revision="ext1000"
)

# Load a decay phase checkpoint 
model = AutoModelForCausalLM.from_pretrained(
 "jhu-clsp/ettin-decoder-400m", 
 revision="decay100"
)

This checkpoint availability enables detailed analysis of training dynamics, loss curves, and capability emergence across the complete 2T token training process.

🔬 Research Applications

What Makes Ettin Unique

Ettin provides the first controlled comparison of encoder vs. decoder architectures:

• Identical Training Data: Same 2T token mixture across all models
• Matched Architectures: Only attention patterns and objectives differ
• Open Everything: Training data, model weights, and batch-level training order
• Multiple Scales: Fair comparison from 17M to 1B parameters
• 250+ Checkpoints: Complete training trajectory analysis

Use Cases for Researchers

• Architecture Studies: Compare encoder vs decoder capabilities fairly
• Training Dynamics: Analyze 250+ checkpoints with batch-level data ordering
• Scaling Laws: Study how architectural advantages change with scale
• Transfer Learning: Investigate cross-objective training effectiveness
• Replication Studies: First open replication of ModernBERT training recipe

Reproducibility

All training artifacts are publicly available:

• Training data with exact batch ordering
• Model checkpoints every 8.5B tokens
• Complete hyperparameter configurations
• Training code and evaluation scripts

Training Details

Data: High-quality mixture including DCLM, Dolma v1.7, scientific papers, code, and curated sources totaling 2T+ tokens

Architecture: Transformer with RoPE, GLU activations, and prenorm layers

Training Phases:

• Pre-training: 1.7T tokens with diverse data mixture
• Mid-training: 250B tokens with higher-quality filtered data and context extension to 8K
• Decay phase: 100B tokens with premium data sources

Key Features:

• Context length: Up to 8K tokens
• Vocabulary: 50,368 tokens (ModernBERT tokenizer)
• Deep but efficient architectures following MobileLLM principles

Model Architecture

Usage Examples

Encoder: Masked Language Modeling

Decoder: Text Generation

Fine-tuning Examples

Encoders

Decoders

Citation

If you use Ettin models in your research, please cite our work:

@misc{weller2025seqvsseqopen,
 title={Seq vs Seq: An Open Suite of Paired Encoders and Decoders}, 
 author={Orion Weller and Kathryn Ricci and Marc Marone and Antoine Chaffin and Dawn Lawrie and Benjamin Van Durme},
 year={2025},
 eprint={2507.11412},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2507.11412}, 
}

A collection of SOTA, open-data, paired encoder-only and decoder only models ranging from 17M params to 1B. See the paper at https://arxiv.org/abs/250 • 30 items • Updated Mar 2 • 31