🌐 Essential-Web: Complete 24-Trillion Token Dataset

🏆 Website | 🖥️ Code | 📖 Paper | ☁️ AWS

📋 Dataset Description

Essential-Web is a 24-trillion-token web dataset with document-level metadata designed for flexible dataset curation. The dataset provides metadata including subject matter classification, web page type, content complexity, and document quality scores for each of the 23.6 billion documents.

Researchers can filter and curate specialized datasets using the provided metadata, reducing the need for custom preprocessing pipelines and domain-specific classifiers.

🔍 Free Decimal Correspondence (FDC) Taxonomy

Essential-Web uses the Free Decimal Correspondence, a Dewey Decimal-inspired open taxonomy with 12 main categories for classifying web content. This systematic approach enables precise domain filtering and dataset curation.

For help navigating FDC codes, see: https://www.librarything.com/mds

⚙️ Dataset Creation

Essential-Web was created using a comprehensive processing pipeline starting from Common Crawl data:

📥 Source Data

• DCLM Pool: 89 resiliparse-extracted Common Crawl WARC snapshots (CC-MAIN-2013-20 to CC-MAIN-2022-49)
• Additional Snapshots: 12 additional snapshots extracted from CC-MAIN-2023-06 to CC-MAIN-2024-38 using resiliparse
• Total: 101 Common Crawl snapshots processed

🔧 Processing Pipeline

1. Document ID Generation: Using xxhash.xxh3_64_intdigest for unique document identification
2. Global Deduplication: Hash-based deduplication across all 101 snapshots
3. Minhash LSH Deduplication: Snapshot-level deduplication with Jaccard threshold of 0.7 (14 bands, 9 rows per band)
4. Quality Annotation: Statistical and model-based quality signals using RedPajama-Data-V2 pipeline variant, including DCLM-baseline fastText classifier
5. Quality Filtering: Manual tuned filters to retain high-quality English documents while preserving math and code content
6. Taxonomy Labeling: Classification of every document using EAI-Taxonomy-0.5b (~90,000 AMD MI300x GPU-hours)

🎯 Performance & Validation

We've curated example domain-specific datasets from Essential-Web using simple metadata filters, showing competitive performance relative to top performing web-curated datasets:

• 🧮 Math: within 8.0% of web-curated baselines
• 💻 Web Code: 14.3% above web-curated baselines
• 🔬 STEM: 24.5% above web-curated baselines
• 🩺 Medical: 8.6% above web-curated baselines

Note: These represent initial examples with significant room for further curation and improvement. Comparisons are against web-sourced datasets rather than specialized synthetic datasets.

🚀 Related Datasets & Models

Domain-Specific Datasets

We've curated high-quality domain-specific datasets from Essential-Web:

• Math: EssentialAI/eai-taxonomy-math-w-fm
• Code: EssentialAI/eai-taxonomy-code-w-dclm
• Medical: EssentialAI/eai-taxonomy-med-w-dclm
• STEM: EssentialAI/eai-taxonomy-stem-w-dclm

Classification Model

• EAI-Taxonomy-0.5b: EssentialAI/eai-taxonomy-0.5b - The efficient classifier used to label Essential-Web documents

🎯 Intended Use

Essential-Web enables researchers to:

• 🚀 Rapid Curation: Create multi-billion-token domain-specific datasets in minutes using SQL-like filters
• 🔍 Flexible Exploration: Explore web content across subjects, quality levels, and content types
• 🏗️ Custom Pipelines: Build specialized training corpora without custom classification infrastructure
• 🔄 Iterative Improvement: Easily modify and refine dataset composition based on training results
• 📊 Quality Control: Filter out low-quality content (ads, product listings) while preserving reasoning-dense documents

Dataset Schema Documentation

Overview

This dataset contains web-crawled text data with comprehensive metadata, quality signals, and taxonomic classifications. Each record represents a document extracted from web archives with detailed provenance tracking and quality assessment metrics.

Core Fields

EAI Taxonomy Classification

Comprehensive hierarchical classification system with primary and secondary labels - the most important feature of this dataset. The taxonomy is designed to provide detailed subject categorization, document type identification, content quality assessment, and extraction quality indicators.

How to Load the Dataset

This section provides examples of how to load the EssentialAI/essential-web-v1.0 dataset using different Python libraries and frameworks.

Using Hugging Face Datasets (Standard Method)

The simplest way to load the dataset is using the Hugging Face datasets library:

from datasets import load_dataset

# Load the entire dataset
dataset = load_dataset("EssentialAI/essential-web-v1.0")

# View dataset structure
print(dataset)
print(f"Number of examples: {len(dataset['train'])}")

You can also load the dataset in streaming mode to avoid downloading the entire dataset at once:

from datasets import load_dataset

# Load in streaming mode
dataset = load_dataset("EssentialAI/essential-web-v1.0", streaming=True)
data_stream = dataset["train"]

# Iterate through examples
for example in data_stream.take(5):
 print(example)

Using PySpark

For large-scale distributed processing, you can load the dataset using PySpark with the pyspark_huggingface library:

# First install the required library:
# pip install pyspark_huggingface

import pyspark_huggingface
from pyspark.sql import SparkSession

# Initialize Spark session
spark = SparkSession.builder.appName("EAI-Taxonomy-Web").getOrCreate()

# Load the dataset using the "huggingface" data source
df = spark.read.format("huggingface").load("EssentialAI/essential-web-v1.0")

# Basic dataset exploration
print(f"Dataset shape: {df.count()} rows, {len(df.columns)} columns")
df.show(10)
df.printSchema()

# Load only specific columns for efficiency
df_subset = (
 spark.read.format("huggingface")
 .option("columns", '["column1", "column2"]') # Replace with actual column names
 .load("EssentialAI/essential-web-v1.0")
)

# Run SQL queries on the dataset
df.createOrReplaceTempView("eai_web_dataset")
result = spark.sql("""
 SELECT COUNT(*) as total_examples
 FROM eai_web_dataset
""")
result.show()

Using Daft

Daft provides a modern DataFrame library optimized for machine learning workloads. You can load the dataset directly from Hugging Face:

import daft

# Load the entire dataset
df = daft.read_parquet("hf://datasets/EssentialAI/essential-web-v1.0")

# Basic exploration
print("Dataset schema:")
df.schema()

print("First 5 rows:")
df.show(5)

If you need to access private datasets or use authentication:

import daft
from daft.io import IOConfig, HTTPConfig

io_config = IOConfig(http=HTTPConfig(bearer_token="your_token"))
df = daft.read_parquet("hf://datasets/EssentialAI/essential-web-v1.0", io_config=io_config)

Installation Requirements

Make sure you have the required libraries installed:

# For Hugging Face datasets
pip install datasets

# For PySpark with Hugging Face integration
pip install pyspark_huggingface

# For Daft
pip install daft

📜 License

Essential-Web-v1.0 contributions are made available under the ODC attribution license; however, users should also abide by the Common Crawl - Terms of Use. We do not alter the license of any of the underlying data.

📝 Citation

@misc{ai2025essentialwebv1024ttokens,
 title={Essential-Web v1.0: 24T tokens of organized web data}, 
 author={Essential AI and : and Andrew Hojel and Michael Pust and Tim Romanski and Yash Vanjani and Ritvik Kapila and Mohit Parmar and Adarsh Chaluvaraju and Alok Tripathy and Anil Thomas and Ashish Tanwer and Darsh J Shah and Ishaan Shah and Karl Stratos and Khoi Nguyen and Kurt Smith and Michael Callahan and Peter Rushton and Philip Monk and Platon Mazarakis and Saad Jamal and Saurabh Srivastava and Somanshu Singla and Ashish Vaswani},
 year={2025},
 eprint={2506.14111},
 archivePrefix={arXiv},
 primaryClass={cs.CL},
 url={https://arxiv.org/abs/2506.14111}, 
}

10 items • Updated Jun 18, 2025 • 10