Ai Training Data Curator

Pricing

Rating

Developer

Actor stats

Categories

Share

Curate high-quality, deduplicated training data for LLM fine-tuning. Extract clean text from any website OR process your own documents with automatic quality scoring, deduplication, and format conversion.

Features

• Smart Content Extraction: Automatically detects and extracts main content, filtering out navigation, ads, and boilerplate
• Bring Your Own Data (BYOD): Process your own text documents without crawling - perfect for existing datasets
• Quality Scoring: Scores each document based on vocabulary diversity, sentence structure, and content density
• Deduplication: Uses MinHash/Jaccard similarity to remove near-duplicate content
• Flexible Crawling: Single page, same domain, same subdomain, or follow all links
• Document Chunking: Split long documents into training-ready chunks with configurable overlap
• Multiple Output Formats: JSONL (OpenAI compatible), JSON, Parquet, CSV, or HuggingFace Datasets format
• Language Filtering: Filter content by language (ISO 639-1 codes)
• Privacy Features: Optionally remove emails and URLs from extracted text

Use Cases

• LLM Fine-tuning: Collect domain-specific training data for fine-tuning language models
• RAG Systems: Build high-quality document collections for retrieval-augmented generation
• Knowledge Bases: Create clean text corpora from documentation sites
• Research: Gather datasets from academic or technical resources
• Data Cleaning: Clean and deduplicate existing text datasets for ML training

Input Configuration

Mode Selection

The actor supports two modes - provide either start_urls (for crawling) or documents (for BYOD):

BYOD (Bring Your Own Data) Settings

Crawl Settings

Content Extraction

Quality & Deduplication

Output Settings

Chunking

Text Cleaning

Performance

Output Format

Each document in the output contains:

{
"text":"The cleaned document text content...",
"doc_id":"abc123def456",
"source_url":"https://example.com/page",
"word_count":1523,
"quality_score":0.847,
"language":"en",
"title":"Page Title",
"description":"Meta description",
"content_type":"documentation",
"scraped_at":"2024-01-15T10:30:00Z"
}

If chunking is enabled, additional fields are included:

{
"chunk_index":0,
"total_chunks":5,
"parent_doc_id":"abc123def456"
}

Quality Metrics

The quality scorer evaluates documents based on:

• Word count: Penalizes very short documents
• Sentence length: Flags very short (fragments) or very long sentences
• Vocabulary diversity: Ratio of unique words to total words
• Boilerplate ratio: Detection of common web boilerplate patterns
• Character composition: Penalizes excessive uppercase, digits, or special characters

Documents with scores below min_quality_score are automatically filtered out.

Example Input

Crawl Python Documentation

{
"start_urls":[
{"url":"https://docs.python.org/3/tutorial/"}
],
"crawl_mode":"same_subdomain",
"max_pages":500,
"content_selectors":[".document",".body"],
"exclude_selectors":[".sphinxsidebar",".related","footer"],
"output_format":"jsonl",
"chunk_documents":true,
"chunk_size":1024
}

Build Knowledge Base from Blog

{
"start_urls":[
{"url":"https://example.com/blog/"}
],
"crawl_mode":"same_domain",
"max_pages":100,
"content_selectors":["article",".post-content"],
"quality_filter":true,
"min_quality_score":0.6,
"deduplicate":true,
"output_format":"parquet"
}

BYOD: Process Your Own Documents

{
"documents":[
"This is a plain text document that will be processed...",
{
"text":"This document has metadata attached to it...",
"source_id":"doc_001",
"metadata":{
"title":"My Document",
"author":"John Doe",
"language":"en"
}
}
],
"deduplicate":true,
"quality_filter":true,
"min_quality_score":0.5,
"output_format":"jsonl"
}

BYOD: Clean Existing Dataset

{
"documents":[
{"text":"First document from your dataset..."},
{"text":"Second document from your dataset..."},
{"text":"Third document from your dataset..."}
],
"byod_text_field":"text",
"deduplicate":true,
"dedup_threshold":0.85,
"chunk_documents":true,
"chunk_size":512,
"output_format":"jsonl"
}

Tips for Best Results

1. Use specific content selectors: Better extraction with precise CSS selectors for your target site
2. Set appropriate word counts: Filter out navigation pages and indexes with min_word_count
3. Enable deduplication: Prevents training on repetitive content (common on content farms)
4. Adjust quality threshold: Lower for technical content, higher for prose
5. Use chunking for long documents: Better for training context windows
6. Start small: Test with max_pages: 20 before large crawls

Pricing

• $0.01 per document - charged for each cleaned document (both crawled and BYOD)

Additional costs:

• Proxy: ~$0.001-0.005 per request (if enabled)
• Storage: ~$0.0001 per document

Support

• Apify Documentation
• Report Issues
• Crawlee Documentation