LLM-Ready Web Scraper

Pricing

Rating

Developer

Actor stats

Categories

Share

Converts web pages to clean, LLM-friendly formats. Perfect for building AI applications.

Use Cases

• RAG Pipelines: Get chunked content ready for vector databases
• Fine-tuning Datasets: Export as JSONL for LLM training
• Knowledge Bases: Build AI chatbot training data
• Content Extraction: Clean text without ads, menus, or clutter

Features

• Automatic content extraction (removes ads, navigation, footers)
• Multiple output formats: Markdown, JSON, JSONL
• Optional chunking with overlap for RAG
• Batch URL processing
• Metadata extraction (title, description, domain)

Output Formats

Markdown

---
title:"Page Title"
url: https://example.com/page
domain: example.com
scraped_at:2024-01-15T10:30:00Z
---
Clean page content here...

JSON

{
"url":"https://example.com",
"success":true,
"content":"Clean text content...",
"metadata":{
"title":"Page Title",
"description":"Meta description"
},
"word_count":1500
}

JSONL (Fine-tuning)

{
"prompt":"Content from Page Title:",
"completion":"Clean text content..."
}

With Chunks (RAG-ready)

{
"chunks":[
{"chunk_id":0,"text":"First chunk...","word_count":500},
{"chunk_id":1,"text":"Second chunk...","word_count":500}
],
"chunk_count":5
}

Input Parameters

Example Input

{
"urls":[
"https://docs.python.org/3/tutorial/",
"https://docs.python.org/3/library/"
],
"outputFormat":"json",
"includeChunks":true,
"chunkSize":500
}

Pricing

Pay only for what you use. Typical cost: $0.01-0.05 per URL depending on page size.