LLM Web Scraper

Pricing

Rating

Developer

Actor stats

Categories

Share

Turn any website into structured JSON data using OpenAI (GPT) or Anthropic (Claude) models. This Actor fetches HTML, cleans it, and uses a Large Language Model (LLM) to extract specific data fields based on a schema you provide.

What This Does

Traditional scraping requires writing brittle CSS selectors or Regex for every field. This Actor uses semantic understanding to locate and extract data, making it resilient to layout changes.

1. Fetches: Downloads the webpage (supports User-Agent rotation).
2. Cleans: Prunes scripts, styles, and ads to minimize token usage.
3. Extracts: Sends the content to an LLM alongside your JSON schema.
4. Validates: Returns structured JSON matching your definition.

Use Cases

• E-commerce: Extract product details (price, specs, availability) from diverse layouts.
• News Aggregation: Normalize article content, authors, and dates into a standard format.
• Lead Generation: Extract contact info and company details from "About Us" pages.
• Financial Data: Parse unstructured tables and reports into usable JSON.

Input

You must provide a target URL and a JSON Schema defining the data you want. You must also provide either an OpenAI or Anthropic API key.

{
"url":"https://news.ycombinator.com",
"schema":{
"stories":[
{
"title":"string",
"points":"number",
"author":"string"
}
]
},
"openaiApiKey":"sk-...",
"model":"gpt-4o",
"maxTokens":2000,
"selector":"table.itemlist"
}

Configuration Parameters

*One of the two API keys is required.

Output

The Actor outputs a JSON object containing the extracted data and metadata about the run.

{
"url":"https://news.ycombinator.com",
"success":true,
"tokensUsed":1450,
"model":"gpt-4o",
"data":{
"stories":[
{"title":"Rust vs C++","points":156,"author":"dev_user"},
{"title":"New AI Model","points":42,"author":"ai_researcher"}
]
}
}

Optimization & Costs

LLM scraping involves two costs: the Apify run cost and your external LLM API usage. To minimize both:

1. Use Selectors: Always provide a CSS selector (e.g., div.product-details) if possible. This discards headers, footers, and sidebars before sending text to the AI, significantly reducing your token bill.
2. Choose the Right Model: gpt-4o and claude-3-5-sonnet offer the best balance of speed and intelligence. Smaller models are cheaper but may struggle with complex schemas.
3. Clean Input: The Actor automatically removes <script>, <style>, and <nav> tags to ensure high-quality context for the AI.