Pricing

Rating

Developer

Actor stats

Categories

Share

LLM-Guided Corporate Website Scraper

An advanced Apify actor that uses LLMs (Large Language Models) to identify and extract high-value business contact information from corporate websites.

🚀 Overview

This scraper goes far beyond traditional crawling. It:

• Uses GPT (OpenAI) to intelligently rank internal URLs based on their relevance to contact data
• Maximizes content extraction, including hidden and modal content
• Parses and validates contact fields using LLMs and custom regex preprocessing
• Aggregates data across multiple pages for higher confidence

💡 Key Features

• 🧰 LLM-based URL Evaluation: Scores and selects only the most promising URLs per domain
• 🔍 Maximum Content Extraction: Scrapes visible and hidden elements, emails, phone numbers, and text sections
• 🔧 Custom Prompt Engineering: Tailored prompts for URL scoring and field extraction
• 📊 Smart Aggregation: Merges multiple extractions into one confident, enriched result per domain
• 🚪 Resilient Parsing: Handles edge cases, malformed responses, and fallback scoring
• ✅ GDPR-friendly Proxy Support: With optional German residential proxies

⚙️ Input

This actor expects the following input:

{
"urls":["https://example.com"],
"openaiApiKey":"sk-...",
"maxRequests":50,
"useProxy":true,
"enableUrlEvaluation":true,
"aggregateResults":true,
"includeExtendedFields":true,
"costLimit":1.0
}

🔄 Workflow

1. Main page is loaded
2. LLM evaluates internal links for contact relevance
3. Top N URLs are crawled (contact, impressum, team, etc.)
4. Content is extracted (even from modals, hidden fields, footers)
5. Text is preprocessed for LLM efficiency
6. LLM parses the data into a structured JSON object
7. Data is validated, weighted, and aggregated into one high-confidence result

🌐 Output Format

Each record pushed to the dataset contains:

{
"executive_name":"Max Mustermann",
"executive_title":"Geschäftsführer",
"company_email":"info@example.com",
"company_phone":"+41 44 123 45 67",
"company_address":"Musterstrasse 1, 8000 Zürich",
"confidence_score":0.92,
"sources":[...],
"aggregated_from_pages":6,
"domain":"example.com"
}

📈 Performance & Cost

• Average ~40 websites for 0.07 $ (at gpt-3.5-turbo rates)
• Each domain result is based on up to 8 evaluated subpages
• Internal cost tracking included

🔐 Notes

• Requires valid OpenAI API key (gpt-3.5-turbo)
• Proxy use is optional, but recommended for stable scraping
• Works well for DE/CH/Austria-based companies (Impressum detection)

🚪 Limitations

• Not optimized for dynamic SPAs
• Some LLM responses may still need fallback handling (included)

🚧 Future Improvements

• Add multilingual prompt switching (based on targetLanguage input)
• Upgrade to gpt-4-turbo for more robust data quality
• Add custom scoring model for aggregation weighting

🌟 Created by Timo Sieber — for smarter, LLM-powered scraping at scale.