Article Extractor & News Scraper

Pricing

Rating

Developer

Actor stats

Categories

Share

👁 Apify Actor
👁 Python 3.12
👁 License: MIT

Extract articles, news content, and blog posts from any website. Get clean, structured data with title, full text, authors, publication date, images, keywords, and metadata—powered by 7 specialized extraction engines.

🔗 Run this Actor on Apify | 📖 API Documentation

Table of Contents

• Features
• Use Cases
• How It Works
• Extraction Engines Comparison
• Quick Start
• Input Configuration
• Output Format
• Example Outputs
• Anti-Blocking Features
• Performance Tips
• Integrations
• Troubleshooting
• FAQ
• Changelog

✨ Features

Core Capabilities

• 🔍 7 Specialized Extraction Engines — Choose from Newspaper4k, Trafilatura, Boilerpy3, News-Please, Goose3, Article Parser, and JusText
• 🌐 Universal Website Compatibility — Works with news sites, blogs, magazines, and any article-based content
• 📊 Complete Content Extraction — Captures title, description, full text, authors, publication date, images, keywords, and metadata
• 🔄 Smart Fallback System — Automatically tries alternative extractors if the primary one fails

Anti-Blocking Technology

• 🎭 Browser Fingerprint Generation — Uses browserforge for realistic browser headers
• 🔀 Proxy Rotation — Automatic proxy rotation with support for residential proxies
• ⏱️ Intelligent Rate Limiting — Domain-specific delays and concurrency control
• ☁️ CloudScraper Integration — Bypasses Cloudflare and similar protections
• 📦 Google Cache Fallback — Retrieves content from Google's cache when direct access fails

Output Options

• 📝 Plain Text — Clean, extracted article text
• 🔖 Article HTML — Preserved formatting with links and media
• 📄 Full Page HTML — Complete webpage source for custom processing
• 📋 Structured JSON — All metadata in a standardized format

🎯 Use Cases

⚙️ How It Works

graph LR
 A[Input URLs] --> B[Fetch Pages]
 B --> C{Anti-Bot Check}
 C -->|Blocked| D[Rotate Proxy/Headers]
 D --> B
 C -->|Success| E[Extract Content]
 E --> F{Extraction OK?}
 F -->|No| G[Try Fallback Engine]
 G --> E
 F -->|Yes| H[Output JSON]

1. Input Processing — Accepts a list of article URLs
2. Smart Fetching — Uses randomized browser headers and proxy rotation
3. Anti-Bot Evasion — Detects and bypasses blocking with CloudScraper and fingerprint rotation
4. Content Extraction — Applies the selected extraction engine
5. Fallback Logic — Automatically tries alternative engines if extraction fails
6. Output Generation — Returns structured JSON with all extracted data

📊 Extraction Engines Comparison

Recommended Engines by Content Type

• 📰 News Sites → Newspaper4k or Trafilatura
• 📝 Blog Posts → Trafilatura or Goose3
• 📚 Long-form Articles → Newspaper4k (with NLP for summarization)
• 🖼️ Image-heavy Content → Goose3
• ⚡ High-volume Scraping → Boilerpy3 or Trafilatura
• 🔤 Non-English Content → JusText (40+ languages supported)

🚀 Quick Start

Run on Apify Platform

{
"startUrls":[
"https://www.nytimes.com/2024/01/15/technology/ai-developments.html",
"https://www.theguardian.com/world/2024/jan/15/breaking-news"
],
"extractorEngine":"newspaper4k"
}

Run Locally with Apify CLI

# Install Apify CLI
npminstall-g apify-cli
# Clone and run
apify pull article-extractor-news-scraper
cd article-extractor-news-scraper
apify run --input='{"startUrls": ["https://example.com/article"]}'

Call via API

curl-X POST "https://api.apify.com/v2/acts/YOUR_USERNAME~article-extractor-news-scraper/runs"\
-H"Authorization: Bearer YOUR_API_TOKEN"\
-H"Content-Type: application/json"\
-d'{
 "startUrls": ["https://www.bbc.com/news/world-12345"],
 "extractorEngine": "newspaper4k"
 }'

📝 Input Configuration

Full Input Example

{
"startUrls":[
"https://www.nytimes.com/2024/01/15/world/article.html",
"https://www.theguardian.com/world/2024/jan/15/story",
"https://www.bbc.com/news/world-12345678"
],
"extractorEngine":"newspaper4k",
"useFallbackExtractors":true,
"saveHtml":false,
"saveArticleHtml":true,
"maxRetries":15,
"useHeaderGenerator":true,
"headerGeneratorOptions":{
"browsers":["chrome","firefox","safari","edge"],
"devices":["desktop"]
},
"customHeaders":{},
"proxyConfiguration":{
"useApifyProxy":true,
"apifyProxyGroups":["RESIDENTIAL"]
}
}

📤 Output Format

Each extracted article produces a JSON object with the following fields:

Common Fields (All Engines)

Extended Fields (Engine-Dependent)

Metadata Fields

📋 Example Outputs

🛡️ Anti-Blocking Features

This Actor includes advanced anti-blocking technology to maximize success rates:

Browser Fingerprint Generation

Uses browserforge to generate realistic browser fingerprints including:

• Chrome, Firefox, Safari, and Edge user agents
• Proper sec-ch-ua client hints
• Consistent platform and viewport data
• Session-based fingerprint persistence

Proxy Rotation

• Automatic proxy rotation on 403/429 errors
• Support for residential, datacenter, and custom proxies
• Domain-specific proxy strategies

Intelligent Rate Limiting

• Per-domain concurrency control
• Adaptive delays based on site response
• Strict mode for heavily protected sites

CloudScraper Integration

• Bypasses Cloudflare browser verification
• Handles JavaScript challenges
• Automatic cookie management

Google Cache Fallback

When direct access fails after all retries, the Actor attempts to retrieve content from Google's cache as a last resort.

⚡ Performance Tips

For Maximum Speed

{
"extractorEngine":"boilerpy3",
"maxRetries":5,
"useFallbackExtractors":false
}

For Maximum Success Rate

{
"extractorEngine":"newspaper4k",
"maxRetries":15,
"useFallbackExtractors":true,
"proxyConfiguration":{
"useApifyProxy":true,
"apifyProxyGroups":["RESIDENTIAL"]
}
}

For Rich Metadata

{
"extractorEngine":"newspaper4k",
"saveArticleHtml":true,
"useFallbackExtractors":true
}

🔌 Integrations

Python

from apify_client import ApifyClient
client = ApifyClient("YOUR_API_TOKEN")
run_input ={
"startUrls":["https://www.example.com/article"],
"extractorEngine":"newspaper4k"
}
run = client.actor("YOUR_USERNAME/article-extractor-news-scraper").call(run_input=run_input)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(f"Title: {item['title']}")
print(f"Text: {item['text'][:200]}...")

JavaScript/Node.js

import{ ApifyClient }from'apify-client';
const client =newApifyClient({token:'YOUR_API_TOKEN'});
const run =await client.actor('YOUR_USERNAME/article-extractor-news-scraper').call({
startUrls:['https://www.example.com/article'],
extractorEngine:'newspaper4k'
});
const{ items }=await client.dataset(run.defaultDatasetId).listItems();
items.forEach(item=>{
 console.log(`Title: ${item.title}`);
 console.log(`Text: ${item.text.substring(0,200)}...`);
});

Webhooks

Configure webhooks to receive results automatically:

{
"webhooks":[
{
"eventTypes":["ACTOR.RUN.SUCCEEDED"],
"requestUrl":"https://your-server.com/webhook"
}
]
}

Zapier / Make (Integromat)

Use the Apify integration in Zapier or Make to connect extracted articles to:

• Google Sheets
• Notion databases
• Slack notifications
• Email newsletters
• CRM systems

🔧 Troubleshooting

Common Issues

Reporting Issues

If you encounter persistent issues:

1. Check if the URL works in a regular browser
2. Try different extraction engines
3. Open an issue with:
   • The problematic URL
   • Your input configuration

❓ Frequently Asked Questions

📝 Changelog

v1.0.0 (December 2025)

• ✨ Initial public release
• 🔍 7 extraction engines: Newspaper4k, Trafilatura, Boilerpy3, News-Please, Goose3, Article Parser, JusText
• 🛡️ Advanced anti-blocking with browserforge fingerprinting
• 🔄 Automatic fallback extraction
• ☁️ Google Cache fallback for blocked pages
• 📊 Multiple dataset views (Overview, Content, Metadata)
• ⚙️ Configurable concurrency and retry settings

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Built with ❤️ for the data extraction community

Keywords: article extractor, news scraper, web scraping, content extraction, newspaper4k, trafilatura, apify actor, python scraper, text extraction, metadata extraction, NLP, news monitoring, content aggregation