Pricing

Rating

Developer

Actor stats

Categories

Share

📰 Google News Scraper Actor

A fast and scalable Apify Actor for scraping Google News articles by keywords or topics, with full article extraction, deduplication, and structured dataset output.

It supports:

• 🔍 Keyword-based search
• 🧭 Topic-based news scraping
• 🌍 Region + language control
• ⚡ Fast parallel scraping
• 🧠 Article extraction (title, snippet, image)
• 📦 Clean Apify dataset output

🚀 Features

• Google News RSS scraping
• Full article decoding (Google News redirect resolver)
• HTML article parsing
• Block detection fallback
• Threaded scraping for speed
• Apify dataset integration (Actor.push_data)
• Supports multiple topics in one run

⚙️ Input Configuration

🔹 Mode: Keywords

Search news using keywords.

Example input

{
"mode":"keywords",
"keywords":["ai","openai"],
"maxArticles":20,
"region_language":"US:en",
"timeframe":"1d"
}

🔹 Mode: Topics

Scrape full Google News categories.

Available topics

• WORLD
• NATION
• BUSINESS
• TECHNOLOGY
• ENTERTAINMENT
• SPORTS
• SCIENCE
• HEALTH

Example input

{
"mode":"topics",
"topics":["BUSINESS","TECHNOLOGY"],
"maxArticles":20,
"region_language":"US:en",
"timeframe":"1d"
}

📤 Output Format (Dataset Item)

Each scraped article is stored in the Apify dataset.

Example output

{
"position":7,
"title":"A fragile jihadist-separatist alliance in Mali",
"link":"https://www.france24.com/en/africa/20260501-example",
"domain":"www.france24.com",
"source":"www.france24.com",
"snippet":"A fragile alliance between jihadist and separatist groups in Mali is evolving...",
"thumbnail":"https://example.com/image.jpg",
"date_utc":"2026-05-02T16:36:39.390391+00:00"
}

📊 Dataset Fields

🔄 How It Works

Step 1

Fetch Google News RSS feed:

https://news.google.com/rss/search?q=QUERY

Step 2

Decode Google redirect links → real article URL

Step 3

Scrape article HTML:

• title
• meta description / og:description
• first paragraphs fallback
• og:image

Step 4

Push results to Apify dataset

⚡ Performance

• Multi-threaded scraping (ThreadPoolExecutor)
• 8 workers by default
• Fast RSS parsing
• Block detection fallback

🛡 Block Handling

The scraper detects:

• JS requirement pages
• “Access denied”
• “Subscribe to continue”
• adblock walls

If blocked:

• Falls back to RSS snippet

📌 Example Use Cases

• AI news monitoring
• Financial news tracking
• Competitor intelligence
• Media dashboards
• Research automation
• Trending topic tracking

🚀 Example Output Flow

Input:

{
"mode":"keywords",
"keywords":["elon musk","openai"],
"maxArticles":10
}

Output:

• 10 cleaned articles
• deduplicated
• full URLs
• structured dataset

🧠 Notes

• Google News RSS returns ~100 items per query
• Final output is limited by maxArticles
• Topics are aggregated before scraping
• Keywords use OR logic

📈 Future upgrades

• sentiment scoring
• AI clustering by topic
• full-text extraction fallback (Readability)
• proxy rotation
• webhook export (Zapier / Supabase)