Pricing

Rating

Developer

Actor stats

Categories

Share

Twitter/X Scraper (Python)

A powerful Apify actor to scrape Twitter/X data without requiring authentication. Uses Playwright for browser automation and extracts data directly from the Twitter/X website.

Features

✅ No Authentication Required - Works without login cookies
✅ Search Tweets - Find tweets by keywords, hashtags
✅ User Timelines - Get tweets from any public user
✅ User Profiles - Extract profile information
✅ Media & Replies - Filter for specific content types
✅ Engagement Metrics - Likes, retweets, replies, views
✅ Anti-Detection - Built-in stealth features

How It Works

1. Playwright Browser - Uses a real browser to access Twitter
2. Guest Token - Obtains guest credentials automatically
3. DOM Extraction - Parses tweets directly from the page
4. Infinite Scroll - Automatically scrolls to load more tweets

Input Parameters

Search Configuration

Scraping Options

Filters

Output Format

Tweet Data

{
"tweet_id":"1234567890",
"text":"Tweet content here...",
"author_name":"Elon Musk",
"author_username":"elonmusk",
"likes":50000,
"retweets":10000,
"replies":5000,
"views":1000000,
"created_at":"2024-01-15T10:30:00.000Z",
"url":"https://x.com/elonmusk/status/1234567890",
"is_retweet":false,
"is_reply":false,
"media":[{"type":"image","url":"..."}],
"hashtags":["#tech"],
"mentions":["@OpenAI"],
"scraped_at":"2024-01-15T12:00:00.000Z"
}

Profile Data

{
"name":"Elon Musk",
"username":"elonmusk",
"bio":"...",
"location":"Austin, TX",
"website":"tesla.com",
"join_date":"Joined June 2009",
"followers":"170M",
"following":"500",
"is_verified":true,
"profile_image_url":"...",
"url":"https://x.com/elonmusk",
"scraped_at":"2024-01-15T12:00:00.000Z"
}

Local Development

Prerequisites

• Python 3.9+
• pip

Setup

# Navigate to the actor directory
cd twitter-scraper-python
# Create virtual environment
python -m venv venv
# Activate (Windows)
.\venv\Scripts\activate
# Activate (Linux/Mac)
source venv/bin/activate
# Install dependencies
pip install-r requirements.txt
# Install Playwright browsers
playwright install chromium

Run Locally

# Using Apify CLI
apify run
# Or directly
python -m src.main

Test Input

Edit apify_storage/key_value_stores/default/INPUT.json:

{
"usernames":["elonmusk"],
"max_tweets":10
}

Deployment to Apify

# Login to Apify
apify login
# Push to Apify
apify push

Limitations

• Rate Limits: Twitter may rate-limit after many requests
• Login Walls: Some content requires authentication
• Dynamic Changes: Twitter's DOM may change

Tips

• Use residential proxies for better success rates
• Start with small limits to test
• Use Apify Proxy when deploying to Apify platform

License

ISC License