Google Scholar Scraper

Pricing

Rating

Developer

Actor stats

Categories

Share

Apify Actor to scrape Google Scholar search results with advanced filtering options.

Features

• Search by keyword: Find academic papers, articles, and books
• Author filtering: Filter results by specific authors
• Year range: Limit results to specific publication years
• Sort options: Sort by relevance or date
• Citation data: Extract citation counts and related articles
• PDF links: Automatically detect available PDF downloads
• Rate limiting: Built-in 5-10 second delays to respect Google Scholar
• Robust parsing: Handles various result formats (articles, books, citations)

Input Parameters

Output Format

Each result contains:

{
"title":"Paper title",
"articleUrl":"https://example.com/paper.pdf",
"pdfUrl":"https://example.com/download.pdf",
"authors":"John Doe, Jane Smith",
"year":2023,
"journal":"Journal of Machine Learning Research",
"abstract":"This paper presents...",
"citationCount":42,
"citedByUrl":"https://scholar.google.com/scholar?cites=...",
"relatedArticlesUrl":"https://scholar.google.com/scholar?q=related:...",
"allVersionsCount":3,
"isBook":false,
"isCitation":false,
"isPdf":true
}

Usage Example

Input

{
"searchQuery":"deep learning natural language processing",
"author":"Yoshua Bengio",
"yearFrom":2020,
"yearTo":2024,
"sortBy":"date",
"maxResults":50
}

Run Locally

# Install dependencies
npminstall
# Build TypeScript
npm run build
# Run actor (requires input.json in root or Apify environment)
npm start

Important Notes

Rate Limiting

Google Scholar is very strict about automated access:

• Actor uses 5-10 second delays between requests
• Realistic User-Agent rotation
• Proper HTTP headers to mimic browser behavior
• Automatic CAPTCHA detection and graceful shutdown

Recommendation:

• Keep maxResults under 100 for reliability
• Use longer delays for larger scrapes
• Consider using Google Scholar API alternatives for production use

CAPTCHA/Blocking

If Google Scholar detects automation:

• Actor logs a warning and stops gracefully
• No partial results are lost (already scraped data is saved)
• You can retry with longer delays or from a different IP

Legal Considerations

• Respect Google Scholar's Terms of Service
• Use for research/academic purposes
• Do not overload their servers
• Consider API alternatives for commercial use

Development

Build

$npm run build

Local Testing

$npm run dev

Docker Build

docker build -t google-scholar-scraper .
docker run -eAPIFY_INPUT='{"searchQuery":"machine learning"}' google-scholar-scraper

Troubleshooting

No Results Found

• Check if query has typos
• Try broader search terms
• Verify year range is valid

CAPTCHA Detected

• Reduce maxResults
• Run actor less frequently
• Use different IP address
• Consider Google Scholar API

Parser Errors

• Google Scholar HTML structure may change
• Open an issue with example query
• Actor will skip unparseable results

License

Apache-2.0

Support

For issues or questions, please open a GitHub issue or contact the Apify support team.