arXiv Research Paper Scraper

Pricing

Rating

Developer

Actor stats

Categories

Share

Extract comprehensive research paper metadata from arXiv — the premier open-access preprint server for physics, mathematics, computer science, and more. 🎓📚

Features

• Full paper metadata — arXiv ID, title, authors, abstract, categories, dates
• PDF & abstract links — direct links to papers
• Pagination — automatically iterates through pages to reach maxItems
• Deduplication — no duplicate papers across pages
• Flexible search — search by all fields, title, author, abstract, category, etc.
• Sorting — sort by relevance, submission date, or last updated date
• No anti-bot issues — arXiv is an open academic resource

Input Parameters

Example INPUT.json

{
"query":"large language models",
"searchType":"all",
"sortBy":"submittedDate",
"sortOrder":"descending",
"maxItems":50
}

Output Fields

Example Output

{
"position":1,
"arxivId":"2501.12345",
"title":"Scaling Laws for Neural Language Models",
"authors":["Jared Kaplan","Sam McCandlish"],
"abstract":"We study empirical scaling laws for language model performance...",
"primaryCategory":"cs.LG",
"categories":["cs.LG","cs.CL","stat.ML"],
"submittedDate":"15 January, 2025",
"updatedDate":null,
"abstractUrl":"https://arxiv.org/abs/2501.12345",
"pdfUrl":"https://arxiv.org/pdf/2501.12345",
"comments":"35 pages, 14 figures",
"journalRef":null,
"doi":null,
"searchQuery":"large language models",
"scrapedAt":"2025-05-01T12:00:00.000Z"
}

Pagination

arXiv returns 25 results per page. The scraper automatically navigates through pages using the start offset parameter until maxItems is reached or no more results are available.

Use Cases

• Academic research monitoring — track new papers in your field
• Trend analysis — identify emerging topics and research directions
• Author profiling — collect all papers by specific authors
• Citation database — build reference datasets for research tools
• Competitive intelligence — monitor publications from research groups
• AI/ML dataset creation — collect paper abstracts for NLP training

Notes

• arXiv is a free, open-access resource — no authentication needed
• Results may vary slightly based on arXiv's real-time indexing
• The abstract field contains the full abstract text
• Use searchType: "au" to search by author name (e.g. "Hinton, Geoffrey")
• Use searchType: "cat" with category codes like "cs.AI", "math.CO", "hep-th"