AI Training Data Scraper (Substack / Medium)

Pricing

Rating

Developer

Actor stats

Categories

Share

Extract clean, structured text data from Substack and Medium publications — formatted as Markdown or Plain Text — ready for LLM fine-tuning, RAG pipelines, and content analysis.

Why this scraper?

LLM training requires massive volumes of high-quality, long-form text. Substack and Medium are the internet's richest sources of expert-written articles, but scraping them manually is tedious and the raw HTML is full of noise (ads, popups, subscribe widgets).

This Actor thoughtfully cleans the content, stripping out:

• Subscribe popups & paywall gates
• Navigation headers & footers
• Author bio cards & social share buttons
• Script/style tags and embedded widgets

What you get is pure, clean content — exactly what AI training pipelines need.

What does it extract?

Input Parameters

Sample Input

{
"publicationUrls":[
"https://newsletter.banklesshq.com",
"https://medium.com/@exampleauthor"
],
"maxArticlesPerPublication":5,
"outputFormat":"markdown"
}

Sample Output

{
"url":"https://newsletter.banklesshq.com/p/the-future-of-defi",
"title":"The Future of DeFi",
"subtitle":"Where decentralized finance is headed next",
"author":"Bankless",
"date":"2024-01-15",
"content":"# The Future of DeFi\n\nDecentralized finance has come a long way..."
}

Use Cases

• LLM Fine-Tuning: Build domain-specific training datasets from expert writers
• RAG Pipelines: Populate vector databases with high-quality knowledge bases
• Content Analysis: Analyze publication trends, writing styles, and topic coverage
• Research: Systematically collect articles on specific subjects