Pricing

Rating

Developer

Actor stats

Categories

Share

Lobsters Scraper — Tech Community Stories, Comments & Tags

Extract clean JSON from Lobste.rs, the invite-only technology community known for high-signal programming, security, infrastructure, and open-source discussions. This Actor uses Lobsters' public JSON endpoints, so it is fast, lightweight, and does not need a browser.

Why use it?

• Track developer conversations — monitor what experienced engineers are discussing before it spreads to larger networks.
• Build research datasets — collect story titles, URLs, scores, submitters, tags, timestamps, and comment counts.
• Find niche technical content — filter by tags such as programming, security, linux, rust, devops, or ai.
• Power RAG and trend pipelines — export structured items to datasets, webhooks, Google Sheets, vector DBs, or your own API.

Input examples

1. Hottest stories

{
"mode":"hottest",
"limit":50
}

2. Newest submissions

{
"mode":"newest",
"limit":30
}

3. Stories by tag

{
"mode":"tag",
"tag":"security",
"limit":40
}

4. Single story deep scrape

{
"mode":"story",
"storyId":"abc123"
}

5. Available tags

{
"mode":"tags"
}

Input fields

Output

Each dataset item is normalized JSON. Story records can include:

Example output item:

{
"short_id":"abc123",
"url":"https://example.com/article",
"comments_url":"https://lobste.rs/s/abc123/example_article",
"title":"Example technical article",
"created_at":"2026-04-24T12:00:00.000Z",
"score":42,
"comment_count":18,
"tags":["programming","security"],
"submitter_user":"example_user"
}

Use cases

• Developer trend monitoring — daily scrape hottest/newest stories and alert on specific keywords.
• Security research — track security, privacy, and infrastructure discussions.
• Competitive intelligence — monitor how engineering communities react to new tools and releases.
• Content discovery — find high-quality links for newsletters, blogs, podcasts, or social posts.
• LLM/RAG ingestion — feed curated technical links and discussions into knowledge bases.

Integrations

The Actor works with standard Apify integrations:

• Webhooks for automatic downstream processing.
• Google Sheets / Make / Zapier / n8n for no-code workflows.
• Apify API, JavaScript SDK, and Python client for custom apps.
• Dataset export as JSON, CSV, Excel, XML, or RSS.

JavaScript SDK

import{ ApifyClient }from'apify-client';
const client =newApifyClient({token: process.env.APIFY_TOKEN});
const run =await client.actor('Helpermara/lobsters-scraper').call({
mode:'tag',
tag:'security',
limit:50,
});
const{ items }=await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python client

from apify_client import ApifyClient
client = ApifyClient('YOUR_APIFY_TOKEN')
run = client.actor('Helpermara/lobsters-scraper').call(run_input={
'mode':'hottest',
'limit':50,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

Notes and limits

• Uses Lobsters public JSON endpoints; no login or invite is required for public story data.
• Keep limits reasonable. The Actor is designed for targeted collection, not aggressive crawling.
• Some fields depend on what Lobsters returns for a specific endpoint and story.

FAQ

Does it scrape comments?
The Actor is focused on story, tag, and story metadata. story mode retrieves the public story JSON available from Lobsters, including discussion metadata when exposed by the endpoint.

Do I need a Lobsters account?
No. Public JSON endpoints are used.

Can I filter by multiple tags?
Run the Actor once per tag and merge datasets downstream, or schedule several runs with different inputs.

Is this browser-based?
No. It uses direct HTTP requests, which makes it faster and cheaper than browser automation.

Keywords

Lobsters scraper, lobste.rs API, tech news scraper, developer community scraper, programming links, security news, engineering trends, Apify Actor, JSON API scraper, RAG dataset, developer intelligence.

Changelog

• 2026-04-24 — Rebuilt Actor implementation around real Lobsters JSON endpoints and expanded Store documentation.