Pricing

Rating

Developer

Actor stats

Categories

Share

RAG Web Crawler — Clean Markdown + Token-Sized Chunks, Pay-Per-Result

Turn any website into embeddings-ready chunks with citations and predictable per-chunk pricing. No CSS tuning, no runaway compute bills.

Generic crawlers hand you raw pages and make you build the RAG pipeline yourself. This actor hands you clean, token-sized, deduplicated, citable chunks — at a fixed price per chunk you keep.

What it does

• Clean LLM-ready Markdown — @mozilla/readability strips nav, footers, ads, and cookie banners; turndown + GFM converts the cleaned DOM to Markdown with heading hierarchy, fenced code blocks, and tables preserved.
• Structure-aware, token-budgeted chunking — splits on the heading tree, then recursively sub-splits oversized sections to your token budget (default 512) with overlap (default 75). Code blocks and tables are kept intact, never split mid-block.
• Rich per-chunk provenance — every chunk ships source URL + deep anchor, page title, full headings path, content hash, token count, content type, and language for metadata-filtered vector search and deep-link citations.
• Dedup + junk filtering — exact content-hash dedup plus 64-bit SimHash near-duplicate collapsing, and low-information / nav-residue chunk filtering.
• Four output formats — chunks-jsonl (one record per chunk), markdown (one record per page), langchain (drop-in {page_content, metadata} Document JSON), and jsonl-bulk (flat one-record-per-chunk for DB/COPY/pgvector).
• Incremental / delta sync — on scheduled re-runs, only NEW or CHANGED pages are re-emitted (and billed). Makes daily/weekly crawls cheap.
• Budget guarantee — maxPages is a hard ceiling; billing is per emitted result, so a runaway crawl can never produce a runaway bill.

Incremental / delta sync — cheap scheduled re-runs

Turn on Incremental sync (incremental: true) and schedule the actor to run daily or weekly. The first run does a full crawl and seeds a per-URL content-hash state in a named key-value store. Every later run crawls the site, but only re-emits chunks for pages that are new or changed — unchanged pages cost nothing. A typical weekly docs re-crawl re-emits a handful of pages instead of hundreds.

• State is automatic. The state store name defaults to a deterministic hash of your start URLs, so a scheduled task reuses its own prior state with zero config. Set stateStoreName explicitly to share state across tasks/schedules.
• forceFullCrawl: true re-emits everything and rebuilds the baseline — use after changing chunking settings or to refresh a stale index.
• emitDeletions: true writes a tombstone record ({ deleted: true, url, ... }) to a separate deletions dataset for every URL that disappeared since the last run, so downstream vector stores can purge stale vectors. Tombstones are not billed.
• When incremental is ON, each emitted record carries a change_status (new | changed) in its metadata.

The run summary (OUTPUT key-value record) includes a delta block: pages_new, pages_changed, pages_unchanged, pages_deleted, chunks_skipped_unchanged (the spend you saved), state_store, prior_run_id.

When all incremental options are OFF (the default), behavior and output are byte-for-byte identical to v1.0.

Output (chunks-jsonl)

{
"id":"a1f3c9e29b2c4d10",
"url":"https://docs.example.com/guide/install",
"title":"Getting Started — Example Docs",
"chunkIndex":3,
"chunkTotal":11,
"headingsPath":["Getting Started","Setup","Installation"],
"text":"## Installation\n\nInstall via npm:\n\n```bash\nnpm install crawlee\n```",
"tokenEstimate":498,
"fetchedAt":"2026-06-20T14:02:11Z",
"content_hash":"sha256:...",
"metadata":{
"source_url":"https://docs.example.com/guide/install",
"deep_link":"https://docs.example.com/guide/install#installation",
"anchor":"installation",
"canonical_url":"https://docs.example.com/guide/install",
"page_title":"Getting Started — Example Docs",
"char_count":2104,
"content_type":"mixed",
"language":"en",
"last_modified":null,
"crawl_timestamp":"2026-06-20T14:02:11Z"
}
}

Each record maps 1:1 to a vector-DB upsert: { id, values=embed(text), metadata }.

Output (langchain)

One record per chunk, drop-in for LangChain — [Document(**r) for r in dataset]:

{
"page_content":"## Installation\n\nInstall via npm...",
"metadata":{
"id":"a1f3c9e29b2c4d10",
"source":"https://docs.example.com/guide/install",
"title":"Getting Started — Example Docs",
"deep_link":"https://docs.example.com/guide/install#installation",
"canonical_url":"https://docs.example.com/guide/install",
"headings_path":["Getting Started","Setup","Installation"],
"chunk_index":3,
"chunk_total":11,
"content_type":"mixed",
"language":"en",
"token_estimate":498,
"char_count":2104,
"content_hash":"sha256:...",
"last_modified":null,
"crawl_timestamp":"2026-06-20T14:02:11Z"
}
}

Output (jsonl-bulk)

Fully flat one-record-per-chunk for generic bulk import (DB COPY / pgvector):

{
"id":"a1f3c9e29b2c4d10",
"text":"## Installation\n\nInstall via npm...",
"source_url":"https://docs.example.com/guide/install",
"deep_link":"https://docs.example.com/guide/install#installation",
"canonical_url":"https://docs.example.com/guide/install",
"title":"Getting Started — Example Docs",
"headings_path":"Getting Started > Setup > Installation",
"chunk_index":3,
"chunk_total":11,
"content_type":"mixed",
"language":"en",
"token_estimate":498,
"char_count":2104,
"content_hash":"sha256:...",
"last_modified":null,
"crawl_timestamp":"2026-06-20T14:02:11Z"
}

Input

See .actor/input_schema.json. Key fields: startUrls, crawlScope, maxCrawlDepth, maxPages, renderJs, outputFormat, chunkSize, chunkOverlap, dedupNearDuplicates, filterJunkChunks, and the incremental sync fields incremental, forceFullCrawl, stateStoreName, emitDeletions.

Pricing

Pay-Per-Event. Billable unit = one emitted dataset item. Deduped and junk-filtered chunks are not billed.

Run locally

npminstall
npm run build
apify run # reads .actor/INPUT.json

Roadmap (v1.2+)

Inline embeddings, direct vector-DB push (Pinecone/Qdrant/Weaviate/pgvector), missedGraceRuns before tombstoning, Standby low-latency mode.