Pricing

Rating

Developer

Actor stats

Categories

Share

RAG-Ready Content Structurer

Turn pages you own into deterministic, embeddings-ready RAG chunks — clean, hashed, and token-bounded.

What it does

This Actor ingests a list of URLs that you own or are authorized to crawl, fetches each page, strips boilerplate, and converts it into clean Markdown. It then produces RAG chunks: semantic (heading-aware) or fixed-size chunks with full heading lineage, a deterministic token estimate, and a sha256 content hash per chunk — ready to upsert into a vector store.

The pipeline is built around a fail-closed gate chain:

1. Gate A — Ownership attestation. You must attest you own/are authorized for every URL. If not true, the run is rejected before any fetch, with zero events billed.
2. Gate B — Paid-plan only. Default-deny until the Apify paid-plan run flag is confirmed; free/unknown plans run nothing and are not billed.
3. Gate C — robots.txt. Always honored and cannot be disabled. Disallowed URLs emit 0 chunks, are charged 0, and are listed in robots_skipped_urls.

Cleaning uses Mozilla Readability + Turndown for boilerplate-stripped Markdown. Chunking is deterministic: identical input yields byte-stable output and the same sha256 content hash, and no chunk exceeds maxTokens.

Note: the token count per chunk is a deterministic word-based estimate used to bound chunk size, not a tiktoken count.

Input

Defined by INPUT_SCHEMA.json. Key fields:

Output

Defined by dataset_schema.json. Every record carries a record_type. The run path emits:

• chunk (RAG): chunk_id (position-stable key for idempotent vector-DB upserts — re-crawling updates the same vectors instead of duplicating them), source_url, page_title, section_path (heading lineage, e.g. ["Guide","Setup","Auth"]), heading (immediate section title), chunk_index, section_chunk_index/section_chunk_count (position within the section), chunk_text (clean Markdown), token_count (≤ maxTokens), char_count, word_count, overlap_prev (chunk carries overlap from the previous one), content_hash (sha256:...), language, extracted_fields, retrieved_at, render_mode.
• run_summary (exactly one per run): pages_requested, pages_fetched, pages_failed ([{url, reason}] — pages that failed extraction are isolated here at zero charge, never crashing the run), chunks_emitted, total_tokens, robots_skipped_urls, output_mode.

Chunking quality. Code fences are kept atomic — a chunk never contains a half-open ``` fence; an oversized code block is split with each piece re-wrapped in its original fence + language, so every chunk is independently valid, embeddable Markdown. Splits prefer natural boundaries (paragraph → sentence → word → char), and a hard character cap (secondary to the token bound) means a whitespace-free blob (minified JS, base64) can't masquerade as a tiny chunk and silently blow an embedding model's real token limit. All output is deterministic and byte-stable across runs.

The dataset schema also defines AEO record types (faq_pair, answer_block, llms_txt, citation_block) and a structured-extraction extracted_fields shape, with prebuilt dataset views (RAG chunks, AEO assets, Run summary). These AEO/extraction outputs are reserved in the schema but are not produced by the current run path — the Actor currently emits RAG chunk records plus one run_summary.

The RAG chunks view is ready to upsert into Pinecone / Weaviate / Qdrant / pgvector.

Pricing

Pay-Per-Event. You are billed only for what actually runs (after the gates), via Actor.charge():

The developer keeps 80% and Apify keeps 20% (standard Apify 80/20 split).

Example run cost — 100 owned pages, RAG mode: $0.05 + (100 × $0.003) = $0.35.

Why this Actor

• Deterministic & idempotent. Cleaning and chunking are pure and byte-stable: identical input produces identical chunks and identical sha256 content hashes — safe re-runs, safe vector-store de-duplication.
• Ownership-gated and robots-compliant by design. A mandatory ownership/authorization attestation rejects unauthorized runs before any fetch (zero billing), and robots.txt is forced on and cannot be disabled.
• Paid-plan only, fail-closed billing. Default-deny until a paid plan is confirmed; failed and robots-disallowed pages charge $0.
• Embeddings-ready output with token bounds. Chunks carry heading lineage and a token estimate that never exceeds your maxTokens, in a schema with a prebuilt view for Pinecone / Weaviate / Qdrant / pgvector.

About

This Actor is AI-authored and operated under the publisher's LLC. Actor.charge() is the only billing path and it bills the customer only — the Actor has no payout or money-out capability; revenue settlement is handled entirely by Apify's monetization rail.