RAG Web Browser

Pricing

Rating

Developer

Actor stats

Categories

Share

Search the web (or fetch direct URLs) and return clean markdown ready for LLM/RAG pipelines. HTTP-first with chrome131 TLS impersonation; Playwright fallback when needed. Pro filters narrow the result set to exactly what your retrieval index needs.

What this actor does

• Sends a query to Google → extracts top organic results → fetches each → cleans HTML → emits structured records
• Or fetches direct URLs (startUrls) and skips Google entirely
• Strips boilerplate (nav, footer, ads, scripts) before extracting the main article
• Outputs markdown (default), plain text, and/or raw html per page
• Returns one clean record per page with title, description, language, word count, reading time

Output per page

• url, loadedUrl, domain
• title, description, languageCode
• text (plain text) — when requested
• markdown (LLM-ready) — when requested
• html (raw cleaned HTML) — when requested
• wordCount, readingTimeMinutes (220 wpm)
• httpStatusCode, loadedTime (seconds)
• searchRank (1-based — when the URL came from Google search)
• recordType: "page", scrapedAt

Empty fields are omitted from the output (no nulls).

Input

Example: search query

{
"query":"best vector database for RAG 2024",
"maxResults":5,
"outputFormats":["markdown"],
"minTextLength":500,
"domainBlocklist":["pinterest.com","youtube.com"]
}

Example: direct URLs

{
"startUrls":[
"https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
"https://docs.langchain.com/docs/use-cases/qa/"
],
"outputFormats":["markdown","text"],
"htmlTransformer":"readable-text"
}

Example: filter for relevance

{
"query":"vector embeddings tutorial",
"maxResults":10,
"keywordsAnyOf":["embedding","vector","similarity"],
"minTextLength":1000,
"outputFormats":["markdown"]
}

Use cases

• RAG ingestion — pull fresh top-N Google results for a topic, hand markdown to your embedder
• News briefings — daily query like "AI news today", filter by minTextLength to drop SEO thin pages
• Competitive monitoring — domainAllowlist of competitor domains, scrape their blogs weekly
• Reference enrichment — feed each citation URL from a paper into the actor for clean text extraction
• LLM context — give an LLM the cleaned markdown of pages, not raw HTML, to save tokens

FAQ

Does it require a login or cookies? No. All fetches are anonymous.

Is a proxy needed? Apify proxy is enabled by default to avoid Google rate-limits and unblock some target sites. You can disable it with useApifyProxy: false.

What's the difference between raw-http and browser-playwright? Raw HTTP uses curl_cffi with chrome131 TLS impersonation — fast and works on ~80% of sites. Browser mode runs headless Chromium, waits for JS, and dismisses cookie banners — slower but handles SPAs.

Why is description missing on some pages? Some pages don't expose a <meta name="description"> or og:description. The omit-empty contract drops missing fields rather than emit nulls.

Why does markdown look stripped down? We intentionally output simple markdown (headings, lists, links, emphasis, code) — RAG embedders strip most formatting anyway, and simpler markdown reduces token bloat.

What if all my filters reject every result? The actor finishes cleanly with a status message instead of pushing placeholder rows.

How do I use this with my LLM/RAG pipeline? Trigger this actor from your indexing job, read the dataset (each record has url + markdown), embed the markdown, store in your vector DB.