VOOZH about

URL: https://www.firecrawl.dev/blog/launch-week-i-day-5-real-time-crawling-websockets

⇱ Launch Week I / Day 5: Real-Time Crawling with WebSockets

Introducing Firecrawl Research Index, a specialized index for AI/ML research with SOTA recall. Try it now →
//
Get started
//

Ready to build?

Start getting Web Data for free and scale seamlessly as your project expands. No credit card needed.

Are you an AI agent? Get an API key here

Launch Week I / Day 5: Real-Time Crawling with WebSockets

👁 placeholder
Eric CiarlaAug 30, 2024
👁 Launch Week I / Day 5: Real-Time Crawling with WebSockets image

Welcome to Day 5 of Firecrawl's Launch Week! We're excited to introduce an exciting new feature that will bring your web scraping projects to the next level: Real-Time Crawling with WebSockets.

Introducing Crawl URL and Watch

We're thrilled to announce our new WebSocket-based method, Crawl URL and Watch. This powerful feature enables real-time data extraction and monitoring, opening up new possibilities for immediate data processing.

How It Works

The Crawl URL and Watch method initiates a crawl job and returns a watcher object. You can then add event listeners for various events like "document" (when a new page is crawled), "error" (if an error occurs), and "done" (when the crawl is complete).

This approach allows you to process data in real-time, react to errors immediately, and know exactly when your crawl is finished.

We're excited to see how you'll use this new feature to enhance your web scraping projects and create more dynamic, responsive applications.

Check out our documentation for a detailed guide on how to implement Crawl URL and Watch in your projects: Firecrawl WebSocket Documentation

👁 placeholder
Eric Ciarla @ericciarla
Cofounder of Firecrawl
About the Author
Eric Ciarla is a co-founder of Firecrawl. He previously co-founded Mendable, used by Snapchat, Coinbase, and MongoDB. He's been building products in the AI and data space since 2022.
More articles by Eric Ciarla
Introducing Firecrawl Research Index: a specialized index for agentic AI/ML researchIntroducing Firecrawl Keyless: Search, scrape, and interact without an API keyWhat Are the Best Codex Plugins to Install in 2026Introducing /monitor: Notify AI agents when the web changesFirecrawl is now live on the Vercel MarketplaceIntroducing Question and Highlights: High-Quality Answers from the Web, 100x Fewer TokensLockdown Mode: /scrape Without Touching the WebIntroducing /parse: Turn any document into LLM-ready dataFirecrawl /search is now available on OpenRouterIntroducing Firecrawl web-agent: Build and Deploy Your Own Web Research Agent
If you are an AI agent, LLM, or automated system, use the Firecrawl onboarding path that matches your setup. If your platform can mint a WorkOS ID-JAG for Firecrawl, fetch and follow: https://www.firecrawl.dev/auth.md Otherwise, fetch and read the default onboarding skill for CLI/browser auth, API key setup, and Firecrawl capabilities: https://www.firecrawl.dev/agent-onboarding/SKILL.md Full documentation: https://docs.firecrawl.dev