How Botpress Populates AI Chatbot Knowledge Bases at Scale with Firecrawl
Eric CiarlaApr 21, 2025
Botpress uses Firecrawl to power knowledge base creation for AI chatbots, letting users pull content from any website in seconds. One integration replaced an entire class of in-house HTML processing.
What is Botpress? Botpress is an all-in-one platform for building AI agents powered by the latest LLMs, letting teams build, deploy, and monitor agents across channels, tools, and data.
When you're building a bot platform, the knowledge base is the foundation. A bot is only as useful as the content it can access. Manually sourcing and formatting that content doesn't scale.
Michael Masson, CTO at Botpress, and his team are focused on removing friction from every part of bot creation. For their knowledge base feature, the goal was straightforward: let users leverage their existing web content without the manual work typically required.
What was Botpress handling in-house before Firecrawl?
Web scraping is core to Botpress's knowledge base feature. Before Firecrawl, the team was handling HTML to Markdown conversion themselves. This demanded additional processing overhead and ongoing maintenance.
It worked, but it wasn't where the team wanted to spend engineering time.
How does Firecrawl fit into Botpress's knowledge base workflow?
With Firecrawl, Botpress can import content from any website directly into a user's knowledge base with minimal effort. The built-in HTML to Markdown conversion handles all the cleanup automatically, no manual parsing required.
What stood out during integration was how little adaptation was needed.
Unlike other solutions we evaluated, Firecrawl intelligently extracted relevant data right out of the box. This saved us substantial development time and resources — we didn't have to manually parse page content to get the data we needed.
— Michael Masson, CTO, Botpress
What has Botpress's production experience with Firecrawl looked like?
The support from the Firecrawl team has been exceptional. When we quickly hit the default usage limit due to our high volume, their team responded immediately and ensured we maintained access during this critical time.
Stability since launch has been consistent, with very few issues encountered.
Firecrawl is the easiest way to extract relevant content from a website.
If Botpress had to stop using Firecrawl tomorrow, the built-in HTML to Markdown conversion is what they'd miss most. That single capability has streamlined their workflow more than any other part of the integration.
Ready to power your AI application with reliable web data? Try Firecrawl and ship faster.
Frequently Asked Questions
How does Botpress use Firecrawl?
Botpress uses Firecrawl to power their knowledge base feature, letting users import content from any website directly into their chatbot knowledge bases. Firecrawl's built-in HTML to Markdown conversion handles all the cleanup automatically.
What problem did Firecrawl solve for Botpress?
Before Firecrawl, Botpress was handling HTML to Markdown conversion in-house, which required additional processing and ongoing maintenance. Firecrawl extracted relevant data right out of the box, saving the team substantial development time.
What made Firecrawl the right fit for Botpress?
Firecrawl required almost no adaptation during integration and delivered structured, usable output immediately, unlike other solutions Botpress evaluated. The responsive Firecrawl team also ensured continued access when Botpress quickly hit default usage limits due to high volume.
