Pricing

Rating

Developer

Actor stats

Categories

Share

Sitemap Generator

Automatically crawl any website and generate XML, HTML, and text sitemaps for SEO optimization. Perfect for submitting to Google Search Console, Bing Webmaster Tools, and improving search engine indexing.

Features

• Automatic Page Discovery: Intelligently crawls websites following internal links.
• Customizable Crawling: Set crawling depth and apply filters to include/exclude specific pages.
• Multiple Sitemap Formats:
  • XML: Standard sitemap format for search engines.
  • HTML: Human-readable sitemap for visitors.
  • Text: Simple list of URLs.
• Advanced Filtering: Use Regex patterns to include or exclude specific URLs.
• Robots.txt Support: Automatically respects robots.txt rules (configurable).
• Monetization: Pay-Per-Event pricing model (charged per page successfully crawled).
• Direct Connection: This Actor uses a direct connection for crawling and does not support proxies. Note that for large-scale crawling, this might be limited by target site rate limits.

Output

The Actor generates the following outputs:

Key-Value Store

1. sitemap.xml: Standard XML sitemap (if selected).
2. sitemap.html: HTML sitemap for users (if selected).
3. sitemap.txt: Plain text URL list (if selected).

Dataset

The dataset will contain the public URLs of the generated sitemap files. For example:

{
"format":"xml",
"url":"https://api.apify.com/v2/key-value-stores/..../records/sitemap.xml"
}

Input Configuration

Required

• Start URLs (startUrls): List of URLs to start crawling from.

Optional

• Max Crawl Depth (maxCrawlDepth): How deep to crawl (default: 3).
• Max Pages Per Crawl (maxPagesPerCrawl): Limit the number of pages (default: 30).
• Include Patterns (includePatterns): Regex patterns for URLs to include.
• Exclude Patterns (excludePatterns): Regex patterns for URLs to exclude.
• Sitemap Formats (sitemapFormats): "xml", "html", "txt" (default: "xml").
• Respect robots.txt (respectRobotsTxt): Whether to respect robots.txt (default: true).
• Change Frequency (changefreq): "always", "hourly", "daily", "weekly", "monthly", "yearly", "never" (default: "weekly").
• Default Priority (defaultPriority): Priority for pages, 0.0 to 1.0 (default: 0.5).
• Include Images (includeImages): Include image URLs in XML sitemap (default: false).

Examples

Basic Usage

{
"startUrls":[{"url":"https://example.com"}],
"maxCrawlDepth":3,
"sitemapFormats":["xml"]
}

Advanced Usage

{
"startUrls":[{"url":"https://example.com"}],
"maxCrawlDepth":5,
"maxPagesPerCrawl":1000,
"excludePatterns":[".*/admin/.*",".*/login.*"],
"sitemapFormats":["xml","html"],
"respectRobotsTxt":true,
"changefreq":"daily",
"priority":0.8
}

Related Actors

Expand your web scraping capabilities with these other powerful Apify Actors developed by the same author:

Website Screenshotter

• URL: https://apify.com/eunit/website-screenshotter
• Description: Capture high-quality website screenshots. Features full-page capture, mobile emulation, custom resolutions, and export to JPG, PNG, or PDF. Fast and reliable. Export your result in any format of your choosing.

LinkedIn Job Postings Scraper

• URL: https://apify.com/eunit/linkedin-job-postings-scraper
• Description: Designed for lead generation and career intelligence, this scraper extracts tailored job titles, company names, locations, and job URLs from LinkedIn. Essential for matching job market trends with real estate data (e.g., analyzing local economic activity).

Target Product Reviews Scraper

• URL: https://apify.com/eunit/target-reviews-scraper
• Description: A tool for market research and e-commerce analysis. Scrape product reviews, star ratings, and consumer sentiment from Target.com, useful for businesses analyzing the retail landscape in target neighborhoods.

Your feedback

We are always working on improving the performance of our Actors. So if you’ve got any technical feedback for the Sitemap Generator Actor or simply found a bug, please create an issue on the Actor’s Issues tab in Apify Console.