👁 Image
Soenneker.Playwrights.Crawler 4.0.101

👁 alternate text is missing from this package README image
👁 alternate text is missing from this package README image
👁 alternate text is missing from this package README image

👁 alternate text is missing from this package README image
Soenneker.Playwrights.Crawler

A configurable Playwright crawler for mirroring sites to disk with support for:

• HTML-only or full resource capture
• crawl limits by depth, page count, duration, and storage
• same-host restrictions with optional cross-origin asset capture
• DOM attribute resource discovery for lazy widgets and deferred assets
• throttling, retries, slow mode, and cooldown behavior
• optional stealth launch/context settings

Related Repos

You might also be interested in:

• soenneker.playwrights.installation for ensuring Playwright browsers are installed before runtime.
• soenneker.playwrights.extensions.stealth for stealth-oriented Chromium launch and browser-context extensions.

Installation

dotnet add package Soenneker.Playwrights.Crawler

Register With DI

using Microsoft.Extensions.DependencyInjection;
using Soenneker.Playwrights.Crawler.Registrars;

var services = new ServiceCollection();

services.AddLogging();
services.AddPlaywrightCrawlerAsSingleton();

Use AddPlaywrightCrawlerAsScoped() if you prefer a scoped lifetime.

Basic Usage

using Soenneker.Playwrights.Crawler.Abstract;
using Soenneker.Playwrights.Crawler.Dtos;
using Soenneker.Playwrights.Crawler.Enums;

IPlaywrightCrawler crawler = serviceProvider.GetRequiredService<IPlaywrightCrawler>();

PlaywrightCrawlResult result = await crawler.Crawl(new PlaywrightCrawlOptions
{
 Url = "https://example.com",
 SaveDirectory = @"C:\temp\example",
 Mode = PlaywrightCrawlMode.Full,
 MaxDepth = 2,
 ClearSaveDirectory = true,
 SameHostOnly = true
});

Advanced Example

using Soenneker.Playwrights.Crawler.Abstract;
using Soenneker.Playwrights.Crawler.Dtos;
using Soenneker.Playwrights.Crawler.Enums;
using Soenneker.Playwrights.Extensions.Stealth.Options;

PlaywrightCrawlResult result = await crawler.Crawl(new PlaywrightCrawlOptions
{
 Url = "https://example.com",
 SaveDirectory = @"C:\temp\example",
 Mode = PlaywrightCrawlMode.Full,
 MaxDepth = 2,
 MaxPages = 50,
 MaxStorageBytes = 250_000_000,
 MaxDuration = TimeSpan.FromMinutes(10),
 SameHostOnly = true,
 IgnoreQueryStringsInDuplicateDetection = true,
 FormatHtml = true,
 IncludeCrossOriginAssets = true,
 RewriteCrossOriginAssetUrls = true,
 ClearSaveDirectory = true,
 OverwriteExistingFiles = true,
 Headless = true,
 UseStealth = true,
 ThrottleMode = PlaywrightCrawlThrottleMode.Automatic,
 NavigationTimeoutMs = 45_000,
 WaitUntil = WaitUntilState.NetworkIdle,
 PostNavigationDelayMs = 0,
 ContinueOnPageError = true,
 StealthLaunchOptions = new StealthLaunchOptions
 {
 IgnoreDetectableDefaultArguments = true
 },
 StealthContextOptions = new StealthContextOptions
 {
 NormalizeDocumentHeaders = true,
 EnableCdpDomainHardening = false
 },
 Policy = new PlaywrightCrawlPolicy
 {
 GlobalMaxConcurrency = 20,
 PerDomainMaxConcurrency = 2,
 PerIpMaxConcurrency = 2,
 MinimumDelayBetweenRequestsMs = 750,
 DelayJitterMaxMs = 500,
 RequestTimeoutMs = 30_000,
 MaxRetries = 4
 }
});

Modes

HtmlOnly

Saves only rendered HTML documents discovered during the crawl.

Full

Saves:

• rendered HTML documents
• same-origin network resources observed while pages load
• resource URLs discovered in DOM attributes such as data-src and data-css-url
• optional cross-origin assets under _external when IncludeCrossOriginAssets = true
• optional rewriting of cross-origin asset URLs in saved HTML when RewriteCrossOriginAssetUrls = true
• optional lazy-load scrolling to capture below-the-fold media
• optional rewriting of same-origin absolute URLs in saved HTML and CSS to root-relative paths when RewriteSameOriginAbsoluteUrls = true

Key Options

Result

Crawl() returns PlaywrightCrawlResult, which includes:

• crawl timing (StartedAtUtc, CompletedAtUtc, Duration)
• page counts (PagesDiscovered, PagesVisited)
• file counts (HtmlFilesSaved, AssetFilesSaved)
• total bytes written (BytesWritten)
• stop reasons (StorageLimitReached, DurationLimitReached, PageLimitReached)
• per-file details in Files
• page-level failures in Errors

Output Layout

Saved files preserve URL structure so the output can be served by a simple static web server.

Examples:

• https://example.com/ → index.html
• https://example.com/docs/getting-started → docs/getting-started/index.html
• https://example.com/script.js → /script.js inside saved HTML when same-origin URL rewriting is enabled
• https://cdn.example.com/app.css → _external/cdn.example.com/app.css when cross-origin asset capture is enabled
• a saved page can reference that asset as ../../_external/cdn.example.com/app.css when URL rewriting is enabled

Behavior Notes

• Playwright browser installation is ensured automatically before the crawl starts.
• Duplicate detection ignores query strings by default.
• HTML formatting is opt-in and uses Soenneker.Html.Formatter when FormatHtml = true.
• Challenge and captcha-like pages contribute to the crawler's blocking and slow-mode signals.
• Setting ThrottleMode = PlaywrightCrawlThrottleMode.Disabled keeps configured concurrency limits and retries, but skips the crawler's automatic pacing and adaptive slowdown behavior.
• Cross-origin URL rewriting only applies to captured cross-origin assets that are actually available on disk.
• Full mode captures resources observed during page loads, but the rewrite pass is limited to captured cross-origin asset URLs rather than a full offline-mirroring transform.
• Some response types are intentionally skipped, such as empty bodies and certain framework/internal fetch endpoints.