Pinned Loading

1. WarcProtocol Public

   Parser for WARC (aka WebArchive) files

   C# 15 4

2. CommonCrawl Public

   Common Crawl's processing tools

   C# 11

3. UrlNormalization Public

   URL normalizer to canonicalize (standardize) the text representation of a URL to determine if differently-formatted URLs are identical

   C# 5

4. SitemapsProtocol Public

   Parsers for sitemap / sitemap index (aka Sitemaps Protocol)

   C#

5. RobotsProtocol Public

   Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag

   C#