spatie/crawler

This package provides a powerful, easy to use class to crawl links on a website. Under the hood, Guzzle promises are used to crawl multiple URLs concurrently.

Because the crawler can execute JavaScript, it can crawl JavaScript rendered sites. Under the hood, Chrome and Puppeteer are used to power this feature.

Here's a quick example:

use Spatie\Crawler\Crawler;
use Spatie\Crawler\CrawlResponse;

Crawler::create('https://example.com')
 ->onCrawled(function (string $url, CrawlResponse $response) {
 echo "{$url}: {$response->status()}\n";
 })
 ->start();

Or collect all URLs on a site:

$urls = Crawler::create('https://example.com')
 ->internalOnly()
 ->depth(3)
 ->foundUrls();

You can also test your crawl logic without making real HTTP requests:

Crawler::create('https://example.com')
 ->fake([
 'https://example.com' => '<html><a href="/about">About</a></html>',
 'https://example.com/about' => '<html>About page</html>',
 ])
 ->foundUrls();

If you need to stop a crawl based on external state, you can register a callback that receives the current crawler instance and is checked before scheduling each next request:

use Spatie\Crawler\Crawler;

$shouldStop = false;

Crawler::create('https://example.com')
 ->shouldStopCallback(function (Crawler $crawler) use (&$shouldStop) {
 return $shouldStop;
 })
 ->onCrawled(function (string $url) use (&$shouldStop) {
 $shouldStop = true;
 })
 ->start();

Support us

👁 Image

We invest a lot of resources into creating best in class open source packages. You can support us by buying one of our paid products.

We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on our contact page. We publish all received postcards on our virtual postcard wall.

Documentation

All documentation is available on our documentation site.

Testing

composer test

Changelog

Please see CHANGELOG for more information on what has changed recently.

Contributing

Please see CONTRIBUTING for details.

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities.

Credits

License

The MIT License (MIT). Please see License File for more information.

URL: https://packagist.org/packages/spatie/crawler

⇱ spatie/crawler - Packagist.org

spatie/crawler

Maintainers

Package info

Fund package maintenance!

Statistics

Security

README

Crawl the web using PHP