How to run Scrapy spiders in Python

In this article, we are going to discuss how to schedule Scrapy crawl execution programmatically using Python. Scrapy is a powerful web scraping framework, and it's often necessary to schedule the execution of a Scrapy crawl at specific intervals. Scheduling Scrapy crawl execution programmatically allows you to automate the process of scraping data and ensures that you have the most up-to-date data.

Required Packages

Install Scrapy and schedule the library.

pip install schedule
pip install scrapy

Schedule Scrapy Crawl

In order to schedule Scrapy crawl execution, we will use the schedule library. This library allows us to schedule a task to be executed at a specific time or interval.

Step 1: Create a new folder

Step 2: Inside the folder, start a new project by the following command:

scrapy startproject <project_name>

Step 3: Import schedule library and create a function that runs the Scrapy crawl.

Step 4: Use the schedule library to schedule the crawl function to run at a specific interval

In this example, the crawl function is scheduled to run every 5 minutes. The schedule.run_pending() method checks if any scheduled tasks are due to be run and the time.sleep(1) method is used to prevent the program from using all the CPU resources. You can also schedule the task at a specific time using schedule.every().day.at("10:30").do(crawl). You can also use schedule.clear() method to clear all the scheduled tasks.

Example 1

Create a new folder. Inside the folder, start a new project(Quotes). Create WikiSpider.py file in this code is using the Scrapy library to create a spider that scrapes data from Wikipedia. The spider, called "WikiSpider", is set to start at the URL "https://en.wikipedia.org/wiki/Database" and is configured with a number of custom settings, such as the user agent, download delay, and a number of concurrent requests. The spider's parse method is called when the spider is done crawling and it gets the title of the page and all the paragraphs from the page and writes it in a txt file named "wiki.txt". The code also uses the schedule library to run the spider every 30 seconds and uses an infinite loop to keep running the scheduled spider until it is stopped manually.

Output: Run the Spider

scrapy runspider WikiSpider.py

On running WikiSpider.py, a wiki.txt file is created which contains contents from https://en.wikipedia.org/wiki/Database scheduled every 30 seconds.

EXAMPLE 2

Here is an example of a Scrapy spider that scrapes quotes from a website and prints the output to the console. The spider is scheduled to run every hour using the schedule library.

Create a new folder. Inside the folder, start a new project(Quotes). Create QuotesSpider.py file in this code is using the Scrapy library to create a spider that scrapes data from a website that contains quotes. The spider, called "QuotesSpider", is set to start at the URL "http://quotes.toscrape.com/page/1/", which is a website that contains quotes.
The spider's parse method is called when the spider is done crawling and it gets the text, author, and tags of each quote and yields it as a dictionary. Also, it checks for the next page and follows the link if it exists. The code also uses the schedule library to run the spider every 30 seconds and uses an infinite loop to keep running the scheduled spider until it is stopped manually.

Output: Run the Spider

scrapy runspider QuotesSpider.py