Scraping Quotes using Python

Web scraping is the process of automatically extracting data from websites and converting it into a structured format such as tables or files. In this article, we will learn how to scrape quotes from a website using Python libraries like Requests and BeautifulSoup and store the extracted data in a DataFrame for analysis.

Prerequisites

Installed the following Python libraries:

Implementation

Step 1: Import Required Libraries and Connect to the Website

We first import the required libraries such as requests and beautifulSoup. It sends a request to the website.

• requests.get(): fetches the webpage.
• BeautifulSoup(): parses the HTML content so we can extract data.

website used in this article: https://quotes.toscrape.com/

Step 2: Extract All Quotes Text

Now, we extract all the quote texts present on the page.

Step 3: Extract Author Names

Finds all author names and stores them inside a list.

Step 4: Extract quote details, tags and author links

In this step, we extract all information related to each quote. Extracts quote text, Extracts author name, Extracts author profile link, extracts all tags, Prints everything for verification.

Output:

Step 5: Store Extracted Data in a List

We now store all extracted values together so they can be converted into a table later.

Step 6: Collect author elements

This step collects all author HTML elements from the page, which can be useful for further inspection or advanced data extraction.

Output:

Step 7: Extract tags

Here, we extract all tags associated with a quote to understand the themes or categories linked to it.

Step 8: Extract only quote text

This step focuses on isolating just the quote text from the HTML structure for clean and direct use.

Output:

A day without sunshine is like, you know, night.

Step 9: Convert data into a DataFrame

Converts scraped data into a table. Makes it easier to analyze and store.

Output:

Step 10: Scrape multiple pages

In this step, we automate the scraping process across multiple pages to gather a larger and more complete dataset.

Step 11: Create final DataFrame

Converts all scraped pages into a DataFrame. Renames columns and builds full author profile URLs.

Output:

Now we have created a dataframe and it can be further used for analysis and model making.