YouTube Data Scraping, Preprocessing and Analysis using Python

YouTube is one of the largest video-sharing platforms, hosting millions of videos across diverse categories and audiences. Analyzing YouTube data helps uncover valuable insights about content performance, viewer engagement and emerging trends. In this article, we will use real YouTube channel data to extract meaningful patterns and visualise key metrics effectively using Python.

Web Scraping

Web scraping is the process of automatically extracting data from websites. Here we focus on scraping YouTube video data using Python tools like requests and BeautifulSoup, which allows us to collect information such as video titles, views and upload dates for analysis or research purposes.

1. Installing Required Libraries

Here we install requests, beautifulsoup4 and xlsxwriter

2. Importing Libraries

Libraries for sending HTTP requests, parsing HTML content, handling JSON data and writing Excel files are imported to support the scraping pipeline.

3. Sending HTTP Request to YouTube Page

An HTTP request is sent to the YouTube channel’s videos page. A User-Agent header is included to mimic a real browser and avoid request blocking.

Here we will be using: https://www.youtube.com/c/GeeksforGeeksVideos/videos

4. Creating the BeautifulSoup Object

The HTML response obtained from the request is parsed using BeautifulSoup. This allows structured navigation through the page and easy extraction of required elements.

5. Extracting Embedded YouTube JSON Data

YouTube loads video data dynamically using JavaScript. The embedded ytInitialData JSON is extracted from the page source to access structured video information.

6. Navigating to Video Metadata

The nested JSON structure is traversed to locate the section containing individual video metadata. This includes information such as video title, view count and duration.

7. Extracting Titles, Views and Durations

Each video entry is processed in a loop to extract relevant fields safely. The extraction is limited to the most recent 30 videos to keep the dataset manageable.

8. Creating the Excel File

An Excel workbook and worksheet are created using XLSXWriter. Column headers are added to clearly represent each extracted data attribute.

9. Writing Data to Excel Sheet

The extracted video data is written row-by-row into the Excel sheet. This ensures that each video details are stored in a structured tabular format.

10. Saving the File

The workbook is closed to save the Excel file properly. This completes the web scraping process and stores the data for further analysis.

Output:

Scraped latest 30 videos successfully! Saved as youtube_videos.xlsx

Data Preprocessing

Scraped data often contains text-based values and inconsistencies. Data preprocessing cleans and standardizes the data to make it suitable for analysis and visualization.

1. Importing the Data

The scraped Excel file is loaded into a pandas DataFrame. This allows efficient data manipulation and preprocessing operations.

Output:

2. Cleaning the Views Column

The Views column contains textual information. First the views suffix is removed. Then views are converted to numeric values:

• Removing the views suffix.
• Converting values ending with K to numeric by multiplying by 1000.
• Converting plain numeric strings to floats.
• Setting any inconsistent or missing entries to None

3. Cleaning the Duration Column

Here we:

• Removes newline characters.
• Converts durations to total seconds.
• Sets invalid or missing entries to None.

4. Categorizing Videos by Duration

Videos are grouped into categories based on their duration. This simplifies analysis and helps compare different types of content.

Output:

Text Preprocessing

This is an important step in preparing textual data such as video titles, for tasks like text analysis, sentiment analysis or machine learning models. Preprocessing ensures that the text is normalized, clean and meaningful reducing noise and irrelevant information.

1. Importing Required Libraries

Libraries like regular expressions and nltk are imported. These tools help clean and process textual data efficiently.

2. Initializing Stopwords and Stemmer

Stopwords are removed to eliminate commonly used but insignificant words. Stemming is applied to reduce words to their base form.

3. Defining the Text Preprocessing Function

The preprocess_text() function performs the following steps for each title:

• Lowercasing: converts all text to lowercase for uniformity.
• Removing URLs: strips out links that do not contribute to analysis.
• Removing non-alphabetic characters: keeps only letters and spaces.
• Tokenization: splits text into individual words.
• Stopword Removal and Stemming: filters out irrelevant words and reduces words to their root form.
• Joining tokens: combines the processed tokens back into a clean sentence.

4. Applying Text Preprocessing on Video Titles

The preprocessing function is applied to all video titles. The cleaned text is stored back in the DataFrame for further analysis.

Output:

Data Visualization

Data visualization helps convert processed data into graphical form. It makes patterns, trends and insights easier to understand.

1. Word Cloud of Video Titles

A Word Cloud provides a visual representation of the most frequent words in video titles. Larger words indicate higher frequency, helping identify trending topics or common keywords in the channel’s content.

Output:

2. Top 3 Most Viewed Videos

A bar plot is used to display the top-performing videos based on view count. This visualization clearly shows which videos attract the most audience engagement.

Output:

3. Video Count by Duration Category

A count plot visualizes the distribution of videos across different duration categories such as Mini-Videos, Long-Videos and Very-Long-Videos. This helps understand the type of content the channel focuses on.

Output:

You can download full code from here