1"""Module defines the main entry point for the Apify Actor.
2
3Feel free to modify this file to suit your specific needs.
4
5To build Apify Actors, utilize the Apify SDK toolkit, read more at the official documentation:
6https://docs.apify.com/sdk/python
7"""
8
9from __future__ import annotations
10
11# Beautiful Soup - A library for pulling data out of HTML and XML files. Read more at:
12# https://www.crummy.com/software/BeautifulSoup/bs4/doc
13# Apify SDK - A toolkit for building Apify Actors. Read more at:
14# https://docs.apify.com/sdk/python
15from apify import Actor
16from bs4 import BeautifulSoup
17
18# HTTPX - A library for making asynchronous HTTP requests in Python. Read more at:
19# https://www.python-httpx.org/
20from httpx import AsyncClient
21
22
23asyncdefmain()->None:
24"""Define a main entry point for the Apify Actor.
25
26 This coroutine is executed using `asyncio.run()`, so it must remain an asynchronous function for proper execution.
27 Asynchronous execution is required for communication with Apify platform, and it also enhances performance in
28 the field of web scraping significantly.
29 """
30asyncwith Actor:
31# Retrieve the input object for the Actor. The structure of input is defined in input_schema.json.
32 actor_input =await Actor.get_input()or{'url':'https://apify.com/'}
33 url = actor_input.get('url')
34ifnot url:
35raise ValueError('Missing "url" attribute in input!')
36
37# Create an asynchronous HTTPX client for making HTTP requests.
38asyncwith AsyncClient()as client:
39# Fetch the HTML content of the page, following redirects if necessary.
40 Actor.log.info(f'Sending a request to {url}')
41 response =await client.get(url, follow_redirects=True)
42
43# Parse the HTML content using Beautiful Soup and lxml parser.
44 soup = BeautifulSoup(response.content,'lxml')
45
46# Extract all headings from the page (tag name and text).
47 headings =[]
48for heading in soup.find_all(['h1','h2','h3','h4','h5','h6']):
49 heading_object ={'level': heading.name,'text': heading.text}
50 Actor.log.info(f'Extracted heading: {heading_object}')
51 headings.append(heading_object)
52
53# Save the extracted headings to the dataset, which is a table-like storage.
54await Actor.push_data(headings)

A template for web scraping  data from a single web page in Python. The URL of the web page is passed in via input, which is defined by the input schema . The template uses the HTTPX  to get the HTML of the page and the Beautiful Soup  to parse the data from it. The data are then stored in a dataset  where you can easily access them.

The scraped data in this template are page headings but you can easily edit the code to scrape whatever you want from the page.

• Apify SDK  for Python - a toolkit for building Apify Actors  and scrapers in Python
• Input schema  - define and easily validate a schema for your Actor's input
• Request queue  - queues into which you can put the URLs you want to scrape
• Dataset  - store structured data where each object stored has the same attributes
• HTTPX  - library for making asynchronous HTTP requests in Python
• Beautiful Soup  - library for pulling data out of HTML and XML files

1. Actor.get_input() gets the input where the page URL is defined
2. httpx.AsyncClient().get(url) fetches the page
3. BeautifulSoup(response.content, 'lxml') loads the page data and enables parsing the headings
4. This parses the headings from the page and here you can edit the code to parse whatever you need from the page

   for heading in soup.find_all(["h1","h2","h3","h4","h5","h6"]):

5. Actor.push_data(headings) stores the headings in the dataset

• BeautifulSoup Scraper 
• Python tutorials in Academy 
• Web scraping with Beautiful Soup and Requests 
• Beautiful Soup vs. Scrapy for web scraping 
• Integration with Make, GitHub, Zapier, Google Drive, and other apps 
• Video guide on getting scraped data using Apify API 
• A short guide on how to build web scrapers using code templates: