Biblioteca BeautifulSoup | Web Scraping with Python: Biblioteca BeautifulSoup

Share on facebook
Share on twitter
Share on linkedin
Share on telegram
Share on whatsapp



unstructured data in a structured format? This is where Web Scraping comes in..

What is Web Scraping?

In plain language, Web scraping, web harvest, O web data extraction is an automated big data collection process (unstructured) of websites. The user can extract all the data on particular sites or the specific data according to the requirement. The collected data can be stored in a structured format for further analysis.

Uses of web scraping:

In the actual world, web scraping has gained a lot of attention and has a wide range of uses. Some of them are listed below:

  1. Social Media Sentiment Analysis
  2. Lead generation in the marketing domain
  3. Market analysis, online price comparison in the e-commerce domain
  4. Collect training and test data in machine learning applications

Steps involved in web scraping:

  1. Find the URL of the web page you want to scrape
  2. Select particular items by inspecting
  3. Writes the code to get the contents of the selected items
  4. Store data in the required format

That simple guys! .. !!

Libraries / Popular tools that are used for web scraping are:

  • Selenium: a framework for testing web applications
  • BeautifulSoup: Python library for getting HTML data, XML and other markup languages
  • Pandas: Python library for data manipulation and analysis

In this article, we will create our own dataset by extracting Domino's Pizza reviews from the website.

We will use requests Y Beautiful soup by scraping and analysis the data.

Paso 1: find the URL of the web page you want to scrape

Open the URL “"And search for Domino's Pizza in the search bar and press Enter.


Below is what our reviews page looks like.


Paso 1.1: Defining the base URL, query parameters

The base URL is the consistent part of your web address and represents the path to the website's search function.

base_url = ""

query parameters represent additional values that can be declared on the page.

query_parameter = "?page="+str(i) # i represents the page number
URL = base URL + query parameter (image by author)

Paso 2: select particular items by inspecting

Below is an image of a sample review. Each review has many elements: the rating given by the user, the username, the date of the review and the text of the review along with some information about how many people you liked.


Our interest is to extract only the text of the review. For that, we need to inspect the page and get the HTML tags, the attribute names of the target element.

To inspect a web page, right-click the page, select Inspect or use the keyboard shortcut Ctrl + Shift + I.

In our case, the revision text is stored in the html tag

of the div with the class name "rvw-bd

Inspecting the target elements

With this, we get acquainted with the website. Let's quickly jump to scraping.

Paso 3: Writes the code to get the contents of the selected items

Start by installing the modules / required packages

pip install pandas requests BeautifulSoup4

Import required libraries

import pandas as pd
import requests
from bs4 import BeautifulSoup as bs

pandas – to create a data frame
requests: to send HTTP requests and access HTML content from the destination web page
BeautifulSoup: is a Python library for parsing structured HTML data

Create an empty list to store all extracted reviews

all_pages_reviews = []

define a scraper function

def scraper():

Inside the scraper function, type a for the loop to run through the number of pages you would like to scrape. I'd like to scrape the five-page reviews.

for i in range(1,6):

Creando una lista vacía para almacenar las reseñas de cada página (of 1 a 5)

pagewise_reviews = []

Construye la URL

url = base_url + query_parameter

Envíe la solicitud HTTP a la URL mediante solicitudes y almacene la respuesta

response = requests.get(url)

Cree un objeto de sopa y analice la página HTML

soup = bs(response.content, 'html.parser')

Encuentre todos los elementos div del nombre de claservw-bdy guárdelos en una variable

rev_div = soup.findAll("div",attrs={"class","rvw-bd"})

Recorra todo el rev_div y agregue el texto de revisión a la lista pagewise

for j in range(len(rev_div)):
			# finding all the p tags to fetch only the review text

Anexar todas las reseñas de páginas a una sola listaall_pages about

for k in range(len(pagewise_reviews)):

Al final de la función, devuelve la lista final de reseñas.

return all_pages_reviews
Call the function scraper() and store the output to a variable 'reviews'
# Driver code
reviews = scraper()

Paso 4: almacene los datos en el formato requerido

4.1 almacenamiento en un marco de datos de pandas

i = range(1, len(reviews)+1)
reviews_df = pd.DataFrame({'review':reviews}, index=i)
Now let us take a glance of our dataset

4.2 Escribir el contenido del marco de datos en un archivo de texto

reviews_df.to_csv('reviews.txt', sep = 't')

With this, terminamos de extraer las reseñas y almacenarlas en un archivo de texto. Mmm, es bastante simple, ¿no?

Código Python completo:

# !pip install pandas requests BeautifulSoup4 
import pandas as pd
import requests
from bs4 import BeautifulSoup as bs
base_url = ""
all_pages_reviews =[]
  1. def scraper():
        for i in range(1,6): # fetching reviews from five pages
            pagewise_reviews = [] 
            query_parameter = "?page="+str(i)
    	url = base_url + query_parameter
    	response = requests.get(url)
    	soup = bs(response.content, 'html.parser') 
    	rev_div = soup.findAll("div",attrs={"class","rvw-bd"}) 
        for j in range(len(rev_div)):
        # finding all the p tags to fetch only the review text
        for k in range(len(pagewise_reviews)):
            return all_pages_reviews
    # Driver code
    reviews = scraper()
    i = range(1, len(reviews)+1)
    reviews_df = pd.DataFrame({'review':reviews}, index=i)
    reviews_df.to_csv('reviews.txt', sep = 't')

Final notes:

At the end of this article, we have learned the step-by-step process of extracting content from any web page and storing it in a text file.

  • inspect the target item using the browser's development tools
  • use requests to download HTML content
  • parsing HTML content using BeautifulSoup to extract the required data

We can further develop this example by scraping usernames, reviewing text. Vectorize clean revision text and group users according to written revisions. We can use Word2Vec or CounterVectorizer to convert text into vectors and apply any of the Machine Learning grouping algorithms.


Biblioteca BeautifulSoup: Documentation, Video Tutorial

DataFrame to CSV

GitHub repository link to download source code

I hope this blog helps to understand web scraping in Python using the BeautifulSoup library. Happy learning !! 😊

The media shown in this article is not the property of DataPeaker and is used at the author's discretion.

Subscribe to our Newsletter

We will not send you SPAM mail. We hate it as much as you.