Take the power of web scraping in your hands
The phrase “we have enough data” does not exist in the language of data science. I have never come across anyone who has voluntarily said no to collecting more data for their machine learning or deep learning project. And there are often situations where the data you have just isn't enough..
That's when the power of web scraping comes to the fore.. It's a powerful technique that any analyst or data scientist should possess that will keep you in good stead in the industry. (And when you're seated for interviews!).
There are a large number of Python libraries available to perform web scraping. But, How do you decide which one to choose for your particular project? Which Python library has the most flexibility? My goal is to answer these questions here, Through the Lens of Five Popular Python Libraries for Web Scraping I Think Every Enthusiast Should Know About.
Python libraries for web scraping
Web scraping is the process of extracting structured and unstructured data from the web with the help of programs and exporting it to a useful format. If you want to learn more about web scraping, here are a couple of resources to get you started:
Very well, Let's check out the web scraping libraries in Python!!
1. Request Library (HTTP for humans) for web scraping
Let's start with the most basic Python library for web scraping. ‘Requests’ allows us to make HTML requests to the website server to retrieve the data on your page. Obtaining the HTML content of a web page is the first and most important step of web scraping.
Petitions is a python library used to make various types of HTTP requests like GET, MAIL, etc. Due to its simplicity and ease of use, comes with the HTTP for humans tagline.
I'd say this is the most basic yet essential library for web scraping. But nevertheless, request library does not parse retrieved HTML data. If we want to do that, we need libraries like lxml and Beautiful Soup (we'll cover them later in this article).
Let's take a look at the pros and cons of the Requests Python library.
Advantage:
- Simple
- Basic authentication / implicit
- URLs and international domains
- Fragmented requests
- HTTP proxy support (S)
Disadvantages:
- Retrieves only the static content of a page
- Cannot be used to parse HTML
- I can't handle websites created exclusively with JavaScript.
2. lxml library for web scraping
We know the requests the library cannot parse the HTML retrieved from a web page. Therefore, we require lxml, a Python library of production quality XML and HTML parsing, incredibly fast and high-performance.
Combine the speed and power of item trees with the simplicity of Python. Works well when our goal is to extract large data sets. Combining requests Y lxml it is very common in web scraping. It also allows you to extract data from HTML using XPath and CSS selectors.
Let's take a look at the advantages and disadvantages of lxml Python Library.
Advantage:
- Faster than most analyzers
- Light
- Use item trees
- API Pythonic
Disadvantages:
- Doesn't work well with poorly designed HTML
- The official documentation is not very suitable for beginners.
3. Beautiful library of soups for web scraping
Beautiful soup it is perhaps the most used python library for web scraping. Create a parse tree to parse HTML and XML documents. Beautiful Soup automatically converts incoming documents to Unicode and outgoing documents to UTF-8.
One of the main reasons the Beautiful Soup library is so popular is that it is easier to work with and suitable for beginners.. We can also combine Beautiful Soup with other analyzers like lxml. But all this ease of use comes at a cost: is slower than lxml. Even while wearing lxml as analyzer, is slower than pure lxml.
One of the main advantages of the Beautiful Soup library is that it works very well with poorly designed HTML and is feature rich. Combining Beautiful soup Y Petitions it is quite common in the industry.
Advantage:
- Requires a few lines of code
- Great documentation
- Easy to learn for beginners
- Robust
- Automatic encoding detection
Disadvantages:
If you want to learn how to copy web pages with Beautiful Soup, this tutorial is for you:
4. Selenium library for web scraping
There is a limitation for all Python libraries that we have discussed so far: we cannot easily extract data from dynamically populated websites. It happens because sometimes the data present on the page is loaded via JavaScript. In simple words, if the page is not static, the python libraries mentioned above struggle to extract the data from it.
That's where selenium comes in..
Selenium is a Python library originally created for automated testing of web applications. Although it was not originally made for web scraping, The data science community changed that pretty quickly!!
It is a web controller created to render web pages, but this functionality makes it very special. Where other libraries are not able to execute JavaScript, Selenium excels. You can click on a page, fill forms, scroll the page and do a lot more.
This ability to run JavaScript on a web page gives Selenium the power to extract dynamically populated web pages. But here's a trade-off. Load and run JavaScript for each page, which makes it slower and not suitable for large-scale projects.
If time and speed are not a concern for you, you can definitely use selenium.
Advantage:
- Suitable for beginners
- Automated web scraping
- Can scrape dynamically populated web pages
- Automate web browsers
- You can do anything on a web page similar to a person
Disadvantages:
- Very slow
- Difficult to configure
- High CPU and memory usage
- Not ideal for large projects.
Here is a wonderful article to learn how Selenium works (including Python code):
5. Scrapy
Now is the time to introduce you to the BOSS of Python web scraping libraries: ¡Scrapy!
Scrapy it's not just a library; is a complete web scraping framework created by the co-founders of Scrapinghub: Pablo Hoffman and Shane Evans. It's a full-blown belt scraping solution that does all the heavy lifting for you.
Scrapy provides spider robots that can crawl multiple websites and extract the data. Con Scrapy, can create your spider robots, host them in Scrapy Hub or as an API. Allows you to create fully functional spiders in minutes. You can also create pipelines using Scrapy.
The best thing about Scrapy is that it is asynchronous. Can make multiple HTTP requests simultaneously. This saves us a lot of time and increases our efficiency. (And don't we all strive for that?).
You can also add plugins to Scrapy to improve its functionality. Although Scrapy can't handle JavaScript like selenium, you can pair it with a library called Splash, a lightweight web browser. With Splash, Scrapy can even extract data from dynamic websites.
Advantage:
- Asynchronous
- Excellent documentation
- Various add-ons
- Create custom middleware and pipelines
- Low CPU and memory usage
- Well designed architecture
- A plethora of online resources available
Disadvantages:
- Steep learning curve
- Excess for easy jobs
- Not suitable for beginners
If you want to learn Scrapy, that I highly recommend, you should read this tutorial:
Whats Next?
Personally, I find these python libraries extremely useful for my requirements. I'd love to hear your thoughts on these libraries or if you use any other Python libraries, let me know in the comment section below.
If you liked the article, share it on your network and keep practicing these techniques.
