How to Scrape Websites with Python

Web scraping has made it easier to gather data from the web. With the help of Python you can gather structured data from the internet. For example, if you want to figure out the top 10 most popular scientists, you can use Python to scrape the web and provide data of the most popular ones. We will use this example to continue with the tutorial.

The setup

We will use Python 3 and the virtual environment for this tutorial. You will need to download BeautifulSoup4 and a request package so you can handle all the processes.

Web Requests

The first step in web scraping is to make a request. You will need the request package to do this, it makes it much simpler. You will need to use the “GET” command to request for the URL. Once the command is entered you should have the HTML content of the website in front of you. Once that happens you can use Beautiful Soup. With it you can select and extract. The select method allows you to locate different elements in the document.

With Beautiful Soup you can select the information you want and it will display it for you. You can then search the information from what is in front of you. So for example, if you are looking for popular scientists, with Beautiful Soup you can get a list of them from the website you are looking at. Then you can search specific names from the list. You can use the command “get_names()” and the program can highlight each time the name appears.

Determining the Popularity Score

To get a page list of the name you want, you need to use the command “get_hits_on_name (name).” This will allow you to get the popularity score of each name. With the popularity score, you can sort the names based on which 10 have the highest score.

Web scraping can be tough. Python allows you to do it with a bit of ease but you need to make sure the data is as clean as possible and there are no errors in the process.