What is Web Scraping?

Imagine a process where you want to acquire data from any resource or website. The first thing that a developer thinks of does the source has a developer account and it provides an API to get the required information, but in the majority of cases, there are no dev accounts and no APIs available, which makes it difficult to acquire information. Here web scraping comes to the rescue. In web scraping, we analyze the HTML of a website and write the data extraction logic programmatically and store the information in either DB or export the data as a spreadsheet or send the data as a part of the API 

Tools & Technologies used for web scraping 

Python language used for web scraping. In Python, we have the Scrappy framework and a library called BeautifulSoup widely used to get solutions for web scraping. Selenium acts as a mode to mimic browsers in case we need to provide user credentials to log in to any website/application, or it is used in cases where the data is loaded only when we scroll to the bottom and extract data. Using Selenium, we can provide click events and mouse movements that mimic the user accessing the website or application.

Step-by-step process of Web scraping 

  • The first step will be to Identify the target website from where we want to extract the data in the case. I have chosen Flipkart to scrape the reviews of any product.
  • Then we need to collect the URLs of the pages which you want to extract data from.
  • Now we need to make a request to these URLs to get the HTML of the page to make the request. We can use the requests library in python.
  • After this, you will have the HTML page of the URLs as text then. We need to use the Beautiful soup library, with the help of which we can easily track the location of the desired data and extract(scrape) only that data out of the HTML.
  • Now we can save the data in a JSON, CSV file, or another structured format.
  • Challenges in Web Scraping

A- Although it is easy to scrape the data for small projects. There are some challenges in most cases, such as tackling the change in the layout of the webpage, which means if the layout of the webpage changes, the logic we have implemented to get the structured data should also be changed accordingly.

B- Since web scraping requires a lot of requests made to a server from an IP address. The server may detect too many requests and block the IP address to stop further scraping. Proxies rotate the IP address, which helps to avoid blocking the scraper and any issues. It also helps hide the machine’s IP address as it creates anonymity.

C- Most websites use CAPTCHA to detect bot traffic. By using CAPTCHA-solving services, we can bypass this extra layer. There are a few CAPTCHA-solving services: 

Implementing Web Scraping in Python with BeautifulSoup 

So here we will see the implementation of building a simple web app using flask to scrape the reviews of the products listed on the Flipkart webpage:

Here the first steps will be to install and import all the required libraries:

g9ABA3zcX1POSqWtxlfxCG71Zdby7KgTSUNgErXhpkJV_nwdeNVSuPs2Z-F0f1UehFHeJ2MbotEl4IuOcX4z91QZdnL7QVrDaTiIrouly9ORlEQYgptCHd9MGLBRosq2wzezCrxM4ahN7R5yGh00Y5wUkJA_-uDlyBhnU2e-dtjdjsvWfCsEGkCT4A What is Web Scraping?

After successfully importing the required libraries, we will write the steps to fetch the reviews in the structured format; below code is to fetch the reviews section data; we just have to pass the product name in the search string from the frontend:

5Us2m35rGN0aDd1XcbsOOc10EbmIH4A8VrNCTKt6n7XM9s1H1h1fEQvLWkSfqwHjvXqzXu7LShvfmLydJraKih5jGkT4xDINr1yPfnx4O2Gq0tN12Z14YdBLattez-gGqXBEj9t6ClSgHMxUJMscH_uoOXXbrwJ-C2shKWld6oF5tRnoP2g1-3C59g What is Web Scraping?

To select the class of any object on the HTML, you have to go to  the HTML page and right-click on the page anywhere, like the below snapshot:

uD-63FasFUmNVGANaJ2hd59LJDLjNbe5cCrMoUlxznbu8OOKSR1lSREodET53SAsy4ccDLjnHpq9zOWEAWsZV1vvLLq3pXfmVjstoEDBm1F6rJdOsP1HiDTOFqhluihFzOghIDtdRYULRZkVyGwnfuEWG61D0nHvxgj_INneYeIKYitssxVcqJNPiw What is Web Scraping?

And click on the inspect option; after this, a dev page will open. You just have to hover the mouse over the object to see the class of that object on the dev page; below is the example snapshot attached:

0R25llVtBDlwe36srTuCjwFEL_CXpApnv_XeWTds6_kMvdIwHWYrxKvn3O1ZrkQZRZETfeBRWYKe4DBd2xXUwO4XY9dJWx3Azti_tYQdOxFGZILgZJ-XgNxidJG3wjxQBiZ6VSE2TUjuFYiG4Dbcx2obCl6kRTMzU2_Ccy8_DAXkjBZvoPqI9x8-Sw What is Web Scraping?

So as you can see, by hovering over the review, it is showing “_16PBlm” as the class name. So I have taken that as the class name in the code.

In the above code, the first step is going to the Flipkart searched product webpage, such as if you search for iPhone in the search box from the frontend. It will go to the below page internally:

tVwBOg4YC2xsi0slfftPS78YPFc2Mog1FAuNpGKZrWkGRlp7kjpLpHGbUgiSMYBgyYxDzQ0rwj8hT6nZGpXo6U72EZKOiE6fE8E_Xn2ZYqEjQVU9VSO_VITbWaInGwcZ5Ed4v3SLECHY29E6TROwDCzPMy8T1UfvuzgVx784ayH6h_pLJ-aCwKxo5A What is Web Scraping?

And will select the first product to fetch the reviews:

kJpwqjLefgb4xTwEueW2jh89f_2IA9qV-9pOB7M_scJ0CQDXS0kCb7ptJMgvLrBYoQaZ2zHtpj0OFJPxttcVgO4GHLsOaaa4JNYc_4rQqvVm1y5aCyHdNkZIK6Z0L0cK7wdFYmf4izFIdtTOteYbi3KYwwS8Ln1foahWzKFsq0K1KsB0OKMqzImW2w What is Web Scraping?

The above image has the reviews that will be scrapped in a table with the header “Product, Customer Name, Rating, Heading and Comment” in a structured format.

Moving further, we will add the respective data to the table and handle the exceptions that may occur. If any of the above fields are blank and the below code is to add the data in the table while handling the exceptions:

gDcF_FzJf8VmUnfrwnGg3g1MmYGefAV4pKAgUgfbU8RQ4BBMatnxhJTK1Yrx_58le_uuWrb0ivU-z6mL8OUqCedCKq1pcfLqIykz9qJ-bMIaYRj4Uqv_Oba6dbG10dVkJ96z52ixBbkjNM86FXo8hWz2phjWUKKQkovIg9Ag2YrO5oFtz0sDnA--Qg What is Web Scraping?

Now, we will run this code and will get the below output:

6GoSSANqjd5Y3QW8DIPQW7qoXwbToB8B6wWKEOTr78UQu4mRsLQYhywrdCTEZS2UrxVfRce9uUNdkN-xuUxlI5D9ymAZauPnG5S3714PAVcZjHB1M4lTcb-jIhTnRrb3nKjB7ZY_UZRauCuMYT6x5lbx0fXIDMAQ6Ng7WKhvWl_i3yxb8IBMviH9zw What is Web Scraping?

Now we can see that our application is running on the localhost By clicking on this HTTP link, we will redirect to the webpage :

6eYywBrCEctaepSPbPqsnoqx439T1471dSLcG1GmCUK7Rp_EUwRK9LLAemVEvk3bcyBiHpFRAjDsRYR4WxrfYxYQAJKYgxLxyxhHP2uuPPq3g5ANAJDCW0qNTL9Zq3vZghQquH9PhyWlUWV21ZOPWkqKctCH63B785SUTTMD1_WNR4P0aH7gxN_Ung What is Web Scraping?

So here, in the search box, we can search with the name of the product, of which we have to scrape the reviews.

The below snapshot is the Output of the search with the product name iPhone. This will fetch you real-time reviews from the webpage:

Qkup3spK6FIxKtHYrmREp7g2Dy0KsqzkI-eD9VZmUe_1JTU-kx4pNQw8mwqWcenKUlZapJFwyThIduNjOPqf-cEAwJfKFyiezjVy69VkM0ljzCMUI6grpCybfT9U37Bkztwz0ZVP0IzzVxPdnr-Mw8989zXnItUK5EHJu_BJtEU95es6LYWOR_-rfg What is Web Scraping?

Web scraping services offered by Thinkitive 

At thinkitive, we have worked on various projects in different domains and built browser extensions to support web scraping. We have a framework that speeds up the time to scrape any resource. We ensure the scraper is not blocked by taking all possible precautions. At Thinkitive, we have built scrapers that serve thousands of requests simultaneously. We have scrapers built for all types of websites, like dynamic web pages, which are rendered infinitely.

For any questions and inquiries related to web scraping services or if you’re looking to hire Python developers, please contact us.

Pankaj Tamhane

Python Engineering Manager

Related Articles


  1. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again soon!

  2. Cool. I spent a long time looking for relevant content and found that your article gave me new ideas, which is very helpful for my research. I think my thesis can be completed more smoothly. Thank you.

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button