How to Convert HTML Tables into CSV Files in Python?

By | February 18, 2021
How to Convert HTML Tables into CSV Files in Python

HTML is the standard language to create web-pages, it generally used to structure the text, images, and video data on the web-page. HTML can represent the Text data in various formats such as links, normal text, tables, lists, etc. Let’s say you have an HTML or you want to grab the HTML web-page from the Internet and wish to extract the table data to analyze or crunch the data.

Here in this Python tutorial, I will walk you through the Python program on How to extract table data from the HTML web-pages and save it locally in CSV files.

Vamware

Before we dive into the Python program let’s discuss and install the libraries we will be using in this Python tutorial.

Required Libraries

Python requests library

We will be using the requests library to send HTTP GET request to the web-page and in response getting HTML text data.

to install the requests library run the following pip command on your terminal or command prompt.

pip install requests

Python beautifulsoup4 Library

The beautifulsoup4 library is an open-source Python HTML & XML data extractor library. We will be using this library to extract data table data from the HTML page using HTML tag names like <th>, <table>, <tr>, and <td>.

You can install this library using the following pip install command.

pip install beautifulsoup4

Python csv Module

CSV(Comma Separated Values) is one of the Python Standard Libraries, you do not need to install it separately. As the Library names suggest, we can use it to write and read between the CSV files.

To know more about How to write CSV files in Python click here.

Note: As in this tutorial, we will be extracting data from HTML tables, I am assuming that you have some knowledge on how HTML <table> tag works or how other tags like <th>, <tr>, and <td> used.

How to Convert HTML Tables into CSV Files in Python?

Let’s begin with importing the modules for our Python program.

import requests
from bs4 import BeautifulSoup
import csv

Now define a Python variable url for the web-page URL

url= r"https://www.techgeekbuzz.com/difference-between-repeater-datalist-and-gridview/"

response = requests.get(url) #send get request
html_page = response.text #fetch HTML page

get()function will send a get request to the url and,

text property will fetch the response HTML web-page.

Now we will parse the html_page using BeautifulSoup() module so we can extract the html_page using BeautifulSoup find_all() function.

page = BeautifulSoup(html_page, 'html.parser')    #parse html_page

As in this tutorial we are only fetching tables data, let’s extract all the tables present in the page .

tables = page.find_all("table")  #find tables

#print the total tables found
print(f"Total {len(tables)} Found on page {url}")

The find_all("table")will return a list of all the <table> tags present in page.

Now we will loop through every table present in tables list, create the new CSV file and write table data on the CSV file.

for index, table in enumerate(tables):
    print(f"\n-----------------------Table{index+1}-----------------------------------------\n")

    table_rows = table.find_all("tr")

    #open csv file in write mode
    with open(f"Table{index+1}.csv", "w", newline="") as file:
        
        #initialize csv writer object
        writer = csv.writer(file)

        for row in table_rows:
            row_data= []

            #<th> data
            if row.find_all("th"):
                table_headings = row.find_all("th")
                for th in table_headings:
                    row_data.append(th.text.strip())
            #<td> data
            else:
                table_data = row.find_all("td")
                for td in table_data:
                    row_data.append(td.text.strip())
            #write data in csv file
            writer.writerow(row_data)
            
            print(",".join(row_data))
    print("--------------------------------------------------------\n")

Now put all the code together and execute

Python program to Convert web-page tables to CSV files

import requests
from bs4 import BeautifulSoup
import csv

url= r"https://www.techgeekbuzz.com/difference-between-repeater-datalist-and-gridview/"

response = requests.get(url)
html_page = response.text

soup = BeautifulSoup(html_page, 'html.parser')
#find <table>
tables = soup.find_all("table")
print(f"Total {len(tables)} Table(s)Found on page {url}")

for index, table in enumerate(tables):
    print(f"\n-----------------------Table{index+1}-----------------------------------------\n")
    
    #find <tr>
    table_rows = table.find_all("tr")

    #open csv file in write mode
    with open(f"Table{index+1}.csv", "w", newline="") as file:

        #initialize csv writer object
        writer = csv.writer(file)

        for row in table_rows:
            row_data= []

            #<th> data
            if row.find_all("th"):
                table_headings = row.find_all("th")
                for th in table_headings:
                    row_data.append(th.text.strip())
            #<td> data
            else:
                table_data = row.find_all("td")
                for td in table_data:
                    row_data.append(td.text.strip())
            #write data in csv file
            writer.writerow(row_data)

            print(",".join(row_data))
    print("--------------------------------------------------------\n")

Output

Total 3 Table(s) Found on page https://www.techgeekbuzz.com/difference-between-repeater-datalist-and-gridview/

-----------------------Table1-----------------------------------------

DataList,Repeater
Object Rendering
Its object rendered as a table,It uses templates to render its object.
Columns generation
It automatically generates columns using the data source.,It cannot generate columns.
Row selection
It can select a row from the data source.,It cannot select rows.
Content Editing
Using DataList control, we can edit object content.,It does not support content editing.
Data arrangement
It has RepeatDirection property which allows it to arrange data vertically and horizontally.,It cannot manually arrange data.
Performance
It is slower than Repeater.,It is very fast because of its lightweight.
--------------------------------------------------------

-----------------------Table2-----------------------------------------

GridView,Repeater
Debut
GridView was introduced in Asp.Net 2.0,The Repeater was introduced in Asp.Net 1.0.
Columns generation
It automatically generates columns using the data source.,It cannot generate columns.
Row selection
It can select a row from the data source.,It cannot select rows.
Content Editing
Using GridView control, we can edit object content.,It does not support content editing.
In-built methods
It comes with built-in paging and sorting methods.,No built-in support for Built-in paging and sorting developer has to code.
Auto formatting and styling
In GridView we get inbuilt auto format and styling feature.,It does not support these features.
Performance
It is slower than Repeater.,Because of its lightweight, it is faster as compared to GridView.
--------------------------------------------------------

-----------------------Table3-----------------------------------------

GridView,DataList
Debut
GridView was introduced in Asp.Net 2.0 version.,DataList was introduced in Asp.Net 1.0 version.
In-built methods
It comes with built-in paging and sorting methods.,No built-in support for Built-in paging and sorting, the developer has to code for these features.
Build-in CRUD operation
It comes with built-in Update and Deletes Operations, so the developer does not need to write code for simple operations.,If developer use DataList then he/she has to write code for the Update and Delete operations.
Auto formatting and styling
In GridView we get inbuilt auto format and styling feature.,It does not support these features.
Customizable Row
We do not get Customizable row separator feature in GridView.,DataList has SeparatorTemplate for customizable row separator.
Performance:
Its performance is the lowest as compared to Repeater and DataList.,It is faster than the GridView.
--------------------------------------------------------

When you execute the above program you will see that it will save the csv file in the same directory where your Python script is located.

Conclusion

In this Python tutorial, we learned “How to convert HTML tables to CSV files in Python?” this tutorial is an small application of web-scrapping with Python. If you want to learn more about extracting data from web-pages, you can read the official documentation of BeautifulSoup4.

Leave a Reply

Your email address will not be published. Required fields are marked *