How to Use Google Custom Search Engine API in Python

By | February 7, 2021
How to Use Google Custom Search Engine API in Python

Google is the most popular search engine, and compare to other search engines it is fast and returns you the best result based on the searched query.

Let’s say you want to fetch the result showed by the google search engine for the search query “Python”, for this you have an option of web-scraping. But web scraping is not efficient, slow, and for some reason, Google also might block your request considering you as a bot.

Vamware

So Python web scraping for fetching Google results is a big No. So do we have another option? -Yes we have. Google itself provides the Custom Search Engine (CSE) API, which can be used to search results from specific websites or overall Google.

Here in this Python tutorial I will walk you through the Google Custom Search Engine API, and write a Python program to interact with Google CSE & fetch google results for our website TechGeekBuzz.

So let’s get started.

Setup the Google CSE API for Python

First of all, we will be setting the Google Custom Search Engine (CSE), and here our main objective will be to create the API Key and set the search option to the specific website, in this case, which is our own website techgeekbuzz.com.

As here we are using the Google API that’s goes without saying that you should have a Gmail ID for the Google Console developer.

Click here to visit the Google Developer Console Dashboard and login with your Gmail ID. Once you do that you will see a similar Dashboard, there you need to create a New Project.

https://console.developers.google.com/

Click on the My First Project, then click on New Project.

Give Name to your Project, for this tutorial we are creating the Project by the name MyNewProject.

After Creating the Project Select the project and Search for the Custom Search API in the Dashboard Search Bar, and click on the first option.

Now Click on the ENABLE button, and this will enable the Custom Search API for our current project that we have just created.

After Enabling the API, you need to go to the Credentials, there click on the CREATE CREDENTIALS button, and Select the Help me choose an option.

Now click on the API Key link to create a new API Key.

Now Select the API restrictions option and there select the Custom Search API, and hit the Create Button.

When you hit the create button it will create a New API for you with a key now copy the key.

Copy the key in your clipboard because we will be using it in the Python program.

Now we are done with setting the API, now we need to create a Custom Search Engine, and to create a Custom Search Engine we need to visit google https://cse.google.com/cse/all.  And click on the Add button.

After Clicking the ADD button, we need to specify the Sites to Search, as for this tutorial we will be searching for the posts in our own website techgeekbuzz.com, so in the Site to search column we will be filling our website address. And we will be giving our custom search engine the name of MyFirstCustomEngine. After filling the Name and Sites to search click the create button.

This will create a Custom Search Engine for you.

From this Newly created Custom Search Engine we need its Search Engine ID.

To get the Search Engine ID, visit the https://cse.google.com/cse/all, click on the newly created Custom Search Engine, look for the Search Engine ID, and copy it on the clipboard because we will be using it in the Python program.

Now we are all done with setting up the API and Creating the Custom Search API. Now we can write the python script, but before it let’s install the required Library.

Install Required Library

When it’s come to Google APIs, we need to install a Python library provided by google itself to interact with Google API’s.

Run the following Python pip install command to install the goolge-api-python-client library.

pip install google-api-python-client

Now we are all set, open your Best Python IDE or Text Editor and Start Coding.

Create Google Custom Search Engine With Python

Let’s start with importing the required modules, and credentials API_KEY and Search Engine ID.

#google api modules
from apiclient.discovery import build

#credentials 
api_key = "AIzaSyCkaDGV3Ca_jndjdhbswMgPE3dMuciuYKw" 
search_engine_id= "42nskjsh426jsn63d2" #here put your own credential

Now we need to create a Resource object which will send the request to the Custom Search Engine. To build a resource we will use the build() model and call cse() method on it.

#build resouce
resource = build("customsearch", "v1", developerKey=api_key).cse()

The build module accepts 3 mandatory parameters, here we are using the Custom Search Engine that’s why we specify the customsearchservice with version v1and its API key.

Now we need to create a request and execute it with the resource object. The request will contain the query qand the Search Engine ID cx.

After creating the query we need to execute it with execute() method.

#create a request
request =resource.list(q="Python", cx=search_engine_id)

#execute request
result = request.execute()

Normally the web results are sent and received in JSON format, but the execute() method will return the result in Dictionary format which is a similar data structure to JSON.

The result would be a dictionary, where the items key holds all the value for all the search results.

We can print the lent of result["items"] to see the total number of results found on the first page.

print("Total",len(result['items']), "results found on 1st page")

By default the Custom Search Engine show only ten results per page.

Now we can loop through the items and grab the link, title, and description for all the result items.

for item in result["items"]:
    print("|-----------------------------------------|")
    print("Title:", item["title"])   #access title
    print("Link:", item["link"])     #access link
    print("Description:",item["snippet"])    #access description
    print("|-----------------------------------------|\n")

Now put all the code together and Execute.

#Python program to Use Google Custom Search Engine API

#google api modules
from apiclient.discovery import build

#credentials
api_key = "AIzaSyCkaDGV3Ca_jndjdhbswMgPE3dMuciuYKw"
search_engine_id= "42nskjsh426jsn63d2" #here put your own credential

#build resouce
resource = build("customsearch", "v1", developerKey=api_key).cse()

#create a request
request =resource.list(q="Python", cx=search_engine_id)

#execute request
result = request.execute()

print("Total",len(result['items']), "results found on 1st page")
print("---------------------------")

for item in result["items"]:
    print("|-----------------------------------------|")
    print("Title:", item["title"])   #access title
    print("Link:", item["link"])     #access link
    print("Description:",item["snippet"])    #access description
    print("|-----------------------------------------|\n")

Output

Total 10 results found on 1st page
---------------------------
|-----------------------------------------|
Title: Python Global, Local and Nonlocal Variables - TechGeekBuzz
Link: https://www.techgeekbuzz.com/python-global-local-nonlocal-variables/
Description: Nov 1, 2020 ... Python Global Variables. All those variables in python which are declared 
outside the function in a global scope are known asa global variable. In ...
|-----------------------------------------|

|-----------------------------------------|
Title: Python Lambda Function - TechGeekBuzz
Link: https://www.techgeekbuzz.com/python-lambda-function/
Description: Nov 1, 2020 ... Python know as lambda function aka anonymous function. we will discuss how 
and where to use it. so let's discuss Python Lambda Function.
|-----------------------------------------|

|-----------------------------------------|
Title: Python Nested Dictionary
Link: https://www.techgeekbuzz.com/python-nested-dictionary/
Description: Feb 23, 2020 ... In this tutorial we will discuss nested dictionaries in python, we will learn how can 
we create and excess nested dictionaries and its elements.
|-----------------------------------------|

|-----------------------------------------|
Title: Python Variables, Constants & Literals - TechGeekBuzz
Link: https://www.techgeekbuzz.com/python-variables-constants-literals/
Description: Oct 31, 2020 ... In this tutorial, we are going to learn what are the Python variables, Constants & 
Literals with their use. In Python, variables are those.
|-----------------------------------------|

|-----------------------------------------|
Title: Best Way to Learn Python | Tips for Learning Python Programming
Link: https://www.techgeekbuzz.com/best-way-to-learn-python/
Description: May 16, 2020 ... Readability counts. Python's syntax is relatively simple and unique than other 
languages which can be learned easily. Python is used for server- ...
|-----------------------------------------|

|-----------------------------------------|
Title: Learn Python Programming Language - TechGekkBuzz
Link: https://www.techgeekbuzz.com/learn-python-programming/
Description: Nov 3, 2020 ... Python is slower than other programming languages such as C++ and Java, but 
still, it is the number one choice for the Data Science and ...
|-----------------------------------------|

|-----------------------------------------|
Title: Python if, if...else, if...elif...else & Nested if Statement
Link: https://www.techgeekbuzz.com/python-if-elif-else-statements/
Description: Nov 1, 2020 ... we are learn the concept of Flow Control and how to use the if, elif and else 
statement statements. so let's discuss Python If else elif.
|-----------------------------------------|

|-----------------------------------------|
Title: Best Python Compiler You Need to Know In 2020 [Ranked]
Link: https://www.techgeekbuzz.com/best-python-compiler/
Description: May 23, 2020 ... Though by default, python termed as an interpreter, there are many software on 
the internet, which are known as Python compilers.
|-----------------------------------------|

|-----------------------------------------|
Title: 5 Best Python Interpreter You Need to Know in 2020 [Updated]
Link: https://www.techgeekbuzz.com/best-python-interpreter/
Description: Nov 20, 2020 ... Computer can translate and execute the python high-level code. When we install 
python in our system, we apparently install the python ...
|-----------------------------------------|

|-----------------------------------------|
Title: Python Operators: Arithmetic, Comparison, Logical and more.
Link: https://www.techgeekbuzz.com/python-operators/
Description: Dec 12, 2020 ... From web development to artificial intelligence Python is everywhere. As like 
other programming languages, Python also has many operators.
|-----------------------------------------|

From the above output, you can see that all the search results are from techgeekbuzz.com for the “Python” query.

Conclusion

In this Python tutorial, you learned how to set up Custom Search Engine API on Google API console, How to Create A Custom Search Engine, and How to write a Python program to interact with Google Custom Search Engine API.

To Interact with the Google CSE you need the CSE API and the Custom Search Engine ID and you all are good to go.

While Creating the API, do not share the API key or ID with anyone, I have only shared the key and Id to make this tutorial. After writing this tutorial I will delete the API key and ID.

Leave a Reply

Your email address will not be published. Required fields are marked *