How to parse an HTTP User-Agent String and Identify Robots in Python

Today we are going to skip the preamble and go straight to our solution process.

We will be using a Cloudmersive API to accomplish our task, so we are going to start with pip installation of its client.

pip install cloudmersive-validate-api-client

And the next step is to write up some code for instancing our API, then calling the user_agent_parse function through that instance.

from __future__ import print_function
import time
import cloudmersive_validate_api_client
from cloudmersive_validate_api_client.rest import ApiException
from pprint import pprint
# Configure API key authorization: Apikey
configuration = cloudmersive_validate_api_client.Configuration()
configuration.api_key['Apikey'] = 'YOUR_API_KEY'
# Uncomment below to setup prefix (e.g. Bearer) for API key, if needed
# configuration.api_key_prefix['Apikey'] = 'Bearer'
# create an instance of the API class
api_instance = cloudmersive_validate_api_client.UserAgentApi(cloudmersive_validate_api_client.ApiClient(configuration))
request = cloudmersive_validate_api_client.UserAgentValidateRequest() # UserAgentValidateRequest | Input parse request
try:
# Parse an HTTP User-Agent string, identify robots
api_response = api_instance.user_agent_parse(request)
pprint(api_response)
except ApiException as e:
print("Exception when calling UserAgentApi->user_agent_parse: %s\n" % e)

And just like that, we are done with our setup. Now if we just enter our HTTP User-Agent string, the API will parse it, returning any robots that it finds. You just saved yourself an afternoon of tedious work!