How to parse an HTTP User-Agent String and Identify Robots in Python

Today we are going to skip the preamble and go straight to our solution process.

We will be using a Cloudmersive API to accomplish our task, so we are going to start with pip installation of its client.

pip install cloudmersive-validate-api-client

And the next step is to write up some code for instancing our API, then calling the user_agent_parse function through that instance.

from __future__ import print_functionimport timeimport cloudmersive_validate_api_clientfrom import ApiExceptionfrom pprint import pprint# Configure API key authorization: Apikeyconfiguration = cloudmersive_validate_api_client.Configuration()configuration.api_key['Apikey'] = 'YOUR_API_KEY'# Uncomment below to setup prefix (e.g. Bearer) for API key, if needed# configuration.api_key_prefix['Apikey'] = 'Bearer'# create an instance of the API classapi_instance = cloudmersive_validate_api_client.UserAgentApi(cloudmersive_validate_api_client.ApiClient(configuration))request = cloudmersive_validate_api_client.UserAgentValidateRequest() # UserAgentValidateRequest | Input parse requesttry:# Parse an HTTP User-Agent string, identify robotsapi_response = api_instance.user_agent_parse(request)pprint(api_response)except ApiException as e:print("Exception when calling UserAgentApi->user_agent_parse: %s\n" % e)

And just like that, we are done with our setup. Now if we just enter our HTTP User-Agent string, the API will parse it, returning any robots that it finds. You just saved yourself an afternoon of tedious work!

There’s an API for that. Cloudmersive is a leader in Highly Scalable Cloud APIs.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store