How to parse an HTTP User-Agent String and Identify Robots in Python

Today we are going to skip the preamble and go straight to our solution process.

We will be using a Cloudmersive API to accomplish our task, so we are going to start with pip installation of its client.

pip install cloudmersive-validate-api-client

And the next step is to write up some code for instancing our API, then calling the user_agent_parse function through that instance.

from __future__ import print_functionimport timeimport cloudmersive_validate_api_clientfrom cloudmersive_validate_api_client.rest import ApiExceptionfrom pprint import pprint# Configure API key authorization: Apikeyconfiguration = cloudmersive_validate_api_client.Configuration()configuration.api_key['Apikey'] = 'YOUR_API_KEY'# Uncomment below to setup prefix (e.g. Bearer) for API key, if needed# configuration.api_key_prefix['Apikey'] = 'Bearer'# create an instance of the API classapi_instance = cloudmersive_validate_api_client.UserAgentApi(cloudmersive_validate_api_client.ApiClient(configuration))request = cloudmersive_validate_api_client.UserAgentValidateRequest() # UserAgentValidateRequest | Input parse requesttry:# Parse an HTTP User-Agent string, identify robotsapi_response = api_instance.user_agent_parse(request)pprint(api_response)except ApiException as e:print("Exception when calling UserAgentApi->user_agent_parse: %s\n" % e)

And just like that, we are done with our setup. Now if we just enter our HTTP User-Agent string, the API will parse it, returning any robots that it finds. You just saved yourself an afternoon of tedious work!

Image for post
Image for post

Written by

There’s an API for that. Cloudmersive is a leader in Highly Scalable Cloud APIs.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store