1
$\begingroup$

I am currently experimenting with user agent strings. My current plan is to tokenize the user agent string and split it in its important parts. Like OS, Browser Name, etc.

After that, does it make sense to just create some sort of map where iOS = 0.1 and Windows = 0.2 where i map the strings to numbers for the operating system feature of my network?

Is that the current best approach to work with such data, or should I try a different approach? The network should predict which category the user will click, and than move that category to a more prominent place.

For example, I expect that an iOS user prefers an iPhone mobile case more than an android one.

Thank you for helping with my experiments.

Improve this question
$\endgroup$
4
  • $\begingroup$ Create two categorical variables: one for ios vs Windows, another for (OS, version). $\endgroup$
    – Emre
    Jun 20, 2017 at 16:45
  • $\begingroup$ I understand that, but I see in the future a small challenge there. What happens when a new Firefox or Chrome version is released? Or less probably, but if a new operating System comes into existence. e.g. Fuchsia from google? If I would use my dictionary or map, I would then just assign a new "number" like 0.31 for a new operating system. Because all my features are float I could have "unlimited" values for my features, or in other words "unlimited" operating systems. $\endgroup$
    – hacker harry
    Jun 20, 2017 at 17:23
  • $\begingroup$ Use feature hashing. $\endgroup$
    – Emre
    Jun 20, 2017 at 17:37
  • $\begingroup$ Your float variables would work, but not usually as well as categorical variables. This is to do with how your model will by default treat distances between values. By using float, you would be feeding the model meaningless distance metric which it would need to overcome. Nonlinear models such as neural networks can do this, so it's not a major issue, but you haven't really given yourself any benefits. Your scenario of adding a new browser is a good example - the number you choose for it will have significant effect on the output before you have any training examples, without any rationale. $\endgroup$
    – Neil Slater
    Jun 20, 2017 at 19:58

2 Answers 2

Reset to default
1
$\begingroup$

iOS, Windows and Android should not be viewed as strings but as categories. Categories should be one-hot encoded, and should not share the same input as they are not related to each other.

So let's assume we work with two operating systems: iOS and Windows.

So for iOS, input is: 1 0 and for Windows input is: 0 1.

But now we want to add another OS: Android.

iOS becomes: 1 0 0, Windows 0 1 0, Android: 0 0 1. Note that by adding android, you do not actually change the input! You only add an extra 0 for iOS and Windows, and 0's don't get computed in neural networks 0*weight = 0.

So by adding an OS, you do not affect what the network has learned for the previously trained OS's.

Improve this answer
$\endgroup$
0
$\begingroup$

I am a bit hesitant to address this question, as I have a conflict of interest and the product is not Open Source (a version of the product is available on the marketplaces of the three major CSPs, though).

WURFL is an API that can be used to turn HTTP user-agent strings (and, more generally, HTTP requests) into device properties, a technology generally called (Mobile) Device Detection. With WURFL, data scientists can map user-agent strings to device data and greatly reduce cardinality in the process. Thousands (or even hundreds of thousands) of different UA strings can be translated into a few hundreds make and models. OS, browsers and their respective versions can be consolidated and easily aggregated. Correlations are more easily explored and determined. Of course, variance can be reduced along a whole lot of different dimensions (i.e. all the properties (AKA capabilities) offered by WURFL), not just model, OS and browser.

The following article shows a practical example of how WURFL can be used for that purpose from within Python, but all major platforms and programming languages are supported.

https://www.scientiamobile.com/wurfl-and-machine-learning/

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.