I am currently experimenting with user agent strings. My current plan is to tokenize the user agent string and split it in its important parts. Like OS, Browser Name, etc.

After that, does it make sense to just create some sort of map where iOS = 0.1 and Windows = 0.2 where i map the strings to numbers for the operating system feature of my network?

Is that the current best approach to work with such data, or should I try a different approach? The network should predict which category the user will click, and than move that category to a more prominent place.

For example, I expect that an iOS user prefers an iPhone mobile case more than an android one.

Thank you for helping with my experiments.

  • $\begingroup$ Create two categorical variables: one for ios vs Windows, another for (OS, version). $\endgroup$ – Emre Jun 20 '17 at 16:45
  • $\begingroup$ I understand that, but I see in the future a small challenge there. What happens when a new Firefox or Chrome version is released? Or less probably, but if a new operating System comes into existence. e.g. Fuchsia from google? If I would use my dictionary or map, I would then just assign a new "number" like 0.31 for a new operating system. Because all my features are float I could have "unlimited" values for my features, or in other words "unlimited" operating systems. $\endgroup$ – hacker harry Jun 20 '17 at 17:23
  • $\begingroup$ Use feature hashing. $\endgroup$ – Emre Jun 20 '17 at 17:37
  • $\begingroup$ Your float variables would work, but not usually as well as categorical variables. This is to do with how your model will by default treat distances between values. By using float, you would be feeding the model meaningless distance metric which it would need to overcome. Nonlinear models such as neural networks can do this, so it's not a major issue, but you haven't really given yourself any benefits. Your scenario of adding a new browser is a good example - the number you choose for it will have significant effect on the output before you have any training examples, without any rationale. $\endgroup$ – Neil Slater Jun 20 '17 at 19:58

iOS, Windows and Android should not be viewed as strings but as categories. Categories should be one-hot encoded, and should not share the same input as they are not related to each other.

So let's assume we work with two operating systems: iOS and Windows.

So for iOS, input is: 1 0 and for Windows input is: 0 1.

But now we want to add another OS: Android.

iOS becomes: 1 0 0, Windows 0 1 0, Android: 0 0 1. Note that by adding android, you do not actually change the input! You only add an extra 0 for iOS and Windows, and 0's don't get computed in neural networks 0*weight = 0.

So by adding an OS, you do not affect what the network has learned for the previously trained OS's.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.