1
$\begingroup$

I am looking to train a neural network to solve a supervised classification task. But one of my input features is a categorical variable that can have more than a few billion possible values.

For example, one crucial input for my problem is the IP address, and I have >4B values in my dataset. I don’t think having a one-hot vector of that size would be feasible.

Is there any way to deal with this type of inputs?

$\endgroup$
1
  • 2
    $\begingroup$ Can you elaborate on how the IP address is important for the problem? $\endgroup$ – noe Jun 16 at 10:36
0
$\begingroup$

As far as I know it's very unlikely that a categorical variable with billions of possible values could be a good predictor for a ML model, but there is certainly some underlying information related to the IP address which are good predictors. So it's a problem of feature engineering not in an technical sense but in a design sense, i.e. using expert knowledge in order to provide the most relevant information to the model.

I don't know the task or the data but you could study what makes the IP useful:

  • location: maybe using a feature representing the country or a more specific location based on the IP would work.
  • historically known IP: for example some boolean features could represent whether this IP connected in the past hour/day/week/...
  • third-part info: features representing whether this IP belongs to some whitelist/blacklist
  • ...
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.