Encoding a categorical variable with a few billion possible values

Question

I am looking to train a neural network to solve a supervised classification task. But one of my input features is a categorical variable that can have more than a few billion possible values.

For example, one crucial input for my problem is the IP address, and I have >4B values in my dataset. I don’t think having a one-hot vector of that size would be feasible.

Is there any way to deal with this type of inputs?

Answer 1

As far as I know it's very unlikely that a categorical variable with billions of possible values could be a good predictor for a ML model, but there is certainly some underlying information related to the IP address which are good predictors. So it's a problem of feature engineering not in an technical sense but in a design sense, i.e. using expert knowledge in order to provide the most relevant information to the model.

I don't know the task or the data but you could study what makes the IP useful:

• location: maybe using a feature representing the country or a more specific location based on the IP would work.
• historically known IP: for example some boolean features could represent whether this IP connected in the past hour/day/week/...
• third-part info: features representing whether this IP belongs to some whitelist/blacklist
• ...