0
$\begingroup$

I'm working on an intrusion detection project, I have many categorical features, for some I used label encoding since I don't have many possible values. But for IP addresses, it's a high cardinality feature, What is the best encoding method to use for features with high cardinality?

I tried using the library ipaddress, here's the code :

 data['dstip'] =int(ipaddress.IPv4Address(data['dstip']))

but it returns : Expected 4 octets in %r

Do you have any better suggestion for encoding features with high cardinality ? Thanks

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Suggestion: don't do it. If you want to have a blacklist of IPs or something like that, do it outside your model. $\endgroup$
    – noe
    Commented Oct 18, 2022 at 14:25
  • $\begingroup$ @noe Why shouldn't I do it please ? $\endgroup$
    – biihu
    Commented Oct 18, 2022 at 14:41
  • $\begingroup$ Because the IP address space is huge and the training signal is potentially very sparse. Of course, there is a certain locality in the maliciousness of the origin IPs, but don't put the burden of identifying those clusters on your model. You'd better understand the clustering patterns of the attack origin IPs (e.g. this) and give your model something like a binary variable "known to be the origin of previous attacks" or/and a numeric measure of "closeness to known previously malicious IP". $\endgroup$
    – noe
    Commented Oct 18, 2022 at 15:34
  • 1
    $\begingroup$ I have created an answer from my original comment so that it can be upvoted and/or marked as correct. $\endgroup$
    – noe
    Commented Oct 19, 2022 at 9:36

1 Answer 1

Reset to default
1
$\begingroup$

My suggestion is this: don't do it. If you want to have a blacklist of IPs or something like that, do it outside your model.

I suggest this because the IP address space is huge and the training signal is potentially very sparse. Of course, there is a certain locality in the maliciousness of the origin IPs, but don't put the burden of identifying those clusters on your model. You'd better understand the clustering patterns of the attack origin IPs (e.g. this) and give your model something like a binary variable "known to be the origin of previous attacks" or/and a numeric measure of "closeness to known previously malicious.

This also allows you to update your list of malicious origin IPs without retraining the model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.