I am using sklearn and python, to build a "malicious" login identifier. Reading some documents and examples, I chose the RandomForest classifier, then I decided to use the following features:
Timestamp (hour)
IP (this is expanded in feature creation adding geolocation info like Coordinates)
Username (hashed)
In order to train the model, I got some data from a real log. I'm assuming here that these are GOOD data. Then I need to train the BAD (malicious) login. For this I got some IPs from some public blacklist and then I create a BAD training file with the same format.
But since I don't know the username for BAD data (since these are not real login), I decided to set it to 0 and than train the model.
I think that this is messing everything up. The model does not work very well and it is not able to detect BAD login, unless I try to query it with a '0' login name. In other words, if I query the model with a BAD IP and a valid username, I always get a "GOOD" result.
May be this is completely expected in this case, but I'd like to understand some things:
- is this way of creating the BAD list actually wrong?
- do I have a way to consider a feature like a "wildcard" without replicating the line in the training for any good username?
- is it possible to set a "priority" in features evaluation, so that some are more important than other? But this should be done by the algorithm itself....or not?