0
$\begingroup$

I am working on financial data where I have a feature(column) with 90% values between 0-1000 (continuous) and 10% values as -1, -2 and -9. (default values)

Default value definition: -1: data not available -2: Partial data available -9: Data available but its too old

My question - As default value are categorical in nature, how should I treat the 10% default cases to separate them from 90% continuous data points, Should I set them to values like "-9999", "-999" ,"-99" or any other suggestion?

I want to use to data to train classification model

$\endgroup$

1 Answer 1

0
$\begingroup$

If you treat the categorical and continuous values as a single continuous feature, then there is an implied numerical relationship between the values used categorically and the continuous values. So unless there is a real basis for such a relationship, then it's unlikely to achieve good results.

One way to handle this sort of data is to split it into two features - one to handle the continuous values and one to handle the categorical defaults. Then one-hot encode the defaults. Your data points with continuous values would have 0's for all the one-hot encoded columns. The continuous feature would need to be set to a (single) default value for all data points with categorical values.

For instance, if the data is:

ID Feature
1 100
2 250
3 -1
4 -2
5 -9
6 75
7 -2
8 0
9 -1
10 125

This could be encoded as:

ID F_continuous F_-1 F_-2 F_-9
1 100 0 0 0
2 250 0 0 0
3 0 1 0 0
4 0 0 1 0
5 0 0 0 1
6 75 0 0 0
7 0 0 1 0
8 0 0 0 0
9 0 1 0 0
10 125 0 0 0
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.