1
$\begingroup$

I have a feature that shows a characteristic of the instances. That characteristic can be present or not. If present it shows an almost normal distribution of values (actually a bit skewed to the right, but with a log transformation it becomes normalized). When the characteristic is not present in the instance, the value of the feature is just 0.

So at the end, I have a distribution with a lot of instances with value 0 and a bit far right from it the almost-normal distribution. I would like to split it in two different features: one that shows the absence/presence of the characteristic (easy), and a second that shows only a normal distribution without the annoying peak around zero.

$\endgroup$
1
$\begingroup$

Aren't you providing the answer? You can split the feature in two, namely, if feature_to_split is the feature you're talking about, you can create feature_to_split_ispresent which will take either 1 or 0 depending on the presence or absence of that specific characteristic, and feature_to_split_value which will take the actual value of that characteristic.

$\endgroup$
3
  • $\begingroup$ Yeah, that I had clear. The problem is: which value do I assign for the feature_to_split_value in the case of absence of the characteristic? The obvious but messy solution is splitting the dataset in two groups one, with the feature_to_split_value feature and another without it. But it is quite annoying $\endgroup$ – Mario Tormo Oct 21 '20 at 12:56
  • $\begingroup$ You could simply assign 0 or NaN value, and use the feature_to_split_ispresent as flag to access the feature_to_split_value only when it is present. $\endgroup$ – Francesco Alongi Oct 21 '20 at 13:12
  • $\begingroup$ Hmm, that would be an option, but i don't know how to tell a regressor to just access the feature when another feature is flagged... $\endgroup$ – Mario Tormo Oct 21 '20 at 13:15
0
$\begingroup$

I don't have a precise answer to that because it depends on what you want to do with that data. Assuming that your task is supervised learning since is the most popular, just extract that feature will be enough for a model to discriminate between different cases.

EDIT:

Models like linear regression or NN works better under normality regime; in this case I would try these options:

  1. Leave 0 because 0 * w = 0 so will be influent into the calculus but still remains the bias term
  2. Replace 0 with the mean of the non-zeros points so your distribution will be normal
  3. Scale non zeros point distribution to a N(0, 1) using standardization
  4. do 2) then 3)
$\endgroup$
3
  • $\begingroup$ Yes, the idea is using it for a regression problem. I want to use different models, and in some of them it is ok if the feature is not normal distributed, but some of them are sensitive to non-normal distributions. Also, I want to transform the feature with a natural log and 0 values are not allowed, so I'm thinking replacing them with a 1. $\endgroup$ – Mario Tormo Oct 21 '20 at 13:19
  • $\begingroup$ ok, edited the answer $\endgroup$ – Mikedev Oct 21 '20 at 13:33
  • $\begingroup$ Yep, that was my idea, but I'm still unhappy with it. The problem I see to that approach is that by replacing the 0 with the mean, if the regressor decides that the flag is not relevant for the model, we are giving the model a huge source of error. My problem with that feature is that, first it appears to be relevant for the prediction according to the published literatur, and second, the zero values represent around the 45% of the dataset, so i can't just drop them. I guess, at the end, I will do what you suggested. $\endgroup$ – Mario Tormo Oct 21 '20 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.