2
$\begingroup$

I have a data set that has some holes in it. I was wondering if I could add two columns for each feature representing this feature has data and this feature doesn't have data for each of my features. Would a CNN be able to take advantage of this or should I just use another method to approximate the result.

Note The reason why I'm looking for an alternative is that I have a very small dataset and I assume extrapolating would insert to great of a bias on whatever method I use determine the missing value.

Sample Data

0.0,0.3,0.2
0.4,Nan,0.2
0.2,0.6,Nan
Nan,0.1,0.5
0.0,0.2,0.0
0.9,Nan,Nan

Would be converted to

0.0,1,0, 0.3,1,0, 0.2,1,0
0.4,1,0, 0.0,0,1, 0.2,1,0
0.2,1,0, 0.6,1,0, 0.0,0,1
0.0,0,1, 0.1,1,0, 0.5,1,0
0.0,1,0, 0.2,1,0, 0.0,1,0
0.9,1,0, 0.0,0,1, 0.0,0,1

Since my dataset is already relatively small I don't particularly want to remove entries. This would reduce my data by roughly a third.

If this is a 'foul ball out of left field' (makes no sense and would cause more errors) how would you deal with a small data set with a decent amount of holes.

EDIT: As per @zachdj suggestion in comments

Would be instead converted to this data set which is much smaller.

0.0,1, 0.3,1, 0.2,1
0.4,1, 0.0,0, 0.2,1
0.2,1, 0.6,1, 0.0,0
0.0,0, 0.1,1, 0.5,1
0.0,1, 0.2,1, 0.0,1
0.9,1, 0.0,0, 0.0,0
$\endgroup$
6
  • $\begingroup$ Can you give some background info what the data is about and for what reasons data might be missing? $\endgroup$
    – Jonathan
    Nov 18, 2019 at 21:02
  • 1
    $\begingroup$ You don't need to one-hot encode the "missing" indicator. It's already binary. For example, your first row could be converted to 0.0,1, 0.3,1, 0.2,1, and your last row would be 0.9,1, 0.0,0, 0.0,0 $\endgroup$
    – zachdj
    Nov 18, 2019 at 21:06
  • $\begingroup$ The missing data is due to a mixture of non inputted user input and Not Applicable. Majority of the holes would be categorized as non inputted user input. The data types I'm looking to deal with are all numerical (price, size, percentage,..). All my missing categorical data have been encoded as all zeros. $\endgroup$ Nov 18, 2019 at 21:09
  • $\begingroup$ Thanks @zachdj that reduces the size greatly. $\endgroup$ Nov 18, 2019 at 21:09
  • $\begingroup$ I don't know much about neural networks, but if you're using such non-linear models, wouldn't it be safer to choose a value other than 0.0 when there is no value? i.e. something clearly out of the range of possible values. Based on this, the model could interpret it to be a particular case; then you would not even need the binary "this feature has no data" column, this would be implicit. $\endgroup$ Nov 18, 2019 at 22:02

1 Answer 1

1
$\begingroup$

If you are using Python, the Imputer class could be a way to fill in the holes. You can use it to define what counts as a missing value and define a method for replacing those (mean, median, mode, constant value). Without knowing what the data is, it's hard to say whether this is an ideal solution, but it is a very quick one.

The Imputer class also includes a MissingIndicator method which would ouput a Boolean array based on the data. This could be another option, and then you could stack the Boolean array onto the original data.

$\endgroup$
4
  • $\begingroup$ Since I'm worried that my lack of data would cause extrapolation by mean, median, mode or constant value to induce a bias error. Some missing points are in feature groups of 5 to 10 entries. Can Imputer output statistical data on how confident it is that the created data is close to the true mean of the feature for the feature group. Can I also tell Imputer which features it should group the data set by to make its calculations or does it just take the mean,median,... of all available data for the missing feature. $\endgroup$ Nov 19, 2019 at 16:24
  • $\begingroup$ You can specify axis in the Imputer so it would either calculate along rows or columns, but I don't believe it would take into account anything other than each specific row or column, and then calculate the statistic on that particular series. To my knowledge there is not a built-in statistic for computing confidence on the group. Another option would be pd.groupby([[columns]].mean() to calculate the mean for the feature group and then use this as the constant fill value. $\endgroup$
    – whege
    Nov 19, 2019 at 16:41
  • $\begingroup$ With missingness indicators, (most?) models are able to properly deal with the raw column however you imputed it. $\endgroup$
    – Ben Reiniger
    Dec 20, 2019 at 21:52
  • $\begingroup$ Link for sklearn's MissingIndicator: scikit-learn.org/stable/modules/generated/… $\endgroup$
    – Ben Reiniger
    Dec 20, 2019 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.