Effects that Empty cells have in Unsupervised Machine Learning

Question

I have a dataframe in a csv file that I would like to perform different unsupervised machine learning algorithms on.

The file itself has some cells that are not filled - they are just empty:

As shown in the picture above, some of the cells are empty. So the question follows - will these empty cells effect the outlier detection process which I would like to use?

Or should I instead work on a code that fills these empty cells with 0's

I have been told that that filling the empty cells has to be done so we don't have data sparsity, even though my csv files is thousands of rows long

Any advice regarding this is more than welcome.

Answer 1

Empty cells can be considered NaN of missing values. There are several ways to work with missing values. You can check this source:

• Encode NAs as -1 or -9999. This works reasonably well for numerical features that are predominantly positive in value, and for tree-based models in general. This used to be a more common method in the past when the out-of-the box machine learning libraries and algorithms were not very adept at working with missing data.

• Casewise deletion of missing data. Here you simply drop all cases/rows from the dataset that contain missing values. In the case of a very large dataset with very few missing values, this approach could potentially work really well. However, if the missing values are in cases that are also otherwise statistically distinct, this method may seriously skew the predictive model for which this data is used. Another major problem with this approach is that it will be unable to process any future data that contains missing values. If your predictive model is designed for production, this could create serious issues in deployment.

• Replace missing values with the mean/median value of the feature in which they occur. This works for numerical features. The choice of median/mean is often related to the form of distribution that the data has. For imbalanced data, the median may be more appropriate, while for symmetrical and more normally distributed data, the mean could be a better choice. Label encode NAs as another level of a categorical variable. This works with tree-based models and other models if the feature can be numerically transformed (one-hot encoding, frequency encoding, etc.). This technique does not work well with logistic regression.

• Run predictive models that impute the missing data. This should be done in conjunction with some kind of cross-validation scheme in order to avoid leakage. This can be very effective and can help with the final model. Use the number of missing values in a given row to create a new engineered feature. As mentioned above, missing data can often have lots of useful signal in its own right, and this is a good way to encode that information.