With me is a dataset collected from IoT sensors with one column labeled “Soil Humidity” measured in percentage. It stands to reason then that all the values be positive percentages, however there’s a mix of negative percentage in there which is undesirable. Is there a way to handle unwanted negative numbers in pandas python before running it through a machine learning model.
1 Answer
$\begingroup$
$\endgroup$
How to handle invalid values like this is an extremely common problem in machine learning, since most datasets contain errors of some kind.
There are a few ways to do it. For example, you could set them all to 0:
df.loc[df.SoilHumidity < 0, 'SoilHumidity'] = 0
Or you could fill them with the avg(SoilHumidity), and create an extra feature to flag to the model that they were missing:
import numpy as np
df['SoilHumidityInvalid'] = np.where(df.SoilHumidity < 0, 1, 0)
df.loc[df.SoilHumidity < 0, 'SoilHumidity'] = df.SoilHumidity.mean()
Or, you can try to impute them somehow. Either by back or forward filling (I.E. taking the value from the next or the previous row in your dataset) or by creating a model that uses the other features of your dataset to predict what these invalid values should be.
The right method can depend; sometimes domain knowledge guides you (i.e. if you know the sensor can mistakenly read negatives when it should read 0, then you know to fill with 0). Failing that, I would just try a couple of methods and use cross-validation to see which improves your model the most.
