0
$\begingroup$

With me is a dataset collected from IoT sensors with one column labeled “Soil Humidity” measured in percentage. It stands to reason then that all the values be positive percentages, however there’s a mix of negative percentage in there which is undesirable. Is there a way to handle unwanted negative numbers in pandas python before running it through a machine learning model.

$\endgroup$
1
$\begingroup$

How to handle invalid values like this is an extremely common problem in machine learning, since most datasets contain errors of some kind.

There are a few ways to do it. For example, you could set them all to 0:

df.loc[df.SoilHumidity < 0, 'SoilHumidity'] = 0

Or you could fill them with the avg(SoilHumidity), and create an extra feature to flag to the model that they were missing:

import numpy as np

df['SoilHumidityInvalid'] = np.where(df.SoilHumidity < 0, 1, 0)

df.loc[df.SoilHumidity < 0, 'SoilHumidity'] = df.SoilHumidity.mean()

Or, you can try to impute them somehow. Either by back or forward filling (I.E. taking the value from the next or the previous row in your dataset) or by creating a model that uses the other features of your dataset to predict what these invalid values should be.

The right method can depend; sometimes domain knowledge guides you (i.e. if you know the sensor can mistakenly read negatives when it should read 0, then you know to fill with 0). Failing that, I would just try a couple of methods and use cross-validation to see which improves your model the most.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.