94
$\begingroup$

I got ValueError when predicting test data using a RandomForest model.

My code:

clf = RandomForestClassifier(n_estimators=10, max_depth=6, n_jobs=1, verbose=2)
clf.fit(X_fit, y_fit)

df_test.fillna(df_test.mean())
X_test = df_test.values  
y_pred = clf.predict(X_test)

The error:

ValueError: Input contains NaN, infinity or a value too large for dtype('float32').

How do I find the bad values in the test dataset? Also, I do not want to drop these records, can I just replace them with the mean or median?

Thanks.

$\endgroup$
2
  • 2
    $\begingroup$ You can use numpys 'np.isfinite()' to create a boolean mask and after indexing by this mask your data $\endgroup$ Feb 1, 2020 at 10:37
  • $\begingroup$ Check if you have taken log() of any feature with zero value in it? $\endgroup$
    – DataFramed
    Mar 4, 2020 at 10:17

11 Answers 11

77
$\begingroup$

With np.isnan(X) you get a boolean mask back with True for positions containing NaNs.

With np.where(np.isnan(X)) you get back a tuple with i, j coordinates of NaNs.

Finally, with np.nan_to_num(X) you "replace nan with zero and inf with finite numbers".

Alternatively, you can use:

  • sklearn.impute.SimpleImputer for mean / median imputation of missing values, or
  • pandas' pd.DataFrame(X).fillna(), if you need something other than filling it with zeros.
$\endgroup$
4
  • $\begingroup$ I prefer identity condition for checking nan, if x!=x return None, many times np.isnan(x) had failed for me, do not remember the reason $\endgroup$
    – Itachi
    Jul 19, 2018 at 6:43
  • 2
    $\begingroup$ It is not advisable to replace NaN values with zeros. NaN values might still have significance in being missing and imputing them with zeros is probably the worst thing you can do and the worst imputation method you use. Not only will you be introducing zeros arbitrarily which might skew your variable but 0 might not even be an acceptable value in your variables, meaning your variable might not have a true zero. $\endgroup$
    – hussam
    May 9, 2019 at 20:04
  • $\begingroup$ I realized that I did not provide any guidance. If you want to impute your data either use a rolling average using .rolling() to replace missing value with the mean value of a rolling window. If you want something more robust use module <b>missingpy</b> you can use MissForest for a randomforest based imputation. $\endgroup$
    – hussam
    May 9, 2019 at 21:32
  • $\begingroup$ Wonderfu methodsl to examine existing dataframes. In my case, when I formed my X vector, extra blank rows were added for some reason and therefore what fit thought were NaNs really should not have been in the dataframe in the first place $\endgroup$
    – demongolem
    Apr 24, 2020 at 15:07
10
$\begingroup$

For anybody happening across this, to actually modify the original:

X_test.fillna(X_train.mean(), inplace=True)

To overwrite the original:

X_test = X_test.fillna(X_train.mean())

To check if you're in a copy vs a view:

X_test._is_view
$\endgroup$
1
  • 4
    $\begingroup$ While this is true technically, it's wrong practically. You can't fill the X_test NAs with the X_test mean, because in real life you won't have the X_test mean when you're predicting a sample. You should use the X_train mean because this is the only data you actually have in hand (in 99% of the scenarios) $\endgroup$
    – Omri374
    Jun 17, 2018 at 11:56
9
$\begingroup$

Assuming X_test is a pandas dataframe, you can use DataFrame.fillna to replace the NaN values with the mean:

X_test.fillna(X_test.mean())
$\endgroup$
1
  • $\begingroup$ X_test is the numpy array. Just updated the df_test in the original question, still got the same error ... $\endgroup$
    – Edamame
    May 26, 2016 at 14:56
6
$\begingroup$

I faced similar problem and saw that numpy handles NaN and Inf differently.
Incase if you data has Inf, try this:

np.where(x.values >= np.finfo(np.float64).max)
Where x is my pandas Dataframe 

This will be giving a tuple of location of places where NA values are present.

Incase if your data has Nan, try this:

np.isnan(x.values.any())
$\endgroup$
6
$\begingroup$

Don't forget

col_mask=df.isnull().any(axis=0) 

Which returns a boolean mask indicating np.nan values.

row_mask=df.isnull().any(axis=1)

Which return the rows where np.nan appeared. Then by simple indexing you can flag all of your points that are np.nan.

df.loc[row_mask,col_mask]
$\endgroup$
4
$\begingroup$

Do not forget to check for inf values as well. The only thing that worked for me:

df[df==np.inf]=np.nan
df.fillna(df.mean(), inplace=True)

And even better if you are using sklearn

def replace_missing_value(df, number_features):

    imputer = Imputer(strategy="median")
    df_num = df[number_features]
    imputer.fit(df_num)
    X = imputer.transform(df_num)
    res_def = pd.DataFrame(X, columns=df_num.columns)
    return res_def

When number_features would be an array of the number_features labels, for example:

number_features = ['median_income', 'gdp']
$\endgroup$
4
$\begingroup$

In most cases getting rid of infinite and null values solve this problem.

get rid of infinite values.

df.replace([np.inf, -np.inf], np.nan, inplace=True)

get rid of null values the way you like, specific value such as 999, mean, or create your own function to impute missing values

df.fillna(999, inplace=True)

or

df.fillna(df.mean(), inplace=True)
$\endgroup$
2
$\begingroup$

Here is the code for how to "Replace NaN with zero and infinity with large finite numbers." using numpy.nan_to_num.

df[:] = np.nan_to_num(df)

Also see fernando's answer.

$\endgroup$
1
$\begingroup$

If your values are larger than float32, try to run some scaler first. It'd be rather unusual to have deviation spanning more than float32.

$\endgroup$
1
$\begingroup$

You can list your columns that had NaN with this function

df.isnull().sum()

and then you can fill these NAN values in your dataset file. (csv or excel file)

$\endgroup$
1
  • $\begingroup$ Yes, then you can fill these values in your dataset file also like that --> df[:] = np.nan_to_num(df) $\endgroup$ Apr 28, 2022 at 17:53
-2
$\begingroup$

Once Check X and y types. Both should be of the same kind.

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.