How can I solve it? : <ValueError: Input contains NaN, infinity or a value too large for dtype('float64').>

Question

def preprocess_features(X):
    output = pd.DataFrame(index = X.index)
    for col , col_data in X.iteritems():
        if col_data.dtype == object :
            col_data = pd.get_dummies(col_data , prefix = col)
        output = output.join(col_data)
    return output
X_a = preprocess_features(X_all)


X_train , X_test , Y_train , Y_test = train_test_split(X_a , Y_all , test_size = 0.15, random_state = 2)
#print(X_train.shape,X_test.shape,Y_train.shape,Y_test.shape)
clf = LogisticRegression()
clf.fit(X_train , Y_train)

When I run this code, I get the following error: How do I resolve it?

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

Answer 1

Check the columns inside your dataframe.

You can try:

X.count()

to see how many data entries each column has.

From there, you can try:

.fillna()

or go back to your data and fill the necessary information or use sklearn.preprocessing.Imputer encode mean / median imputation for missing values.

Make sure that each column doesn't have any string or letter in it. Check also each column and get their max value, maybe you can convert them in much more smaller number (e.g. Normalizing)

Also, try this:

np.isnan(X)  

to get a boolean form of your dataframe to check if there's any blanks