sklearn KNN fit throws out error : value too large for dtype('float64')

Question

I have cleaned the data from nan values and infinite values, the only feature which has a large float is the column 8 (it's a sum)

I have no Idea how to fix this last error, I tried all previous solutions which are related to my question but nothing worked.

This is the result of : df.isnull().sum():

I am following this tutorial (because I'm just starting to learn about machine learning supervised algorithm KNN): Tutorial Link

Please help me out!

Answer 1

You should scale your data before training your data. Try something simple:

df = (df - df.min()) / (df.max() - df.min())

If you're using the L2 distance in your knn model this means some values are squared (which is probably why you max out the float64). Btw float64 max is $2^{31} − 1$ so check the range of the columns just to be sure there might be an outlier.

Answer 2

I'm almost certain that you have missing values, check that and fill them beforehand

Assuming you data frame is called df

df.info() will give you that info

Additionally df.describe() is a good method to validate the maxmimum value is not np.inf

Answer 3

Try converting your dataframe to integer values since its dtype('float64') is throwing the error. Assuming your dataframe is called df, use this:

 df["column_8"] = pd.to_numeric(df["column_8"])

This will convert all values in that column to integers.