74 votes
Accepted

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

With np.isnan(X) you get a boolean mask back with True for positions containing NaNs. With ...
user avatar
  • 856
10 votes

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

For anybody happening across this, to actually modify the original: X_test.fillna(X_train.mean(), inplace=True) To overwrite the original: ...
user avatar
9 votes

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

Assuming X_test is a pandas dataframe, you can use DataFrame.fillna to replace the NaN values with the mean: ...
user avatar
  • 191
7 votes
Accepted

Python stemmer for Georgian

I don't know any Georgian stemmer or lemmatizer. I think, however, that you have another option: to use unsupervised approaches to segment words into morphemes, and use your linguistic knowledge of ...
user avatar
  • 16k
6 votes

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

Don't forget col_mask=df.isnull().any(axis=0) Which returns a boolean mask indicating np.nan values. ...
user avatar
  • 181
6 votes

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

I faced similar problem and saw that numpy handles NaN and Inf differently. Incase if you data has Inf, try this: ...
user avatar
5 votes
Accepted

One Hot Encoding for any kind of dataset

I would recommend to use the one hot encoding package from category encoders and select the columns you want to using pandas select dtypes. ...
user avatar
4 votes

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

Do not forget to check for inf values as well. The only thing that worked for me: df[df==np.inf]=np.nan df.fillna(df.mean(), inplace=True) And even better if you ...
user avatar
  • 141
4 votes
Accepted

Pandas dataframe, create columns depending on the row value

First we use DataFrame.explode to unnest your lists to rows. Then we use DataFrame.pivot_table to pivot your dataframe from ...
user avatar
  • 186
4 votes
Accepted

How is the GridsearchCV Score calculated?

The score is based on the scorer defined in the scoring argument. Meaning, the scorer can be any of the default metrics, such as precision, accuracy or F1-score (e.g., this); or a custom scorer. For a ...
user avatar
  • 172
4 votes
Accepted

Filling missing values for Embedded List in Python3

According to the suggestion, @bkshi gave to me I come up with a solution here below: Also since texts_to_sequences() function convert my list to sequences starting ...
user avatar
3 votes

How to obtain unique count of categorical variable based on another categorical variable?

Try nunique(). That should do it. Here is a toy example: ...
user avatar
  • 4,877
3 votes
Accepted

Feature Importance from GridSearchCV

I think that you just need: feature_importances = rf_gridsearch.best_estimator_.feature_importances_ This provides the feature importance for all the attributes in ...
user avatar
3 votes

Is it alright to split a GridSearchCV?

First my understanding of your problem. You want to find the best hyperparameters for a Random Forest. For that, you want to first adjust n_estimators parameter and then the rest of parameters in ...
user avatar
3 votes

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

In most cases getting rid of infinite and null values solve this problem. get rid of infinite values. df.replace([np.inf, -np.inf], np.nan, inplace=True) get ...
user avatar
3 votes

How to use Cosine Distance matrix for Clustering algorithms like mean-shift, DBSCAN, and optics?

Several scikit-learn clustering algorithms can be fit using cosine distances: ...
user avatar
3 votes
Accepted

How to present Market Basket Analysis Results?

The best way to understand multiple association rules is to visualize them. This makes it even easier to present. This paper covers multiple approaches for visualizing association rules. Go through ...
user avatar
  • 2,155
3 votes

How to combine and separate test and train data for data cleaning?

Add an indicator column while concatenating the two dataframes, so you can later seperate them again: ...
user avatar
  • 186
3 votes

Python stemmer for Georgian

If absolutely necessary, You could build your own stemmer. It is fairly simple programming, but takes some studying of the Georgian language in the process, there are however plenty tutorials around ...
user avatar
  • 131
3 votes
Accepted

ValueError: Input 0 of layer conv2d is incompatible with the layer: expected axis -1 of input shape to have value 1 but received input with shape

The error message means that the input shape of Conv2D layer should be (128,128,1) which is consistent with your model summary. However, in the actual input the shape it finds is (128,128,3), hence ...
user avatar
  • 347
3 votes
Accepted

Beginning my data science journey

I too have a masters degree in physics so maybe you can relate! The first thing to do would be to get your fundamentals in python strong. Pick up a python beginners tutorial from Youtube and learn all ...
user avatar
  • 1,273
3 votes

Beginning my data science journey

I'm also a newbie in the DS world but one thing that I find really helpful which I highly recommend is to do a lot of work "by hand". All the model that you can use in DS or Machine Learning ...
user avatar
3 votes
Accepted

PCA followed by UMAP then go into Random Forest

They are 3 different algorithms: they work better in parallel, rather than in series because they have different purposes. In addition to that, their output always brings some uncertainty (overall PCA)...
user avatar
2 votes

ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

Here is the code for how to "Replace NaN with zero and infinity with large finite numbers." using numpy.nan_to_num. df[:] = np.nan_to_num(df) Also see fernando's ...
user avatar
  • 131
2 votes
Accepted

Why does my GridSearchCV always break up?

First, you are fitting $5 \cdot 3\cdot2\cdot2\cdot2\cdot5=600$ models and n_estimator=500 is quite big. Of course, this depends on your dataset and in your computing power. My first guess will be ...
user avatar
2 votes

How many trees does a Random Forest need?

My 2 cents: I'm fan of defying the max_leaf_nodes (in this example 5) and then visualizing it. I suggest starting at 3 and then increasing it slightly (the same applies for your Random Forest). In ...
user avatar
2 votes

How many trees does a Random Forest need?

By other posts and this one seems what you don't have a clear intuition of the n_estimators of the random forest. I am going to assume that you are referring to the n_estimators (from this other ...
user avatar
2 votes

Is it alright to split a GridSearchCV?

Edit: oh, now I think I see why @CarlosMougan said no. You said ...start the same GridsearchCV with the same parameter and just change... If you mean use the optimal values for all ...
user avatar
  • 9,751
2 votes
Accepted

Validation Loss is not decreasing - Regression model

the network architecture above is a very strange choice. When you have only 6 input features, it is weird to have so much Dense layers stacked. if network is overfitting, WHERE IS DROPOUT? Why not ...
user avatar
  • 278
2 votes
Accepted

How to Minimize mean square error using Python

For this kind of problem, I would definitely start with scipy.otpimize methods. I reproduce here an example on how to use it in your context: ...
user avatar
  • 136

Only top scored, non community-wiki answers of a minimum length are eligible