77
votes
Accepted
ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
With np.isnan(X) you get a boolean mask back with True for positions containing NaNs.
With ...
- 886
14
votes
Accepted
What tokenizer does OpenAI's GPT3 API use?
Tokenizer for GPT-3 is the same as GPT-2:
https://huggingface.co/docs/transformers/model_doc/gpt2#gpt2tokenizerfast
linked via:
https://beta.openai.com/tokenizer
UPDATE March 2023
For newer models, ...
- 256
10
votes
ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
For anybody happening across this, to actually modify the original:
X_test.fillna(X_train.mean(), inplace=True)
To overwrite the original:
...
- 109
9
votes
ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
Assuming X_test is a pandas dataframe, you can use DataFrame.fillna to replace the NaN values with the mean:
...
- 191
7
votes
Accepted
Python stemmer for Georgian
I don't know any Georgian stemmer or lemmatizer. I think, however, that you have another option: to use unsupervised approaches to segment words into morphemes, and use your linguistic knowledge of ...
- 23.8k
6
votes
ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
Don't forget
col_mask=df.isnull().any(axis=0)
Which returns a boolean mask indicating np.nan values.
...
- 181
6
votes
ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
I faced similar problem and saw that numpy handles NaN and Inf differently.
Incase if you data has Inf, try this:
...
5
votes
Accepted
Pandas dataframe, create columns depending on the row value
First we use DataFrame.explode to unnest your lists to rows.
Then we use DataFrame.pivot_table to pivot your dataframe from ...
- 241
5
votes
Accepted
One Hot Encoding for any kind of dataset
I would recommend to use the one hot encoding package from category encoders and select the columns you want to using pandas select dtypes.
...
- 6,082
4
votes
ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
Do not forget to check for inf values as well. The only thing that worked for me:
df[df==np.inf]=np.nan
df.fillna(df.mean(), inplace=True)
And even better if you ...
- 141
4
votes
ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
In most cases getting rid of infinite and null values solve this problem.
get rid of infinite values.
df.replace([np.inf, -np.inf], np.nan, inplace=True)
get ...
- 141
4
votes
Accepted
How is the GridsearchCV Score calculated?
The score is based on the scorer defined in the scoring argument. Meaning, the scorer can be any of the default metrics, such as precision, accuracy or F1-score
(e.g., this); or a custom scorer.
For a ...
- 172
4
votes
Accepted
Filling missing values for Embedded List in Python3
According to the suggestion, @bkshi gave to me I come up with a solution here below:
Also since texts_to_sequences() function convert my list to sequences starting ...
- 191
4
votes
How to combine and separate test and train data for data cleaning?
Add an indicator column while concatenating the two dataframes, so you can later seperate them again:
...
- 241
4
votes
Accepted
ValueError: Input 0 of layer conv2d is incompatible with the layer: expected axis -1 of input shape to have value 1 but received input with shape
The error message means that the input shape of Conv2D layer should be (128,128,1) which is consistent with your model summary. However, in the actual input the shape it finds is (128,128,3), hence ...
- 387
4
votes
Sentence tokenization for sentence without punctuation
You can apply a previous step to add punctuation and proper casing to the text, and then segment the sentences. For this, you may use Re-punctuate. When applied to your text, Re-punctuate gives the ...
- 23.8k
3
votes
Accepted
Feature Importance from GridSearchCV
I think that you just need:
feature_importances = rf_gridsearch.best_estimator_.feature_importances_
This provides the feature importance for all the attributes in ...
- 501
3
votes
Is it alright to split a GridSearchCV?
First my understanding of your problem. You want to find the best hyperparameters for a Random Forest.
For that, you want to first adjust n_estimators parameter and then the rest of parameters in ...
- 6,082
3
votes
How to obtain unique count of categorical variable based on another categorical variable?
Try nunique(). That should do it. Here is a toy example:
...
- 5,360
3
votes
How to use Cosine Distance matrix for Clustering algorithms like mean-shift, DBSCAN, and optics?
Several scikit-learn clustering algorithms can be fit using cosine distances:
...
- 20.5k
3
votes
Accepted
How to present Market Basket Analysis Results?
The best way to understand multiple association rules is to visualize them. This makes it even easier to present. This paper covers multiple approaches for visualizing association rules. Go through ...
- 2,215
3
votes
Python stemmer for Georgian
If absolutely necessary, You could build your own stemmer. It is fairly simple programming, but takes some studying of the Georgian language in the process, there are however plenty tutorials around ...
- 131
3
votes
How can I solve it ,TypeError: cannot pickle 'dict_keys' object?
In your spatial_dataset class, dict.keys() is called to get the keys. This is known to cause pickling errors such as the one you ...
3
votes
Accepted
Beginning my data science journey
I too have a masters degree in physics so maybe you can relate!
The first thing to do would be to get your fundamentals in python strong. Pick up a python beginners tutorial from Youtube and learn all ...
- 1,906
3
votes
Beginning my data science journey
I'm also a newbie in the DS world but one thing that I find really helpful which I highly recommend is to do a lot of work "by hand". All the model that you can use in DS or Machine Learning ...
- 181
3
votes
Accepted
PCA followed by UMAP then go into Random Forest
They are 3 different algorithms: they work better in parallel, rather than in series because they have different purposes.
In addition to that, their output always brings some uncertainty (overall PCA)...
- 4,589
2
votes
ValueError: Input contains NaN, infinity or a value too large for dtype('float32')
Here is the code for how to "Replace NaN with zero and infinity with large finite numbers." using numpy.nan_to_num.
df[:] = np.nan_to_num(df)
Also see fernando's ...
- 141
2
votes
Accepted
Why does my GridSearchCV always break up?
First, you are fitting $5 \cdot 3\cdot2\cdot2\cdot2\cdot5=600$ models and n_estimator=500 is quite big.
Of course, this depends on your dataset and in your computing power.
My first guess will be ...
- 6,082
2
votes
How many trees does a Random Forest need?
My 2 cents: I'm fan of defying the max_leaf_nodes (in this example 5) and then visualizing it. I suggest starting at 3 and then increasing it slightly (the same applies for your Random Forest). In ...
- 1,057
2
votes
How many trees does a Random Forest need?
By other posts and this one seems what you don't have a clear intuition of the n_estimators of the random forest.
I am going to assume that you are referring to the n_estimators (from this other ...
- 6,082
Only top scored, non community-wiki answers of a minimum length are eligible