# Tag Info

71

With np.isnan(X) you get a boolean mask back with True for positions containing NaNs. With np.where(np.isnan(X)) you get back a tuple with i, j coordinates of NaNs. Finally, with np.nan_to_num(X) you "replace nan with zero and inf with finite numbers". Alternatively, you can use: sklearn.impute.SimpleImputer for mean / median imputation of missing ...

11

For anybody happening across this, to actually modify the original: X_test.fillna(X_train.mean(), inplace=True) To overwrite the original: X_test = X_test.fillna(X_train.mean()) To check if you're in a copy vs a view: X_test._is_view

9

Assuming X_test is a pandas dataframe, you can use DataFrame.fillna to replace the NaN values with the mean: X_test.fillna(X_test.mean())

6

I don't know any Georgian stemmer or lemmatizer. I think, however, that you have another option: to use unsupervised approaches to segment words into morphemes, and use your linguistic knowledge of Georgian to devise some heuristic rules to identify the stem among them. This kind of approach consists of a model trained to identify morphemes without any ...

5

Don't forget col_mask=df.isnull().any(axis=0) Which returns a boolean mask indicating np.nan values. row_mask=df.isnull().any(axis=1) Which return the rows where np.nan appeared. Then by simple indexing you can flag all of your points that are np.nan. df.loc[row_mask,col_mask]

5

I faced similar problem and saw that numpy handles NaN and Inf differently. Incase if you data has Inf, try this: np.where(x.values >= np.finfo(np.float64).max) Where x is my pandas Dataframe This will be giving a tuple of location of places where NA values are present. Incase if your data has Nan, try this: np.isnan(x.values.any())

5

I would recommend to use the one hot encoding package from category encoders and select the columns you want to using pandas select dtypes. import numpy as np import pandas as pd from category_encoders.one_hot import OneHotEncoder pd.options.display.float_format = '{:.2f}'.format # to make legible # make some data df = pd.DataFrame({'a': ['aa','bb','...

4

Do not forget to check for inf values as well. The only thing that worked for me: df[df==np.inf]=np.nan df.fillna(df.mean(), inplace=True) And even better if you are using sklearn def replace_missing_value(df, number_features): imputer = Imputer(strategy="median") df_num = df[number_features] imputer.fit(df_num) X = imputer.transform(...

4

The score is based on the scorer defined in the scoring argument. Meaning, the scorer can be any of the default metrics, such as precision, accuracy or F1-score (e.g., this); or a custom scorer. For a scorer (by convention), higher value is better. The value is not necessarily a percentage, but is often normalized between 0 and 1.

4

According to the suggestion, @bkshi gave to me I come up with a solution here below: Also since texts_to_sequences() function convert my list to sequences starting from 1, I could use pad_sequence() and use 0 instead of a string value. This solution satisfies my requirements so I used a number as padding instead of a string value. import pandas as pd ...

3

Each of those selected clustering algorithms can be fit using cosine distances in scikit-learn: from sklearn.cluster import DBSCAN, MeanShift, OPTICS from sklearn.metrics.pairwise import cosine_distances # Define clustering algorithms algorithms = [DBSCAN, MeanShift, OPTICS] # Placeholder for results results = dict.fromkeys((a.__name__ for a in algorithms)...

3

Try nunique(). That should do it. Here is a toy example: import pandas as pd df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar', 'foo', 'bar', 'foo', 'foo'], 'B' : ['one', 'one', 'two', 'three', 'two', 'two', 'one', 'two']}) print(df) gives A B 0 foo one 1 bar one ...

3

First we use DataFrame.explode to unnest your lists to rows. Then we use DataFrame.pivot_table to pivot your dataframe from rows to column to get your desired result: dfn = df.assign(countries=df['countries'].str.split(',')).explode('countries') dfn['numbers'] = df.assign(numbers=df['numbers'].str.split(',')).explode('numbers')['numbers'] dfn = ( dfn....

3

I think that you just need: feature_importances = rf_gridsearch.best_estimator_.feature_importances_ This provides the feature importance for all the attributes in your dataset. For more information on this as well as other options, you may also refer to the Scikit-learn official documentation.

3

First my understanding of your problem. You want to find the best hyperparameters for a Random Forest. For that, you want to first adjust n_estimators parameter and then the rest of parameters in different runs. Before answering to your question, you will only want to do a thorough search of hyperparameters when you want to have an improvement of around 1%...

3

The best way to understand multiple association rules is to visualize them. This makes it even easier to present. This paper covers multiple approaches for visualizing association rules. Go through its references. They also suggest their tool, but it is in R. If you want resources for python try searching for "association rules visualization python" and you'...

3

Try using pandas melt dataT = pd.DataFrame({"userID":[1,2,3],"date":["2020-01-01","2019-01-02","2020-08-12"],"-7":[0,1,1],"-6":[1,0,0],"-5":[0,0,0]}) Input: dataT.melt(value_vars= ["-7","-6","-5"], value_name="count") Output: Update By ...

2

In most cases getting rid of infinite and null values solve this problem. get rid of infinite values. df.replace([np.inf, -np.inf], np.nan, inplace=True) get rid of null values the way you like, specific value such as 999, mean, or create your own function to impute missing values df.fillna(999, inplace=True) or df.fillna(df.mean(), inplace=True)

2

Here is the code for how to "Replace NaN with zero and infinity with large finite numbers." using numpy.nan_to_num. df[:] = np.nan_to_num(df) Also see fernando's answer.

2

First, you are fitting $5 \cdot 3\cdot2\cdot2\cdot2\cdot5=600$ models and n_estimator=500 is quite big. Of course, this depends on your dataset and in your computing power. My first guess will be that you have not enough RAM memory on your laptop(if you are running it there) and that is why it is collapsing. If the error is this one, I recommend sampling ...

2

My 2 cents: I'm fan of defying the max_leaf_nodes (in this example 5) and then visualizing it. I suggest starting at 3 and then increasing it slightly (the same applies for your Random Forest). In general, at around 5 I see overfitting. With your large dataset, you might need a bit more (i.e. max_leaf_nodes = 10?). Why? Or the answer to your question... ...

2

By other posts and this one seems what you don't have a clear intuition of the n_estimators of the random forest. I am going to assume that you are referring to the n_estimators (from this other question). n_estimators is the number of trees that your 'forest' has. Not the depth of your tree. That is another parameter. If you are referring to max_depth = ...

2

Edit: oh, now I think I see why @CarlosMougan said no. You said ...start the same GridsearchCV with the same parameter and just change... If you mean use the optimal values for all hyperparameters except n_estimators and now search only on that one hyperparameter, then Carlos is right, and for the right reason. Below, I interpreted your suggestion as ...

2

the network architecture above is a very strange choice. When you have only 6 input features, it is weird to have so much Dense layers stacked. if network is overfitting, WHERE IS DROPOUT? Why not trying some regularizers, if the latter does not help? +1 for David Waterworth - correlation/causal analysis is not everything yet. Does linear regression provide ...

2

TensorSpec is mostly used by tf.function to specify input signature. tf.function will create a graph for different input shapes and datatypes, but it is possible that your function graph is compatible with different shapes. As a performance optimization, You can optionally provide a signature so that no unnecessary graphs are created.

2

Label encoding is not a good idea if the nature of categories are not ordinal (it is actually not my favorite anyways). Use one-hot encoding and see how it works. You may apply a feature extraction on top of it, e.g. PCA, to reduce the noise coming from sparsity. The other idea is to label categories by their fraction in the feature, for example: [a,b,b,c,a,...

2

There's no algorithm intended specifically for this task, you need to design the process yourself (like for most tasks btw). Given that the goal would be to use a person's name as an indication, I'd suggest you represent a name as a vector of characters n-grams in the features. Example with bigrams ($n=2$): "Braund" = [ #B, Br, ra, au, un, nd, d# ] ...

2

If absolutely necessary, You could build your own stemmer. It is fairly simple programming, but takes some studying of the Georgian language in the process, there are however plenty tutorials around the web for building a stemming process.

2

Note that I don't know nmslib and I'm not familiar with search optimization in general. However I know Okapi BM25 weighting. How do they both (bm25, nmslib) differ? These are two completely different things: Okapi BM25 is a weighting scheme which has a better theoretical basis than the well known TFIDF weighting scheme. Both methods are intended to score ...

2

You set the input shape to (1500, 2) whereas your data only contains a single feature. You should therefore change the shape to (1,) or (None, 1) to match the shape of the input data.

Only top voted, non community-wiki answers of a minimum length are eligible