Hot answers tagged

12

Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document embeddings: not all words equally represent the meaning of a particular sentence. And here different weighting strategies are applied, TF-IDF is one of them, and, ...


11

scikit-learn's FeatureUnion concatenates features from different vectorizers. An example of combining heterogeneous data, including text, can be found here.


10

The main difference is that HashingVectorizer applies a hashing function to term frequency counts in each document, where TfidfVectorizer scales those term frequency counts in each document by penalising terms that appear more widely across the corpus. There’s a great summary here. Hash functions are an efficient way of mapping terms to features; it doesn’t ...


9

Train a tfidfvectorizer with your corpus and use the following code: tfidf = Tfidfvectorizer () dict(zip(tfidf.get_feature_names(), tfidf.idf_))) Now you have a dictionary with words as its keys and weights as the corresponding values. Let me know if it worked.


7

The most accepted idea is that bag-of-words, Tf-Idf and other transformations should be left as is. According to some: Standardization of categorical variables might be not natural. Neither is standarization of Tf-Idf because according to stats stack exchange: (it's) (...) usually is a two-fold normalization. First, each document is normalized to ...


7

Yes, your method is valid and it has been studied before it is known as Mean of Word Embeddings (MOWE) or Sum of Word Embeddings (SOWE), although your method is more a weighted mean of vectors. I think that a good starting point for knowing more about the academics of the method is this paper: How Well Sentence Embeddings Capture Meaning. It discusses your ...


6

The HashingVectorizer has a parameter n_features which is 1048576 by default. When hashing, they don't actually compute a dictionary mapping terms to a unique index to use for each one. Instead, you just hash each term and use a large enough size that you don't expect there to be too many collisions: hash(term) mod table_size. You can make the returned ...


5

Usually, if possible, you'd want to keep your matrice sparse as long as possible as it saves a lot of memory. That's why there are sparse matrices after all, otherwise, why bother? So, even if your classifier requires you to use dense input, you might want to keep the TFIDF features as sparse, and add the other features to them in a sparse format. And then ...


3

What is wrong with your implementation is that you are passing a dataframe directly to tfidf vectorizer. If you check your data, it would look like this - >>> x_train_tf.toarray() array([[ 0.70710678, 0. , 0.70710678], [ 0. , 1. , 0. ]]) If you check what features you are getting, you will see - >>> ...


3

Weighting scheme 2 in table Recommended TF-IDF weighting schemes in tf–idf | Wikipedia should solve your problem.


3

From the way the TfIdf score is set up, there shouldn't be any significant difference in removing the stopwords. The whole point of the Idf is exactly to remove words with no semantic value from the corpus. If you do add the stopwords, the Idf should get rid of it. However, working without the stopwords in your documents will make the number of features ...


3

Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document embeddings; not all words equally represent the meaning of a particular sentence. And here different weighting strategies are applied, TF-IDF is one of those ...


3

Formally the problem of topic modelling is a clustering problem: given a collection of text documents, group together the documents which are topically similar. So technically it can indeed be done with a TF-IDF representation of documents as follows: Collect the global vocabulary across all the documents and calculate the IDF for every word. Represent ...


3

You can use pandas.Dataframe.sparse.from_spmatrix. It will create a Dataframe populated by pd.arrays.SparseArray from a scipy sparse matrix. Pandas used to have explicit sparse dataframes, but in more modern versions there is no such concept. Only normal pd.Dataframe populated by sparse data.


3

You could use pandas pivot_table() to transform your data frame into a count matrix, and then apply sklearn TfidfTransformer() to the count matrix in order to obtain the tf-idfs. import pandas as pd from sklearn.feature_extraction.text import TfidfTransformer # input data df = pd.DataFrame({ 'RepID': [1, 1, 1, 2, 2, 5684, 5684, 5684], 'Word': ['...


2

The list of 20,000 most common words in English is avaiable here. By using Zipf's law, we can obtain the probability of these words as below. Zipf's Law In the English language, the probability of encountering the rth most common word is given roughly by P(r)=0.1/r for r up to 1000 or so. The law breaks down for less frequent words, since the harmonic ...


2

I don't believe that there are any precalculated idf values out there. Inverse Document Frequency (idf) is the inverse of the number of documents in which a particular word appears in your corpus. If you only have one document, I'm afraid that value is simply 1. However, if you are looking to get rid of words such as the, as, and it which don't carry much ...


2

We do something similar for financial news classification which I suspect is similar to what you're trying to do, the problem we hit when using completely automated classification was there are a number of rules you have to consider i.e. there are two Tesco's -the UK super market chain and a tractor company based in the states. Instead, we built ...


2

The usual rule is to ask only one question per post. I will answer your first question. Build a dictionary, once, in advance. With a bit of effort you should be able to construct a dictionary that has all the words that are likely to appear in the future. Take all news articles over the past year, or all of Wikipedia, or some other very large collection, ...


2

I had a similar problem. What I was doing was this: I loaded a pretrained tfidf vectorizer. And on the new data to be predicted, I called vectorizer.fit_transform() and I got similar error. What solved the issue was calling vectorizer.transform(). It is because, fit_transform() will fit the current data in the model, which is not what we are seeking ...


2

I am still not quite sure how you are solving in the problem with tf-idf for a QA system. However, there have been lots of improvements that has been done in the QA domain over the years, with the usage of deep learning for natural language processing. I would urge you to look at the following approaches that might help you to reach the accuracies you are ...


2

It is common for TFIDF to be a strong model. People constantly get high places in Kaggle competitions with TFIDF models. Here is a link to the winning solution that used TFIDF as one of its features (1st place Otto product classification). You will most likely get a stronger model if you combine the TFIDF and RNN into one ensemble. Other results from Kaggle: ...


2

Not a very clever solution though. But I managed to do some trick to make it work. I am not fully satisfied by the result but the algorithm is able exactly predict the Item based on given Composition column. from io import StringIO import json col = ['Item', 'Composition'] df = dfmin[col] df['Item'] = df['Item'].apply(lambda x: ''.join(str(x).strip('[]') if ...


2

Normalised vectors have magnitude 1, so it doesn't matter if you explicitly divide by the magnitudes or not. It's mathematically equivalent either way. I see no reason that you couldn't use normalised vectors in TS-SS, but it seems that the main motivation for using TS-SS in the first place is that it makes more sense for vectors that may have different ...


2

The issue is due to your lamda function with the tokenizer key word argument. >>> from sklearn.feature_extraction.text import TfidfVectorizer >>> from joblib import dump >>> t = TfidfVectorizer() >>> dump(t, 'tfidf.pkl') ['tfidf.pkl'] No issues. Now let's pass a lambda function to tokenizer >>> t = ...


2

They all sound like interesting approaches. The first one is better I think because it allows for unseen hyphenated words to be somewhat understood (as e.g. well + known ~= well-known). For a tfidf BOW model, you might get good performance from any of the above. For a model that is sensitive to word order I would certainly go with the first option and might ...


2

The GridsearchCV object in sklearn does a cross-validation on the data you feed it during your fit. In your case you have specified cv=5: this means GridSearchCV splits your data into train/test splits 5 times and reports on the mean performance over those 5 trials to be 0.868. You asked why GridSearchCV knows the best parameters without feeding it testing ...


2

To complement my comment I'm taking those paragraphs from data camp tutorial in which they explain this in a very clear way


2

I have had to deal with huge data frames as you mention, in mi case the problem was "solved" by storing the data frame as pickle pd.to_pickle() and not as csv. The memory usage reduced by 60% I also heard recently about a format named feather For reference: https://towardsdatascience.com/the-best-format-to-save-pandas-data-414dca023e0d


2

It depends on the problem you are trying to solve. If you know the signal in the dataset already, the words which decide your decision then go with Bag of Words. This is useful when you are doing something like text classification. On the other hand, TF-IDF is useful when you don't know the signal in the dataset. If you want to do text similarity, then, this ...


Only top voted, non community-wiki answers of a minimum length are eligible