Hot answers tagged

9

Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document embeddings: not all words equally represent the meaning of a particular sentence. And here different weighting strategies are applied, TF-IDF is one of them, and, ...


8

The main difference is that HashingVectorizer applies a hashing function to term frequency counts in each document, where TfidfVectorizer scales those term frequency counts in each document by penalising terms that appear more widely across the corpus. There’s a great summary here. Hash functions are an efficient way of mapping terms to features; it doesn’t ...


6

scikit-learn's FeatureUnion concatenates features from different vectorizers. An example of combining heterogeneous data, including text, can be found here.


6

The HashingVectorizer has a parameter n_features which is 1048576 by default. When hashing, they don't actually compute a dictionary mapping terms to a unique index to use for each one. Instead, you just hash each term and use a large enough size that you don't expect there to be too many collisions: hash(term) mod table_size. You can make the returned ...


6

Yes, your method is valid and it has been studied before it is known as Mean of Word Embeddings (MOWE) or Sum of Word Embeddings (SOWE), although your method is more a weighted mean of vectors. I think that a good starting point for knowing more about the academics of the method is this paper: How Well Sentence Embeddings Capture Meaning. It discusses your ...


4

The most accepted idea is that bag-of-words, Tf-Idf and other transformations should be left as is. According to some: Standardization of categorical variables might be not natural. Neither is standarization of Tf-Idf because according to stats stack exchange: (it's) (...) usually is a two-fold normalization. First, each document is normalized to ...


3

Usually, if possible, you'd want to keep your matrice sparse as long as possible as it saves a lot of memory. That's why there are sparse matrices after all, otherwise, why bother? So, even if your classifier requires you to use dense input, you might want to keep the TFIDF features as sparse, and add the other features to them in a sparse format. And then ...


3

What is wrong with your implementation is that you are passing a dataframe directly to tfidf vectorizer. If you check your data, it would look like this - >>> x_train_tf.toarray() array([[ 0.70710678, 0. , 0.70710678], [ 0. , 1. , 0. ]]) If you check what features you are getting, you will see - >>> ...


3

From the way the TfIdf score is set up, there shouldn't be any significant difference in removing the stopwords. The whole point of the Idf is exactly to remove words with no semantic value from the corpus. If you do add the stopwords, the Idf should get rid of it. However, working without the stopwords in your documents will make the number of features ...


2

The list of 20,000 most common words in English is avaiable here. By using Zipf's law, we can obtain the probability of these words as below. Zipf's Law In the English language, the probability of encountering the rth most common word is given roughly by P(r)=0.1/r for r up to 1000 or so. The law breaks down for less frequent words, since the harmonic ...


2

I don't believe that there are any precalculated idf values out there. Inverse Document Frequency (idf) is the inverse of the number of documents in which a particular word appears in your corpus. If you only have one document, I'm afraid that value is simply 1. However, if you are looking to get rid of words such as the, as, and it which don't carry much ...


2

We do something similar for financial news classification which I suspect is similar to what you're trying to do, the problem we hit when using completely automated classification was there are a number of rules you have to consider i.e. there are two Tesco's -the UK super market chain and a tractor company based in the states. Instead, we built ...


2

The usual rule is to ask only one question per post. I will answer your first question. Build a dictionary, once, in advance. With a bit of effort you should be able to construct a dictionary that has all the words that are likely to appear in the future. Take all news articles over the past year, or all of Wikipedia, or some other very large collection, ...


2

Train a tfidfvectorizer with your corpus and use the following code: tfidf = Tfidfvectorizer () dict(zip(tfidf.get_feature_names(), tfidf.idf_))) Now you have a dictionary with words as its keys and weights as the corresponding values. Let me know if it worked.


2

I had a similar problem. What I was doing was this: I loaded a pretrained tfidf vectorizer. And on the new data to be predicted, I called vectorizer.fit_transform() and I got similar error. What solved the issue was calling vectorizer.transform(). It is because, fit_transform() will fit the current data in the model, which is not what we are seeking ...


2

I am still not quite sure how you are solving in the problem with tf-idf for a QA system. However, there have been lots of improvements that has been done in the QA domain over the years, with the usage of deep learning for natural language processing. I would urge you to look at the following approaches that might help you to reach the accuracies you are ...


2

It is common for TFIDF to be a strong model. People constantly get high places in Kaggle competitions with TFIDF models. Here is a link to the winning solution that used TFIDF as one of its features (1st place Otto product classification). You will most likely get a stronger model if you combine the TFIDF and RNN into one ensemble. Other results from Kaggle: ...


2

Normalised vectors have magnitude 1, so it doesn't matter if you explicitly divide by the magnitudes or not. It's mathematically equivalent either way. I see no reason that you couldn't use normalised vectors in TS-SS, but it seems that the main motivation for using TS-SS in the first place is that it makes more sense for vectors that may have different ...


2

The issue is due to your lamda function with the tokenizer key word argument. >>> from sklearn.feature_extraction.text import TfidfVectorizer >>> from joblib import dump >>> t = TfidfVectorizer() >>> dump(t, 'tfidf.pkl') ['tfidf.pkl'] No issues. Now let's pass a lambda function to tokenizer >>> t = ...


2

Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document embeddings; not all words equally represent the meaning of a particular sentence. And here different weighting strategies are applied, TF-IDF is one of those ...


1

One important factor to take into account is how you use the numerical representation of words / embeddings from either TF-IDF or Word2Vec to then compute sentiments. Without knowing how you do this, it is difficult to give a concrete answer. Also, which task are you working on, what does a result of 90% mean? Regardless of how you compute TF-IDF (there are ...


1

It doesn't seem that non_negative is an argument in some versions. Try using decode_error = 'ignore'. If you're working with a large dataset, this error could also be resulting from hash collisions, which can be solved by increasing the number of features: vect = HashingVectorizer(decode_error = 'ignore', n_features = 2**21, ...


1

You need to ensure that the hashing vector doesn't purpose negatives. The way to do this is via HashingVectorizer(non_negative=True).


1

In my opinion, there is no way to deal out-of-vocabulary terms in TF-IDF as it only works using the test corpus. It's like you are trying to find certain word like technology in sports corpus.


1

As I see it, TF-IDF is very poor (more or less like a category feature) and does not take the words' precedence (approximately 80% of cases have 2 or more words). One possible approach is to embed Word2Vect => Doc2Vect for every sentence. A good implementation is the gensim library: https://radimrehurek.com/gensim/models/doc2vec.html Doc2Vect will provide ...


1

It depends what it's going to be used for, but in general it can make sense to use TF-IDF with short sentence. The main difference with the more standard case of long sentences is that TF (Term Frequency) won't play any role since the frequency will almost always be 1. IDF can still be useful though, assuming it's relevant to assign more weight to rare words ...


1

I don't think there is a clear-cut criterion to decide on what method to use. From what I see, you have a good amount of data and the problem is "complex" (language). This are reasons to go for "deep" learning such as neural nets or boosting. The reason is that both can handle "non-linearity" very well. Another thing that comes to my mind is that when you ...


1

welcome to ML and data science. This is a classic situation where a RNN would be usefull. You could either train one from Keras yourself which would help you learn and get a better model but might take more time. Or you could use a more pre-made soloution if this is your first project maybe something like TextGenRNN (will still recquire a bit of modification)...


1

Yes, Cosine TF-IDF is quite transparent so it's usually reasonably easy to visualize the words which contribute the most to a score. Cosine is defined as the dot product divided by the product of the norms, so you can isolate the terms: dotproduct(d_1,d_2) = tfidf(w1,d1) * tfidf(w1,d2) + tfidf(w2,d1) * tfidf(w2,d2) + ... + tfidf(wN,dN) Ranking the words ...


1

Provided limited information & context you have provided, I would suggest you to look for feature selection when each dimension belongs to a word. Feature selection will give you most important words. Most important words in the sense, words deciding the decision surface of the model.


Only top voted, non community-wiki answers of a minimum length are eligible