Hot answers tagged

6

Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document embeddings: not all words equally represent the meaning of a particular sentence. And here different weighting strategies are applied, TF-IDF is one of them, and, ...


6

The main difference is that HashingVectorizer applies a hashing function to term frequency counts in each document, where TfidfVectorizer scales those term frequency counts in each document by penalising terms that appear more widely across the corpus. There’s a great summary here: https://spark.apache.org/docs/latest/mllib-feature-extraction.html Hash ...


5

scikit-learn's FeatureUnion concatenates features from different vectorizers. An example of combining heterogeneous data, including text, can be found here.


5

The HashingVectorizer has a parameter n_features which is 1048576 by default. When hashing, they don't actually compute a dictionary mapping terms to a unique index to use for each one. Instead, you just hash each term and use a large enough size that you don't expect there to be too many collisions: hash(term) mod table_size. You can make the returned ...


5

Yes, your method is valid and it has been studied before it is known as Mean of Word Embeddings (MOWE) or Sum of Word Embeddings (SOWE), although your method is more a weighted mean of vectors. I think that a good starting point for knowing more about the academics of the method is this paper: How Well Sentence Embeddings Capture Meaning. It discusses your ...


3

The most accepted idea is that bag-of-words, Tf-Idf and other transformations should be left as is. According to some: Standardization of categorical variables might be not natural. Neither is standarization of Tf-Idf because according to stats stack exchange: (it's) (...) usually is a two-fold normalization. First, each document is normalized to ...


3

What is wrong with your implementation is that you are passing a dataframe directly to tfidf vectorizer. If you check your data, it would look like this - >>> x_train_tf.toarray() array([[ 0.70710678, 0. , 0.70710678], [ 0. , 1. , 0. ]]) If you check what features you are getting, you will see - >>> ...


2

Usually, if possible, you'd want to keep your matrice sparse as long as possible as it saves a lot of memory. That's why there are sparse matrices after all, otherwise, why bother? So, even if your classifier requires you to use dense input, you might want to keep the TFIDF features as sparse, and add the other features to them in a sparse format. And then ...


2

I don't believe that there are any precalculated idf values out there. Inverse Document Frequency (idf) is the inverse of the number of documents in which a particular word appears in your corpus. If you only have one document, I'm afraid that value is simply 1. However, if you are looking to get rid of words such as the, as, and it which don't carry much ...


2

I had a similar problem. What I was doing was this: I loaded a pretrained tfidf vectorizer. And on the new data to be predicted, I called vectorizer.fit_transform() and I got similar error. What solved the issue was calling vectorizer.transform(). It is because, fit_transform() will fit the current data in the model, which is not what we are seeking ...


2

We do something similar for financial news classification which I suspect is similar to what you're trying to do, the problem we hit when using completely automated classification was there are a number of rules you have to consider i.e. there are two Tesco's -the UK super market chain and a tractor company based in the states. Instead, we built ...


2

The usual rule is to ask only one question per post. I will answer your first question. Build a dictionary, once, in advance. With a bit of effort you should be able to construct a dictionary that has all the words that are likely to appear in the future. Take all news articles over the past year, or all of Wikipedia, or some other very large collection, ...


2

I am still not quite sure how you are solving in the problem with tf-idf for a QA system. However, there have been lots of improvements that has been done in the QA domain over the years, with the usage of deep learning for natural language processing. I would urge you to look at the following approaches that might help you to reach the accuracies you are ...


2

It is common for TFIDF to be a strong model. People constantly get high places in Kaggle competitions with TFIDF models. Here is a link to the winning solution that used TFIDF as one of its features (1st place Otto product classification). You will most likely get a stronger model if you combine the TFIDF and RNN into one ensemble. Other results from Kaggle: ...


2

Normalised vectors have magnitude 1, so it doesn't matter if you explicitly divide by the magnitudes or not. It's mathematically equivalent either way. I see no reason that you couldn't use normalised vectors in TS-SS, but it seems that the main motivation for using TS-SS in the first place is that it makes more sense for vectors that may have different ...


2

From the way the TfIdf score is set up, there shouldn't be any significant difference in removing the stopwords. The whole point of the Idf is exactly to remove words with no semantic value from the corpus. If you do add the stopwords, the Idf should get rid of it. However, working without the stopwords in your documents will make the number of features ...


1

urllib parse seems like the function for you. With this you are able to extract keywords from the net location and the path separately if you desire to process them separately or even if you want to join them back again later. The result should look something like this: from urllib.parse import urlparse o = urlparse('https://www.forbes.com/sites/...


1

TfidfVectorizer will by default normalize each row. From the documentation we can see that: norm : ‘l1’, ‘l2’ or None, optional (default=’l2’) Each output row will have unit norm, either: * ‘l2’: Sum of squares of vector elements is 1. The cosine similarity between two vectors is their dot product when l2 norm has been applied. * ‘l1’: Sum of absolute ...


1

In my opinion, there is no way to deal out-of-vocabulary terms in TF-IDF as it only works using the test corpus. It's like you are trying to find certain word like technology in sports corpus.


1

I don't think there is a clear-cut criterion to decide on what method to use. From what I see, you have a good amount of data and the problem is "complex" (language). This are reasons to go for "deep" learning such as neural nets or boosting. The reason is that both can handle "non-linearity" very well. Another thing that comes to my mind is that when you ...


1

Yes, Cosine TF-IDF is quite transparent so it's usually reasonably easy to visualize the words which contribute the most to a score. Cosine is defined as the dot product divided by the product of the norms, so you can isolate the terms: dotproduct(d_1,d_2) = tfidf(w1,d1) * tfidf(w1,d2) + tfidf(w2,d1) * tfidf(w2,d2) + ... + tfidf(wN,dN) Ranking the words ...


1

Provided limited information & context you have provided, I would suggest you to look for feature selection when each dimension belongs to a word. Feature selection will give you most important words. Most important words in the sense, words deciding the decision surface of the model.


1

I would say 1000 documents is a bit less to draw any conclusion about the vectorization technique, Neither an increase in True positive by 1 would matter. As the size of the vocabulary increases, TfidfVectorizer would be better able to differentiate rare words and commonly occurring words while Countvectorizer would still give equal weight to all words which ...


1

Yes. Stacking is essentially feeding the predictions of the base learners to a meta learner. Sort of like a model of the models. Here's a good explanation of that. Bagging,Boosting and Stacking


1

Not a very clever solution though. But I managed to do some trick to make it work. I am not fully satisfied by the result but the algorithm is able exactly predict the Item based on given Composition column. from io import StringIO import json col = ['Item', 'Composition'] df = dfmin[col] df['Item'] = df['Item'].apply(lambda x: ''.join(str(x).strip('[]') if ...


1

One important factor to take into account is how you use the numerical representation of words / embeddings from either TF-IDF or Word2Vec to then compute sentiments. Without knowing how you do this, it is difficult to give a concrete answer. Also, which task are you working on, what does a result of 90% mean? Regardless of how you compute TF-IDF (there are ...


1

Train a tfidfvectorizer with your corpus and use the following code: tfidf = Tfidfvectorizer () dict(zip(tfidf.get_feature_names(), tfidf.idf_))) Now you have a dictionary with words as its keys and weights as the corresponding values. Let me know if it worked.


1

TFIDF decreases as term frequency will be decreased linearly and idf increases log linearly. Document similarity will decrease as value of tfidf vectors should decrease as reputation of bigrams are more less than each single word.


1

The list of 20,000 most common words in English is avaiable here. By using Zipf's law, we can obtain the probability of these words as below. Zipf's Law In the English language, the probability of encountering the rth most common word is given roughly by P(r)=0.1/r for r up to 1000 or so. The law breaks down for less frequent words, since the ...


Only top voted, non community-wiki answers of a minimum length are eligible