New answers tagged tfidf
1
Universal encoding and TF-IDF are two different beasts. I assume you mean the Vector Space Model transformed by TF-IDF. Either way: Neither tell you directly what the similarity of two texts are. Usually You'll use something like cosine distance to do that.
For the VSM there are scores of techniques to transform it. To name a few: Rocchio Transformation, ...
0
TF-IDF is the most simple and starting point of training the embedding for paragraph. SIF and doc2vec provide alternative methods for the embeddings too. Skip thought use encoder to train the embeddings. There are multiple ways of getting the embeddings.
1
TF-IDF is a vectorization technique used to convert documents (a single tweet in your case is a document) to vectors. After you train the TF-IDF model, the only words/vocabulary it has learnt, would be from the set of documents (aka corpus, the entire set of 3k tweets).
Since you mentioned that there were 570 unique feature words after TF-IDF, that would be ...
2
Word2Vec algorithms (Skip Gram and CBOW) treat each word equally,
because their goal to compute word embeddings. The distinction
becomes important when one needs to work with sentences or document
embeddings; not all words equally represent the meaning of a
particular sentence. And here different weighting strategies are
applied, TF-IDF is one of those ...
2
The issue is due to your lamda function with the tokenizer key word argument.
>>> from sklearn.feature_extraction.text import TfidfVectorizer
>>> from joblib import dump
>>> t = TfidfVectorizer()
>>> dump(t, 'tfidf.pkl')
['tfidf.pkl']
No issues. Now let's pass a lambda function to tokenizer
>>> t = ...
Top 50 recent answers are included
