0
$\begingroup$

I'm transforming a text in tf-idf from sklearn. I made the model:

from sklearn.feature_extraction.text import TfidfVectorizer
corpus = words
vectorizer = TfidfVectorizer(min_df = 15)
tf_idf_model = vectorizer.fit_transform(corpus)

And now I'm making vectors for different sets of words (documents), like:

word_set = ['dog', 'cat', 'foo']
v = vectorizer.transform(word_set)

But I want just one vector of these words, to compare to other documents. But when I use transform, the shape of v becomes:

<3x56492 sparse matrix of type '<class 'numpy.float64'>'
    with 3 stored elements in Compressed Sparse Row format>

I want a vector with shape 1x56492, and not 3x56492.. I'm certainly missing something here. Maybe you guys have some tips?

Thank you very much in advance.

$\endgroup$
  • $\begingroup$ word_set should be a list of lists(list of vectors). try putting [['dog', 'cat', 'foo']] $\endgroup$ – yoav_aaa May 1 '19 at 13:08
  • $\begingroup$ @yoav_aaa I did t = [['dog', 'cat', 'foo']] and v = vectorizer.transform(t) and that gave me AttributeError: 'list' object has no attribute 'lower' $\endgroup$ – why_not May 1 '19 at 13:13
0
$\begingroup$

As far as I can tell it's interpreting your new word_set word_set = ['dog', 'cat', 'foo'] as three separate documents containing one word each, whereas if you did word_set = ['dog cat foo'] it would interpret this as a single new document containing those words.

What behavior are you expecting from this function? Is corpus = words a list of document strings, or a list of single words? If it's the latter, this is likely not doing what you think it is doing, and you should instead make corpus a list of document strings.

$\endgroup$
  • $\begingroup$ Hm, you are right! Thanks. I made the model with documents like: ['dog cat foo', 'dog boo'.. etc]. When I try to do the transform vectorizer now, it gives me 1x(...) vector. Ultimately, i want to check the similarity of the documents with cosine similarity (and also new documents, that's why I needed transform) $\endgroup$ – why_not May 1 '19 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.