1
$\begingroup$

I have a basic doubt with regards to conversion of text to numbers and feeding it to LSTM. I am aware of the different methods such as OneHot, CountVectorizer, TfIDF, Word2vec etc. My doubt is, If we use a Count Vectoriser or Tfidf, Then in LSTM, we have to pass through the entire vocabulary of words for each sentence since that's how TFIDF and count vectoriser encodes the sentences. Am I right?

My second doubt is, If we use TFIDF or COuntVectorizer, Each word will have different value based on its occurrence and frequency. This is in contrast to Word2Vec where a embedding is learned and used. If each time the LSTM model sees different values for a particular word, How can it learn? Like in a sentence if the word "Hi" appears 6 times, Its encoded with the number 6 in its appropriate index, And in another sentence if it appears 4 times, we encode with the value 4. How does this work? It doesn't make sense.

Improve this question
$\endgroup$
2
  • $\begingroup$ You might be interested in this question about traditional representation of text. $\endgroup$
    – Erwan
    Apr 8, 2022 at 15:44
  • $\begingroup$ Hi, I am aware of the answer posted there, Could you kindly resolve the specific doubt I am facing. Thanks! $\endgroup$
    – mewbie
    Apr 8, 2022 at 15:49

1 Answer 1

Reset to default
1
$\begingroup$

The difference between the traditional bag of words representation and the word embedding representation is that:

  • bag of words: every index of a vector represents a specific word. Since there must be an index for every possible word, the dimension of every vector must indeed be the full vocabulary size.
  • embedding: words and sentences are represented in an (usually pre-trained) embedding space. The dimension of this space is predefined and arbitrary, and there's no way to know directly what every index represents. Indirectly, it can be proved that indexes can represent quite precise semantic concepts.

Anyway, in both cases the features values (which can vary) don't carry the meaning, it's always the fixed indexes which represent a particular semantic concept.

In your example, say the word "Hi" has index 1234: the fact that this specific index contains 6 or 4 allows the model to recognize a similarity between these 2 sentences. Note that in an embedding representation it's also the indexes which carry the concept. For example maybe "Hi" would have a important value for the dimension related to "salutations" and this would allow the model to find a similarity with worlds like "hello", "dear X", etc.

Improve this answer
$\endgroup$
11
  • $\begingroup$ But if I am doing a machine Translation, And I have a list of sentences to translate, How does it make sense to use Tf-idf since each sentence will be encoded to full vocabulary size. $\endgroup$
    – mewbie
    Apr 8, 2022 at 18:49
  • $\begingroup$ And also, If suppose I have a sentence "It is Raining" and "Theres a storm here", and I have to predict 0/1. If I use word embeddings, I need to only pass in the given words embeddings and obtain the output label. If I use Tf-idf, I have to pass the entire vocabulary for each sentence to my LSTM to obtain thee output. And moreover, The order of words is not preserved in TFIDF, So, How will it work? $\endgroup$
    – mewbie
    Apr 8, 2022 at 18:52
  • $\begingroup$ Again, If I have a word embedding, Each word will have some specific dimension vector. So If I have "It is Raining" "It is snowing" The word "It" appears twice but will get the same word embedding in both the sentences. But in tiff, the word "it" will have different value for each sentence. How will the model know ? $\endgroup$
    – mewbie
    Apr 8, 2022 at 18:55
  • $\begingroup$ @mewbie machine translation (MT) is a whole different problem, because it's not mainly about representing the text, it's about finding the alignment from one language to another. I'm not up to date with recent MT models. But do you mean that you want to train a MT model, or just apply a model to find the translation? In any case you would probably have to ask a different question. Also imho it's easier to understand the basics of text representation (for example in classification) first, before something advanced like MT. $\endgroup$
    – Erwan
    Apr 8, 2022 at 19:44
  • $\begingroup$ point 2: that's not totally exact that you only pass the word embeddings, because you still need a way to represent the sentence as a whole in the embedding space. The deep network may do this job for you, but at some level there's a representation of the full sentence somewhere, and since the values are real numbers this representation is actually more complex and more acccurate than a sparse array. The number of dimensions is only one part of the complexity. And yes, by definition the order of the words is lost in "bag of words" (hence the term). $\endgroup$
    – Erwan
    Apr 8, 2022 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.