2
$\begingroup$

I am working on a multiclass text classification problem. I want to use the top k features based on mutual information (mutual_info_classif) for training my model.

I started this project on ML models:

I used tfidf for feature extraction and then used mutual_info_classif for feature selection. 

    svc = Pipeline([('tfidf', TfidfVectorizer()),
                   ('ig', SelectKBest(mutual_info_classif,k=1000)),
                   ('clf',LinearSVC(multi_class='ovr')),
              ])
    svc.fit(X_train, y_train)
    y_pred = svc.predict(X_test)

This was pretty straight-forward.

Next, I started to work on RNN (more specifically a simple LSTM model)and that is where I am having a problem.

I have used pre-trained word embeddings from GloVe (300d) to get features from my data. The embedding matrix I feed into the embedding layer of my RNN has the shape (4293,300), 4293 is the number of unique words found in my data and 300 is the dimension.

My questions are: Is there any way that I can use the top 1000 words (features) out of these 4293 based on mutual_info_classif? Is it even possible to do so? If yes, then should it be done before making the embedding_matrix or after?

Improve this question
$\endgroup$
1
$\begingroup$

When using a RNN, you don't feed all the data at once, you usually have a seq2seq model. The models are created with an encoder-decoder architecture. The LSTM is used in the encoding phase.

So, let's say you have a text of 78 words. You will feed the embedding vector (size 300) of those 78 words, 1-by-1 into your LSTM and in the end you will get a hidden vector which represents your sentence.

Then, you can take this hidden vector and use it for classification (with a feedforward neural network, for example).

So, it doesn't matter that you have 4293 unique words in your data. You need to feed your LSTM a sequence of size [<number of words in sentence> × 300].

Improve this answer
$\endgroup$
2
  • $\begingroup$ Agreed. But, what if for that particular sentence, I want that out of those 78 words, only the top 30 words (based on mutual info) to be fed into the LSTM? Is that even possible? Suppose, I somehow reduce my embedding matrix from (4293,300) to (1000,300) by picking the top 1000 words using mutual_info. Now, when that 78-word sentence will be fed to the LSTM, will not only those words (30 out of 78) be fed which are present in the embedding matrix currently? If yes, then I come back to my original question: How to get the embedding matrix from (4293,300) to (1000,300) using mutual_info? $\endgroup$ – Akshat Shrivastava Dec 29 '19 at 17:35
  • $\begingroup$ Ok. If you have, 1) a lot of smaller sentences, then I would you suggest you score each word in a sentence based on their TF-IDF Score, and if you have 2) fewer but large sentences (e.g. size >4000) then I would suggest you use a summarization algorithm, However, with either approach, keep the original words sequence, because an LSTM expects that sequence to be there. $\endgroup$ – Bruno Lubascher Dec 30 '19 at 7:37
0
$\begingroup$

You lose the point of word matrix if you do that.

Whole point is, very roughly speaking, when you pre-deterimened your word dictionary to calculate distributed representation of words. In other words information of one word will be in other words-vectors also.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.